Error: unable to proceed without entity mappings including remote targets: error mapping Remote Storage Targets

[scaleoutsean]

In v8.0.1 I got remote sync enabled (or so I think), but get this on push:

$ sudo beegfs remote status /mnt/beegfs/archive/flights-1m.csv 
Error: unable to proceed without entity mappings including remote targets: error mapping Remote Storage Targets: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:9010: connect: connection refused"

The thing is, I don't set 127.0.0.1 anywhere. Maybe I did have it at one point, but I've since restarted beegfs-sync and beegfs-remote, so it's running on my external IP.

$ netstat -ant | grep 9010
tcp        0      0 192.168.1.198:9010      0.0.0.0:*               LISTEN     

So, where is this supposed 127.0.0.1 coming from?

$ cat /etc/beegfs/beegfs-remote.toml | grep 9010 
# address = "s198:9010"
address = "192.168.1.198:9010"

This is an all-in-one enviro - very simple, one VM runs MD, Storage, Mgmt, Client, etc.

/etc/beegfs/beegfs-sync.toml:

mount-point = "/mnt/beegfs"

[log]
level = 3
# type = 'syslog'
type = 'logfile'
file = '/var/log/beegfs/beegfs-remote.log'

[server]
address = "192.168.1.198:9011"
tls-cert-file = "/etc/beegfs/cert.pem"
tls-key-file = "/etc/beegfs/key.pem"
tls-disable = false

[manager]
journal-db = "/var/lib/beegfs/sync/journal.badger"
job-db = "/var/lib/beegfs/sync/job.badger"

[remote]
tls-cert-file = "/etc/beegfs/cert.pem"
tls-disable-verification = false
tls-disable = false

/etc/beegfs/beegfs-remote.toml:

[log]
level = 3
type = 'syslog'

[server]
# address = "s198:9010"
address = "192.168.1.198:9010"
tls-cert-file = "/etc/beegfs/cert.pem"
tls-key-file = "/etc/beegfs/key.pem"
tls-disable = false

[job]
path-db = "/var/lib/beegfs/remote/path.badger"

#
# --- Section 1.2: [Worker Settings] ---
#

[[worker]]
id = "1"
name = "s198"
type = "beesync"
address = "s198:9011"
tls-cert-file = "/etc/beegfs/cert.pem"
tls-disable-verification = false
tls-disable = false

[[remote-storage-target]]
id = "3"
name = "minio"
policies = { FastStartMaxSize = 10485760 } # I removed one zero at the end
s3.endpoint-url = "https://s3:443"
s3.region = "us-east-1"
s3.bucket = "beegfs"
s3.access-key = "abc"
s3.secret-key = "abc"

Sync service:

$ journalctl -u beegfs-sync
May 17 05:10:04 aio systemd[1]: Started beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:34:52 aio systemd[1]: Stopping beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer)...
May 17 05:34:52 aio systemd[1]: beegfs-sync.service: Deactivated successfully.
May 17 05:34:52 aio systemd[1]: Stopped beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:34:52 aio systemd[1]: beegfs-sync.service: Consumed 11.969s CPU time, 14.1M memory peak, 0B memory swap peak.
May 17 05:34:52 aio systemd[1]: Started beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:36:26 aio systemd[1]: Stopping beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer)...
May 17 05:36:26 aio systemd[1]: beegfs-sync.service: Deactivated successfully.
May 17 05:36:26 aio systemd[1]: Stopped beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:36:26 aio systemd[1]: Started beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:43:44 aio systemd[1]: Stopping beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer)...
May 17 05:43:44 aio systemd[1]: beegfs-sync.service: Deactivated successfully.
May 17 05:43:44 aio systemd[1]: Stopped beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).
May 17 05:43:44 aio systemd[1]: beegfs-sync.service: Consumed 3.163s CPU time, 16.8M memory peak, 0B memory swap peak.
May 17 05:43:44 aio systemd[1]: Started beegfs-sync.service - BeeGFS Sync (Remote Storage Targets Syncer).

Remote service:

May 17 05:47:15 aio systemd[1]: Stopped beegfs-remote.service - BeeGFS Remote (Remote Storage Targets Coordinator).
May 17 05:47:15 aio systemd[1]: Started beegfs-remote.service - BeeGFS Remote (Remote Storage Targets Coordinator).
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: <=== BeeRemote Initialized ===>
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: start-of-day{"application": "beegfs-remote", "version": "8.0.1", "commit": "79c836a572adcc325dfeb4b0d13d731baa780600", "built": >
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: checking BeeGFS mount point
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: downloaded license from BeeGFS management node{"certificate": "TMP-1992142438", "licensedTo": "NetApp", "viaPartner": "ThinkParQ>
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: connecting to node{"component": "workermgr", "nodeID": "1", "nodeName": "s198"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: All 2 tables opened in 0s{"component": "job", "database": "pathDB"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: Discard stats nextEmptySlot: 0{"component": "job", "database": "pathDB"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: Set nextTxnTs to 32{"component": "job", "database": "pathDB"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: Deleting empty file: /var/lib/beegfs/remote/path.badger/000015.vlog{"component": "job", "database": "pathDB"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: listening on local network address{"component": "server", "address": "192.168.1.198:9010"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: now accepting job requests and work responses{"component": "job"}
May 17 05:47:15 aio /opt/beegfs/sbin/beegfs-remote[50212]: serving gRPC requests{"component": "server"}
May 17 05:47:16 aio /opt/beegfs/sbin/beegfs-remote[50212]: connected to node{"component": "workermgr", "nodeID": "1", "nodeName": "s198"}

Another suspicious behavior was that my target ID 1 was shown unavailable. I changed it twice, incrementing ID to 2 and then 3. Even with a valid TLS cert, it still shows as unavailable. mc shows it as available and I can PUT files from BeeGFS into the bucket.

$ sudo beegfs entry info /mnt/beegfs/archive/flights-1m.csv 
PATH                     ENTRY_ID      TYPE  META_NODE  META_MIRROR   STORAGE_POOL  STRIPE_PATTERN  STORAGE_TARGETS  STORAGE_MIRRORS  REMOTE_TARGETS   
/archive/flights-1m.csv  0-68275F27-1  file  m:1        (unmirrored)  archive (2)   RAID0 (4x512K)  s:102            (unmirrored)     3-not-available 

$ mc ls s3/ | grep bee
[0001-01-01 00:00:00 UTC]     0B beegfs/

$ mc cp archive/flights-1m.csv s3/beegfs/
...beegfs/archive/flights-1m.csv: 40.98 MiB / 40.98 MiB ┃▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓▓┃ 59.31 MiB/s 0s
$ mc rm s3/beegfs/flights-1m.csv
Removed `s3/beegfs/flights-1m.csv`.

1. There should probably be some instructions regarding S3 target validation because I think checking with mc seems reasonable enough, but apparently it's not and some BeeGFS-specific validation steps are required on top of that. Is HTTPS not supported? Or path-style?
   Alternatively, debug logs for S3 should be available so that we can see by ourselves what is failing.

2. transport: Error while dialing: dial tcp 127.0.0.1:9010: connect: connection refused - where is this set (beegfs-remote.toml > address doesn't seem to be the whole story) and is it required to restart more than just beegfs-remote and beegfs-sync to have that update take effect?

3. I noticed these messages, but they aren't clear. What is one supposed to do here? I thought incrementing worker ID (new ID results in a new worker/remote worker) and restarting service would fix it, but that doesn't seem to have effect. Should worker/remote worker name also be changed?
   Also, the nodeName value: can be an arbitrary string (basically a tag) or is this supposed to be something else (hostname, something from beegfs node list or some other string)? Since worker address must be provided, having to provide a name would be superfluous (unless this is meant to be a feature for multi-homed workers, but in that case the docs should elaborate on that; I'd still prefer to assign a DNS address to particular interface (e.g. s198-int) than have to provide s198 as nodeName and then also manually specify an IP address). If it's not a name but a unique tag, it would have been better if it was called that way. Note: nodeID is now 2 as 1 set earlier didn't work, so I modified its name, incremented nodeID and restarted service, but that didn't help. aio is localhost name, s198 is its DNS name on LAN (also used in TLS cert).

2025-05-17T06:44:50.409903+00:00 aio /opt/beegfs/sbin/beegfs-remote[54338]: unable to connect to
node (unable to retry){"component": "workermgr", "nodeID": "2", "nodeName": "s198", "error": 
"FAILURE configure update on node with message error applying updated configuration: updating RST 
configuration after it was initially set is not currently supported"}

[iamjoemccormick]

$ sudo beegfs remote status /mnt/beegfs/archive/flights-1m.csv
Error: unable to proceed without entity mappings including remote targets: error mapping Remote Storage Targets: rpc error: code = > Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 127.0.0.1:9010: connect: connection refused"

That error is actually from the beegfs tool. By default the --remote-addr is 127.0.0.1:9010 thinking that would make it slightly easier to try things out where Remote and CTL are just on the same machine and you are using the default 0.0.0.0:9010 Remote server address to simply listen on all addresses.

In your case just do:

$ export BEEGFS_REMOTE_ADDR=192.168.1.198:9010
$ beegfs remote list

You could also add it to .bashrc to make it persist beyond the current shell session.

target ID 1 was shown unavailable

That is another side effect of the Remote address not being set in CTL. The ID is set on the entry but the name is set in Remote, however CTL not being able to talk to Remote is not a fatal error for the entry info mode.

I noticed these messages, but they aren't clear. What is one supposed to do here?

Currently in addition to Remote you also have to restart the Sync node(s) to change the RST configuration. If you did that and it still doesn't work let me know! Not needing to do that is on the list of future improvements :)

I'll make that and the other error more intuitive as to how to fix the situation.

[scaleoutsean]

Thanks for throwing light on these mysteries.

In your case just do:
$ export BEEGFS_REMOTE_ADDR=192.168.1.198:9010

I don't remember seeing that anywhere.

Maybe it'd be good to add environmental variables such as BEEGFS_REMOTE_ADDR (as well as --remote-addr) to the docs (or maybe it's just the search feature?)?

With BEEGFS_REMOTE_ADDR I can see the RST, thanks!