beeond: Management did not accept target free space info message, Waiting until all nodes have registered with mgmtd...forever

[trothgsi]

Hi all,
I'm just starting with beeond, which seems to start, mounts the file system on the starting node and then gets stuck waiting for the others.

I have 32 nodes running Rocky 9.5. They can communicate with each other - I did some MPI jobs on them previously.

For beeond, I followed the docu step by step. Command line reads beeond start -n nodefile -d /srv/beeond -c /mnt/beeond

On node 1, where I issued the beeond start command, the mount succeeds, giving me a 1.3T /mnt/beeond, which seems reasonable if all 32 nodes contribute.

But for > 12 hours now, the start command paints dots onto the screen, Waiting until all nodes have registered with mgmtd

None of the other nodes has beeond mounted, and there, the /var/log/beegfs-storage_2025...log mostly repeats
XNodeSync [XNodeSync] >> Management did not accept target free space info message.

Seems to be a basic communication issue?

Regards,
Thomas

[iamjoemccormick]

That error happens if a storage node is unable to update the management with its available target capacity (which happens periodically). That specific error typically indicates some sort of communication/network issue as you suspected. This could be BeeGFS misconfiguration, notably if connection authentication is not setup properly (or disabled fully). But in your case since the mount succeeded on node 1 it seems more likely some network/other issue happened later on.

Did your management node log anything interesting?

[trothgsi]

Since this is supposed to be a test of beeond, there is no connection authentication.
It was just dnf install beeond, create the nodefile, run beeond start -n nodefile -d /srv/beeond -c /mnt/beeond.

The output of dsmpi01:~# beeond start tells me that reachability of all hosts has been checked.

Then, it started mgmtd: Starting beegfs-mgmtd on host: dsmpi01

Then, beegfs-storage is started on all hosts.

Then, Starting beegfs-meta on host: dsmpi01

Finally, Waiting until all nodes have registered with mgmtd - infinitely
And the log on the management node says

Main [MessagingTk.cpp:25] >> Retrying communication. peer: beegfs-mgmtd management [ID: 1]; message type: RegisterNode (1039)
Main [MessagingTk.cpp:447] >> Unable to connect, is the node offline? node: beegfs-mgmtd management [ID: 1]; Message type: RegisterNode (1039)

If I interrupd this everlasting start command, I find a /mnt/beeond mounted with the size of 32*disk-size of the 32 nodes participating. However, no access to that mount point Remote I/O error.

On any of the other nodes, I find

(1) 15:26:10 XNodeSync [XNodeSync] >> Management did not accept target free space info message.

And if I try beegfs --mgmtd-addr=dsmpi01 node list from one of the other nodes, it tells me

Error: unable to proceed without a working management node: reading certificate file failed: open /etc/beegfs/cert.pem: no such file or directory

Seems like this no-security approach isn't working so well?
It would not be a problem to set up an /etc/beegsfs/connauthfile on all nodes. If only the docu had told me so.

[scaleoutsean]

So you go this to work by creating a connauthfile?

I don't know if you've stepped on one (or more?) of the same landmines, but I struggled with this in v8 as well. Maybe you can skim thru this (contains some clarifications) and see if any of this helps.
#42