Skip to content

Conversation

@smklein
Copy link
Collaborator

@smklein smklein commented Mar 28, 2025

Partial fix of #7874 and #4203

This PR causes new points to be created as immutable directories, preventing files from begin created
in the "parent filesystem", underneath the intended mountpoint.

@morlandi7 morlandi7 added this to the 14 milestone Mar 28, 2025
@smklein smklein requested review from jgallagher and leftwo March 31, 2025 21:25
@leftwo
Copy link
Contributor

leftwo commented Mar 31, 2025

I'm going to take this for a spin on Dublin.
I'm installing r13 on there now and will put some data on it, then try mupdating to this.

// property on it. This prevents the mountpoint from being used as anything other than a
// mountpoint.
//
// NOTE: This must only be called on mountpoints for datasets which ARE NOT MOUNTED.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if this is called and the dataset is mounted? Do we need to guard against that?

Edit: after reading below, I think if we did this, we'd make the mounted directory immutable, make a temporary directory, then fail in the rename() calls (since rename won't cross filesystems, right?). The renames failing is good, but making the mounted directory immutable seems pretty bad...

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is always a little bit racy, but I can do something better in this case. It's easy to query zfs get -Hpo value mountpoint <mountpoint>, and bail if things already look mounted.

I'll do that.

if !mounted && want_to_mount {
Self::mount_dataset(name)?;
if !zoned && !mounted {
if let (CanMount::On, Mountpoint::Path(path)) =
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What other kinds of Mountpoints are there? (And what would it mean if we had CanMount::On, MountPoint::SomethingElse(_)?)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://illumos.org/man/8/zfs

     mountpoint=path|none|legacy
       Controls the mount point used for this file system.  See the Mount Points
       section for more information on how this property is used.

       When the mountpoint property is changed for a file system, the file
       system and any children that inherit the mount point are unmounted.  If
       the new value is legacy, then they remain unmounted.  Otherwise, they are
       automatically remounted in the new location if the property was
       previously legacy or none, or if they were mounted before the property
       was changed.  In addition, any shared file systems are unshared and
       shared in the new location.

My expectation is that Path is the only value where we actually want to do this mountpoint setup.

@leftwo
Copy link
Contributor

leftwo commented Apr 1, 2025

I'm going to take this for a spin on Dublin. I'm installing r13 on there now and will put some data on it, then try mupdating to this.

After installing r13, I created some resources on the rack., then rebooted each node by itself. I then created and deleted more resources on the rack. The reboot caused the crypt/debug datasets to not be mounted any longer. The additional activities resulted in logs being rotated to crypt/debug directories (not the dataset, just the directory).

After mupdate to this PR and a schema update, things came back online.

I can see the results here where crypt/debug directories were moved aside, and crypt/debug datasets are all now mounted.

root@oxz_switch0:~# pilot host exec -c 'hostname && ls -ld /pool/ext/*/crypt/old*' 0-31
14  BRM42220026        ok: BRM42220026
drwxr-xr-x  26 root     root          26 Dec 28  1986 /pool/ext/3a88433a-4d00-457a-8090-97425c59ed38/crypt/old-under-mountpoint-debugLH2P1r
15  BRM27230037        ok: BRM27230037
drwxr-xr-x  28 root     root          28 Dec 28  1986 /pool/ext/278b9ed4-99e1-43b1-8180-d56ab43f2da3/crypt/old-under-mountpoint-debugTpJYl8
drwxr-xr-x  15 root     root          15 Dec 28  1986 /pool/ext/2ae96717-8422-4e7e-9f23-3acdcf81bbb1/crypt/old-under-mountpoint-debugazHYZx
16  BRM23230018        ok: BRM23230018
drwxr-xr-x  25 root     root          25 Dec 28  1986 /pool/ext/acacdc94-295c-403a-ae59-af062fb0dc07/crypt/old-under-mountpoint-debugCocT6m
17  BRM23230010        ok: BRM23230010
drwxr-xr-x  33 root     root          33 Dec 28  1986 /pool/ext/2bf80d15-daef-4712-8fc2-443393d52048/crypt/old-under-mountpoint-debugqaUZcs

Copy link
Contributor

@leftwo leftwo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

github really struggled with the diff here. I believe I've looked at the most important parts and it all looks good. If you hid some easter egg in nexus_db_schema then good for you, I can't find it.

}

// If it doesn't exist, make it.
// If the dataset doesn't exist, create it.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth a comment here that zoned datasets are mounted elsewhere?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@smklein
Copy link
Collaborator Author

smklein commented Apr 3, 2025

github really struggled with the diff here. I believe I've looked at the most important parts and it all looks good. If you hid some easter egg in nexus_db_schema then good for you, I can't find it.

I went ahead and rebased, hopefully this is easier to read now!

Base automatically changed from do-more-mounting to main April 3, 2025 20:38
@smklein smklein enabled auto-merge (squash) April 3, 2025 21:45
@smklein smklein merged commit 9163dbf into main Apr 4, 2025
18 checks passed
@smklein smklein deleted the immutable-mountpoints branch April 4, 2025 01:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants