PV Protection breaks deleting

[rmb938]

If a user deletes a PV while it has a PVC with a pv protection finalizer the CSI never gets alerted to delete the volume when the pvc is deleted.

[rmb938]

kubernetes-sigs/sig-storage-lib-external-provisioner#6 has been merged so just waiting for a release to make a pr to update the library

[msau42]

Thanks for finding this issue. Could you also look into adding an e2e test for this in k/k? This may be a good place: https://sourcegraph.com/github.com/kubernetes/kubernetes/-/blob/test/e2e/storage/csi_volumes.go

Also, I think this is an issue for in-tree volumes too. Should we consider fixing in-tree behavior? cc @jsafrane @wongma7

[rmb938]

@msau42 I will look into adding tests, I am not too familiar with the kubenretes e2e tests but I will try and figure it out!

[msau42]

feel free to ping me if you need help. Try following this guide: https://github.com/kubernetes/community/blob/master/contributors/devel/e2e-tests.md

[msau42]

Hi, looking back at this PR, I'm not sure if this is the right way to fix the issue.

I think this issue also exists for in-tree volumes, so we should see if a fix is possible at the PV controller level. At the very least, we should make this behavior opt-in and maybe alpha flag gated at the CSI provisioner level while we investigate other solutions.

cc @jsafrane @wongma7

[wongma7]

I'll make it opt-in.

I don't think the *exact same issue occurs but I agree this needs another look. The fix is a bit heavy-handed, it fixes behaviour for this specific issue so that it's the same for in-tree & out-of-tree, but also makes other behaviour different which I didn't follow up on...

If a user deletes a PVC+PV while they are Bound, the PV's protection finalizer will not be removed until the PVC is deleted, PV leaves Bound, and goes to Released.

• The in-tree controller will always see the PVC has been deleted first, then change the PV's state to Released, then try to reclaim the PV. Maybe the in-tree controller is wrong to try to reclaim even if there is already a deletionTimestamp on the PV? If that reclaim attempt fails, and the PV is actually deleted in the meantime, does it retry? @jsafrane
• The out-of-tree controller had this optimization/check to avoid immediately doing a reDelete https://github.com/kubernetes-sigs/sig-storage-lib-external-provisioner/pull/6/files#diff-18de8fd9ec37aa97da8831f9408ce6aeL860, and so unlike in-tree didn't try to reclaim PVs if they already had a deletionTimestamp. Now it does, so in that way it's like in-tree, but the finalizer also means users can't immediately delete PVs, unlike in-tree.

I liked the idea of having a finalizer because it makes it clear that the provisioner is responsible for deleting the back-end volume and users can still delete the PV by removing the finalizer first. Why they want to delete the dynamically provisioned PV manually when the provisioner will do it automatically, IDK (it could be an honest mistake/accident, like @rmb938 kind of suggests in kubernetes-sigs/sig-storage-lib-external-provisioner#5), but I think in any case they expect the back-end volume to be deleted when they delete the PV because the ReclaimPolicy of the PV is Delete. We document that you must delete the PVC first users should delete only the PVC, but clearly the ReclaimPolicy can cause confusion.

Anyway, all that said, I like consistency with in-tree more than I like the finalizer so I hope @jsafrane can help.

[rmb938]

Yea, this whole issue came about when I accidentally deleted the PV while the PVC was still around. Then when I went to go and delete the PVC the CSI never got called to clean up the backing volume for the PV.

I actually never tested this issue on the in-tree provisioners so my expectations of this not happening with them could have been wrong.

However if this issue also happens with in-tree provisioners I think we should have a discussion to change this behaviour so the backend volumes always get cleaned up when the ReclaimPolicy is Delete. I certainly do not want tons of leftover volumes lying around when someone accidentally deletes the PV resource first.