Semantics of destructive operations

[mhausenblas]

If an ACK user deletes a Kubernetes namespace that has, say, an S3 bucket custom resource in it, is the S3 bucket also deleted or not?

The answer to this question (and the respective UX) should follow the "no surprises" principle. IOW: opting into (forcing) destructive operations rather than silently cascading the delete from the Kubernetes cluster perimeter to AWS services.

[donpinkster]

In EC2 you can instances tag them as "unterminatable", so you have to remove a flag first before you can terminate EC2 instances. It would be great to have something like this for the ACK resources which have state in them.

For example RDS, if some rogue component, a wrong GitOps commit or whatever reason a delete event happens on the RDS object in the K8s API I would like to have an option to prevent the actual deletion of the associated resource in AWS.

[mhausenblas]

Thanks @donpinkster and indeed I was thinking along that line, potentially on the namespace level. There are a couple of questions like is it opt in or force deletion, but keep it coming, all of this should give us enough input for a design. Any volunteers? ;)

[justinfiore]

I agree. Deletion should either be a 2 step process or something that must explicitly be configured on the custom resource.
A 2-step process could be implemented in the following way:
Have a property on the resource like cascadeDelete which defaults to false.
You could then edit the resource, set that to true, then delete the resource.

Or if you didn't want a two-step process, just set that to true from the start.

I think the behavior should be the same across all resource types. Not just ones that are stateful, as that would be surprising behavior.

[joberdick]

I would make this an option on the controller side. It being per object would be good. Like maybe we want ECR Repos to be removed with the namespace, but not an s3 bucket.

[jaypipes]

@joberdick Yes, that's what I'm leaning towards as a long-term solution.

[amitkarpe]

We can use annotations or labels, which will declare whether this object is expected to delete or not.

For example, helm uses "annotations" like:
helm.sh/resource-policy: keep

For example, skaffold uses "labels" like:
skaffold.dev/cleanup: "false"

[szuecs]

If you have an annotation, how do you solve if you have 2 references with different annotation value?

In general I agree don’t delete data just like that.

[ravishtiwari]

If an ACK user deletes a Kubernetes namespace that has, say, an S3 bucket custom resource in it, is the S3 bucket also deleted or not?

The answer to this question (and the respective UX) should follow the "no surprises" principle. IOW: opting into (forcing) destructive operations rather than silently cascading the delete from the Kubernetes cluster perimeter to AWS services.

I agree to the sentiments of No surprises.
Storage or any other location where we store:

• Data
• State

shouldn't removed silently, instead, users should be informed about residual and what they should do in order to may be get a clean slate.

we can either have a config like:

auto_clean_on_delete: ["s3", "ecr", ...]

or force to add a similar tag for each of the resource, like: auto_remove_enabled:false some similar to what @donpinkster said, so that we know what are going to remove and what will automatically be cleaned.

Cheers,

[dcherman]

This sounds very similar to the reclaim policy feature of PVs:

https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming

Unless there's a reason not to do so, it seems like it'd help reduce confusion if we re-used the same/similar field naming and terminology as the built-in PV object

reclaimPolicy: Retain | Delete

[Quentin-M]

The existing reclaim policy, which defaults to Retain and must be overridden to Delete / Recycle is nice indeed, but it may not work as well as it does for PV/PVCs because a PV that was formerly attached to a deleted PVC in Retain mode will still show in the PV list (as Released ) and therefore be cleaned up easily as necessary by an administrator a posteriori, whereas the S3 CR will be gone for good, making tracking/cleaning up harder (would have to go into AWS and rely on some heuristics or AWS Tags marking the staleness).

Rather, I'd suggest using the one Kubernetes feature designed specifically for those use-cases, the ValidatingAdmissionWebhook. With just about 15 lines of YAML, you can instruct Kubernetes to ask the question to your operator via its service endpoint whether the command should be honored or not. In your operator's code, you can then make an informed decision (command accepted/honored vs. command denied) based on the policy specified within the custom resource. By default, the CRD should set the policy of any new stateful CR to something safe. For instance, in the case of an S3 bucket, similarly to what the S3 UI has, the default policy could be "can't delete if the bucket is not empty", the operator can easily check whether the bucket is empty or not, and accept/deny the request. If the tenant wants to bypass that and make the operator delete the bucket & its content entirely, the tenant can issue a quick CR modification (e.g. kubectl edit/patch) to change to policy & issue the command again.

That'd also reduce the likelihood of a tenant having to re-import a CR after mistakenly and actually removing it from Kubernetes (but retaining it in AWS thankfully).. which is "painful" enough to do with a PV/PVC as they have to bind together. That does not imply that an "import/reconciliation" feature should not be written, which could also be very useful to migrate existing AWS resources (e.g. Terraform managed) into AWS Controller's management on Kubernetes, without downtime.