Add DeletionPolicy as an option for S3 Custom Resources

[joerocklin]

I'm not sure if this is an issue with the service operator or with something in cloudformation, but here's what I'm seeing.

1. Create an S3 bucket with logging disabled, versioning enabled, and accessControl Private. All here works fine.
2. Delete the S3 resource from k8s (no files added to the bucket, no actions performed at all other than looking at it to verify proper creation parameters) and the CF stack skips the delete with no reason provided. After the SKIP message, the errors are emitted from the operator (log below) regarding missing information.
3. The S3 resource is removed from k8s, but the ConfigMap and Service are still present and the bucket is still in place.

I'm happy to provide any information which might be of interest in solving this problem. I'm still looking into details myself but want to see if others are experiencing the same problem.

Here is a partial log of the output with some values altered to protect some information:

aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:27Z" level=info msg="deleted s3bucket 'rocklin-aws-so-test'" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:37Z" level=debug msg="&{Type:Notification TopicArn:arn:aws:sns:us-east-2:123456789012:dt-ue2-s3bucket Message:StackId='arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e'\nTimestamp='2018-10-19T11:42:27.639Z'\nEventId='0de67df0-d394-11e8-9ddf-0aa0cd9d9f9c'\nLogicalResourceId='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\nNamespace='123456789012'\nPhysicalResourceId='arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e'\nPrincipalId='AROAISVCX6PVI5CBBLZ3K:kiam-kiam'\nResourceProperties='null'\nResourceStatus='DELETE_IN_PROGRESS'\nResourceStatusReason='User Initiated'\nResourceType='AWS::CloudFormation::Stack'\nStackName='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\nClientRequestToken='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\n ParsedMessage:map[Timestamp:2018-10-19T11:42:27.639Z EventId:0de67df0-d394-11e8-9ddf-0aa0cd9d9f9c LogicalResourceId:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops PhysicalResourceId:arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e PrincipalId:AROAISVCX6PVI5CBBLZ3K:kiam-kiam ResourceStatus:DELETE_IN_PROGRESS ClientRequestToken:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops StackId:arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e Namespace:123456789012 ResourceProperties:null ResourceStatusReason:User Initiated ResourceType:AWS::CloudFormation::Stack StackName:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops] Namespace: ResourceName: ResourceProperties:{Tags:[]} Updatable:false}" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:37Z" level=debug msg="stackID arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e updated status to DELETE_IN_PROGRESS" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=debug msg="&{Type:Notification TopicArn:arn:aws:sns:us-east-2:123456789012:dt-ue2-s3bucket Message:StackId='arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e'\nTimestamp='2018-10-19T11:42:29.259Z'\nEventId='S3bucket-DELETE_SKIPPED-2018-10-19T11:42:29.259Z'\nLogicalResourceId='S3bucket'\nNamespace='123456789012'\nPhysicalResourceId='rocklin-aws-so-test'\nResourceProperties='{\"BucketName\":\"rocklin-aws-so-test\",\"LifecycleConfiguration\":{\"Rules\":[{\"Status\":\"Enabled\",\"Transitions\":[{\"StorageClass\":\"Glacier\",\"TransitionInDays\":\"0\"}],\"ExpirationInDays\":\"365\",\"Id\":\"GlacierRule\",\"Prefix\":\"Archive\"}]},\"VersioningConfiguration\":{\"Status\":\"Enabled\"},\"AccessControl\":\"Private\",\"Tags\":[{\"Value\":\"projectName-ops\",\"Key\":\"Namespace\"},{\"Value\":\"36749881\",\"Key\":\"ResourceVersion\"},{\"Value\":\"rocklin-aws-so-test\",\"Key\":\"ResourceName\"},{\"Value\":\"dt-ue2\",\"Key\":\"ClusterName\"},{\"Value\":\"operator.aws\",\"Key\":\"Heritage\"}]}'\nResourceStatus='DELETE_SKIPPED'\nResourceStatusReason=''\nResourceType='AWS::S3::Bucket'\nStackName='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\nClientRequestToken='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\n ParsedMessage:map[ResourceStatus:DELETE_SKIPPED ResourceType:AWS::S3::Bucket Timestamp:2018-10-19T11:42:29.259Z EventId:S3bucket-DELETE_SKIPPED-2018-10-19T11:42:29.259Z Namespace:123456789012 ResourceProperties:{\"BucketName\":\"rocklin-aws-so-test\",\"LifecycleConfiguration\":{\"Rules\":[{\"Status\":\"Enabled\",\"Transitions\":[{\"StorageClass\":\"Glacier\",\"TransitionInDays\":\"0\"}],\"ExpirationInDays\":\"365\",\"Id\":\"GlacierRule\",\"Prefix\":\"Archive\"}]},\"VersioningConfiguration\":{\"Status\":\"Enabled\"},\"AccessControl\":\"Private\",\"Tags\":[{\"Value\":\"projectName-ops\",\"Key\":\"Namespace\"},{\"Value\":\"36749881\",\"Key\":\"ResourceVersion\"},{\"Value\":\"rocklin-aws-so-test\",\"Key\":\"ResourceName\"},{\"Value\":\"dt-ue2\",\"Key\":\"ClusterName\"},{\"Value\":\"operator.aws\",\"Key\":\"Heritage\"}]} StackName:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops ClientRequestToken:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops StackId:arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e LogicalResourceId:S3bucket PhysicalResourceId:rocklin-aws-so-test ResourceStatusReason:] Namespace:projectName-ops ResourceName:rocklin-aws-so-test ResourceProperties:{Tags:[{Key:Namespace Value:projectName-ops} {Key:ResourceVersion Value:36749881} {Key:ResourceName Value:rocklin-aws-so-test} {Key:ClusterName Value:dt-ue2} {Key:Heritage Value:operator.aws}]} Updatable:true}" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=error msg="error getting s3buckets" error="s3buckets.service-operator.aws \"rocklin-aws-so-test\" not found" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=error msg="error processing message" error="s3buckets.service-operator.aws \"rocklin-aws-so-test\" not found" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=debug msg="stackID arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e updated status to DELETE_SKIPPED" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=debug msg="&{Type:Notification TopicArn:arn:aws:sns:us-east-2:123456789012:dt-ue2-s3bucket Message:StackId='arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e'\nTimestamp='2018-10-19T11:42:29.822Z'\nEventId='0f356c20-d394-11e8-8b3f-023ea1ad1aec'\nLogicalResourceId='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\nNamespace='123456789012'\nPhysicalResourceId='arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e'\nPrincipalId='AROAISVCX6PVI5CBBLZ3K:kiam-kiam'\nResourceProperties='null'\nResourceStatus='DELETE_COMPLETE'\nResourceStatusReason=''\nResourceType='AWS::CloudFormation::Stack'\nStackName='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\nClientRequestToken='dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops'\n ParsedMessage:map[ResourceProperties:null ResourceStatusReason: ClientRequestToken:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops Namespace:123456789012 PrincipalId:AROAISVCX6PVI5CBBLZ3K:kiam-kiam EventId:0f356c20-d394-11e8-8b3f-023ea1ad1aec LogicalResourceId:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops PhysicalResourceId:arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e ResourceStatus:DELETE_COMPLETE ResourceType:AWS::CloudFormation::Stack StackName:dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops StackId:arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e Timestamp:2018-10-19T11:42:29.822Z] Namespace: ResourceName: ResourceProperties:{Tags:[]} Updatable:false}" hostname=aws-service-operator-587fff54f4-cx474
aws-service-operator-587fff54f4-cx474 aws-service-operator time="2018-10-19T11:42:39Z" level=debug msg="stackID arn:aws:cloudformation:us-east-2:123456789012:stack/dt-ue2-s3bucket-rocklin-aws-so-test-projectName-ops/e393d610-d393-11e8-b80e-0264de9b3f5e updated status to DELETE_COMPLETE" hostname=aws-service-operator-587fff54f4-cx474

[prashantchitta]

I am getting the same error as well. Unable to delete the s3 bucket.
Here is my s3.yaml file that i used to create

apiVersion: service-operator.aws/v1alpha1
kind: S3Bucket
metadata:
  name: prashant-operator
spec:
  versioning: true
  accessControl: PublicRead
  website:
    enabled: true
    indexPage: index.html
    errorPage: 500.html
  logging:
    enabled: false
    prefix: "archive"

time="2018-10-19T22:17:13Z" level=info msg="deleted s3bucket 'prashant-operator'" hostname=aws-service-operator-69c869846d-j69k2
time="2018-10-19T22:17:31Z" level=error msg="error getting s3buckets" error="s3buckets.service-operator.aws \"prashant-operator\" not found" hostname=aws-service-operator-69c869846d-j69k2
time="2018-10-19T22:17:31Z" level=error msg="error processing message" error="s3buckets.service-operator.aws \"prashant-operator\" not found" hostname=aws-service-operator-69c869846d-j69k2

I still see the bucket exists in s3, Configmap and Service exists

[christopherhein]

There are two things going on here, as of right now the s3 template doesn't tear down the bucket on deletion, Honestly, I'm not sure if there is something that I set up in the CFN to do that or if that is standard practice for buckets created by CFN. Something to look into for sure.

Second, you are hitting: #41 which I've been a little undecided about from an implementation perspective. I have a second issue which is related that will give the dependency tree a different way of handling it #84 this is a standard component in Kubernetes and might help this implementation in the long run.

@joerocklin and @prashantchitta can you put a note on #41 with your thoughts for if it should handle this or leave it to you to clean up. My thinking it purely around not removing a potential dependency for a running application, IE, if you delete the ConfigMap and an app, uses that ConfigMap the app would then be in an undesirable state.

[christopherhein]

Also changing the title to reflect the first issue in the above comment.

[prashantchitta]

@christopherhein I just checked the s3 CFN that you created has DeletionPolicy: Retain

 Tags:
        - Key: Namespace
          Value: !Ref Namespace
        - Key: ResourceVersion
          Value: !Ref ResourceVersion
        - Key: ResourceName
          Value: !Ref ResourceName
        - Key: ClusterName
          Value: !Ref ClusterName
        - Key: Heritage
          Value: operator.aws
      VersioningConfiguration: !If 
        - UseVersioning
        - Status: Enabled
        - !Ref 'AWS::NoValue'
    DeletionPolicy: Retain

Any reason you did that? If we delete the s3 CRD, I expect it that it deletes the S3 bucket as well. May be you can introduce an option in CRD to say if the user wants to delete or retain the S3 bucket when the CRD is deleted. But the default should be Delete unless the user says to Retain. That is how Dynamic Volumes works in K8 using Storage Classes where you can specify if you want to Retain or Delete the volume when the pvc is deleted

Also, Anything created with s3 should be deleted along with s3 deletion. So the configmaps and services should be deleted as well. This is how K8 design is. In k8 you can delete configmap even if its mounted to another pod. The pod will continue to work since the configmap is mounted. If the pod dies and tries to come back up, it will fail. So here the design should be similar and in-line to K8's implementation. Thoughts?

[christopherhein]

There it is… honestly, that template was pulled from the https://github.com/awslabs/aws-servicebroker then modified to support the static website attributes. We could should make that something configurable via the CRD. Best option to give some flexibility both.

For the additional resources we can track that in #41

Thanks @prashantchitta