Patching status after removing finalizer

[surajkota]

Describe the bug

2021-09-06T17:58:55.473Z	DEBUG	ackrt	removed resource from management	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.473Z	DEBUG	ackrt	<< r.setResourceUnmanaged	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.473Z	INFO	ackrt	deleted resource	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.473Z	DEBUG	ackrt	< r.deleteResource	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.473Z	DEBUG	ackrt	> r.patchResourceStatus	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.473Z	DEBUG	ackrt	>> kc.Patch (status)	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.474Z	DEBUG	ackrt	<< kc.Patch (status)	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3, "error": "models.sagemaker.services.k8s.aws \"xgboost-model\" not found"}
2021-09-06T17:58:55.474Z	DEBUG	ackrt	< r.patchResourceStatus	{"kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 2, "account": "", "role": "", "region": "us-west-2", "kind": "Model", "namespace": "default", "name": "xgboost-model", "generation": 3}
2021-09-06T17:58:55.474Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "model", "request": "default/xgboost-model"}
2021-09-06T17:58:55.474Z	DEBUG	controller-runtime.controller	Successfully Reconciled	{"controller": "model", "request": "default/xgboost-model"}

See the Logs above. Controller is trying to patch status after finalizer is removed and gets an error "error": "models.sagemaker.services.k8s.aws \"xgboost-model\" not found"}. This bug was introduced in aws-controllers-k8s/code-generator#175 when rm.Delete() started returning a resource passed through updateConditions instead of nil post successful delete.

https://github.com/aws-controllers-k8s/code-generator/blob/a866a846b883a772c4671061783ccc21a48f9412/templates/pkg/resource/manager.go.tpl#L157-L160

Steps to reproduce
generate controller with the latest code-gen and delete a resource

Expected solution

1. return nil, nil instead of latest, nil after finalizer is removed - https://github.com/aws-controllers-k8s/runtime/blob/24e0b8ddeaaf7df4bf55ef1a75e4608ea5ec01fb/pkg/runtime/reconciler.go#L508
2. Change https://github.com/aws-controllers-k8s/code-generator/blob/a866a846b883a772c4671061783ccc21a48f9412/templates/pkg/resource/manager.go.tpl#L160 to return nil, nil. When delete is successful and returns nil resource, pass the same to the caller instead of returning the resource passed to this method.
3. Check if the resource is managed before patching in handleReconcileError - https://github.com/aws-controllers-k8s/runtime/blob/24e0b8ddeaaf7df4bf55ef1a75e4608ea5ec01fb/pkg/runtime/reconciler.go#L647

Recommended solution - Implement both 1&2. Please let me know your thoughts. I have opened PRs for solutions 1&2.

[jaypipes]

@surajkota would this problem go away if we simply removed the finalizer in setResourceUnmanaged but did not call kc.Patch right afterwards?

[surajkota]

@jaypipes yes. The PRs linked to the issue are helping do same thing. Lmk if you are proposing something different

aws-controllers-k8s/runtime#50

aws-controllers-k8s/code-generator#182