All records were deleted and recreated ONCE when ingress pods were restarted

[max-rocket-internet]

Not this is not related to other issues where the records are continuously recreated

We update our nginx-controllers with new request/limit settings. This started a rolling restart of the nginx-ingress controllers. During this time, external-dns deleted and recreated all our DNS records (about 30 records). It was quite disruptive:

time="2019-11-21T10:52:39Z" level=info msg="All records are already up to date"                                               
time="2019-11-21T10:53:39Z" level=info msg="All records are already up to date"                                               
time="2019-11-21T10:54:39Z" level=info msg="Desired change: DELETE aaaa.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]"
time="2019-11-21T10:54:39Z" level=info msg="Desired change: DELETE bbbb.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]"
time="2019-11-21T10:54:39Z" level=info msg="Desired change: DELETE ccccc.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]" 
time="2019-11-21T10:54:39Z" level=info msg="30 record(s) in zone xxx-yyy.com. [Id: /hostedzone/XXXXXXXXXXXX] were successfully updated"                  
time="2019-11-21T10:55:39Z" level=info msg="Desired change: CREATE aaaa.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]"   
time="2019-11-21T10:55:39Z" level=info msg="Desired change: CREATE bbbb.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]" 
time="2019-11-21T10:55:39Z" level=info msg="Desired change: CREATE cccc.xxx-yyy.com A [Id: /hostedzone/XXXXXXXXXXXX]"
time="2019-11-21T10:55:39Z" level=info msg="30 record(s) in zone xxx-yyy.com. [Id: /hostedzone/XXXXXXXXXXXX] were successfully updated"                  
time="2019-11-21T10:56:39Z" level=info msg="All records are already up to date"                                               
time="2019-11-21T10:57:39Z" level=info msg="All records are already up to date"                       

I know we use set sync mode to upsert-only but do want external-dns to remove records for ingress resources when they are deleted.

We've used both extern-dns and nginx-ingress for a long time and never seen anything like this.

None of our ingress resources were removed during this process. They are all showing an age of between 22-380 days. So I am not sure how or why external-dns would decide to delete the DNS records.

k8s version: 1.13
external-dns version: docker.io/bitnami/external-dns:0.5.16-debian-9-r0
nginx-ingress version: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.25.1

Any ideas? Or how I could debug this?

[rajatjindal]

not sure, but can you double check if you have disabled update-status-on-shutdown

from https://kubernetes.github.io/ingress-nginx/user-guide/cli-arguments/

--update-status-on-shutdown
Update the load-balancer status of Ingress objects when the controller shuts down. Requires the update-status parameter. (default true)

[max-rocket-internet]

@rajatjindal

not sure, but can you double check if you have disabled update-status-on-shutdown

No we have not disabled this. So it would default to true.

Why would this affect external-dns though?

[rajatjindal]

If I understand it correctly, it will remove the status from Ingress object, and that will trigger external-dns to remove the DNS entry.

https://github.com/kubernetes/ingress-nginx/blob/4847bb02f05ab323afb578600e83e4879a9791c7/internal/ingress/status/status.go#L107

How many replicas of ingress-nginx are you running? This will happen if your ingress-nginx leader shuts down (or if you have only one replicas).

Let me know if that does (or does not) make sense.

[max-rocket-internet]

Hmmm interesting.

How many replicas of ingress-nginx are you running?

Between 10 and 20. This happened when there was a rolling restart.

Let me know if that does (or does not) make sense.

I guess it makes sense but how or why would this behaviour be good for any reason?

Perhaps I'll create an issue in the ingress-nginx repo 🙂

[max-rocket-internet]

@rajatjindal

You are right, setting --update-status-on-shutdown=false is the fix: kubernetes/ingress-nginx#4774