Recurrent logs from Traefik "Provider connection error unexpected EOF"

[kassyss]

Hi, we are using docker-socket-proxy with Traefik to protect our docker environment.
In traefik v2.0 logs there are many error logs like these :

time="2019-09-30T14:10:37+02:00" level=error msg="Provider connection error unexpected EOF, retrying in 263.019463ms" providerName=docker
time="2019-09-30T14:20:38+02:00" level=error msg="Provider connection error unexpected EOF, retrying in 372.070307ms" providerName=docker
time="2019-09-30T14:30:38+02:00" level=error msg="Provider connection error unexpected EOF, retrying in 726.287014ms" providerName=docker

the docker-socket-proxy configuration is the minimum

 - CONTAINERS=1

Do you think theses errors appear because docker-socket-proxy is compatible with Docker API version 1.37 and we are using version 1.40?

Another question, does docker-socket-proxy is still maintained?

Thanks you.

[yajo]

Yes, this is till maintained, although it's quite stable and has not much patches lately.

It might be related with the api version, maybe we need to update this image for newer versions...

Also I didn't test Traefik v2 yet.

[kassyss]

Hello Yako. Thank you very much for your answer. I happy to learn that this efficient application is still maintained because there are not many simple ways to get Traefik (community edition) secure.
There are chances that the problem i mentioned earlier are related to the docker api version évolution but i am not sure that’s why i opened up this issue. I will stay tuned on docker-socket-proxy next version and give some feedback.

Best regards,

[clownba0t]

I'm seeing this too, and I think it might be due to the haproxy configuration that docker-socket-proxy uses under the hood.

I noticed errors in the Traefik logs for my installation almost every 10 minutes, and it just so happens that the client and server timeouts in the haproxy config are also 10 minutes. It may be a coincidence, but reading the haproxy documentation, I don't think so.

As far as I know, amongst other things, Traefik connects to docker to fetch events. This is a persistent connection that continues to stream events as they occur. If no events are being generated because nothing interesting is happening with the docker installation that docker-socket-proxy is wrapping, no data will be sent on the connection, and haxproxy will time it out and close it down. I'm guessing that in this case Traefik attempts to recover by retrying the connection, which I assume almost always succeeds immediately, and is then able to keep it for another 10 minutes or so.

If this is indeed the root cause of the issue, I don't think there's any easy way to work around it immediately because the timeout values are hardcoded into the haproxy configuration. The easiest way to work around this might be to modify docker-socket-proxy to use an environment variable for the timeouts, which would allow clients to set a timeout appropriate to their use case.

Another option is to find a way to remove the timeouts from the configuration altogether, which would essentially disable them. However, the haproxy documentation rightly points out that this is not recommended for various reasons such as connections getting stuck and never being cleaned up.

[yajo]

One idea that floats over my mind is to use a different traefik to proxy the docker socket, instead of HAProxy.

I'm not sure if that'd fix this, but for sure most of our user base uses traefik and would be more able to contribute patches.

[1flyride]

I'm running into this problem as well with Docker API 1.40, and have the collateral issue that some Traefik-served containers then lose connection to Traefik and don't recover without container restarts. I can only solve the problem by reverting Traefik to direct socket access.

How would one go about configuring another Traefik instance to provide proxy services for the socket?

[yajo]

How would one go about configuring another Traefik instance to provide proxy services for the socket?

Basically you'd have to redo this project, but based on Traefik.

[koshatul]

Verifying what @clownba0t said.

I've narrowed this down to timeout server in the haproxy.cfg.

Traefik does a request against the events API which blocks and returns events as they happen, if you aren't getting any docker events in 10 minutes the connection times out and restarts.

So as it stands it isn't an error since Traefik will just reconnect in less than a second.

I upped the timeout to 1h in my own config, but it just means the error is now every hour instead of every 10 minutes since the containers are pretty stable and there's no events even in an hour.

The cleanest option would be to use TCP keepalives and option clitcpka/option srvtcpka while disabling haproxy timeouts, but since modifying the TCP keepalive settings inside the docker container isn't feasible (either running the container as privileged or manually specifying --sysctl options), we're stuck with 10m timeouts.

I don't think converting to Traefik would solve the issue, it has an idleTimeout default of 360s, so that message would be in the logs every 6 minutes instead of 10 minutes, also Traefik doesn't support unix socket backends.

It boils down to how can you tell a connection is still there if it is open but has no traffic.

[yajo]

Very interesting insight. Makes a lot of sense.

I'd say then that the best fix here would be to add a note in the README, right?

[koshatul]

Yeah, I'm willing to be wrong, but I can't think of a way to fix it without introducing issues, maybe timeouts of 1-2 weeks, since most connections should be coming from a relatively "local" network, so they should get terminated properly, one rogue service could consume all the connections though.

But if your services are really stable, then you'd still get the error every 1-2 weeks.

Real world example:

$ docker ps -a --format='{{.Status}}' | sort -u
Up 5 weeks

I'd still have gotten the message at least twice.

[MarkErik]

What do you think about a temporary workaround where a small container is created that sleeps for 8 minutes and then exits, and is restarted by Docker?

In the same docker-compose file that I have the docker-socket-proxy I added this container:

  keepalive:
    image: busybox:latest
    container_name: keepalive
    restart: unless-stopped
    command: sh -c 'sleep 480; exit 0'

[shcorya]

What about implementing a Docker healthcheck and getting rid of the HAProxy timeout entirely?

[yajo]

What about implementing a Docker healthcheck and getting rid of the HAProxy timeout entirely?

How would that fix the problem?

[MarkErik]

My hope with the "keepalive" container was to avoid what @1flyride mentioned "collateral issue that some Traefik-served containers then lose connection to Traefik and don't recover without container restarts"

I don't see the Traefik error messages in the logs when using the "keepalive" container.

Why I picked this option of creating a small container and using the command option with sleep, and not modifying the timeout on the docker-socket-proxy was so that I wouldn't have to customise any Dockerfiles and could just pull the images directly.

Again, my workaround isn't ideal, and I hoped that by sharing it, we can keep this discussion going.

[shcorya]

What about implementing a Docker healthcheck and getting rid of the HAProxy timeout entirely?

How would that fix the problem?

Going off what @koshatul posted above, the proxy times out if a client subscribes to the Docker events endpoint yet no events fire within the specified timeout period. If we remove the timeout server option entirely, the client would not time out even when no events are fired for an indefinitely long period.

That considered, removing the timeout from HAProxy would also remove any check that the proxy is actually able to reach the Docker socket. This could be mitigated by adding a HEALTHCHECK option to the Dockerfile, something along the lines of:

HEALTHCHECK CMD curl -f http://localhost:2375/_ping || exit 1

By periodically hitting the Docker API's _ping endpoint with a healthcheck, we could ensure that the proxy is able to reach the Docker socket and that clients are able to reach the Docker API through the proxy without relying on HAProxy's internal timeouts.