Memory leak with Consul Config Watch

[nickcodefresh]

I raised the following issue on Stackoverflow because I was experiencing my Spring Boot app just eating away at the heap

https://stackoverflow.com/questions/37137358/huge-org-springframework-boot-loader-launchedurlclassloader-memory-usage

After investigation is appears as though Consul is returning a changing X-Consul-Index header value when requesting KVs that do not exist.

For example http://consul:8500/v1/kv/mio/application,server/?recurse&wait=2s&index=281722, where mio/application,server is the default content and one of the active Spring profiles. 281722 was the index value returned from the previous call, and Consul may now return a completely different value.

A changing X-Consul-Index value causes the Spring Cloud to refresh the context (via org.springframework.cloud.consul.config.ConfigWatch#watchConfigKeyValues), which in turn creates a new instance of org.springframework.cloud.consul.config.ConsulPropertySourceLocator. For whatever reason, the old instance of ConsulPropertySourceLocator aren't garbage collected, leading to a memory leak.

I'm using Spring Boot c1.3.2.RELEASE, Spring Cloud vBrixton.RC2 and Consul v0.6.4.

At the moment the only way round this for us is disable config watch via spring.cloud.consul.config.watch.enabled

[dsyer]

Can you debug your app and put a break point in the ConsulPropertySourceLocator constructor. Capture the stack when that is called (after startup), and attach it here. Or we could do that if you send us a simple sample to reproduce the issue. I don't see any calls to the constructor once my sample app is up and running.

[nickcodefresh]

Yep sure, I'll do that tomorrow. It'll have to be the stack, as i don't have the time to create a sample app alas.

[dsyer]

Actually, nevermind about the breakpoint: your analysis already points at the config watch as a problem (but our samples don't get pinged with that X-Consul-Index thing), and you can simulate the same thing just by POSTing to /refresh. What we need is to check your classpath for old versions of stuff, because we don't think there is a leak in the latest spring-cloud-context.

This issue looks like it belongs in spring-cloud-commons, but we can keep pushing on it here till we understand what's going on.

[nickcodefresh]

I'm not sure how you want me to check the classpath, but here's the Maven dependency tree output

output.txt

[dsyer]

Thanks, that's useful, but rules out a classpath issue I think. Still it would be a good idea for you to try with the latest code (Brixton.RELEASE is all but done, so try a snapshot and tell us what happens).

[choppedpork]

You should be able to reproduce a fairly quickly changing X-Consul-Index by using Consul as a docker overlay network backend - we've had it set up with a single (unused) overlay network and I think this is what was causing the constant index increase (since I've removed it the index hasn't changed for 12+ hrs).

edit: you don't even need an overlay network, just connect one docker daemon to Consul using --cluster-store=consul://x.x.x.x:8500/network --cluster-advertise=eth0:2375

[nickcodefresh]

We tried using Spring Cloud Brixton.BUILD-SNAPSHOT but we had the same issue of an ever increasing classes loaded metric.

[dsyer]

Did you try triggering a manual garbage collection? Is this with or without the X-Consul-Index forcing a refresh all the time?

[nissel]

I noticed a similar memory leak, with lots of cglib enhanced MethodProxy classes but using the Eureka, every Eureka registry update / healthcheck created some more classes, one for each registered instance in Eureka.

I was going to raise an issue but the updating to the latest version of spring boot 1.3.6 fixed the problem for me. Not sure if it was 1.3.4 or 1.3.5 where I was seeing the issue.

[spencergibb]

Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.

[dahamr]

We have run into something that may be similar. I've put together a simple service to reproduce.

https://github.com/dahamr/springCloudConsulTest

If I understand the additional information you are requesting:

• yes I forced a manual gc in visualvm before taking the heap dump
• I did not use the /refresh endpoint to reproduce

[spencergibb]

@dahamr can you be more detailed about what you can "reproduce"? Do you see unbounded memory growth that never gets reclaimed?