startup performance is downgraded significantly when add opentelemetry agent in spring project

[gangxie112]

Describe the bug
A clear and concise description of what the bug is.

We have a quick test against the startup performance ot the opentelemetry agent. It seems that it increases the startup time a lot, almost 1/3. We tested 1.10 and 1.26. And there is no improvement with latest verion 1.26. Is this as expect？And any way to improve it?

ver. | 1 | 2 | 3 | 4 | 5 | avg.
1.10 | 1m22.587s | 1m22.453s | 1m22.978s | 1m18.716s | 1m33.138s | 81.7s
1.26 | 1m28.510s | 1m17.919s | 1m36.579s | 1m27.459s | 1m25.587s | 84.9s
no agent | 1m0.820s | 1m1.225s | 1m5.130s | 1m0.072s | 1m0.041s | 61.4s

Steps to reproduce
If possible, provide a recipe for reproducing the error.

What did you expect to see?
A clear and concise description of what you expected to see.

What did you see instead?
A clear and concise description of what you saw instead.

What version are you using?
(e.g., v0.9.0, 393e4a2, etc)

Environment
Compiler: (e.g., "AdoptOpenJDK 11.0.6")
OS: (e.g., "Ubuntu 20.04")
Runtime (if different from JDK above): (e.g., "Oracle JRE 8u251")
OS (if different from OS compiled on): (e.g., "Windows Server 2019")

Additional context
Add any other context about the problem here.

[mateuszrzeszutek]

Hey @gangxie112 ,

Yes, this is more or less to be expected. During startup the agent has to scan all loaded classes, and if any of them matches any of the instrumentations included in it, transform them. This is very computationally intensive, and the bigger your app is the more work the javaagent has to do.

You can expect the startup times to improve a bit after #8525 gets merged, but they will always be noticeably larger than without instrumentation.

[gangxie112]

Hey @mateuszrzeszutek ,
Thanks for the quick response. I noticed an inpromenet which also named "Faster type matching" 5724, which is introduced in 1.13. Doesn't it resolve the problem? What's the difference? do I miss something?

[mateuszrzeszutek]

Doesn't it resolve the problem? What's the difference? do I miss something?

It did make the matchers a bit faster, but it is simply not possible to "resolve the problem". The agent will always add some startup time, we can make some optimizations (which we do) but it's never going to be any close to 0.

[gangxie112]

Another idea is to allow the user to do "delayed init", like after the spring app is up. Afterall, startup time is much more sensitive and critical than the tracing info.

[mateuszrzeszutek]

Another idea is to allow the user to do "delayed init", like after the spring app is up.

That's not feasible, for multiple reasons. E.g. we don't really know when the app is up (the javaagent can be used to instrument more than just spring apps); retransforming/redefining classes has a set of restrictions that do not apply when transforming on class load.

[laurit]

@gangxie112 Could you try with the latest snapshot from https://oss.sonatype.org/content/repositories/snapshots/io/opentelemetry/javaagent/opentelemetry-javaagent/1.27.0-SNAPSHOT/ and let us know whether there is any improvement in startup time

[breedx-splk]

Another idea is to allow the user to do "delayed init", like after the spring app is up. Afterall, startup time is much more sensitive and critical than the tracing info.

There are definitely users who want/need telemetry during early application lifecycle and would NOT want it deferred. Just different use cases for different users IMO.

[trask]

Linking to related discussion / decision about supporting deferred / dynamic "attach": #1932

[dreaminghk]

In my use case, we have been looking into using the auto-instrumentation. However, given some of the applications are deployed with less cpu resources (<2 cores). We observe significant startup overhead and we cannot use it.

If delay-init can be supported, it will be very great.

[trask]

@dreaminghk just checking that you've tried with 1.27.0? some people have seen good improvement with that version due to #8525

[dreaminghk]

@trask Not yet. I have planned to try that soon. Will let you know if it helps!

[dreaminghk]

Hi @trask ! The followings are the testing result. There are performance improve with default settings with java agent 1.26. We get the best performance with java agent 1.25 plus only enable 3 libraries to get auto-instrumentation. Our use case is small size container, maybe auto-instrumentation do not fit in this case.

====================================================
Testing environment:
EKS
CPU: 300m-500m
RAM: 1G

JDK17 with -XX:InitialRAMPercentage=75 -XX:MaxRAMPercentage=75 -XX:MinRAMPercentage=75

====================================================

Test result (startup time):

No auto-instrument => 30s

With java agent 1.25:

	Auto-instrument => 91s
	Auto-instrument  (OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false, OTEL_INSTRUMENTATION_MONGO_ENABLED=true, OTEL_INSTRUMENTATION_JERSEY_ENABLED=true, OTEL_INSTRUMENTATION_JETTY_ENABLED=true)=> 50s

With java agent 1.26

	Auto isntrument => 62s
	Auto-instrument  (OTEL_INSTRUMENTATION_COMMON_DEFAULT_ENABLED=false, OTEL_INSTRUMENTATION_MONGO_ENABLED=true, OTEL_INSTRUMENTATION_JERSEY_ENABLED=true, OTEL_INSTRUMENTATION_JETTY_ENABLED=true)=> 61s

[trask]

@dreaminghk what about 1.27? that's the release that has #8525