Skip to content

Conversation

@jamescrosswell
Copy link
Collaborator

@jamescrosswell jamescrosswell commented Sep 19, 2023

#skip-changelog

Overview

When load testing an ASP.NET Core application that creates a lot of spans, GetLastActiveSpan was responsible for a significant proportion of memory allocation:

                    // The endpoint being load tested
                    endpoints.MapGet("/spans", async context =>
                    {
                        for (int i = 0; i < 1000; i++)
                        {
                            var span = SentrySdk.GetSpan()?.StartChild($"child {i}");
                            span?.Finish();
                        }
                        await context.Response.WriteAsync("Hello Spans!");
                    });

image

This appears to be due to the call to Spans.OrderByDescending:

Spans.OrderByDescending(x => x.StartTimestamp).FirstOrDefault(s => !s.IsFinished);

Iteration 1

By replacing this with a for loop, memory consumption in our benchmark dropped by 65.68% from 2109.85 KB to 724.02 KB

Before

Method SpanCount Mean Error StdDev Gen0 Gen1 Allocated
'Create spans for scope access' 1 45.39 us 13.71 us 0.752 us 5.6152 1.7700 19.77 KB
'Create spans for scope access' 10 205.46 us 276.93 us 15.179 us 23.9258 4.3945 85.51 KB
'Create spans for scope access' 100 4,055.16 us 5,006.05 us 274.399 us 601.5625 85.9375 2109.85 KB

After

Method SpanCount Mean Error StdDev Gen0 Gen1 Allocated
'Create spans for scope access' 1 52.05 us 242.80 us 13.309 us 5.2490 1.6479 18.92 KB
'Create spans for scope access' 10 164.61 us 55.00 us 3.015 us 12.2070 4.1504 63.88 KB
'Create spans for scope access' 100 2,287.86 us 1,098.36 us 60.205 us 183.5938 50.7813 724.02 KB

Iteration 2

By instead keeping all the spans on a Stack and inspecting the most recent one(s) to return the last active span when required we were able to reduce memory usage for the 100 spans scenario drops by a further 63.25% from 724.02 KB to 266.09 KB. So about an ~87% reduction vs the original benchmarks.

After

Method SpanCount Mean Error StdDev Gen0 Gen1 Allocated
'Create spans for scope access' 1 28.45 us 13.270 us 0.727 us 4.6387 1.5869 16.64 KB
'Create spans for scope access' 10 84.98 us 6.361 us 0.349 us 9.3994 2.5635 39.17 KB
'Create spans for scope access' 100 657.54 us 54.581 us 2.992 us 57.6172 16.6016 266.09 KB

@jamescrosswell jamescrosswell changed the title Cache sorted spans in the TransactionTracer Memory optimisation for GetLastActiveSpan Sep 20, 2023
@jamescrosswell
Copy link
Collaborator Author

Important Note

I merged main into this branch and annoyingly it seems to contain a bunch of the changes from #2637... I'm not sure why but it makes this PR really hard to review.

As such, I created a new branch and PR for these changes... will close this PR in favour of that branch.

See #2642 instead.

@jamescrosswell jamescrosswell deleted the last-active-span-memory branch November 22, 2023 19:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants