Why agent cost half time in flame graph?

[wangsikuan]

My environment:
macOS Bigsur
go1.13.1

When I add agent code in my go program, I found something strange.

pyroscope agent exactly cost half time

Maybe something went wrong or there is some knowledge I don't get?

[petethepig]

This is interesting. Could you share your program with us?

[wangsikuan]

https://gist.github.com/wangsikuan/4ac234cbb8c45d3f187ad81e281bc03e

OK, I put my code in this gist.

[petethepig]

Thanks for the report @wangsikuan!

I was able to reproduce the issue on go1.13.1, but not on the later versions.

Background

Our profiler looks at the current stacktrace of each goroutine 100 times per second. When it does that it inevitably sees the goroutine that this is happening in. We don't want to include this information in the final profiler output, so we have a list of functions we exclude from the profile. One of them is runtime.GoroutineProfile. For some reason, this function is not included in the stacktrace we get on go1.13.1.

Steps taken to fix this

The fix turned out to be pretty simple: a565c52. I just added (s *GoSpy) Snapshot to the exclude list.

Further steps

@wangsikuan it would be helpful if you could confirm this works:

go get -u github.com/pyroscope-io/pyroscope/pkg/agent/profiler
go run issue-50.go

[petethepig]

I'm gonna close this for now, but feel free to re-open

[wangsikuan]

Hi, thanks for your explaination!

When I run the command, I got same result.

go get -u github.com/pyroscope-io/pyroscope/pkg/agent/profiler

And I checked my go pkg, I found agent pkg already updated, like below

➜  agent pwd
/Users/solmyr/gopath/pkg/darwin_amd64/github.com/pyroscope-io/pyroscope/pkg/agent
➜  agent ll
total 40
-rw-r--r--  1 solmyr  staff    18K  2  2 21:23 profiler.a

I know that upgrade my go version would solve it, but I just curious what I am doing wrong, because look like you already solved it.

And I am willing for contributing some work for this project.

Thanks

[petethepig]

We changed the way the profiler works in v0.0.30 so this is probably no longer relevant