felixge
Joined 17 years ago
Karma
1,709
Submissions
295
Member since
Dec 7
Recent submissions
Comment · 7 months ago
We had reached out to y'all last year to explore taking ideas from your format. It definitely looks interesting! But IIRC nobody from your team ended up making it to one of our SIG meetings?
Comment · 7 months ago
OTel Profiling SIG maintainer here: I understand your concern, but we’ve tried our best to make things efficient across the protocol and all
involved components.
Please let us know if you find any issues with what we are shipping right now.
Comment · 1 year ago
Do you think the GC roots alone (goroutine stacks with goroutine id, package globals) would be enough?
I think in many cases you'd want the reference chains.
The GC could certainly keep track of those, but at the expense of making things slower. My colleagues Nick and Daniel prototyped this at some point [1].
Alternatively the tracing of reference chains can be done on heap dumps, but it requires maintaining a partial replica of the GC in user space, see goref [2] for that approach.
So it's not entirely trivial, but rest assured that it's definitely being considered by the Go project. You can see some discussions related to it here [3].
Disclaimer: I contribute to the Go runtime as part of my job at Datadog. I can't speak on behalf of the Go team.
[1] https://go-review.googlesource.com/c/go/+/552736
[2] https://github.com/cloudwego/goref/blob/main/docs/principle....
Comment · 1 year ago
I'd love to hear more! What kind of profiling issues are you running into? I'm assuming the inuse memory profiles are sometimes not good enough to track down leaks since they only show the allocation stack traces? Have you tried goref [1]?. What kind of memory pressure issues are you dealing with?
[1] https://github.com/cloudwego/goref
Disclaimer: I work on continuous profiling for Datadog and contribute to the profiling features in the runtime.
Comment · 1 year ago
Hacking into the Go runtime with eBPF is definitely fun.
But for a more long term solution in terms of reliability and overhead, it might be worth raising this as a feature request for the Go runtime itself. Type information could be provided via pprof labels on the allocation profiles.
Comment · 1 year ago
+1. In particular []byte slice allocations are often a significant driver of GC pace while also being relatively easy to optimize (e.g. via sync.Pool reuse).
Comment · 2 years ago
This is great, thank you.
Comment · 2 years ago
I'm skeptical that it's worth it myself, this was just a fun research project for me. But once hardware shadow stacks are available, I think this could be great.
To answer your first question: For most Go applications, the average stack trace depth for profiling/execution tracing is below 32 frames. But some applications use heavy middleware layers that can push the average above this limit.
That being said, I think this technique will amortize much earlier when the fixed cost per frame walk is higher, e.g. when using DWARF or gopclntab unwinding. For Go that doesn't really matter while the compiler emits frame pointers. But it's always good to have options when it comes to evolving the compiler and runtime ...
Comment · 2 years ago
That seems to be windows only? My main target OS is Linux.
Comment · 2 years ago
Thank you so much, this is very helpful and interesting. I'll try to experiment with this at some point.
Blog: https://blog.felixge.de/ Old Blog: http://felixge.de/ Twitter: http://twitter.com/felixge GitHub: http://github.com/felixge