benfrederickson
Joined 16 years ago
Karma
1,767
Submissions
66
Member since
Jun 22
Recent submissions
Comment · 5 months ago
Comment · 5 years ago
Sorry about leaving your program suspended - I believe that was because of a bug that should be fixed now https://github.com/benfred/py-spy/issues/390 =(
Comment · 6 years ago
> you can either race with the program and hope that you read its memory to get the function stack before it changes what function it's running (and you're likely to win the race, because C is faster than Python), or you can pause the program briefly while taking a sample.
Py-spy defaults to blocking because the results can be pretty wrong otherwise: https://github.com/benfred/py-spy/issues/56 . You can see this problem profiling a program like https://github.com/benfred/py-spy/blob/master/tests/scripts/... with or without the nonblocking flag in py-spy - the nonblocking version produces garbage output.
Somewhat interestingly, this problem doesn't seem to occur with Ruby - and rbspy can get away without pausing the target program with only minor errors seen when profiling a similar function. I suspect this is because of differences between how the Ruby and Python interpreters store call stack information, but haven't had a chance to dig into the specifics.
Comment · 6 years ago
Interesting article. While I definitely think you should be profiling your code to figure out the hot spots, cProfile has some limitations for profiling: cProfile doesn't give you line numbers, doesn’t work with threads, and significantly slows your program down.
I wrote a tool py-spy (https://github.com/benfred/py-spy) that is worth checking out if you’re interesting in profiling python programs. Not only does it solve those problems with cProfile - py-spy also lets you generate a flamegraph, profile running programs in production, works with multiprocess python applications, can profile native python extensions etc.
Comment · 7 years ago
Author here. It's worth noting that since I wrote this post, py-spy has gained the ability to profile multiprocess python applications - and can also now show local variables in the dump command.
Comment · 8 years ago
I wrote something that will get you the python interpreter stack from any running cpython process : https://github.com/benfred/py-spy/ , and rbspy can do the same for ruby https://github.com/rbspy/rbspy
Comment · 8 years ago
the name pyspy was taken on pypi already : https://github.com/tdfischer/pyspy =)
Comment · 8 years ago
Thanks! Both of your suggestions totally make sense. I've created an issue to track the poll() issue here https://github.com/benfred/py-spy/issues/13 - I think that should be an easy fix.
Comment · 8 years ago
Not yet - but I'm hoping to have a version that supports this next week. Will update this issue when it's done: https://github.com/benfred/py-spy/issues/3
One thing to note is that there are some differences in blocking behaviour of the target process. Py-spy blocks by default and profiling.sampling doesn't. I wrote a bit about why py-spy blocks by default here https://www.benfrederickson.com/why-python-needs-paused-duri... - the first version of py-spy also didn't block and since we got incredibly misleading results at times this was one of the first changes I made to py-spy