Statistical Profiling in Python, Part 3: Speaking pprof
The first two posts in this series were about collecting data: CPU and wall-clock samples from a signal handler, and heap samples from inside the allocator. Both produce a pile of stack traces with counts attached.
A pile of stack traces is not an answer. To be useful it has to be reachable from outside the process, and it has to arrive in a format that something can draw. Neither of those is intellectually interesting, and both are the reason the project was worth doing at all — so this last post is about the plumbing.
pypprof adds HTTP endpoints to a running Python application, modeled directly on Go’s net/http/pprof.
Copying an interface on purpose
Go’s profiling story is good for one reason above all others: a Go service already has the endpoints. You do not add a dependency, redeploy, or reproduce the problem locally. You point a tool at a process that is misbehaving right now and ask it questions.
from pypprof.net_http import start_pprof_server
start_pprof_server(port=8081)
That starts a small HTTP server on a daemon thread, serving:
| Endpoint | What it gives you |
|---|---|
/debug/pprof/profile?seconds=30 | CPU profile |
/debug/pprof/wall?seconds=30 | Wall-clock profile |
/debug/pprof/heap?gc=1 | Live heap snapshot |
/debug/pprof/thread?debug=1 | Every thread’s stack |
/debug/pprof/goroutine | The same thing, under a name Go’s tools expect |
/debug/pprof/cmdline | The process’s argv |
The paths, the query parameters, and the ?debug=1 plain-text mode are all copied from Go rather than designed. That is the point. Every one of them is a detail that some existing tool already knows, and matching them means:
$ go tool pprof -http=:8088 :8081/debug/pprof/profile
$ go tool pprof :8081/debug/pprof/heap
$ curl localhost:8081/debug/pprof/thread?debug=1
works against a Python process. The flame graph, the call graph, the source view, the diff between two profiles — none of it had to be written, because the Python service is now indistinguishable from a Go one as far as go tool pprof is concerned. The goroutine alias is the most shameless example: Python has no goroutines, but tools go looking for that path, and answering with thread stacks is more useful than being pedantic.
The seams
A few details in the implementation are worth pulling out, because they are the kind of thing that only shows up once the pieces have to work together.
Signals have to be registered from the main thread. Python will only let you call signal.signal() from the main thread, and the wall profiler needs a SIGALRM handler. But the profiling server runs on a background thread, and by the time a request arrives it is far too late to install one. So start_pprof_server registers the handler eagerly, before spawning the server thread, purely so the capability exists later:
# Signal handlers can only be registered on the main thread.
# So do it now before spawning the background thread.
_wall_profiler.register_handler()
This has a side effect worth knowing about, which is that importing pypprof takes over SIGALRM for the process whether or not you ever request a wall profile. That is an unfriendly thing for a library to do, and the alternative — failing at request time with an error the user cannot act on — is worse.
Heap profiling can fail in a way that is the user’s fault. The heap endpoint needs mprofile installed and tracing already started, since a sampling heap profiler can only report allocations it was present for. You cannot retroactively enable it and get an answer. Both cases return 412 Precondition Failed with a message saying which one it is, rather than an empty profile that looks like a program using no memory.
The thread dump needs no machinery at all. sys._current_frames() returns a frame for every running thread, and formatting them is a dozen lines of pure Python. It is by far the least sophisticated thing in the three repositories and quite possibly the one I have used most, because “what is every thread doing right now” answers a large fraction of production questions — particularly the ones where the answer is “all forty of them are waiting on the same lock.”
The format is the product
All four profile types converge on the same output: a gzipped pprof Profile protobuf. A shared builder interns strings, functions, and source locations, assigns them stable IDs, and emits samples referring to those IDs. Each profile type differs only in what it puts in the value fields — nanoseconds for CPU and wall, bytes and object counts for the heap, a count of one per thread for thread dumps.
Writing a protobuf builder is dull work. It is also the single highest-leverage part of the project, and it is worth being explicit about why, because the lesson generalizes well beyond profiling.
Each of these three components is individually modest. A fork of Google’s sampler with the cloud bits removed. An adaptation of heapprof onto CPython’s allocator API. A few hundred lines of BaseHTTPRequestHandler. What makes them add up to something is that the output speaks a format with an ecosystem behind it. I did not build a UI, a storage layer, a diffing tool, a flame-graph renderer, or a symbol browser, and users of these libraries get all of them.
The alternative — a bespoke format and a small custom viewer — would have been more fun to write and would have produced a tool nobody used, including me. Choosing to be compatible instead of novel is usually the higher-value decision, and it is almost always the less enjoyable one.
Update: Python caught up
Python has not stood still since 2019, and the gap I was writing about has largely closed from the other direction.
tracemalloc, introduced by PEP 454, remains the standard-library answer for tracing Python allocations and is still the right tool when exact allocation histories matter. PEP 669, “Low Impact Monitoring for CPython”, added a much cheaper monitoring interface for tools that need interpreter events without turning every line into a callback — the thing that made cProfile-style tooling so expensive.
The bigger change is statistical execution profiling arriving in the standard library. Python 3.15 adds the profiling package and profiling.sampling, a built-in sampler that can attach to an already running process and produce pstats, flame graphs, and other views. That is squarely the problem this series was about, solved properly, by people with the ability to change the interpreter rather than work around it — which is a much better place to solve it from. It addresses where time goes rather than where heap bytes live, so mprofile is not entirely redundant, but I would reach for the built-in tools first now.
There is a satisfying symmetry in that. I wrote these libraries because Go had production-friendly profilers and Python did not, and the workarounds they required — racy reads of interpreter internals, a documented list of Python versions where wall profiling might break your Ctrl-C — were all symptoms of being outside the interpreter looking in. Python has since grown its own, and the hacks can be retired.
The engineering question underneath has not changed at all: collect enough information to explain a real program, while disturbing that program as little as possible. Everything in these three posts is a consequence of taking that second clause seriously.
