ivanyu
Joined 11 years ago
Joined 11 years ago
1. The current bring-your-own-key.
2. A central summary storage, filled by me.
3. A central summary storage, crowdsourced.
4. A paid subscription, where I effectively run some LLM proxy.
I wanted something low overhead and be just the right size for yet another weekend project which I could drop at any moment. Supporting some infrastructure, having moderation headaches, let alone receiving payments ruled out pretty much everything but the current approach.
1. After the recent successful growth of Antithesis [1], I'm diving into the topic of deterministic simulation testing. There are some cases (and ready-to-use libraries) where people are doing this in Rust, C++, Go, I'm interesting in this in Java. So I'm up to some experiments. I've also started an "awesome list" of resources about this topic [2]
2. I've made a generated serialization/deserialization library for the Kafka wire protocol in Rust, tested against the original Java implementation. I'll add 3.9 support once it's released and don't see much upcoming changes to the library, apart from maybe working on the Go version.
[2] https://github.com/ivanyu/awesome-deterministic-simulation-t...
PyHeap uses GDB to attach to the target CPython process and inject a piece of Python code that writes the heap dump. The dump will not 100% complete, because the dumper relies only on what it can find in the run time using the standard modules `sys` and `gc` without support from CPython. For example, various native extensions like NumPy must be explicitly supported (not at the moment). However, it's good enough for many use cases.
GDB doesn't need to be installed "near" the target, only "near" the dumper itself. It mean, it will work even with e.g. dockerized Python apps without modification. Linux namespaces are generally supported.
A word on motivation. Java has nice tooling for saving (dumping) the content of the process' heap into a file. A running JVM or a JVM failing with out-of-memory (at least, HotSpot) can do a dump in a portable format. There are standard CLI (jmap, jcmd) and GUI (VisualVM) tools to initiate this [1]. Tools like Eclipse Memory Analyzer [2] or VisualVM [3] can read and query (OQL) these files. Literally see the values of fields, local variables, thread states, etc. This is a super powerful debugging technique. In my career I've figured out many tricky bugs by poring over a heap dump. I miss this debugging experience badly in the Python world. I decided to fill the gap and provide something similar to jmap + Eclipse Memory Analyzer or VisualVM heap dump viewer.
[1] https://www.baeldung.com/java-heap-dump-capture