There is a lot of detail here, but the higher-level takeaway is this: the Python C API is a very "wide" API. It exposes lots of details of CPython: the good, the bad, and the ugly. You get direct access to pointers to the underlying objects. You manually manage reference counts.
For the polar opposite of this, consider the Lua API. You don't get pointers to any VM data structures except a single pointer to the "Lua state". You do not perform any manual memory management.
Lua's approach has yielded amazing results. LuaJIT is not only source-compatible with pre-existing Lua 5.1 extensions, it is binary compatible with them. You can take a .so that you built before LuaJIT ever existed and it will work with LuaJIT without recompiling. This is astounding to me.
Moral of the story: keep your interfaces as narrow and encapsulated as possible!
For the polar opposite of this, consider the Lua API. You don't get pointers to any VM data structures except a single pointer to the "Lua state". You do not perform any manual memory management.
Lua's approach has yielded amazing results. LuaJIT is not only source-compatible with pre-existing Lua 5.1 extensions, it is binary compatible with them. You can take a .so that you built before LuaJIT ever existed and it will work with LuaJIT without recompiling. This is astounding to me.
Moral of the story: keep your interfaces as narrow and encapsulated as possible!