rraval

We use it in production to pregenerate a pool of certificate requests for SSL key pinning. Rotating a key requires m-of-n cooperation.

CPython in particular has ad-hoc refcounting semantics where references can either be borrowed or stolen and you have to carefully verify both the documentation and implementation of functions you call because it's the wild west and nothing can be trusted: https://docs.python.org/3.9/c-api/intro.html#reference-count...

This ad-hoc borrowed vs stolen references convention bleeds into cffi as well. If you annotate an FFI function as returning `py_object`, cffi assumes that the reference is stolen and thus won't increment the ref count. However, if that same function instead returns a `struct` containing a `py_object`, cffi assumes the reference is borrowed and will increment the ref count instead.

So a harmless looking refactoring that changes a directly returned `py_object` into a composite `struct` containing a `py_object` is now a memory leak.

Memory leaks aren't so bad (even Rust treats them as safe after the leakpocalypse [1] [2]). It's when you go the other way and treat what should have been a borrowed reference as stolen that real bad things happen.

Here's a quick demo that deallocates the `None` singleton:

    Python 3.9.13 (main, May 17 2022, 14:19:07)
    [GCC 11.3.0] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import sys
    >>> sys.getrefcount(None)
    4584
    >>> import ctypes
    >>> ctypes.pythonapi.Py_DecRef.argtypes = [ctypes.py_object]
    >>> for i in range(5000):
    ...     ctypes.pythonapi.Py_DecRef(None)
    ...
    0
    0
    0
    0
    0
    [snip]
    Fatal Python error: none_dealloc: deallocating None
    Python runtime state: initialized

    Current thread 0x00007f28b22b7740 (most recent call first):
      File "<stdin>", line 2 in <module>
    fish: Job 1, 'python3' terminated by signal SIGABRT (Abort)

Previous HN discussion: https://news.ycombinator.com/item?id=25524472

[PDF]: https://www.cs.nott.ac.uk/~pszgmh/fold.pdf

Section 5.1 walks the reader through implementing foldl in terms of foldr after laying down the ground work and gradually introducing things one concept at a time.

For me, the eye-opening insight was using foldr to generate intermediate functions as values, which is the sort of thing "functional" programming does without thinking twice, but is mind bending for someone coming from a more traditional procedural language background.

Motivated by a slightly different use case (seamless syncing across multiple machines), I built a custom tool that solves this concern within `git`: https://github.com/rraval/git-nomad

You can toss `git nomad sync` into a systemd timer (or cronjob if you prefer) and forget about it.

See https://github.com/rraval/zeroindexed/tree/master/packages/t...

The blog explains expanded motivation: https://zeroindexed.com/privacy

Here's the Show HN that never picked up steam: https://news.ycombinator.com/item?id=27175347

By sheer happenstance, I blogged earlier this week about one particular killer feature that doesn't get enough air time: https://news.ycombinator.com/item?id=27344677

My not-flake-yet configuration can be found at https://github.com/rraval/nix

For example I have this alpha quality rust binary that I'm developing but I also want a stable version installed at the OS level. I write a Nix package and simply compose it into my overall NixOS configuration alongside the more official Nixpkgs: https://github.com/rraval/nix/blob/master/git-nomad.nix

> like a source code tarball where you would traditionally run configure && make && make install?

Nix has a bunch of defaults that make a conventional package like this straightforward.

Here's a package for a vanilla C binary + library that does the `autoreconf && ./configure && make && make install` dance: https://github.com/NixOS/nixpkgs/blob/master/pkgs/tools/secu...

It's almost a little misleading because the actual steps are largely inherited from the defaults, you can read more about `stdenv.mkDerivation` here: https://nixos.org/guides/nix-pills/fundamentals-of-stdenv.ht...

I've published a correction: https://zeroindexed.com/nix-ejection-problem#on-gentoo