Funny, I did quite a deep dive on this same issue about two years back, and came to exactly the opposite conclusion: keep ASDF, and lean on Nix to do pinning. I now maintain my own Common Lisp package repository, testing daily against nightly SBCL etc, and I've learned that a lot of CL stability is just because test suites are never run. Nix goes a long way to shield me from the haphazard breakages that other commenters mentioned, but of course it comes with a huge downside: you need to use Nix :)
(just kidding)
Code at https://github.com/hraban/cl-nix-lite if someone is interested.
Obviously since this was written by Fukamachi you know it will be good. Much respect to the man.