Nice! Thank you for doing this. I am not surprised PCA holds up. It comes with some pretty strong theoretical guarantees. The problem with it has always been scalability. The O(n^3) complexity makes it infeasible to use for massive datasets.
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.
That is where something like Matryoshka embeddings has appeal. You trade a little bit of performance for a guarantee of training + validation set coverage.