"it works fine and this is not a bottleneck. For large projects, other parts are much slower": I don't think this is true, especially for large projects. I just ran your tool on the Kubernetes repo with 1536-dim embeddings. The isolated similar-pair search took ~130s and peaked at ~4.8 GB RSS, and the total runtime was ~250s with the same memory peak.
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.
"In larger sets, all vectors are not loaded into memory at once": this is also not correct, at least in the implementation you shared. The similarity matrix is processed in blocks, but the embeddings themselves are loaded all at once and stacked into one NumPy matrix, hence the memory peak.
So in larger projects, more than half of the time is spent on the similarity search step, and almost all the memory is spent there as well.