rkochanowski

Thanks, I will look at this in more detail.

To give more context, the current version is already an optimized one I considered good enough and didn't spend more time on it. In the first attempt, I used a vector database with indexes, trying to query for similar vectors. This was uselessly slow even in medium-sized repos. The brute-force NumPy solution is a significant improvement, making it faster than other calculations like clustering.

"vectors are not loaded into memory at once" is not true, I had in mind splitting computation into blocks.

One possible simple optimization is to use 16-bit floats in vector instead of 32-bit. I used this in a different project (halfvec in pgvector) without affecting results.

1536-dim embeddings from your case also can be reduced. This large vector usually doesn't give much benefit compared to smaller ones. And this is something I will compare in my own tests.

Do you mean to verify if similar code units produce the same result?

The goal of the tool is to also detect code that is similar and behaves differently. There are not ideal duplicates, but still a code that can be refactored, abstracted, or fixed (because the variance may be the result of bug).

PHP was added in the latest release.

Elixir was added in the latest release.

Function level only. I will add more granular chunking in next versions.

It's the opposite.

jscpd is advertised as "Copy/paste detector", Slopo is advertised as "non-exact code duplication".

Slopo also detects copy/pasted code, but this is not the main goal and the report focuses more on similar code units.

Finding similar code is something different than deduplication, even when the final goal looks similar.

Deduplication backend is the easiest part of the tool and it doesn't need any additional libraries. Just calculate embeddings and find close pairs. The complexity is everything around.

Using local models is worth considering and the tool already uses the LiteLLM wrapper, allowing it to configure different models, including local. I left this part for the user.

My solution for false positives is simpler:

1. The tool uses only cosine similarity plus boost depending on distance in the codebase.

2. Classification with LLM. This can be done by coding agent used with project giving better results than integrating this pass in the tool. LLMs used for coding are pretty good.

I assumed that this is not a problem I need to solve inside the tool. I'm aware this is not deterministic, but this is by design.

Regarding information about raw similarity: currently, the score (raw similarity + boost) is visible in the report, so this value can be configured based on data. The raw similarity threshold can also be configured, but it's not displayed. I will think about how to handle this.

I just focused on embeddings without comparing them to deterministic solutions.

But I plan to do my own analysis of different embedding models in the context of code similarity detection. Including BM25 in the comparison is a very good idea.

Currently, only whole functions (including function-like constructs depending on language) are considered as unit.

Skipping the extraction of conditional branches was my decision to not overcomplicate the first versions, which was intended to validate the idea. I will add this in future versions because I agree it's needed for large functions.

I don't think it needs configurable granularity. In the current version, there is an analogous mechanism: when functions are nested, both outer and inner are embedded separately. When both are similar to each other, this pair is excluded. Inner or outer functions can appear in results depending on similarity to other units.

Regarding comments, they are removed and I will think about handling them. The challenge is not with extraction, but with how to present this in a report. This may be a nice addition because coding agents often add comments.

Thanks for the feedback.