liliuleo93
Joined 3 years ago
Karma
12
Submissions
20
Member since
Apr 6
Recent submissions
Comment · 3 years ago
Comment · 3 years ago
I apologize for the repeated posts. The reason I posted it 6 times is that I aim to announce every release and significant commit. The reason my entire history is centered around this benchmark is because I wanted to introduce my project to the community, potentially with some bias. I began my Hacker News journey at that time and wanted to share what I was working on.
Comment · 3 years ago
Fully agree with this idea. All tricks can be a real world strategy and it is impossible for anyone to claiming that they have an absolute fair benchmark.
So the only way we can do to approaching it is to provide more real-world-like cases and forget all tricks vendors might play inside their systems.
Also, people will concern the representative of the cases benchmarks provide. So we plan to make this benchmark more like a framework to support customized cases in the next step.
Comment · 3 years ago
Yes, of course vendors have bias. But IMHO, if a benchmark is reproducible and the use cases can match users' needs, then we can say it can somehow help decision making.
Comment · 3 years ago
Plz feel free to try the benchmark tool and make an issue with more details if it is not reproducible. Then we can dig it to see if there is any misusage or bug in the benchmark code. :)
" With respect to recall vs Performance, your idea is indeed correct. However, several reasons have guided us to our current approach:
1. We are not solely benchmarking open-source systems; we are also focusing on cloud services. Some of these services, such as Zilliz and Pinecone, don't allow users to customize their parameters to tune the recall, aiming to simplify their usage. Consequently, creating a recall vs Performance graph is not feasible. Also this benchmark allow users to customize their parameters for systems allowing tuning to get their own result to do comparison.
2. There already exists a number of benchmarks doing what you've suggested, which target individuals with ANN search backgrounds. Our goal is to make this benchmark as straightforward as possible and to assist people who lack understanding about the inner workings of each system.
3. Concerning reproducibility, generating a recall vs QPS graph that you mentioned, would require conducting a multitude of tests to obtain enough data points, which considerably reduces reproducibility. "
the link is: https://github.com/zilliztech/VectorDBBench/issues/200#issue...