As others have pointed out in another HN discussion (1), Mongodb's reply is definitely questionable, if only by its tone.
I already replied to metheus on Twitter (2) in a thread where we asked for a way to repro their claims. I found their reply and comments very inappropriate, similar to the comment in here. Arrogant and derogatory to OnGres.
Anyway, I was writing this to note that OnGres has replied to Mongo's reply setting an example of how tech discussions should happen: without derogatory and arrogant comments, open to valid criticism (i.e. with something more than words and numbers that cannot be reproduced) and transparency.
Check it out:
https://ongres.com/blog/benchmarking-do-it-with-transparency...
In there you'll see how Mongo consistently mis-interpreted (or mis-represented?) the results. They kept mixing the benchmarks and constantly talked about an experimental driver and missing connection pooling. In fact, they did use the official Mongo Lua driver and the official Java driver for different benchmarks and they did some of the benchmarks with and without connection pooling and published both results.
It's really sad to see Mongo reply to a thorough benchmark like this. It probably has its flaws but instead of correcting them or publishing a better benchmark like the one they did (to magically get 240x...) they chose to mischaracterize the work of others, spreading FUD and accusing them of cheating and being dishonest.
Hopefully they'll turn around and fix it. All it takes is to publish how they got they amazing numbers so that others can comment, repro or dispute the benchmark.
(1) https://news.ycombinator.com/item?id=20479670
(2) https://twitter.com/javiermaestro/status/1151849279226556417
I already replied to metheus on Twitter (2) in a thread where we asked for a way to repro their claims. I found their reply and comments very inappropriate, similar to the comment in here. Arrogant and derogatory to OnGres.
Anyway, I was writing this to note that OnGres has replied to Mongo's reply setting an example of how tech discussions should happen: without derogatory and arrogant comments, open to valid criticism (i.e. with something more than words and numbers that cannot be reproduced) and transparency.
Check it out: https://ongres.com/blog/benchmarking-do-it-with-transparency...
In there you'll see how Mongo consistently mis-interpreted (or mis-represented?) the results. They kept mixing the benchmarks and constantly talked about an experimental driver and missing connection pooling. In fact, they did use the official Mongo Lua driver and the official Java driver for different benchmarks and they did some of the benchmarks with and without connection pooling and published both results.
It's really sad to see Mongo reply to a thorough benchmark like this. It probably has its flaws but instead of correcting them or publishing a better benchmark like the one they did (to magically get 240x...) they chose to mischaracterize the work of others, spreading FUD and accusing them of cheating and being dishonest.
Hopefully they'll turn around and fix it. All it takes is to publish how they got they amazing numbers so that others can comment, repro or dispute the benchmark.
(1) https://news.ycombinator.com/item?id=20479670
(2) https://twitter.com/javiermaestro/status/1151849279226556417