francoismassot

it's tantivy :)

Indeed. They benefit from a discount, but we don't know the discount figure.

To further reduce the storage costs, you can use S3 Storage Classes or cheaper object storage like Alibaba for longer retention. Quickwit does not handle that, so you need to handle this yourself, though.

They have 181 trillion logs

Good question.

Let's estimate the costs of compute.

For indexing, they need 2800 vCPUs[1], and they are using c6g instances; on-demand hourly price is $0.034/h per vCPU. So indexing will cost them around $70k/month.

For search, they need 1200 vCPUs, it will cost them around $30k/month.

For storage, it will cost them $23/TB * 20000 = $460k/month.

Storage costs are an issue. Of course, they pay less than $23/TB but it's still expensive. They are optimizing this either by using different storage classes or by moving data to cheaper cloud providers for long term storage (less requests mean you need less performant storage and usually you can get a very good price on those object storages).

On quickwit side, we will also improve the compression ratio to reduce the storage footprint.

[1]: I fixed the num vCPUs number of indexing, it was written 4000 when I published the post, but it corresponded to the total number of vCPUs for search and indexing.

But you don’t have fast search on those files stored on object storage.

Well, MongoDB was under AGPL v3.0 :)

This is awesome; we need this kind of alternative to overpriced software like Splunk. We built and open-sourced Quickwit to see this kind of tool built on top of it.

We will follow Tracecat closely. I'm convinced this will impact our roadmap, and I'm happy to receive any feedback so you can get the most out of Quickwit for Tracecat.

Good luck, guys!!!

tantivy, not tantivity!!!!!