Just browsing the Quickwit documentation it seems like the general architecture here is to write JSON logs but stores them compressed. Is this just something like gzip compression? 20% compressed size does seem to align to ballpark estimates of JSON GZIP compression. This is what Quickwit (and this page) calls a "document": a single JSON record (just FYI).
Additionally you need to store indices because this is what you actually search. Indices have a storage cost when you write them too.
When I see a system like this my thoughts go to questions like:
- What happens when you alter an index configuration? Or add or remove an index?
- How quickly do indexes update when this happens?
- What about cold storage?
Data retention is another issue. Indexes have config for retention [1]. It's not immediately clear to me how document retention works, possibly from S3 expiration?
So, network transfer from S3 is relatively expensive ($0.05/GB standard pricing [2] to the Internet, less to AWS regions). This will be a big factor in cost. I'm really curious to know how much all of this actually costs per PB per month.
IME you almost never need to log and store this much data and there's almost no reason to ever store this much. Most logs are useless and you also have to question what the purpose is of any given log. Even if you're logging errors, you're likely to get the exact same value out of 1% sampling of logs than you are with logging everything.
You might even get more value with 1% sampling because your query and monitoring might be a whole lot easier with substantially less data to deal with.
Likewise, metrics tend to work just as well from sampled data.
This post suggests 60 day log retention (100PB / 1.6PB daily). I would probably divide this into:
1. Metrics storage. You can get this from logs but you'll often find it useful to write it directly if you can. Getting it from logs can be error-prone (eg a log format changes, the sampling rate changes and so on);
2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;
3. "Offline" data, which you would generally only query if you absolutely had to. This is particularly true on S3, for example, because the write costs are basically zero but the read costs are expensive.
Additionally, you'd want to think about data aggregation as a lot of your logs are only useful when combined in some way
[1]: https://quickwit.io/docs/overview/concepts/indexing
[2]: https://aws.amazon.com/s3/pricing/
Additionally you need to store indices because this is what you actually search. Indices have a storage cost when you write them too.
When I see a system like this my thoughts go to questions like:
- What happens when you alter an index configuration? Or add or remove an index?
- How quickly do indexes update when this happens?
- What about cold storage?
Data retention is another issue. Indexes have config for retention [1]. It's not immediately clear to me how document retention works, possibly from S3 expiration?
So, network transfer from S3 is relatively expensive ($0.05/GB standard pricing [2] to the Internet, less to AWS regions). This will be a big factor in cost. I'm really curious to know how much all of this actually costs per PB per month.
IME you almost never need to log and store this much data and there's almost no reason to ever store this much. Most logs are useless and you also have to question what the purpose is of any given log. Even if you're logging errors, you're likely to get the exact same value out of 1% sampling of logs than you are with logging everything.
You might even get more value with 1% sampling because your query and monitoring might be a whole lot easier with substantially less data to deal with.
Likewise, metrics tend to work just as well from sampled data.
This post suggests 60 day log retention (100PB / 1.6PB daily). I would probably divide this into:
1. Metrics storage. You can get this from logs but you'll often find it useful to write it directly if you can. Getting it from logs can be error-prone (eg a log format changes, the sampling rate changes and so on);
2. Sampled data, generally for debugging. I would generally try to keep this at 10TB or less;
3. "Offline" data, which you would generally only query if you absolutely had to. This is particularly true on S3, for example, because the write costs are basically zero but the read costs are expensive.
Additionally, you'd want to think about data aggregation as a lot of your logs are only useful when combined in some way
[1]: https://quickwit.io/docs/overview/concepts/indexing
[2]: https://aws.amazon.com/s3/pricing/