mvijaykarthik

We store these files on S3 and directly read them in the query path using lambda functions. Parquet would work as a storage format for logs and traces, but not for metrics. At oodle we built the metrics engine first, so we designed a hybrid row/column format with a more S3 friendly indexing strategy - fewer lookups required to locate relevant chunks within the file. The hybrid row/column format was necessary to store multiple "samples" in the same row. This also allowed us to get compression ratios in the order of 600x.

When we extended the engine to logs and traces, we took the approach of tweaking the same file format so that we could re-use a majority of the engine we had already built.

We price per GB of ingested trace - $0.3/GB, goes lower as you scale.

Traditional APM spans are ~2KB - this would come to ~$0.6/million spans.

How we arrived at $10 is that we assumed each agent span is ~35KB. Which means 1M spans is ~35GB - which comes to ~$10. Agent spans are larger because they have large prompts.

If your LLM spans are smaller, the cost would be lower. For our customers we see this vary between 20-30KB, and we took a conservative 35KB number.

1. We store high cardinality metrics in object storage and only keep most recent data in memory. We use serverless functions (lambdas) for querying these objects which helps us scale.

2. We do not have a global index - each hour of metrics have their own index. This helps with ephemeral labels as size of the index remains small

Thanks for your feedback, and great questions. 1. We create serverless functions to process each file and then combine the results, optimized for columnar file formats. 2. This is one of our core innovations :) We created custom representations of WALs which help us with query performance and ingesting them quickly. 3. Once a WAL is ingested, it is available for query within a few seconds. So far it has been reliable and we have not had issues with missing data.

We are reasonably close to the latest version of Grafana. We periodically pull in new changes.

We are still early in our journey, and are currently working on leveraging LLMs for incidents and query / dashboard generation.

We do use pre-LLM-era AI and statistical analysis to provide insights and auto create dashboards for alerts (currently in alpha).

Technically you are correct, the scraper will still exist. However, the hard part is scaling the query and storage layers which we replace.

We use a local disk to temporarily stage data before putting it on S3. We have smaller WAL (write ahead log) objects, and a periodic compaction process which creates read-optimized files on S3.

Great question! Vijay here, I'm one of the co-founders of Oodle. Compared to thanos 1. We use object store (S3) for all queries - even recent time ranges. Object store is not just an archival solution 2. Customized indexing to minimize memory usage. Index is also on object storage. 3. Custom columnar file format optimized for storing metrics on object storage 4. Serverless functions for achieving good query performance. This helps us break down and parallelise queries without impacting cost with pre-provisioned compute. 5. No downsampling. Downsampling is not required to improve query performance or reduce costs with serverless and object storage

About Oodle:

Oodle is a fully managed, serverless metrics observability platform. It is a drop-in replacement for Prometheus that delivers fast performance at high scale. Oodle is 3x cheaper than Managed Prometheus and 10x cheaper than other commercial vendors.