We store these files on S3 and directly read them in the query path using lambda functions. Parquet would work as a storage format for logs and traces, but not for metrics. At oodle we built the metrics engine first, so we designed a hybrid row/column format with a more S3 friendly indexing strategy - fewer lookups required to locate relevant chunks within the file. The hybrid row/column format was necessary to store multiple "samples" in the same row. This also allowed us to get compression ratios in the order of 600x.
When we extended the engine to logs and traces, we took the approach of tweaking the same file format so that we could re-use a majority of the engine we had already built.
When we extended the engine to logs and traces, we took the approach of tweaking the same file format so that we could re-use a majority of the engine we had already built.