Was this before BigQuery/Presto/Trino? To me it seems like those technologies would have been a good fit.
They don't really work with indexes but instead regular files stored in partitions (where date is typically one of them).
This means that they only have to worry about the data (e.g. dates) that you are actually querying. And they scale up to the number of CPUs that particular calculation needs. They rarely choke on big query sizes. And big tables are not really an issue as long as you query only the partitions you need.
They don't really work with indexes but instead regular files stored in partitions (where date is typically one of them).
This means that they only have to worry about the data (e.g. dates) that you are actually querying. And they scale up to the number of CPUs that particular calculation needs. They rarely choke on big query sizes. And big tables are not really an issue as long as you query only the partitions you need.