Comment Comment by arconis987 would like to see work like this, but for datasets in the hundreds of TB or single-digit PBbut i definitely agree about this point> Cluster fatigue is realimo, the concept of “extremely ephemeral query workers” is under-exploredstateless, maintenance-free, burstable fleets of query workers is what I would like to see more of in the future.it’s how we do it, and it gives us full-text search on multi-hundred terabyte data sets in S3, where queries finish in a handful of seconds. our approach: https://docs.scanner.dev/scanner/what-and-why/how-it-works/h...anyone else doing ephemeral query workers fleets? parent arconis987 11 months ago
but i definitely agree about this point
> Cluster fatigue is real
imo, the concept of “extremely ephemeral query workers” is under-explored
stateless, maintenance-free, burstable fleets of query workers is what I would like to see more of in the future.
it’s how we do it, and it gives us full-text search on multi-hundred terabyte data sets in S3, where queries finish in a handful of seconds. our approach: https://docs.scanner.dev/scanner/what-and-why/how-it-works/h...
anyone else doing ephemeral query workers fleets?