The issue that I frequently run into is not that I'm looking for a fancy distributed/sharded database because of reasons of performance, but because I need to store large amounts of data in a way that allows me to grow this datastore by "just adding boxes" while still retaining a few useful database features. I'd love to use Postgres but eventually my single server will run out of disk space.
Now, one approach is to just dismiss this use-case by pointing at DynamoDB and similar offerings. But if for some reason you can't use these hosted platforms, what do you use instead?
For search, ElasticSearch fortunately fits the bill, the "just keep adding boxes" concept works flawlessly, operating it is a breeze. But you probably don't want to use ElasticSearch as your primary datastore, so what do you use there? I had terrible experiences operating a sharded MongoDB cluster and my next attempt will be using something like ScyllaDB/Cassandra instead since operations seem to require much less work and planning. What other databases would offer that no-advance-planning scaling capability?
Somewhat unrelated, by I often wonder what one were to use for a sharded/distributed blob store that offers basic operations like "grep across all blobs" with different query-performance than a real-time search index like ElasticSearch. Would one have to use Hadoop or are there any alternatives which require little operational effort?
Now, one approach is to just dismiss this use-case by pointing at DynamoDB and similar offerings. But if for some reason you can't use these hosted platforms, what do you use instead?
For search, ElasticSearch fortunately fits the bill, the "just keep adding boxes" concept works flawlessly, operating it is a breeze. But you probably don't want to use ElasticSearch as your primary datastore, so what do you use there? I had terrible experiences operating a sharded MongoDB cluster and my next attempt will be using something like ScyllaDB/Cassandra instead since operations seem to require much less work and planning. What other databases would offer that no-advance-planning scaling capability?
Somewhat unrelated, by I often wonder what one were to use for a sharded/distributed blob store that offers basic operations like "grep across all blobs" with different query-performance than a real-time search index like ElasticSearch. Would one have to use Hadoop or are there any alternatives which require little operational effort?