> I've had people tell me all of the extra bloat/slowness in modern software is because it has more features
Lately I've been making something of a hobby of replacing big scale-out Apache Spark ETL jobs with stuff that was hand-coded to use something other than Spark. Generally I'm finding that, with a little care, I can get a lot of these jobs done in less time on a single machine than I can running it on the Spark cluster. (Using code that's easier to reason about, too.) And even when I can't, I'm not sure it's enough slower to warrant the use of the cluster, on the principle that using half a rack worth of compute resources to get the job done in 45 minutes when you could do it in 60 on a single box isn't really a good tradeoff.
All that long-windedness by way of saying, these big data tools are kind of slow and bloated. They're slow and bloated precisely because they have more features. All the extra work they need to do to achieve reliability when you're scaling out has a cost. But that doesn't necessarily mean that the cost is justifiable.
A box truck is more capable for running errands than my Kia Soul, because I can fit half the grocery store in it. But I think we'd all agree that it would be an obscene waste of money to default to using a box truck to buy groceries and make Ikea runs, and justify that decision by pointing out that some small fraction of the time I'll need to transport more stuff than will fit in a Kia Soul, and it's better to standardize on a single model of car. In a computing setting, though, we do that sort of thing all the time.
Lately I've been making something of a hobby of replacing big scale-out Apache Spark ETL jobs with stuff that was hand-coded to use something other than Spark. Generally I'm finding that, with a little care, I can get a lot of these jobs done in less time on a single machine than I can running it on the Spark cluster. (Using code that's easier to reason about, too.) And even when I can't, I'm not sure it's enough slower to warrant the use of the cluster, on the principle that using half a rack worth of compute resources to get the job done in 45 minutes when you could do it in 60 on a single box isn't really a good tradeoff.
All that long-windedness by way of saying, these big data tools are kind of slow and bloated. They're slow and bloated precisely because they have more features. All the extra work they need to do to achieve reliability when you're scaling out has a cost. But that doesn't necessarily mean that the cost is justifiable.
A box truck is more capable for running errands than my Kia Soul, because I can fit half the grocery store in it. But I think we'd all agree that it would be an obscene waste of money to default to using a box truck to buy groceries and make Ikea runs, and justify that decision by pointing out that some small fraction of the time I'll need to transport more stuff than will fit in a Kia Soul, and it's better to standardize on a single model of car. In a computing setting, though, we do that sort of thing all the time.