The best work I've ever done was for one of the biggest energy/agg companies in the world. They wanted an outside contractor to build something from scratch and ignore their normal corporate bureaucracy. They thought it was insane how much better the system we (I) built for them was than their normal in house tooling. The crazy part was the solutions were mostly just normal cloud tech. Postgres, DynamoDB, some AWS tools. They paid over a million dollars (nothing to them) for a set of tools that weren't technically advanced at all.
In my experience, the problems are a result of the manufactured constraints on the physical hardware/sensor devices. Mountings and power and connectivity. Receiving data from an array of devices into a shared storage system for analysis is a well solved problem. Receiving data from multiple sensors on a fixed interval, when the sensors may be made by different companies/work totally differently, is where the complexity lives. Combined with each company trying to build their own awful closed source proprietary data system on top of their sensors, you've got a really terrible time.
> SQL turns out to be really slow for this kind of data
I think this is just poor modeling. SQL is just fine for the work you're talking about.
In my experience, the problems are a result of the manufactured constraints on the physical hardware/sensor devices. Mountings and power and connectivity. Receiving data from an array of devices into a shared storage system for analysis is a well solved problem. Receiving data from multiple sensors on a fixed interval, when the sensors may be made by different companies/work totally differently, is where the complexity lives. Combined with each company trying to build their own awful closed source proprietary data system on top of their sensors, you've got a really terrible time.
> SQL turns out to be really slow for this kind of data
I think this is just poor modeling. SQL is just fine for the work you're talking about.