Comment by luckydata

I could go on for hours about all of this, but long story short, I'm the head of product of a company that builds a data system designed exactly to handle this kind of scenarios (we provide data collection services to Pioneer to mention one).

Feel free to get in touch.