At my old company, Hadoop, JSON, Parquet were the solutions looking for a problem to solve. Yes we handled lots of data, but none of it was time critical. No matter, we had BIG DATA™. Bow when I say that. We weren't Google or Twitter. We were not surgically retrieving data, and while we had lots of data, we didn't have huge data centers; Dremel wasn't relevant to us.
It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry.
"You're supposed to use Hadoop."
"You wanted to know what happened yesterday."
"I did!"
"If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done."
<Crickets>
It's funny! It would take our Hadoop team three weeks to get data together for our use. I often didn't have three weeks. In those times when I needed to use the data from the previous day, I'd just grab the raw data, organize it, process it, and be done with it in a few hours, using Unix/Linux tools and a bit of mathemagical wizardry.
"You're supposed to use Hadoop."
"You wanted to know what happened yesterday."
"I did!"
"If you want it from Hadoop, it will be ready in three weeks. Probably. That's if they have everything done."
<Crickets>