Overall, I agree with much of this post, but there are several caveats:
1) Mongo is a bad point of reference.
The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc.
2) Plan for success
95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it.
That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]).
But the key lessons are correct:
- Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB)
- Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future.
- Cost of data is increasing with regulatory.
1) Mongo is a bad point of reference.
The one lesson I've learned is that there is nothing Mongo does which postgresql doesn't do better. Big data solutions aren't nosql / mongo, but usually things like columnar databases, map/reduce, Cassandra, etc.
2) Plan for success
95% of businesses never become unicorns, but that's the goal for most (for the 5% which do). If you don't plan for it, you won't make it. The reason to architect for scalability when you have 5 customers is so if that exponential growth cycle hits, you can capitalize on it.
That's not just architecture. To have any chance of becoming a unicorn, every part of the business needs to be planned for now and for later: How do we make this practical / sustainable today? How do we make sure it can grow later when we have millions of customers? A lot of this can be left as scaffolding (we'll swap in [X], but for now, we'll do [Y]).
But the key lessons are correct:
- Most data isn't big. I can fit data about every person in the world on a $100 Chromebook. (8 billion people * 8 bits of data = 8GB)
- Most data is rarely queried, and most queries are tiny. The first step in most big data jobs I've done is taking terabytes of data and shrinking it down to the GB, MB, or oven KB-scale data I need. Caveat: I have no algorithm for predicting what I'll need in the future.
- Cost of data is increasing with regulatory.