I worked at a self-described "data-driven" company, and the analogy senior leadership liked to make was that the company was like a machine learning algorithm, using data (particularly A/B tests) to do "gradient descent" of the product into its optimal form.
My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive any key metric you want in the direction you want to. Short-term, it seems to work great.
Long-term, it fails. Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. You can
(and should) invest in metrics that are more indicative of the long-term. And you should use data to help guide and improve your intuition.
But data is not a substitute for good judgment, or for a deep understanding of your users and their problems, or of "where the puck is going". It's just a tool. It's a very powerful tool, but if it's your main or only tool, you will lose.
My first take-away was that using data to make decisions is tremendously, tremendously powerful. A/B tests, in particular, can help determine causality and drive any key metric you want in the direction you want to. Short-term, it seems to work great.
Long-term, it fails. Being purely data-driven without good intuition and long-term bets (that can't be "proven" with data), and the product loses its soul. You can (and should) invest in metrics that are more indicative of the long-term. And you should use data to help guide and improve your intuition.
But data is not a substitute for good judgment, or for a deep understanding of your users and their problems, or of "where the puck is going". It's just a tool. It's a very powerful tool, but if it's your main or only tool, you will lose.