I think it is somewhat like git's creation story. Sometimes a senior dev sees a tool that is close to ideal but needs to work a little differently than what the industry has built.
Databases are up there with encryption. Don't roll your own... mentality.
But sometimes they don't fit the problem your solving. Sometimes the data never changes so why have infrastructure for updates.
Having a big DB running all the time could be too expensive for your business model
Also it is good to be curious about "what is an index" and how does a parquet file look in hex editor. Why can't I write the underlying db table outside of postgres. Why are joins hard..
And then you discover your tools give you a competitive edge
Most of the time there are existing tools, but sometimes they don't.
Databases are up there with encryption. Don't roll your own... mentality.
But sometimes they don't fit the problem your solving. Sometimes the data never changes so why have infrastructure for updates.
Having a big DB running all the time could be too expensive for your business model
Also it is good to be curious about "what is an index" and how does a parquet file look in hex editor. Why can't I write the underlying db table outside of postgres. Why are joins hard..
And then you discover your tools give you a competitive edge
Most of the time there are existing tools, but sometimes they don't.