This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:
* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)
* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?
* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?
I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/
This is an interesting problem both for public datasets like the ones you're showing, and for internal datasets created and exposed by teams within an org. There are a lot of moving pieces to consider over and above the basics of getting data into and out of systems:
* How do you communicate the data schema in a way that provides both strong guarantees (data you see WILL match the advertised schema), while still being adaptable to change and unexpected circumstances (schemas WILL change)
* How do you deal with transformations/data cleanup in a non-hacky way? Then how do you scale them?
* How do you deal with data ownership? What if one data product consumes another in a nontrivial way -- who owns what?
I'm working on a team building a product to solve these problems! We recently opened beta signups so if you're interested, check us out: https://www.estuary.dev/
I'm happy to answer any questions :)