Wow - that sunroof thing is cool, I bet that was a fun area to work on.
There’s a lot to be said for data engineering when done at scale - systems design, devops, cloud engineering etc.
I’m not sure that’s going yet…
Building the individual units (reliable ETL pipelines) might be on the chopping block in the near future - I’m not convinced there’s 100 competing ways to build these.
Still - best way to get started is probably to get stuck in with running a data pipeline using a locally deployed Airflow instance. Read some data from some api and write it to a local database deployment (postgres?).
May I ask the same of web application development?
There’s a lot to be said for data engineering when done at scale - systems design, devops, cloud engineering etc.
I’m not sure that’s going yet…
Building the individual units (reliable ETL pipelines) might be on the chopping block in the near future - I’m not convinced there’s 100 competing ways to build these.
Still - best way to get started is probably to get stuck in with running a data pipeline using a locally deployed Airflow instance. Read some data from some api and write it to a local database deployment (postgres?).
May I ask the same of web application development?