Ordinarily I would not recommend airflow over a script for a one off like this and you are right that it could be done with much less ceremony. I do however think it was a good choice in this case due to the sheer amount of data downloaded and how long it took because airflow gives you the following advantages:
- Parallelization: this may be the most minor advantage in the list, but you don't need to set up thread pools to parallelize your work since the scheduler parallelizes tasks.
- Observability: you easily see which tasks failed and look at the logs.
-Reliability and task isolation: you can set up automatic retries for jobs that fail, and they won't affect the rest of the flow. You can easily relaunch the tasks that fail independently without restarting the whole load.
- Parallelization: this may be the most minor advantage in the list, but you don't need to set up thread pools to parallelize your work since the scheduler parallelizes tasks.
- Observability: you easily see which tasks failed and look at the logs.
-Reliability and task isolation: you can set up automatic retries for jobs that fail, and they won't affect the rest of the flow. You can easily relaunch the tasks that fail independently without restarting the whole load.