nooorofe

it was fixed with workaround, in newer versions, they copy statically compiled nodejs to the server and it works.

Airflow is far from perfect, but I don't understand your concerns. I work in a big and messy company and even messier department. We have jobs running in Databricks, Snowflake, sometimes we read data from API end points, or even files uploaded to SharePoint (my group is not building DW). Airflow lets me organize it in a single workflow. At least I know that every failed job is reported by email and I don't need to search multiple systems - all starts from Airflow.

> Why should biz logic that just needs to run Spark and interact with S3 now need to run a web server?

Webserver is mostly UI. Scheduler service triggers the jobs.

We have groups which run everything as Bash Operator, no dependency issues that way.

You maybe have a very specific use case in mind, the main points of using Airflow for me

* Single orchestration center: manual job control (stop, pause, rerun), backfill; automated scheduler/retry; built-in notification

* Framework built around "reporting period" - it enforces correct abstraction, if a data batch is broken, I can rerun it and rerun all dependent downstream. How do you fix data in event driven workflow?

* managing dependencies

In most cases all Airflow does is running your job with passing it "date" parameter. You can test your code without Airflow - just pass it a date and run from command line.

    from zoneinfo import ZoneInfo
    import pendulum
    import datetime

    def pendulum_to_datetime(pendulum_dt: pendulum.DateTime) -> datetime.datetime:
        return datetime.datetime.fromtimestamp(pendulum_dt.timestamp(), 
    ZoneInfo(pendulum_dt.timezone_name))
    
    # test 
    
    df = pd.DataFrame([[1, 2], [1, 2]], columns=['a', 'b'])
    df["time_column"] = pendulum_to_datetime(pendulum.now())
    
    print(df)

output

       a  b                      time_column
    0  1  2 2023-11-19 18:14:16.027777-05:00
    1  1  2 2023-11-19 18:14:16.027777-05:00


    >> df.dtypes

    a                                         int64
    b                                         int64
    time_column    datetime64[ns, America/New_York]
    dtype: object

I doubt it was .exe, probably .msi (installer) or zip file.

I think the article and most of the commentators are missing a real problem. The real problem is that there is an underlying complexity, which has no simple solution - collaboration of teams/groups of people with different expertise and even different interests is a universally unsolved problem. When a company passes the "startup" stage and gets real customers, it isn't a startup anymore. It has more tasks to handle and needs more people to work, people with different expertise. There are sales, operations, developers (with multiple expertise), lawyers, coordinators (of all kinds). Innovation is not a self-evident target, at some point startup needs to cash innovation. There are frameworks which attempted to address that complexity, for example scrum or more flexible superset "agile manifesto" or even waterfall. They all may work or fail, but there is no guarantee, no recipe.

Term "toxic" is toxic by itself.

Periods when engineer don't understand the problem should be spent on analysis of the problem domain. "Now I am working on defining the problem domain" - is an activity to work "I don't understand the problem" task. During that period probably zero code will be written.

> That author only encounters problems that have been fully solved before

He doesn't, otherwise there would be no talk about "plan B" and risks. When you actively write a project code, you should know that solution is possible. Having plan doesn't mean "problems have been fully solved before". You may have POC which doesn't end in resolution, but it should be clear what is POC for and a failure is possible outcome.

Apache Airflow is tool for *analytical data* pipelines. If you want to create a daily/monthly data feed from your operational DB, move to it into Data warehouse, run jobs to process/aggregate that data and create daily/weekly/monthly reports, Airflow may help with all those.

very basic use cases:

* run job for given reporting_period * backfill data for reporting_period between N and M * lookup failed jobs

Nothing in Databricks supports that.