waylonwalker

Get them floating pop ups

I feel ya. Before using kedro I was using a version of the cookiecutter for data science template that I modified to have something kind of like their data catalog. Before that it was so wild west, every project was its own unique thing.

Maybe a better word is that it comes with an opinionated template? The core parts of Kedro can all be used individually as you wish. Only want a DAG, you can use just the pipeline, Want a catalog setup so you can store intermediate steps, use the Catalog. You can even make kedro project using all the features in a single file with surprisingly little boilerplate.

By adding a catalog entry for each of your datasets with a filepath argument, it will save them to that place. All their datasets use fsspec under the hood so this path can be an s3 bucket or a local path and it all just works.

My experience with McK came in two phases. Phase one started with Kedro, and phase 2 was a cash grab, move as hot and fast as you can. In my experience phase 2 got off the ground quickly, but after week the notebook was riddled with run these sections, but not these, 4 devs on the team were hanging out having coffee much of the day because they were waiting for time in the notebook to implement their changes. There was no version management so days were lost to, well someone deleted something they shouldn't have now we need to rewrite it. The lack of code review, linting, and formatting tools left them swimming in messy code that they would clean up later, but later never came.

The kedro project is still running nearly 3 years after it started, the notebooks are long forgotten.

Notebooks are fine for single contributors if that is what they are comfortable with. If that is what they are comfortable with its probably because they have not experienced engagements like you are bringing to them that require them to collaborate as a larger team. If you plan to effectively run projects that last longer than a few weeks with more than one data scientist, I'd really challenge them to lean into kedro. The long term productivity of the project will greatly benefit from it. You will be showing the team a more sustainable way of creating pipelines that will lead to continued success after you leave. This leaves a better name for you than a quick cash grab that gets long forgotten in my opinion.

I use it quite heavily at my day job. The core pieces of kedro are indispensable on large projects (Pipeline, Catalog, DataSets, Runner, hooks). The marketing has a lot of focus on the template, It's a great template, but not the key feature that brings me back to it. Really like that I never have to worry about io, the catalog abstracts this away. I don't worry about long runs to get started with my work. It saves all intermediate steps automatically, and the Pipeline (DAG) lets me run just the parts that I need for my task. Before using kedro I would need to scaffold out a way to save all of these intermediate steps, or suffer from long run times to get started on my task. Lastly the DAG is indispensible during environment migrations, it can quickly tell me all of its edges that I need to be concerned with picking up and moving with me.

Unlike similar projects kedro is just a python framework. It lets me build, deploy, and orchestrate however works best for my team.