dancrystalbeach

I was recently sucked into doing something like a caveman … writing Rust by hand with the wonderful Zed IDE to solve a Python problem. Ahh … what a wonderful world we live in. It felt strangely good to do such things; it brought me back to the days pre-LLM when I would fight tooth and nail with that borrow checker and compiler. Like it or not, those days are mostly done as far as the future of Software Development goes … I’m sure there will be little hobbit holes of bearded Rust developers hacking away by hand long into the future, but that will be the exception, not the rule.

As the days and weeks trickle by and fall to the ground like the autumn leaves on the maple tree behind the house, my mind wanders back to the good ole’ days when I had nothing in front of me except Notepad++, a terminal, and a problem to solve by myself. Those days are long gone, and Sir Claudius is threatening to turn me into a milquetoast programmer. Some days I feel like screaming (in all caps) at Claudius… “Don’t you know I used to hand-roll Perl before you were born??!!!” Alas, I must find other ways to rage against the machine.

Dealing with Customer Data sucks.

TDLR: Databricks’ new FILE type brings unstructured assets (multimodal data) ((video, audio, etc)) into the Lakehouse with the ease that we expect in this chaotic and agentic age we find ourselves in. I’ve been thinking more and more about multimodal data ever since Tyler brought it to my attention early on. On top of that, finding myself doing more and more unstructured data things these days tends to change how one thinks, such as building agentic solutions at the $day_job. One has to imagine that this type of data will only increase in popularity and, in fact, become more of a requirement for many data engineers in the future. By “this”, I mean munging unstructured or untraditional data into the Lakehouse that was once ruled by tabular data.

Who can write the fastest pipelines?

What is even DuckDB Labs???

I’ve been watching this story play on repeat for some time now; the death of tools like Spark and Kafka has been heralded from the treetops for a decade now, yet here we are … living in a Spark and Kafka world. Someone once told someone else that it is hard to kick against the goad. Indeed it is. I don’t see this as any sort of failure or shortcoming, though; it’s just part of the circle of life, and tech. It’s natural.

A few things have not changed in the age of AI, where the LLM has replaced 3/4 of the brain power of a whole generation of engineers who once found their meaning in the code they wrote. I mean, I hate to say it’s true, but a lot of folk should have been finding their meaning in the code they decided NOT to write. It’s the classic mix-up of smart engineers and developers thinking that their “use” or “importance” to a company or product was the amount and cleverness of the code they put out. They worshipped at the feet of their own fingers, enamored with staring at themselves in the mirror, convincing themselves they were smarter and could write better code than those around them.