otter-in-a-suit

I take passkeys over the unbearable cargo cult that follows LLM companies, namely companies' newly found obsession to "sign in via email".

Ideally optionally followed by a 2FA (naturally, via text, delivered straight to my Mac, even further diluting the questionable security of the whole exercise) and naturally, to be repeated every 2 days or so, since "stay logged in" is the biggest lie after "I've read and accepted the ToS".

Your phone (which is probably what, 80% of relevant traffic these days?) likely has a perfectly fine password manager built in. This "sign in via email" trend must be every scammer and phishers biggest dream come true...

I recommend the Eisenhorn books if you want to get into the lore. It's a good story and covers a lot of ground without expecting a ton of prior knowledge.

Very sad to hear. Rest in peace.

I was at Dollywood a few weeks ago. I highly recommend a visit if you ever get the chance. She has a fantastic legacy in more ways than one.

There's a few options.

Clickhouse, as I mentioned, can be a good final layer, as can postgres.

You can still use duckdb for intermediate transformations, even if the final data lives elsewhere.

duckdb can also access various external sources, such as s3, so you could use duckdb for transformations and write "classic" parquet files to S3 and query them with an engine of your choice (which, again, could also be duckdb, but nothing stopping you from using Trino or something along those lines).

All a question of scale, complexity, cost, and latency. For reasonably low latency, shipping a duckdb file to the edge is fine, I think. Makes CI/deployments more complicated. Or you could assemble the actual duckdb file on site - probably easier with K8s and an init container that can scale? Something like that, I don't use K8s for SkaldMaps, but I have experimented a bit.

For SkaldMaps, the backend is written in go and has an abstraction to plug in a different presentation data store, so I would just need to re-wire data platform to write the final tables to e.g. CH instead of duckdb.

This is a fantastic primer.

A few suggestions:

- Add a "last updated" note, since this space changes often (see the prefect/dagster situation that just happened)

- Add a note about MCPs and other LLM-driven tools and features are becoming more and more important (e.g. hex.ai or the various MCPs shipped with some of the tools you mention, such as OpenMetadata)

- Maybe organize the various tools you mention briefly by their license/model (lots of them can be self hosted, some are SaaS only), since a fully self-hosted data platform is (at least for now...) very much feasible

I also wish more people would talk more about the "engineering" part of "data engineering". I've seen way too many people who claim a title like "data engineer" but lack the fundamentals of building software and are really just copy-paste scripts together.

What I'd love more DEs to think about are things like {unit,integration,e2e,performance} tests, deployments, infrastructure, networking, monitoring (you do touch on that), and all the other things a regular SWE is expected to have at least basic competency in at a certain level. For instance, tools like dbt natively support tests, but people need to write them. Or how you don't have to click-ops Airbyte, there's a terraform provider etc.

I had dinner with Evan at Scala Days last year and had no idea who he was until he casually mentioned Elm. He is delightful and had some great insights on the philosophy behind Elm and on programming languages in general to share.

I have little personal opinion on the state of Elm in prod (the little front end I do is as boring as it gets), but I’m glad to hear the language is still active.

I'm sure they've done the math. Mac has ~8% revenue share for Apple and I (naively) assume they'll just account for a 20% drop in sales with 20% higher prices. Personally, if my Mac were to die right now, I'd scream and shout (well, I'd use Apple Care...), but I won't go back to a Linux laptop, since I'm too deep into the ecosystem. And I suspect I'm not alone.

fwiw, I don't hate the thin-client model for dev work (via ssh, certainly not RDP - I've done both), but I despise the implications of _having_ to do it.