viccis

When it comes to platforms like Databricks, integration tests are honestly the only useful ones. Grug remains vindicated as always.

Managed Spark. That is still what they are, all the other nonsense aside.

Spark is such a headache to run that people would rather pay Databricks to do it for them. They also have some secret sauce like Photon (C++ version of the Spark engine) and the new RT Lakehouse warehouses (10x or so speed up on read-only work flows in my experience) as well as good proprietary optimization features like Liquid Clustering that keep your Parquet files nicely sized and indexed. Their AI features are terrible and I can't tell you how many Zoom calls I've joined to rescue people from wild goose chases their model took them on.

Their main competitor is Snowflake, which imo has nicer UI/UX and snappier SQL execution, but the trade off is that Snowflake has a bit more lock-in, whereas Databricks is easier to kind of just build on top of your existing infrastructure. Snowflake's AI features are also much better in my experience.

>I feel like we might have some a tech theranos on our hands here.

Based on what I've seen some customers paying them, they are absolutely making heaps of money. A "small" SQL warehouse cluster is $8.40 an hour (~$73k a year if you leave it on) and the default size of X-Large is $56 an hour or $490k a year. It goes up to 5X-Large at $729 an hour or $6.3M a year. So they do absolutely let people with horribly optimized data just pay their way out of dealing with the problem, which is something I've seen multiple times.

Praying for the poor Ducklake employees at AWS to keep whatever demon works there at bay who insists that every single data product has to be bolted onto AWS Glue (and its various Data Catalog, Data Lake, etc. partners in crime) in the most clumsy way possible. And please keep it away from Athena, possibly the worst major query engine in terms of support for query features and file formats. I don't want to see AWS-era Ducklake become another failure like S3 Table Buckets are (imo at least).

I'm tempted to use this but the rub with Databricks is that there are always a bunch of different restrictions on what features work in what products (for example RT//Lakehouse looks great but runs a more limited regex parser than other warehouses), and so I'm basically forced to rely heavily on integration tests to catch weird behavior. This would need to copy it closely enough to catch those.

The other problem is that Databricks renames everything like every 6 months so keeping up with them is non-trivial. Their DLT product alone has gone on a jazz odyssey of renames.

Hmm, wonder if Databricks trademarked their Genie AI product.

Just for agents. I track bigger picture work items in a normal ticket tracker (Linear, etc.) and the lightweight one (I use kata) is just for coordinating work and memory between agents.

Anecdotally I only stuggle with this with Claude Code. Codex has been great at auto managing its context and, when combined with how I handle memory outside of the session, I haven't thought about context window size in ages when using Codex.

It does, and, when combined with Claude's stingy cache TTL of 5 minutes (unless you up it to 1h at a sizeable increase in token cost) for API billing, means that some sessions cost like 10x what they should because of cold caches.

I use a lightweight ticket system. Planning creates very detailed tickets with acceptance criteria. Implementation finishes with updating any info relevant to future work on the tickets as comments. The planner searches for previous related work both to avoid past pitfalls but also to get any potentially relevant info into its context. Works well with subagents ramping up quickly.

I agree, and most of my prof friends in math have switched from graded homework to weekly small quizzes. Graded homework was a way to use the stick as motivation because we've given up on the carrot (the learning of the material itself as an internal reward for completing homework).

Some of them keep the homework grade to like ~5% and mostly grade on completion because, when the student's homework proofs use advanced concepts and proof techniques not covered yet but their quizzes get sub-50 point F grades with barely coherent thoughts, it's a clear sign that they are not actually spending time using the resources the class has given them to work on the material.

It seems like all of them have given up on punishing students for it, largely because they're pretenure and the teaching part of tenure review here is primarily student evals drive (HINT: this is a MAJOR part of many broader problems). But to me it's kinda nuts that in one year or so, we have decided that it's actually ok to admit students into very competitive elite institutions, catch them passing off work that isn't theirs as theirs, and not treating that the exact same as we would have a few years ago if we caught them all sharing solutions.

"A pi session and the change it made, scoped to an isolated git-worktree workspace."

Yep. The "A $thing, $elaboration." has joined "$title. $thing, $past_verb." in my "this team didn't care enough to do a basic pass over the main page introducing their product" tells.

Is this not simply the tendency of the rate of profit to fall, just a bit extreme?

Earlier this year we had prospective customers do comparisons against our competitors but also against prompting their own. All of them, without exception, found that to be a bad idea.

If it were just about the ability to edit images, Gimp would have obsoleted Photoshop decades back. Photoshop excels because of its capabilities but even more because of its ease of use and the stickiness of the skillset its users have spent developing.

If you found Schopenhauer's thoughts on it interesting, I'd recommend giving The Birth of Tragedy a read to see where a young Nietzsche took it, relating it to poetry.

It's crazy how bad Dot has been with stuff like this. You'd think it would be able to (and need to be able to) edit a lot of environment settings, tunnels, plugins, etc., but I keep running into issues. For example, I wanted to make a Codex plugin to give it the ability to steer Codex agents running within my own infrastructure, tunneling in via Tailscale. Ended up just giving up and going with a different approach. Because, for example, every time I'd add a new tool to the plugin, I'd have to manually restart Dot (something that wasn't even obvious that it was possible until you fumble around menu dropdowns) and wait because it can't just refresh tools.

Everything I'm describing is more power user stuff than its most basic use cases ("Do this thing for me with your browser") but all the roadblocks I've run into involve one part of OpenAI's product not being able to talk to the other. I'd imagine Claude Duet is the same.