machinelearning

Training is expensive so I wouldn't necessarily call it "pragmatic".

- the tool's goal is actually to provide a lightweight, practical way to avoid wasting training cycles on bad data.

Evals for robotics are also expensive.

- validation loss is a poor proxy of robot performance because success is underconstrained by imitation learning data

- most robot evals today are either done in sim (which at best serves as a proxy) or by a human scoring success in the real world (which is expensive).

It's great if you have evals and want to backtrack (we're building tools for that too) but you definitely don't want to discover you have bad data after all that effort (learned that the hard way, multiple times).

The metrics the tool scores vary from tedious to impossible for a human to sanity check so there's some non-obvious practical value in automating some of it.

"Most ways of constructing accurate models mathematically don’t actually work" > This is true for almost anything at the limit, we are already able to model spatiotemporal dynamics to some useful degree (see: progress in VLAs, video diffusion, 4D Gaussians)

"We’ve known for well over a decade that you cannot cram real-world spatial dynamics into those models. It is a clear impedance mismatch" > What's the source that this is a physically impossible problem? Not sure what you mean by impedance mismatch but do you mean that it is unsolvable even with better techniques?

Your whole third paragraph could have been said about LLMs and isn't specific enough, so we'll skip that.

I don't really understand the other 2 paragraphs, what's this "dark state-of-the-art computer science" you speak of and what is this "area of research became highly developed over two decades but is not in the public literature" how is "the computer science required is so different from classical AI research"?

> The reason is that it makes the essay easier to read. It's less work to read writing that flows well. How does that help the writer? Because the writer is the first reader

Note that the writer's judgement only serves as an initial proxy for how well the essay reads. This implies that the reader, whoever that is, is the true judge of how well it reads. My point is that that group is ill defined.

If it were sufficient for the writer to be the only judge of how well something reads, surely PG wouldn't feel the need to have other proofread his essays. And surely it is not sufficient for someone who lacks taste to judge their own writing as good.

The way I read that statement is the same as the startup advice of "build what you would yourself want". However you still have to validate that the market exists and is big.

There is really nothing profound in that paragraph anyway, all it is saying is that a writer should edit and proofread their work. That whole paragraph could be deleted honestly. It is obvious table stakes for one to edit their work. What differentiates good from bad is a matter of taste + who is judging it.

So one thing to note is that the essay mentions that it refers specifically to "writing that is used to develop ideas" vs. "writing meant to describe others ideas".

The way I interpret this is that it refers to claims that build on each other to come to a conclusion. So the way to test for truth is to somehow test each claim and the conclusion, which could vary in difficulty based on the kind of claims being made.

As this essay exemplifies, it is difficult to test for truth if you make broad claims that are so imprecise that they can't be verified or don't tell you anything interesting when verified using reasonable assumptions.

The issue with this article is that it is very imprecise.

Are the standards for whether something “sounds bad” based on the average person’s reading or the intended audience.

In its most general form (how the median article sounds to the median person), the argument is pretty vacuous.

Most writing discusses simple ideas and they should sound good (familiar, easy, pleasurable) to the median person.

But the most valuable kind of writing could sound tedious and filled with incomprehensible terminology to the median person but concise and interesting to the intended audience.

The current way the idea is stated doesn’t sound correct because you can convincingly defend all 4 quadrants of the truth table.

> “So here we have the most exciting kind of idea: one that seems both preposterous and true.”

Am I missing something or is the “seems true” part taking too many liberties here?

If anything, as described in the previous few sentences, the premise seems false, not true.

Kind of ironic since the line sounds right but isn’t rigorously right, so it undercuts the main argument.

What's a use case are you interested in re: video?

This is a good problem to solve but the approach is wrong imo.

It has to be done in a hierarchical way to know what you attended to + full context.

If the differential vector is being computed with the same input as the attention vector how do you know how to modify the attention vector correctly

No, you've only discussed the Retrieval part of RAG, not the generation part.

The current workflow is to use the embedding to retrieve documents then dump the text corresponding to the embedding into the LLM context for generation.

Often, the embedding is from a different model from the LLM and it is not compatible with the generation part.

So yea, RAG does not pre-compute the KV for each document.

Nope that’s not how most rag systems work today. I looked at pgvector and couldn’t find anything similar.

Do you have a link? Or maybe you misunderstood what I was taking about