dmos62

Kudos. The proposals are actually interesting to browse through. Good paper name by the way.

What is COM?

How do you reuse traces as benchmarks (eval datasets)? Is it about being able to synthesise an agent session from the trace, and having another agent watch the first agent's behavior? Having trouble wrapping my mind around all the little details.

Edit: I initially asked also about general tips for observing agents, but later, noticing that there were no interactions with my comment yet, edited my comment to remove that question, because I felt that it was too broad and would detract from my main question. To my surprise, both were answered by op after my edit; kudos!

I wonder if it's the kind of thing where, if it's good at self-sustenance, then you don't know about it. And, if you do, then it wasn't.

That's very insightful. This hadn't occurred to me.

I've seen a graphical designer struggle to use an analogous product. They tried prompts like "remove background from this photo of dancers, don't change their face": the model couldn't do it: it always changed the faces at least a bit. In other words, the model was not removing the background, instead it was generating the whole picture anew, but without the background, which not only seemed inefficient, but was also way too lossy.

I wonder if this model could finish that task.

The memory invalidation based on tying memories to pieces of code (e.g. functions) is the big thing here, I think, though I see why it wouldn't be front and center.

I've always found memory systems icky, because it's difficult to tell whether a memory is still valid. Also, discoverability is always a concern (when is a memory relevant). This being both a code-graph and a memory system (seemingly unrelated things), actually addresses this in a novel way.

Great video!

True. The kind of guarantees you can get when staying within the database can often solve whole categories of problems. But, DX of having non-trivial logic inside Postgres is not great either: I feel that I'm making a significant trade-off. I've also seen interesting languages that compile to SQL, would consider those in some cases.

I presume you're not talking about autonomous agents, right? Because then you could give it a set of tasks and check its success rate. But, even so, are you not giving it tasks? Are you principally interacting with it through discussions that are more difficult to quantify? Even question answering has benchmarks. I'm having trouble imagining how you use them (or "how people use them" in your words).

Why? Sounds like you don't have a way to support your viewpoint.

What you're saying is sus. If you have a harness that's a tier above frontier labs' offerings, link to it.

I'm asking if there's something like Sky Map, where you point your phone up and it tells you what sattelite you're looking at.

Tangential, how do you identify what is a satellite in the night sky? They're not shown on Sky Map, the app I use.