visarga

The new copyright protects vibes

I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.

Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.

If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.

It would not be inside Wikipedia, but separate, as a collection used in the training corpus.

A billion humans use AI daily, they write about 0.5T tok and read about 2T tok (estimated). In a week they amount to more than the whole training set of GPT-4. Human language is being shaped back by AI language.

I think this is exactly what we should do. Send agents to collect evidence and write deep research reports, then train on those reports. DR reports are derived from hundreds of sources and can map out the distribution, what are the opinions on a topic, not necessarily what is the one correct answer. Agents could write one piece about every human, place, topic, event or entity there is under the sun. They could also try to generate the same without sources (closed-book mode) and report on LLM biases on that question.

We don't have to fall off a cliff to learn to drive on a mountain road, we can imagine the consequences for cheaper. If we didn't have imagination life would be much more dangerous.

> is just profoundly goofy shit

Why not expect it? It's what most model releases do.

The SOTA model dilemma is that you have to pay more and more to train models that focus on less and less of the AI task market, because 95% of tasks can be done by cheap small models.

The context is that which can pay - a human or bank account. It's the liability sink which is immune to AI, nobody cares if your AI failed or you.

I think the article is insightful. I want to contribute my perspective:

1. when an AI starts working on a task, it is within a human context, someone pays and decides to assign that work to AI

2. while it performs work on the task it continuously needs to interact with the context in order to act and observe the feedback

3. at the end of work, the outcomes remain in context, including gains, losses, costs, risks and liability.

So AI is great but can't do the beginning, middle and end of a task. They belong to context, which is us, the AI users. Context is maximally distributed, every human, team, or company that uses AI is a context, they are relational and indexical. You can't have my context, for example - you can't eat for me, or learn for me. Some things must loop back. Especially consequences of AI work. That is irreducible, AI can't improve to grow a skin like us. Its own skin is user base gain/loss, competition with other AI providers and dancing for investors. Every AI usage context is not theirs and they can't own it.

What happens to companies is that their investors expect AI usage already, their competitors use it to compete with them and fast follow, and customers use AI agents to arbitrate their buying opportunities. So even if they did nothing AI related they find themselves in a new world where.

1. anything visible can be easily imitated - code, text, media, and AI agents - I call it the fast follow effect

2. any apparent arbitrage opportunity will be quickly crowded by fast followers removing profit from it

3. AI it self is competitive and the whole stack is fungible - models, harnesses, tools, they can all be replaced easily, what matters more is the project context data which is usually a git backed folder on local disk

So my theory is that AI won't make anyone "rich". Instead we have a musical chairs moment. AI removes distinctiveness from everyone of us. Every time I use AI for myself it is just great and amazing. When others use it, they make our own work seem little and not as impressive. Companies are caught in a Red Queen effect, have to work harder just to stay in place.

And here I find the answer to what will we do with all our free time - compete, run for our own musical chair. We need to find our own niches again, they are now under rapid change. We will invest as much energy as needed to do it, we are ready to pay a lot of effort to differentiate back. But we can't predict it just as you can't predict where the ball in a football match will go 1 minute later, it depends on what every other player is doing.

You can lessen the human review burden by pitting multiple agents against each other. Ask 3 agents to implement the same solution, then set them up against each other in differential tests. This can find bugs in them or clarify the spec. When tokens get cheap it is worth having multiple replicas, taking advantage of different programming languages, libraries and agents. You don't keep one version, you have a population, and it tests itself. It's easier to implement again than to imagine testing scenarios anyway.

You can "solve" 1 and 2 by N-version programming. Have the same code implemented twice, or more times, you get free error testing by fuzzing or running replicas in shadow mode. In order to make replicas maximally different choose a different language, framework, agent harness and LLM.

There remain some nitty details related to tracking randomness and choosing what interfaces to log. But even having your code testable by replicas would help organize it better and expose different assumptions made by the language or library stack.

When you get different outputs from replicas it can be 1. one of them has a bug, or 2. the spec is unclear -> fix the bug or the spec. So this method catches both sides. It also might be faster than traditional code based testing, you can build replicas in parallel. A replica is an open ended test, unlike coded tests it can be tried on open ended input values.

The core idea is that it is easier to implement twice than to imagine bugs. We already solved implementation, but bug finding means running code in your head, it touches the halting problem issues - can we really imagine behavior from code? I don't think we can do it well. That is why we should just make replicas instead, tokens are getting cheaper and cheaper by the month.

> The second form of regulation should be on excessive energy consumption. The US Department of Energy needs to declare energy to be a vital human resource, and exorbitant uses for computing to be curtailed.

If you allow AI to use as much as it needs we get much higher capacity and cheaper energy down the line. Growing pains.

I see AI like the coming of a baby in the family - suddenly you need to get a larger house, baby room, ensure babysitting and education, an effort for the whole family. This is the initial cost, which is high for both a baby and AI.

In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.

I thought the way Jev was presented was sneaky - you can't say "we calibrated the model for you" and skip having a task. specific calibration set. They couldn't have known all our future tasks.

Could be the missing link in the sovereign stack - an agent with a search stub library running locally with higher privacy than using dedicated search providers.