kloud

Calling open-weights as open-source in marketing materials is the usual misrepresentation. But now with the restriction on commercial use (which is against opensource definition) it is not even open-weights, technically it would be more accurate to call it weights-available.

The problem is there is no good metric to describe code quality, it cannot be RLd and that's likely why latest models have such problems with slop. It is great to raise the conversation so that labs focus on this more.

Picking specific metrics will probably not work, it would be a mix of Goodhart's law with Bitter lesson. Maybe picking and labeling quality repos, having whole suite of metrics as input features and training some traditional AI classifiers to steer the LLM training.

GitHub is really two separate products: a code forge for teams and a social coding platform to collaborate on open source.

When I use the social platform, it is mostly as a read-only website. I am occasionally commenting, opening issues, PRs etc. The slowness and annoyances do not matter that much.

But I live in the team tool, where the annoyances and paper cuts are painful. Besides poor craftsmanship, there is often tension between the simpler design for social which sacrifices usability for pro use cases.

The social platform, which has huge network effects and is really hard to replace, is basically a loss leader for marketing the team product. I wonder if it would be viable for another company to create a team tool that stands on its own by outclassing Github like Linear outclasses Jira? That should be easier to replace. And Microsoft could keep running the social platform for the time being, where the adoption of alternatives is much slower and requires a much longer time frame.

This is quite puzzling. Back in the day I saw the breakthrough with Opus 4.5, and since then was very lightweight on CLAUDE.md, trying to declare intent, rather than prescribe imperative instructions. If it was off the mark, I would see clear mismatch, which gave me opportunity to correct, usually to clear and strip context.

Starting with Fable 5, if it goes off the rails, it is more difficult to correct it, because it is overall wrong, but covers its tracks with plausible sounding arguments, so it is hard to pin point and correct.

Now, this article points out techniques that were useful to rely on models more, but those peaked at 4.6. Now according to benchmarks Opus 5 is on the frontier. But when when it has looser reigns, it ends up gaslighting me even more with abstract word soup than any model before.

Gotcha, yeah if a prompt does not produce the result you want, you need to add more details to it, depending on the scope it can be a lot. When it includes all sufficient details it becomes a spec.

You can reference other specs in a prompt. Video codecs are well specified with hundreds of pages of specs, they are probably in some form in the weights already, but you could also reference them and hash them as inputs directly to create a functional alternative to ffmpeg.

This resonates, I can imagine the development process as a cascade: we get business goals and user feedback, product is defined which can be pretty fuzzy, designers define UI, engineering works data models and tech constraints. Inputs at the top can be fuzzy, but each node in the graph applies some intelligence and clarifies it.

Related to the friction you mention, since the flow graph of the context is not preserved when it gets transformed into code, I find LLMs getting confused and making worse changes, because the explicit decisions with good reasons are getting mixed with implicit decisions from LLM surfaced during designing/planning. Sometimes when things change the code should change radically to match the new reality, but LLMs often "play it safe" by accumulating code and adding to tech debt. With a flow graph when change is made in the upper levels, it could propagate and allow decisions in lower levels to be revisited.

I agree, another issue I have with existing SDD frameworks I've seen like spec-kit or openspec, is they seem to miss the mark by coupling specification (what the software should be) and execution (how to get there). If the description of execution is needed, it means the spec is weak. As a rule of thumb if a framework defines execution concepts like tasks or tickets, that is a red flag for me.

As a counter argument for the compiler perspective: what Cursor did is basically transpiling a project to another language, which is a form of compilation.

If you distill the feedback into a spec (probably a combination of both docs and tests) which comprehensively covers the functional requirements, then you could treat the code as cached materialization. The value of generating code again is targeting different non-functional requirements, like ability to run in different environments or catering to different performance characteristics.

Specs might become one solution for coping with the need to review increased volume of code. A spec is a higher level of abstraction than code, which is a higher level of abstraction than machine code. The industry made the transition to higher-level once, paradigm is changing so it might happen again.

The workflow I imagine is either deriving specs from the conversation or reverse engineering the code to spec, review and edit the spec which should be tighter and much more compressed, then deterministically compile to code. Of course we don't want to be spec-first only, that would be going back waterfall, but doing iterations back and forth.

Now, Claude is not a compiler because it is closed and non-deterministic (they do opaque processing on server, hiding reasoning tokens), but LLMs might be. We refer to a piece of code from npm/pip by name to get some code by downloading it. We then have lockfiles with hashes to ensure integrity. Currently we are vibing it, but in the future we could refer to a piece of code by prompt/spec and getting the code by inferring it. To ensure integrity, the lockfile would be hashes of open weights and inference code (and ironing out implementation details like non-determinism due to GPU scheduling, etc.).

Would be interesting to see coding performance on SWE benchmarks.

Fusion of frontier models beating Fable, or cheaper models matching Fable performance at half the cost. Great announcement timing.

What is missing in the article is the reasoning/effort levels, so it is not ruled out the results differ just due to different reasoning budgets.

I would also be interested in seeing coding performance on SWE benchmarks.

Fair point, given it is more unconventional design, it is a riskier way in order to get something that works.

Cool project, I like the idea of having tmux-compatible CLI. I used Zellij to get better UX, but many agent tools integrate with tmux. This way agent tools can still integrate tmux as a defacto standard for programmatic interface while having better interface for users.

But I wonder if tmux/rmux design is suboptimal since it couples session persistence and window management together. Do you have an opinion the design which separates the responsibilities? An example that pioneered this approach is abduco, and libghostty-based zmx being a modern implementation.

Reviewing large volume of code is a problem. In the pre-LLM era, as a workaround to occasionally review large PRs, I used to checkout the PR, reset commits, and stage code as I would review. In the first pass I would stage the trivial changes, leaving the "meat" of the PR that would need deeper thinking for later passes.

With the increased volume of code with agentic coding, what was once occasional is now a daily occurrence. I would like to see new kinds of code review to deal with larger volume of code. The current Github review interface does not scale well. And I am not sure Microsoft has organizational capacity to come up creative UI/UX solutions to this problem.