arkmm

> Our hands are a marvel of evolution allowing us to do extremely precise tasks while still being adaptable to all kind of movement

It's easy to overlook, but to me it's amazing that humans are able to type! Subtle, precise movements by our fingers to express language were presumably not skills our ancestors had an evolutionary advantage for.

Very cool! Can you say a little bit about the size of the DPO training examples and how long training took?

"Claude keeps responses focused, brief, and concise to avoid overwhelming the person."

Claude and I must have a different idea of what brief and concise mean.

Since reasoning tokens are just text, I think the models have learned to squeeze in some computation in their output writing as well. So they're incentivized to be correct but long-winded, as it gives them more time to think. It's kind of the equivalent of filler words for humans, except LLMs can actually word-vomit something intelligible.

I'd also be really curious about the cost to run something like this, and what things you think it's particularly helpful for?

Building a simple sandboxed way to run open weight models over a copy of all my personal data (email, docs, messaging, photos, etc). Think it'd be cool to have a self hosted AI "chief of staff" that has full read access to my personal info.

And the human has to explain it at 2 tokens/sec (average speaking speed ~130-150 wpm). That's another constraint against slop - humans need to be able to explain their code succinctly.

I think part of the problem is the context windows for humans are actually much smaller than what an LLM can keep track of today. The small context window of humans is a feature that forces modularity and abstraction in software engineering so that you can decompose what you're working on into something that can fit into your head. But since LLMs can fit so much more in their head, so to speak, they don't have this same incentive, and you get the unorganized mess of spaghetti code that current agents often produce.

Seems like LLMs embrace that last point as well.

everything is computer

Didn't know this technique had a name, but I would think a modern compiler could make this optimization on its own, no?

You can fine tune a small LLM with a few thousand examples in just a few hours for a few dollars. It can be a bit tricky to host, but if you share a rough idea of the volume and whether this needs to be real-time or batched, I could list some of the tradeoffs you'd think about.

Source: Consulted for a few companies to help them finetune a bunch of LLMs. Typical categorical / data extraction use cases would have ~10x fewer errors at 100x lower inference cost than using the OpenAI models at the time.

Can you share more details about your use case? The good applications of fine tuning are usually pretty niche, which tends to make people feel like others might not be interested in hearing the details.

As a result it's really hard to read about real-world use cases online. I think a lot of people would love to hear more details - at least I know I would!

Payment fees are crazy when you think about them from the perspective of a merchant in a low margin business. E.g. in retail or restaurants, margins aren't much better than ~10%. If they didn't have to pay ~3% credit card fees, they'd have 30% more profit!