colingauvin

It's definitely not a solved problem, and it has been kicking my ass. I don't just want metadata tagging/categorization, I want something that identifies the networks between the information, graduates topics to actual objects, and routinely consolidates/synthesizes narratives that can be easily looked up.

I initially tried annotating everything three times, then judging the annotation with the same model. That failed predictably, so yeah just logprobs right now, judged on a hand-annotated dataset. Mostly correlated - ~80%.

Yes, once it's in LocalAI, I intend to try it out.

Seeing the fine-tuning guides that are coming out with LFM3, I am now considering it :)

Would be a fun experiment.

One thing I have been using LFM 2.6 for is annotating emails, text messages, and other metadata. I keep it all ingesting into one giant database and annotate it all as it comes in on an RTX A4500. The LFM annotations are basically triage, and then slowly annotations trickle into my halo model based on lowest confidence (right now that's DeepSeek 4.1 Flash 1017) for an upgraded annotation. (Agents can then search this database for context).

Excited to roll out LFM 3 to replace it, looks like a big step up. Right now LFM 2.6 gets trustworthy annotations on just over 40% of data points. Just waiting for these to get into LocalAI.

No weekends. We need you to get back to work on the next 8B.

What? We know exactly what it takes to run: 2.8 TB VRAM for FP8...1.4 TB VRAM for FP4.

Of course we'd have no idea if Claude or ChatGPT were doing this because we can't see their stream of thought.

Good article.

>I do think there is a real but moderate Jevons effect here: because inference is so cheap now, I’m much more likely to implement marginal automations than I would have been before. But it isn’t unbounded. In practice, it’s maybe 2x or 3x as much code as I would have wanted at the old price, because I still have to specify what to build, review the output, and maintain it. When the price per task drops 20x, and my usage only goes up 2x to 3x, total dollar revenue into the ecosystem (and net GPU demand per user) still collapses by 70% to 85%.

I don't think this checks out. Jevon's paradox doesn't say when electricity becomes 20x cheaper, someone already using electricity will use 20x more on the same thing they already do, or even 20x more at all. It means a combination of more people and more enterprises will use it, and use more of it.

Concretely - at Claude API prices I'm not going to have AI run a bunch of things that I can do without it. But at DeepSeek API prices, I might.

Fundamentally, I think that we aren't even imagining a number of the use cases for AI, in large part because they don't exist yet. But I do think they will eventually exist in the next 20+ years. Right now we are strapping AI to everything, not reimagining life with coherent artificial intelligence. Similarly I couldn't have predicted something like Walmart+ or Amazon Prime back in 1995.

I'm running it at 3 b.p.w. and it really feels better than Flash 4 at 4 b.p.w. for me. Some of these EXL3 calibrated quants are like magic.

It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.

I was running DS731 until Qwen 3.8 Flash Next (so what, like a couple weeks? The pace has been torrid). I switched to Qwen for image input, then GLM 5.3 Flash. At the time, Qwen felt like a minor step down from DS731, GLM 5.3 Flash feels like a major step up from Qwen, so I'd say slightly better than DS731. I realize that's not exactly what the benchmarks say, but DS731 definitely flailed around on some of the more challenging things that GLM mostly gets for me.

I got this frequently with on GLM 5.3 Flash with GLM's actual plan, I do not get this with self hosting.

Yeah, MiMo Pro is the first Chinese model I can really clearly identify as an upgrade over GLM 5.3 Flash. K3, Qwen 3.8 Pro, GLM 5.3, DS4.1 Flash all feel around equivalent to me. Oddly lots of people have derided MiMo but I have found it able to tackle slightly tougher problems. For instance, I tried GLM 5.3 Flash, DS4.1 Flash, and MiMo Pro on making a new map layer for my Garmin watch, and MiMo was the only one that found the actual best method (.img file for OSM).

It is unreal how good GLM 5.3 Flash is. There's an ~4 bpw EXL3 quant that fits on 2x DGX Sparks. I'm not sure it's worth it now ($7k/Spark) or even when I bought them ($4k/Spark) but there's something about knowing that the whole internet could go down and I could still have this model running at 50-60 tokens/s with 1M context, indefinitely. It's just so good. TensorFold hits 100+ tokens/s in aggregate with 2-3 M total context pool. It's like having Claude at home.