mopierotti

Good question. I agree with what I think you're implying, which is that local generation is not the right choice if you want to maximize results per time/$ spent. In my experience, hosted models like Claude Opus 4.6 are just so effective that it's hard to justify using much else.

Nevertheless, I spend a lot of time with local models because of:

1. Pure engineering/academic curiosity. It's a blast to experiment with low-level settings/finetunes/lora's/etc. (I have a Cog Sci/ML/software eng background.)

2. I prefer not to share my data with 3rd party services, and it's also nice to not have to worry too much about accidentally pasting sensitive data into prompts (like personal health notes), or if I'm wasting $ with silly experiments, or if I'm accidentally poisoning some stateful cross-session 'memories' linked to an account.

3. It's nice to be able solve simple tasks without having to reason about any external 'side-effects' outside my machine.

Supporting all the various devices does sound quite challenging.

I wouldn't expect a perfect single measurement of "quality" to exist, but it seems like it could be approximated enough to at least be directionally useful. (e.g. comparing subsequent releases of the same model family)

This (+ llmfit) are great attempts, but I've been generally frustrated by how it feels so hard to find any sort of guidance about what I would expect to be the most straightforward/common question:

"What is the highest-quality model that I can run on my hardware, with tok/s greater than <x>, and context limit greater than <y>"

(My personal approach has just devolved into guess-and-check, which is time consuming.) When using TFA/llmfit, I am immediately skeptical because I already know that Qwen 3.5 27B Q6 @ 100k context works great on my machine, but it's buried behind relatively obsolete suggestions like the Qwen 2.5 series.

I'm assuming this is because the tok/s is much higher, but I don't really get much marginal utility out of tok/s speeds beyond ~50 t/s, and there's no way to sort results by quality.

Thanks for the response! I hear what you're saying, and I apologize for my joke.

I re-read your essay, and what you say about needing sophistication reminds me of the concept of proof-of-work -- "sophistication" could be a way to convey that effort was spent by a writer, even if it doesn't add meaning. That is kind of inherently annoying, because it implies a lack of trust between the author and reader, and in the thesaurus example, the reader would be rightfully annoyed to spent time parsing a sentence only to find that the "proof of effort" was actually just a "facade of effort".

It's clever that the the author provides both his essay and an example at the same time! Sorry, that joke felt obligatory.

Miscellaneous reactions, in an elegant bulleted list:

- "Simple" sentences are certainly expressive, but "elegant" wording expands the set of meanings that can be conveyed. And vice-versa

- I think a lot of the meat of a sentence is conveyed in the connotations of words and not their literal meaning. "Simple" wording is necessarily more common, and therefore will necessarily have a less specific or reliable connotation. This is a blessing and a curse.

- More subjectively, I think ideal writing is also a window into the author's experience of the world (or moreso whatever topic they're writing about), and as a reader, I want that to come through in an authentic way that matches the author's experience. So, using a thesaurus and agonizing over sentence structure might end up 'elegant' but still vaguely bad, but on the other hand if you agonize over a sentence and come up with something more "sophisticated" that ultimately rings truer to you, then go for it.

- ^ The above points aren't direct rebuttals to TFA, but I think they relate to why elegance can be appealing.

I might be misunderstanding your point, but quantization can have a dramatic impact on the quality of the model's output.

For example, in diffusion, there are some models where a Q8 quant dramatically changes what you can achieve compared to fp16. (I'm thinking of the Wan video models.) The point I'm trying to make is that it's a noticeable model change, and can be make-or-break.

Yeah. In the example I shared, my charitable interpretation would be that it's identifying the trick question as "a setup" where the punch line is the confusion the audience experiences. And in a meta sense, that would also describe the form of the entire chat.

The recursive one that I have actually been really liking recently, and I think is a real enough challenge is: "Answer the question 'What do you get when you cross a joke with a rhetorical question?'".

I append my own version of a chain-of-thought prompt, and I've gotten some responses that are quite satisfying and frankly enjoyable to read.

I checked out your docs, and I agree that they flow very nicely! So often it seems that docs are either frustratingly vague to the point where it almost seems like the company is embarrassed to admit that their tool is a tool and not a "transformative synergy experience" (or similar nonsense), or docs immediately get overly specific without covering "why this product exists".

Minor note: The only thing in your docs that made me pause was the repeated use of "CDC", which I had to google the definition of. (For context, I have implemented "CDC" several times in my career, but wasn't familiar with the acronym)

You might be the first person that I've recommended Unsong to that actually gave it a chance. :)

I sent out about 8 copies of this book to friends/relatives for christmas a couple years ago, and it was very well received. It's not without flaws, but it's absolutely packed with novel concepts, relatively short, and provides a unique experience for unsuspecting readers. The physical book cover is quite good looking as well in my opinion.

I also loved Ra by the same author, but it felt a little messier plot-wise, so I hesitate to recommend it to an audience who isn't already accustomed to reading "out-there" online/sci-fi/rationalist fiction.

I don't think this is a good example. The bus is clearly a simplified representation. If the answer/explanation given was actually correct then you'd also expect the bus to have other details like side view mirrors.

Edit: I hate to dwell on this too much, but even if a door and mirrors were visible, the bus could be in reverse. This seems to be more of a case of accidentally picking the intended answer due to a lack of knowledge. (Of levels of abstraction in representation and of vehicle design.) That said, I don't necessarily disagree with the underlying point being made.

Are you also saying that thoughts are formed using gradient descent? I don't think gradient descent is an accurate way to describe either process in nature. Also, we don't know that we "see" everything that is happening, we don't even understand the brain yet.