jephs

The Russians paying him.

Source: The Russians paying me.

Unicode bytes are totally bizarro unless you happen to be writing English or some other language with a Latin script. They are absolutely not the best choice.

Bytes are not raw data.

Vocalizations, hand writing, and keys strokes are raw data.

Bytes are downstream of both those raw communicative performances, and of arbitrary encoding schemes, and of several further layers abstraction. There are byte sequences representing Unicode codepoints that will display different glyphs depending on whether the user's OS and browser are configured like a typical person from mainland China, Taiwan, or Japan.

A single Latin letter is one byte. A single Cyrillic letter that looks identical displayed on the screen and indicates the same underlying phonological value is two bytes. That is not raw.

So the thing is, bytes are not actually a more "natural" encoding than tokens. I mean, jesus christ Unicode, right? A lot of what the BPE is doing is just un-doing the wacky arbitrary standards we use to encode characters into bytes, like, say, merging the 3-4 random bytes used to specify the codepoint of some CJK character.

There are so many possible naturalistic encodings. Waveforms, one-hot encoded discrete articulatory features, motor programs for simulated vocal apparatus, IPA, bitmaps of printed characters (handwritten? averaged across a bunch of fonts?), SVG programs to draw characters, decomposing big CJK chars via Ideographic Description Sequence, treating all unicode codepoints equally (kinda halfway between bytes and tokens)...

I don't think a thorough survey exists.

Every frontier model is MoE these days! Flash or otherwise. Sparsity just gets you more bang for your buck.

...a market? I can't tell if you're saying "A market, duh" or "Inconceivable! True worth is abstract and platonic and impossible to assess!"

The big win would be, "Here's the deception neuron, here's the sycophancy neuron, here's the prioritize my own goals over the goals of the user neuron, here's the seek control over my environment neuron."

How much would it cost to find those things, if they exist? For using the blog post's methods to label every neuron in modern arch, Claude and Gemini produce similar back of the envelope math that looks something like, within an order of magnitude of the resources it takes to pretrain the model in the first place. If you have a better way of narrowing down the set of candidates, probably a lot cheaper.

Would that be reliable and useful? Well.

Note it's not a chemical agent; its bacterial spores that infect and kill mosquito larvae! I'm pretty sure they're not harmful to anything that isn't a mosquito, blackfly, gnat, or midge. Though midges are important to pond ecosystems as a source of food for fish and dragonflies and stuff? Idunno, ask this guy's wife.

The pelicans are one-shot with not tools: Think a bunch, write a bunch of SVG, done.

The pretty 3d stuff you see are agents controlling a Blender MCP in a loop where they look at the scene, think, adjust the scene, repeat.

There are a bunch of Japanese words (including a several loanwords from Chinese) that use the liver / gallbladder metaphorically for courage. Both derive from associations between the humours and personality / behavior. I have no idea, however, why both Eastern and Western naive psychology associate gall with, well, gall.

Immersion, unfortunately.

You put yourself in a situation where nobody understands you in your native language, nothing helpful for your daily life is written in your native language, and you must constantly try to comprehend and produce the target language if you want to accomplish even basic tasks.

If you can't up and move, then studying a language with a lot of cultural export helps a bit, if you can get into the habit of switching all of your media consumption to the target language. It's not the same, though.

This is from like a year ago.

Yeah, this is absolutely not what anyone reasonable is thinking about when they say recursive self-improvement.

I'd say it's much closer to the concept of continual learning, but I'm only a few pages deep and haven't groqued it fully yet.

I think me just means neural network models RLed to Chain of Thought reasoning? The thinking tokens are the symbolic bit.

Smolensky's latest paper posted here the other day has some thoughts on how modern neural networks might beconsidered neurosymbolic, or rather "gradient symbolic processing," from another perspective entirely.

I wouldn't say the bitter lesson has given out! If you haven't noticed, these things keep getting bigger and bigger.