marviel

It's meant to replace the BPE tokenizer piece, so it isn't a full Language Model by itself.

In fact in Gu's blog post (linked in a post below) it's mentioned that they created a Mamba model that used this in place of the tokenizer.

Thanks for sharing this blog post is a great speculative deep-dive.

it mentions native multimodality somewhere in either the Arxiv or post -- seems like it might handle it well?

No problem! I'm still parsing it myself, but it seems promising in theory, and the result curves are impressive.

> H-Net demonstrates three important results on language modeling:

> 1. H-Nets scale better with data than state-of-the-art Transformers with BPE tokenization, while learning directly from raw bytes. This improved scaling is even more pronounced on domains without natural tokenization boundaries, like Chinese, code, and DNA.

> 2. H-Nets can be stacked together to learn from deeper hierarchies, which further improves performance.

> 3. H-Nets are significantly more robust to small perturbations in input data like casing, showing an avenue for creating models that are more robust and aligned with human reasoning.

how do you account for the success of reasoning models?

I agree these things don't think like we do, and that they have weird gaps, but to claim they can't reason at all doesn't feel grounded.

have often wanted something like "liquid types", but didn't know what it was called! Thanks for this

there is now I suppose! ;)

Yeah it isn't a panacea but it has afforded me less frustration than the alternative of jumping straight in.

> Since what I've been working on is just a mash-up of other people's ideas

Totally, I find most work I do, if I'm honest, is in this bucket. LLMs are pretty good at "filling in the gaps" between two ideas like this

I'm almost to this one in my read through! I'm excited to get to the "information age" arc

Identifying a problem is the first step to solving it.

Changing these buttons to "Rent" would force people to acknowledge what they're getting for their $60 bucks -- effectively a rental.

Once the problem is identified en masse, capitalism can do its work more efficiently.

def cleanup inactive specs over time.

it's only useful if it'll be used!