glomgril

one man's exfiltration is another man's distillation `¯\_(ツ)_/¯`

you could say they're playing by a different set of rules, but distilling from the best available model is the current meta across the industry. only they know what fraction of their post-training data is generated from openai models, but personally i'd bet my ass it's greater than zero because they are clearly competent and in their position it would have been dumb to not do this.

however you want to frame it, they have pushed the field forward -- especially in the realm of open-weight models.

looks like it's there now

Models like this are experimentally pretrained or tuned hundreds of times over many months to optimize the datamix, hyperparams, architecture, etc. When they say "ran parallel trainings" they are probably referring to parity tests that were performed along the way (possibly also for the final training runs). Different hardware means different lower-level libraries, which can introduce unanticipated differences. Good to know what they are so they can be ironed out.

Part of it could also be that they'd prefer to move all operations to the in-house trn chips, but don't have full confidence in the hardware yet.

Def ambiguous though. In general reporting of infra characteristics for LLM training is left pretty vague in most reports I've seen.

He is coming from the perspective of a long-running debate on symbolic versus statistical/data-driven approaches to modeling language structure and use. It seems in recent years he has had trouble coming to terms with the fact that at least for real-world applications of language technology, the statistical approach has simply won the war (or at worst, forms the core foundation on top of which symbolic approaches can have some utility).

I come from the same academic tradition, and have colleagues in common with him. He has been advocating for a quasi-chomskyan perspective on language science for many years -- as have many others working at the intersection of linguistics and psychology/cog sci.

TBH I suspect he himself is a large part of his target audience. A lot of older school academics raised in the symbolic tradition are pretty unsettled by the incredible achievements of the data-driven approach.

Personally I saw the writing on the wall years ago and have transitioned to working in statistical NLP (or "AI" I suppose). Feeling pretty good about that decision these days.

FWIW I do think symbolic approaches will start to shine in the next several years, as a way to control the behavior of modern statistical LMs. But doubtful they will ever produce anything comparable to current systems without a strong base model trained on troves of data.

edit: Worth noting that Marcus has produced plenty of high-quality research in his career. I think his main problem here is that he seems to believe that AI systems should function analogously to how human language/cognition functions. But from an engineering/product perspective, how a system works is just not that important compared to how well it works. There's probably a performance ceiling for purely statistical models, and it seems likely that some form of symbolic machinery can raise that ceiling a bit. Techniques that work will eventually make their way into products, no matter which intellectual tradition they come from. But framing things in this way is just not his style.

Savor it while you can. As a former academic, for me the lack of intrinsic motivation to "create value for shareholders" is the hardest part of working in industry.

As painful as it can be at times, it is a truly beautiful phase of life during which your main obligations are to become an expert in something that interests you and to make enough money to not starve and have a place to live. If you are single, coming directly from the "broke college student" lifestyle, and end up at a university with a good stipend, it won't even feel like you are "poor" and the money is mostly enough. But the life of a grad student in a large public university can come with much more financial instability and heavier teaching loads from day one, with less time for slacking off and letting ideas marinate. Less so if you are in a field/have an advisor with good/consistent funding. The devil is in the details.

Wouldn't change it for the world though, and anecdotally most people I know who ended up finishing the PhD feel the same way.

Main shortcoming of the (American) grad school experience imo is lack of preparation to join the corporate workforce (in my field, there are easily >10x the graduating PhDs each year than there are available university jobs). Academia has done a terrible job preparing grad students for the harsh reality of a non-academic career. Keeping this in mind throughout grad school will help a lot -- you can see the difference in non-academic career trajectory between people who had a backup plan and those who didn't.

This is just brilliant. Brings back memories, some fond others less so. Only addition I'd suggest is a subplot involving teaching/TAing duties and/or money problems.

Good to be occasionally reminded that slacking off is a legitimately important part of the scientific process. Wish this view was more popular in the industry.

That's an insane story. As much as I hate flying, modern aviation infrastructure is one of mankind's most impressive feats.

Yeah that is consistent with my experience for sure. Probably true for lots of others too. I'm pretty neurotic about dumb shit on a day-to-day basis, but the few times something actually extremely serious has gone wrong in/around my life, it's rarely felt particularly scary or panic-inducing -- maybe even less so than the usual "oh my god I probably left the stove on and the building will burn to the ground and it'll be all my fault!"

That's kinda how the pandemic felt to a degree. A happier than average time in my personal life, despite (or perhaps due in part to?!) all the carnage and pandemonium throughout the world. Feels wrong to say, but I will look back on my experience of the pandemic fondly.

Big shouts to visidata -- very underrated tool imo.

The consistency of descriptions is particularly surprising to me. Like you got a roughly circular collection of seemingly random scribbles, but they can tell you exactly which parts of it correspond to the person's nose, hair, arms, eyes, etc. And the descriptions seem to stay the same if you ask about the same picture on different days. Still not sure what to make of this phenomenon but it is fascinating.