kushalc

Not a robotics guy, but to extent that the same fundamentals hold—

I think it's a degrees of freedom question. Given the (relatively) low conditional entropy of natural language, there aren't actually that many degrees of (true) freedom. On the other hand, in the real world, there are massively more degrees of freedom both in general (3 dimensions, 6 degrees of movement per joint, M joints, continuous vs. discrete space, etc.) and also given the path dependence of actions, the non-standardized nature of actuators, actuators, kinematics, etc.

All in, you get crushed by the curse of dimensionality. Given N degrees of true freedom, you need O(exp(N)) data points to achieve the same performance. Folks do a bunch of clever things to address that dimensionality explosion, but I think the overly reductionist point still stands: although the real world is theoretically verifiable (and theoretically could produce infinite data), in practice we currently have exponentially less real-world data for an exponentially harder problem.

Real roboticists should chime in...

Hey folks, OOP/original author and 20-year HN lurker here — a friend just told me about this and thought I'd chime in.

Reading through the comments, I think there's one key point that might be getting lost: this isn't really about whether scaling is "dead" (it's not), but rather how we continue to scale for language models at the current LM frontier — 4-8h METR tasks.

Someone commented below about verifiable rewards and IMO that's exactly it: if you can find a way to produce verifiable rewards about a target world, you can essentially produce unlimited amounts of data and (likely) scale past the current bottleneck. Then the question becomes, working backwards from the set of interesting 4-8h METR tasks, what worlds can we make verifiable rewards for and how do we scalably make them? [1]

Which is to say, it's not about more data in general, it's about the specific kind of data (or architecture) we need to break a specific bottleneck. For instance, real-world data is indeed verifiable and will be amazing for robotics, etc. but that frontier is further behind: there are some cool labs building foundational robotics models, but they're maybe ~5 years behind LMs today.

[1] There's another path with better design, e.g. CLIP that improves both architecture and data, but let's leave that aside for now.

Definitely not just a promotion. We're a mission-driven company and a we've explicitly chosen as a company to share our learnings with folks (even if it reduces our competitive edge a tiny bit) because we believe it's important to good. Not quite open source, but maybe open knowledge?

And while it's nowhere near peer-reviewed academic paper quality (we won't be submitting to Nature or NIPS anytime soon), my personal background is ML — everything we write is backed by cold, hard internal data and we try to stick to the facts.

All of that said, yes, people really like our data-driven insights and it does drive traffic. :)

Hey guys, I'm the author of this post! (And also happen to be TalentWorks CEO.) A friend sent me this link, I'm happy to answer any Qs.

Also, we're hiring. :) If you're sick of spending all your hard-earned education and experience to help Facebook, Google, Amazon, etc. increase ad CTR by 0.001%, we're working on some pretty cool technical problems. Just email me at [email protected].

Thanks for the great critique, geuis.

I'm absolutely in agreement with you about unpredictable markets, the massive power of market forces to effect change and that no one entity has (or should have) control over those forces. No one -- certainly not me -- is advocating any form of socialism and central control over economic resources.

What I'd argue instead, however, is organizations (companies, NGOs, whatever) that create meaningful value for humanity actually create MORE value for themselves. It's really, really hard to foresee what is meaningful value. Facebook was similar to MySpace, but (at least for me) creates infinitely more value. Microsoft was born because IBM miscalculated the value of DOS. The list goes back decades and centuries.

However, I'd wager that today we have more clones than original work. Now, maybe bit.ly 4.0 will be what Facebook was to MySpace, but my guess would be no. Instead, if an entrepreneur chose to work on, say, cheap rockets for space -- something original, regardless of immediate perceived value -- I'd wager that'd create far greater value, probably both for themselves and humanity.

My goal was to less tell an entrepreneur to do something more socially correct -- I'm an entrepreneur myself and I probably wouldn't listen to anyone _telling_ me to do anything -- but rather to try building riskier, original stuff.

You know, I was really trying hard to avoid the debate on patents since it wasn't very related to what I was trying to say. That apparently backfired. :)

I'm pretty familiar with the issues, both on the closed (often corporate) and open (often research, open source) sides -- it just wasn't a place I wanted to go.

I also don't really have an agenda, either with Nathan or anyone else -- you can find me online, I'm pretty much an open book.

The only thing I really do care about here? Building stuff, for- or non-profit, that adds value to humanity.

Back up now.

No, my point was to make meaning -- not trivial stuff.

There's lots of non-humanitarian efforts that create lots of meaning:

* Facebook, helps me stay connected to my friends

* Etsy & Kickstarter, empowers artists around the world

* Reddit & HackerNews, lets me find stuff about topics I like

There's also lots of humanitarian stuff that's just pretty pointless.

But folks can build some pretty amazing things and I wanted to poke the emperor on that.

Am the author -- wasn't expecting HN frontage, yikes! ;) The site just went up a few days ago and is crashing. I'm on it.