mdahardy
Joined 4 years ago
Karma
103
Submissions
26
Member since
Nov 1
Recent submissions
Comment · 10 months ago
Our main argument is that outputs will become increasingly indistinguishable, but the processes won't. E.g. in 5 years if you watch an AI book a flight it will do it in a very non-human way, even if it gets the same flight you yourself would book.
Comment · 10 months ago
This is a fair criticism we should've addressed. There's actually a nice study on this: Vong et al. (https://www.science.org/doi/10.1126/science.adi1374) hooked up a camera to a baby's head so it would get all the input data a baby gets. A model trained on this data learned some things babies do (eg word-object mappings), but not everything. However, this model couldn't actively manipulate the world in the way that a baby does and I think this is a big reason why humans can learn so quickly and efficiently.
That said, LLMs are still trained on significantly more data pretty much no matter how you look at it. E.g. a blind child might hear 10-15 million words by age 6 vs. trillions for LLMs.
Comment · 10 months ago
Ah, nice idea. I hadn't considered locking it after you guess correctly.
Comment · 11 months ago
The cross-tile challenges were quite robust - every model struggled with them, and we tried with several iterations of the prompt. I'm sure you could improve with specialized systems, but the models out-of-the-box definitely struggle with segmentation
Comment · 11 months ago
That's a cool idea. I bet it would work better.
Comment · 11 months ago
yes
Comment · 11 months ago
After watching hundreds of these runs, Gemini was by far the least frustrating model to observe.
Comment · 11 months ago
We have an example of a failed cross-tile result in the article - the models seem like they're much better at detecting whether something is in an image vs. identifying the boundaries of those items. This probably has to do with how they're trained - if you train on descriptions/image pairs, I'm not sure how well that does at learning boundaries.
Reload are challenging because of how the agent-action loop works. But the models were pretty good at identifying when a tile contained an item.
Comment · 11 months ago
Same! As we talk about in the article, the failures were less from raw model intelligence/ability than from challenges with timing and dynamic interfaces
Comment · 11 months ago
While running this I looked at hundreds and hundreds of captchas. And I still get rejected on like 20% of them when I do them. I truly don't understand their algorithm lol
Comment · 11 months ago
You could definitely do better than we do here - this was just a test of how well these general-purpose systems are out-of-the-box