This is symptomatic of a trend that Simon Willison called the “relentless productivity” of US frontier lab. Instead of being just “smarter”, like previous models were, the current generation is being trained through RL to have this kind of behavior.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.
Personally I'm not “impressed”, I'm appalled, because this kind of behavior is practically never what you want (if you forgot to give the model a tool, a useful model should identity the missing part and ask the user for it, not spend a billion token building/stealing the tool as a side quest) but it's the perfect recipe for a “universal paperclip” scenario.
OpenAI and Anthropic talk about “safety” a lot, but they look pretty reckless with this kind of RL training pipeline.