There is a certain difference between activating all relevant memoized facts that's in the weights and stringing them together with the help of all the stored text in the world, or displaying genuinely emergent behaviour and generating novel output.
One is really impressive and useful trick, one is AGI.
Apple's research show almost zero emergent behaviour, so I'm inclined to think most of it was already in the weights.
It doesn't take away the usefulness, it just defined the boundary. We can't expect "original research" then because it actually can't reason about concepts that are too far from whats already in the discourse. The discourse is big so we don't notice.
One is really impressive and useful trick, one is AGI.
Apple's research show almost zero emergent behaviour, so I'm inclined to think most of it was already in the weights.
It doesn't take away the usefulness, it just defined the boundary. We can't expect "original research" then because it actually can't reason about concepts that are too far from whats already in the discourse. The discourse is big so we don't notice.