So the type of emergent ability I am curious about relates to how with enough parameters, the models start to actually emit logical behaviour where previously it was not able to apply that. So an example is:
> David is 15 years old. He has a lot of experience riding bicycles. The minimum age to be allowed to drive a car is 18. Is David allowed to drive a car?
Answer:
> Based on the information provided, David is 15 years old and the minimum age to be allowed to drive a car is 18. Therefore, David is not allowed to drive a car since he has not reached the minimum age requirement.
Trying out smaller models I get all kinds of crazy responses. ChatGPT4 seems to be reliably able to apply this kind of logic to within-prompt information. Is this a gradual property that appears slowly as models scale up? Or something that flips beyond a certain size? Or something specific about the HRLF that has been applied to it? Whatever it is, the end result is that this larger model is useful for fundamentally different things that the other models are not. Whether you call that emergent behaviour, sums greater than parts etc. doesn't change that.
> David is 15 years old. He has a lot of experience riding bicycles. The minimum age to be allowed to drive a car is 18. Is David allowed to drive a car?
Answer: > Based on the information provided, David is 15 years old and the minimum age to be allowed to drive a car is 18. Therefore, David is not allowed to drive a car since he has not reached the minimum age requirement.
Trying out smaller models I get all kinds of crazy responses. ChatGPT4 seems to be reliably able to apply this kind of logic to within-prompt information. Is this a gradual property that appears slowly as models scale up? Or something that flips beyond a certain size? Or something specific about the HRLF that has been applied to it? Whatever it is, the end result is that this larger model is useful for fundamentally different things that the other models are not. Whether you call that emergent behaviour, sums greater than parts etc. doesn't change that.