LLMs have induction heads that store such names as sort of variables and copy them around for further processing.
If you think about it, copying information from inputs and manipulating them is a much more sensible approach v/s memorizing info, especially for the long tail (where not enough "storage" might be worth allocating into network weights)
If you think about it, copying information from inputs and manipulating them is a much more sensible approach v/s memorizing info, especially for the long tail (where not enough "storage" might be worth allocating into network weights)