I have to disagree with the fundamental premise here. LLMs do indeed have an internal "concept" mapping - the embedding layer. There's some fun demos on youtube[1] where you can manipulate embedding vectors (such as king - man + woman = queen), and these manipulations are done entirely in the embedding/concept space.
[1] https://youtu.be/gQddtTdmG_8 (around 14:00)