As a SW person (working with silicon folks though), I was always interested in the same idea. So I just vibecoded a little compiler from LLM (config.json, weights) to RTL (happy to share if you want). Then synthesized it via yosys.
I picked TinyStories-1M, synthesized via yosys against SKY130 PDK. It's GPT-Neo with 8 layers. At int8 it's ~30mm^2 per layer (including kv caches, multipliers and all the attention stuff). So it's 250 mm^2 for the full toy model.
I'm kinda surprised, I spent no time, so if you want to etch a toy model, you can do that (you can probably reduce the area significantly). If you want a mask for that is probably going to cost 100k though.
I picked TinyStories-1M, synthesized via yosys against SKY130 PDK. It's GPT-Neo with 8 layers. At int8 it's ~30mm^2 per layer (including kv caches, multipliers and all the attention stuff). So it's 250 mm^2 for the full toy model.
I'm kinda surprised, I spent no time, so if you want to etch a toy model, you can do that (you can probably reduce the area significantly). If you want a mask for that is probably going to cost 100k though.