Comment by maxaravind

Nope. Even if context can theoretically encode arbitrary computation under fixed weights, this requires the weights to implement a usable interpreter. Random weights almost surely do not. Training is what constructs that interpreter. Without it context has no meaningful computational semantics.

It's kind of like asking if I make a random circuit with logic gates, does that become a universal computer that can run programs.