Matching braces is part of written natural language and would be beyond the power of regular languages wouldn't it? I thought there was something special with transformers allowing to learn that at least within its context window size in a generalizable way.
It also sounds like from the paper transformers are formally proven to capable of it, but the training can't get the weights to make it capable of it, even though there should exist such weights, if I am reading them right.
It also sounds like from the paper transformers are formally proven to capable of it, but the training can't get the weights to make it capable of it, even though there should exist such weights, if I am reading them right.