Comment by Amsozzer

Good intuition, but the enforcement mechanism is a substring blocklist (both "rm -fr /" and "base64 -d | sh" work), and tainting will not work since the model outputs an entirely new string to use as an argument for the tool. It is worth reading the CaMeL paper from DeepMind.