It is important to understand how this technology works. It isn't magic. Your example is solid. Similar to the Chinese Room.
Do you think your legislators or most users understand tokens and predictions and what is going on? Do you think that those who bombed a school using it know this?
Until frontier models can account for the disparity between what they can do and what the person (or other agent) is asking, we will continue to suffer bad information used like good information.
That tokens cannot count letters like that is not a brain-dead gotcha. Ignoring how humans are using these models is lacking in critical understanding of the real world. People are literally going to die due to humans not understanding how this works (and already have). True brain-dead, and body-dead.
Do you think your legislators or most users understand tokens and predictions and what is going on? Do you think that those who bombed a school using it know this?
Until frontier models can account for the disparity between what they can do and what the person (or other agent) is asking, we will continue to suffer bad information used like good information.
That tokens cannot count letters like that is not a brain-dead gotcha. Ignoring how humans are using these models is lacking in critical understanding of the real world. People are literally going to die due to humans not understanding how this works (and already have). True brain-dead, and body-dead.