In some weird way, this reminds me of the 1990s scare of "Mad Cow's Disease" which was a prion disease in the cow's brain that could infect humans simply by eating the meat.
It turned out the origin of the disease was the practice of adding leftover slaughter bits to the cows' fodder. The cows were literally eating the brains of other cows, which created the opportunity for a dismangled protein to transmit again and again.
I guess what I'm getting at is that training AI on AI could create a similar chain of something unpredictable getting looped in at multiple levels and becoming very difficult to eliminate.
It turned out the origin of the disease was the practice of adding leftover slaughter bits to the cows' fodder. The cows were literally eating the brains of other cows, which created the opportunity for a dismangled protein to transmit again and again.
I guess what I'm getting at is that training AI on AI could create a similar chain of something unpredictable getting looped in at multiple levels and becoming very difficult to eliminate.