I see this basic logic in almost AI ethics threads, and it starts with a big assumption: "humans learn from copyrighted source material without copyright violation". This then gets tenuously extended to "ai also learns, so it must not be in violation of copyright law".
The first assumption is highly flawed though. Humans routinely do violate copyright law. Plagiarism is a huge problem in many sectors; un-cited direct copies of people's work in violation of fair use is a regular every day occurrence in the human world. It doesn't matter if you memorized the source material or if you transcribed it or if you copy and pasted it, if it isn't your source material and is someone else's, you've committed a violation of the law. Learning to produce original work and reproducing someone else's work is not the same thing. If an AI is ingesting and perfectly reproducing someone else's copyrighted works, it is in violation of copyright law in the same way a human would be if they reproduce someone else's copyrighted works.
The first assumption is highly flawed though. Humans routinely do violate copyright law. Plagiarism is a huge problem in many sectors; un-cited direct copies of people's work in violation of fair use is a regular every day occurrence in the human world. It doesn't matter if you memorized the source material or if you transcribed it or if you copy and pasted it, if it isn't your source material and is someone else's, you've committed a violation of the law. Learning to produce original work and reproducing someone else's work is not the same thing. If an AI is ingesting and perfectly reproducing someone else's copyrighted works, it is in violation of copyright law in the same way a human would be if they reproduce someone else's copyrighted works.