Spidering and Web scraping in and of itself is permitted by law (eu) or fair use (us). It is not considered illegal and many people do it for many different purposes. There are many common tools and libraries to help with this on linux, mac os and windows. It's even legal to keep the copies in a searchable database. [1]
What is not then permitted is to give other people copies, or publish them on your website, or pretend it's your own work etc...
When it comes to LLMs or image generation models, they don't keep any copies and they don't generate any copies either, so they consider themselves to be well in the clear. [2]
If you want to stop people scraping your stuff anyway, you can always use robots.txt , or put things up behind a login-wall.
Do consider the morality of what you are doing though. Personally I feel that published data should be scrape-able where practical.
[1] https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Googl.... (you're even allowed to do this with physical books)
[2] https://www.uspto.gov/sites/default/files/documents/OpenAI_R... (With apologies for my crude summary of their actual arguments)
What is not then permitted is to give other people copies, or publish them on your website, or pretend it's your own work etc...
When it comes to LLMs or image generation models, they don't keep any copies and they don't generate any copies either, so they consider themselves to be well in the clear. [2]
If you want to stop people scraping your stuff anyway, you can always use robots.txt , or put things up behind a login-wall.
Do consider the morality of what you are doing though. Personally I feel that published data should be scrape-able where practical.
[1] https://en.wikipedia.org/wiki/Authors_Guild%2C_Inc._v._Googl.... (you're even allowed to do this with physical books)
[2] https://www.uspto.gov/sites/default/files/documents/OpenAI_R... (With apologies for my crude summary of their actual arguments)