Comment by jowea

How can open source models respectful of robots.txt possibly perform equally if they are missing information that the other models have access to?