IMO this is an underappreciated advantage for Google. Nobody wants to block the GoogleBot, so they can continue to scrape for AI data long after AI-specific companies get blocked.
Gemini is currently embarrassingly bad given it came from the shop that:
1. invented the Transformer architecture
2. has (one of) the largest compute clusters on the planet
3. can scrape every website thanks to a long-standing whitelist
Gemini is currently embarrassingly bad given it came from the shop that:
1. invented the Transformer architecture
2. has (one of) the largest compute clusters on the planet
3. can scrape every website thanks to a long-standing whitelist