Agreed that it can be a headache, but wanted to offer an alternative perspective.
Personally, I feel that inclusion in Google constitutes public access to the data. As long as I'm not logged into an account on their system, I feel ethically justified about scraping their data.
In other words, I do not feel compelled to respect robots.txt if that file does not also block googlebot.
Legally it may be another issue, but ethically I consider inclusion in Google as an announcement that this information is public.
Personally, I feel that inclusion in Google constitutes public access to the data. As long as I'm not logged into an account on their system, I feel ethically justified about scraping their data.
In other words, I do not feel compelled to respect robots.txt if that file does not also block googlebot.
Legally it may be another issue, but ethically I consider inclusion in Google as an announcement that this information is public.