Ask HN: Can I get in trouble for crawling using the Googlebot user agent?

A lot of sites have IP crawl restrictions, but add exceptions for Googlebot. Could Google or the crawled site legally do something when they find out?