Uh, you don't really "allow" them to visit your website. All you can do is disallow them from visiting by blocking them using robots.txt. If you don't have a robots.txt stating otherwise they will crawl and index your site.
UPDATE: I suppose if you specifically allow the Googlebot via robots.txt, then in that case they could probably argue that you gave them permission to access the site.
UPDATE: I suppose if you specifically allow the Googlebot via robots.txt, then in that case they could probably argue that you gave them permission to access the site.