Google has a separate user-agent identifier in for robots.txt for AI products that can be blocked separately from the search crawler. Many sites block the AI crawler while allowing the google search crawler. AI content is controlled using googlebot-extended user-agent
See https://developers.google.com/search/docs/crawling-indexing/...
In fact NYT does block google from using its content for GenAI products:
User-agent: Google-Extended Disallow: /
As well as chat gpt
User-agent: GPTBot Disallow: /
So idk why this would be unexpected.
See https://www.nytimes.com/robots.txt