The New York Times blocks OpenAI’s web crawler - eviltoast
The NYT doesn’t want OpenAI training AI models with its articles.

The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

    • poke@sh.itjust.worksEnglish
      3·
      1 year ago

      They made a flag specifically for their crawler, so they can say that they do but in the most annoying way possible.