The New York Times blocks OpenAI’s web crawler - eviltoast

The New York Times blocks OpenAI’s web crawler::The New York Times has officially blocked GPTBot, OpenAI’s web crawler. The outlet’s robot.txt page specifically disallows GPTBot, preventing OpenAI from scraping content from its website to train AI models.

  • poke@sh.itjust.works
    link
    fedilink
    English
    arrow-up
    3
    ·
    1 year ago

    They made a flag specifically for their crawler, so they can say that they do but in the most annoying way possible.