Hi, I’m building a personal website and I don’t want it to be used to train AI. In my robots.txt file I blocked:
- ChatGPT-User
- GPTBot
- Google-Extended
- FacebookBot
What bots should I also add? Are there any other ways to block AI bots?
IMPORTANT: I don’t want to block search engine crawlers, only bots that are used to train AI.
Good idea. I will made a invisible link to “traps for bots”. One trap will show random text, one will be redirect loop and one would be random link generator that will link to itself. I will also make every response randomly slow, for example 0,5 to 1,5 seconds.
Good thing is that I can also block search engine crawlers from accessing only the traps.
If you’re interested in traps, you can add a honeypot to your robots.txt. It comes with some risk of blocking legitimate users, though.