My personal website that primarily functions as a front end to my home server has been getting BEAT by these stupid web scrapers. Every couple of days the server is unusable because some web scraper demanded every single possible page and crashed the damn thing
Also it doesn’t respect
robots.txt(the file that tells bots whether or not a given page can be accessed) unlike most AI scrapping bots.My personal website that primarily functions as a front end to my home server has been getting BEAT by these stupid web scrapers. Every couple of days the server is unusable because some web scraper demanded every single possible page and crashed the damn thing
I do the same thing, and I’ve noticed my modem has been absolutely bricked probably 3-4 times this month. I wonder if this is why.
Thankfully they haven’t bricked my modem yet, but it’s possibly worth looking into
I doubt that…
People would be able to tell from the traffic on their websites.