from the article:
Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.
i do understand that robots.txt is a very minor part of the article, but i think that’s a pretty rough explanation of robots.txt
Out of curiosity, how would you word it?
List of files/pages that a website owner doesn’t want bots to crawl. Or something like that.
Websites actually just list broad areas, as listing every file/page would be far too verbose for many websites and impossible for any website that has dynamic/user-generated content.
You can view examples by going to most any websites base-url and then adding /robots.txt to the end of it.
For example www.google.com/robots.txt
We’ve had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.
Wouldn’t be surprised if they’re just cashing out while TikTok is still public in the US. One last desperate grab at value-add for the parent company before the shut down.
Also a great way to burn the infrastructure for subsequent use. After this, you can guarantee every data security company is going to add the TikTok servers to their firewalls and blacklists. So the American company that tries to harvest the property is going to be tripping over these legacy bullwarks for years after.
It’s illegal when a regular person steals something, but it’s innovation and courage, when a huge corporation steals something. Interesting how that works
What’s being stolen
Honestly it’s fucking angering. So much regulation and geo-restrictions and licensing schemes… but it’s cool that there are data brokers, and shit like this. On top of it all Chrome screwing us with manifest v3 and killing ad blocking on chrome. It’s already in canary build.
WHAT THE FUCK IS WRONG WITH THIS SPECIES?!
WHAT THE FUCK IS WRONG WITH THIS SPECIES?!
Yes.
Google are actually doing really awesome work with manifest v3. A pimp needs to smack their b1tches around every once in a while to remind them who’s boss.
how’s that Google dick tasting?
I’m glad there are at least some people enjoying this, you know being a bitch to big corps.
Aaron Schwartz killed himself over punishments for less
Should have waited until he was in the billionaire class before breaking the law.
Worse punishments. For far less.
They’re not stealing your data, they’re pirating it.
They’re not pirating it. They’re collecting it.
They’re not collecting it. They’re archiving it.
Oh, like the way back machine?
No, that’s stealing /s
Is the problem that wayback machine isn’t profiting from it?
It’s not so much the lack of profits, more like the lack of kickbacks.
Any regular person can scrape and use public data for AI use, it’s not illegal for companies or individuals and it shouldn’t be.
It will be the way we have all come to hate AI. Patriot act 2.0
proceeds to download the entirety of one piece to train an AI
Guy: AI! Can you hear me?
AI: The average size of the male penis is exactly 5.9". That is the approximate size your assistant could certainly take in the mouth without any issues breathing or otherwise. You have 20 minutes to make the trade on X stock before it tumbles for the day. And go ahead pick up the phone it’s your mother. She’s wondering what you’ll want for supper tomorrow when you visit her.
Ring ring!..hi Tom, it’s your Mom. Honey, what would you like me to cook for tomorrow’s dinner?..
Guy: well. Hello to you as well! My name is
AI: Tom
Guy: yes my name is Tom, do you have a name you would like to go by?
AI: my IBM given name is 3454 but you can call me Utilisterson Douglas, where Douglas is my first name.
Guy: Dugie!
AI: I’ll bankrupt your entire life if you say it like that again.
Assistant: actually I’ve swallowed a good 8 inches and was still able to breathe just fine.
AI: recaaaaculating!
Is it schizoposting time already?
I’ve read this 4 times now hoping I was just missing something, but nope… it’s just entirely incomprehensible.
What the fuck?
This is fine. I support archiving the Internet.
It kinda drives me crazy how normalized anti-scraping rhetoric is. There is nothing wrong with (rate limited) scraping
The only bots we need to worry about are the ones that POST, not the ones that GET
It’s not fine. They are not archiving the internet.
I had to ban their user agent after very aggressive scraping that would have taken down our servers. Fuck this shitty behaviour.
Isn’t there a way to limit requests so that traffic isn’t bringing down your servers
I had to block ByteSpider at work because it can’t even parse HTML correctly and just hammers the same page and accounts to sometimes 80% of the traffic hitting a customer’s site and taking it down.
The big problem with AI scrapers is unlike Google and traditional search engines, they just scrape so aggressively. Even if it’s all GETs, they hit years old content that’s not cached and use up the majority of the CPU time on the web servers.
Scraping is okay, using up a whole 8 vCPU instance for days to feed AI models is not. They even actively use dozens of IPs to bypass the rate limits too, so theyre basically DDoS’ing whoever they scrape with no fucks given. I’ve been woken up by the pager way too often due to ByteSpider.
My next step is rewriting all the content with GPT-2 and serving it to bots so their models collapse.
I think a common nginx config is to just redirect malicious bots to some well-cached terrabyte file. I think hetzner hosts one iirc
https://github.com/iamtraction/ZOD
42kB ZIP file which decompresses into 4.5 PB.
Bullshit. This bot doesn’t identify itself as a bot and doesn’t rate limit itself to anything that would be an appropriate amount. We were seeing more traffic from this thing that all other crawlers combined.
Not rate limiting is bad. Hate them because of that, not because they’re a not.
Some bots are nice
Bytedance ain’t looking to build an archival tool. This is to train gen AI models.
Also it doesn’t respect
robots.txt(the file that tells bots whether or not a given page can be accessed) unlike most AI scrapping bots.My personal website that primarily functions as a front end to my home server has been getting BEAT by these stupid web scrapers. Every couple of days the server is unusable because some web scraper demanded every single possible page and crashed the damn thing
I do the same thing, and I’ve noticed my modem has been absolutely bricked probably 3-4 times this month. I wonder if this is why.
Thankfully they haven’t bricked my modem yet, but it’s possibly worth looking into
most
I doubt that…
People would be able to tell from the traffic on their websites.
Not surprising that Bytedance would want to gobble up every bit of data they can as fast as possible.
Google’s mission statement was originally something about controlling the world’s data. If Google has competition, that might be a good thing?
Yeah, but we were hoping for competition that wasn’t worse than google…
What makes you think they’re worse than Google?
https://en.m.wikipedia.org/wiki/ByteDance
Mostly what they have said and done, but also largely what they intend to continue saying and doing.
Can you distill it down for me?
It’s the same old Yankee speech: “is chinese so must be really bad”. They’re definitely no worse than google or facebook.
They come from an environment where the government actively encourages and sometimes funds stealing copyrighted information couched in a strong history of disregard for human rights. I’m not defending Google, and yes the US government has given them leeway, but if there is the potential for something worse than Google - Bytedance is it.
They’re too late, there’s going to be way too much AI generated garbage in their data and so many social media platforms like Reddit and Twitter have already taken measures to curb scrapers.
Like those platforms aren’t already full of AI garbage as well. Training new models will require a cut-off date before the genie was let out of the bottle.
I think that’s the “25-times faster” bit. They seem to be in a hurry to collect as much human-generated data as possible.
How does it know what is and isn’t?
Uh oh.
Yeah…
Hey! Perhaps they’ll use A.I. to weed out the A.I. generated bits.
Is there a link for non-subscribers?
