TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI - eviltoast
  • tempest@lemmy.ca
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    The type of request is not relevant. It’s the cost of the request that’s an issue. We have long ago stopped serving html documents that are static and can be cached. Tons of requests can trigger complex searches or computations which are expensive server side. This type of behavior basically ruins the internet and pushes everything into closed gardens and behind logins.