Bots are currently scraping the internet for LLM training data at unprecedented rates[1][2][3], driving up costs and destabilizing public-facing websites. I want to talk about how this has been particularly difficult for wikis, and has gotten much worse in the last few months.
  • ElectricVocalist
    18·
    3 months ago

    but I would assume there’s an arms race going on behind-the-scenes between Cloudflare and the bot developers

    No. CF lost years ago, and the checks can be bypassed easily. It’s just that it blacklists ips generating insane traffic but there is a lot of margin

    • smeg@infosec.pubEnglish
      5·
      3 months ago

      I put my little blog behind Cloudflare because I was tired of it going down due to scrapers overwhelming my little VPS.

      • ElectricVocalist
        1·
        3 months ago

        Crawlers have never been a problem for me as my internet subscription is unlimited. My experience comes from being the one crawling sites and bypassing the CF challenge