I know there are tools that supposedly make it harder for AI to be trained on your images. But what about text or code? Is there any way to share things outside of a small group while minimizing the risk that some fucker will scrape them or otherwise use them for building an LLM?
glue traps are a good solution to badly-behaved scrapers. just list a path in your robots.txt that slowly produces random text and links, all of which go to the same sort of page. if they respect robots.txt, they’re fine. if they don’t, they get stuck ingesting nonsense.
not a catch-all solution but a good tool.
AFAIK it was a glue trap / tarpit that one guy I know tried. Sure, it did result in the scraper getting a lot of useless data. OTOH, they dedicated massive resources to trying to get those non-existent documents, so his website was effectively DDOSed as a result. That’s why he’s now blocking entire countries.
yeah slowing down access and using a cheap generation method is important there.
Even that isn’t enough if they respond to that by just throwing more crawlers at your site.
yeah it’s not the only thing you want on there of course. i run a tarpit, fail2ban, and anubis.
Yeah, at one point he posted that fail2ban was only able to ban about 20 IPs/second, but he was getting 200 requests per second, and the bans weren’t having a meaningful impact on that (i.e. the requests were all coming from different IPs). From what I remember, his system load was over 1000.
oh wow. vps? they tend to live in busy ip blocks.
I think it was running on a VPS, but I’m not sure. As for the traffic, I think it was coming from all over. Amazon-owned netblocks, mysterious ones that were hard to trace, etc.