I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
I’m getting really tired of “verify you’re not a bot” checks everywhere.
I understand these checks help prevent bots, but they seem to appear on so many websites now. Does anyone else find them frustrating?
It’s easy to block most bots. It’s impossible to block bots that target you specifically. I’m not blocking any bots on my websites, I like it better to optimize performance so that I can serve them all and have a lot of headroom
No, it’s not. Had request from a bot network, always changing ip’s, user agent and request url. There is no pattern there I can block.
I created a bot black hole. Specific recursive link that if the click it three times it blocks the IP. And the link is “click if ai” URL at the bottom. Works like a charm.
No; the traffic i get seems like it had stored my sitemap from ~10 years ago because the requested urls there are long gone, and now tries to guess other url’s with a similar pattern. It does not seem to follow links from the frontpage.
That’s fascinating. It almost sounds like they’re training a scraper. Which, really, is far and away the worst possible bot to have.
Training a scraper? Sounds more like they’re just scraping training data.
Maybe they’re dating train scrapings
They’re straining craping dats
You’re entirely right, but I just wanted to post something innocuous to confirm that I picked up a stalker. Sorry for involving you!
I have fail2ban take care if those. 404s and all that.
I already looked into something similar, IP banning does not work. One day i got ~120k unique IPs, but the most active one only made ~100 requests. I would gain nothing if i run any script to lookup the IP with each requests (and using up more resources in the process), if it fails 99% of the time because the next hit is a different IP.
Ok. Sounds like your in a unique situation. Good luck. You seem fucked.
All the above worked for me.
I also have the same problem as who you’re replying to on a few of my websites. The only way to block them is Anubis or Cloudflare WAF.
I think you might be in the unique situation, to be honest. It may only be a matter of time until your solution also stops working.
Agreed. The current situation seems to be a gazillion bots each making a single request. I’ve had some luck blocking whole ASNs (Alibaba, I’m glaring at you), and Anubis. I’ve had pretty good uptime the last few months.
There surely is ja4 but I recognize it requires some setup
I wish Traefik exposed the ja4 fingerprint. There’s an open issue: https://github.com/traefik/traefik/issues/12421