Today I set up a little project website on a new subdomain. It’s not a www subdomain or a newly registered domain, which is easy to detect. We’re talking about:
Within 20 minutes, the anthropic ClaudeBot was on it. I could tell because the nginx access log showed a hit to robots.txt and then a handful of pages.
First off, how the hell did they find it? Next, is my DNS provider, Amazon Route 53 selling this kind of data now? Or is there some kind of DNS wildcard query?
What’s that thing Google is pushing, where the CAs basically push a list of all the certs they issue? Is that live? Maybe Amazon issued you a key, and then published it in a list of “domains I’ve issued keys for”, and they’re just watching that list?
Unless that’s not a thing, or not a thing yet, or I’m fully misremembering…
That thing is called „certificate transparency logs“