Em Adespoton

  • 2 Posts
  • 4.23K Comments
Joined 3 years ago
cake
Cake day: June 4th, 2023

help-circle



  • One other hurdle to note specifically in this time: because of the absolute flood of AI bots that tend to hammer popular sites and ignore robots.txt, many web servers out there have moved to a bot allow list — which means if you’ve got a new properly behaved and reporting index bot, you’ll likely find it gets blocked by default on a LOT of the Internet.