We’ve noticed a significant increase in bad bots crawling our servers, causing high load and ignoring robots.txt directives. Many of these are AI bots, along with some overly aggressive search engine crawlers. This might not be an issue for larger servers, but for smaller setups, it can easily cripple a website. A few things we’ve…