Blocking malicious crawlers or scrapers in Apache

Occasionally we see a customer who has a popular website that often gets people trying to crawl it and copy the lot, This has the unfortunate side effect that its hammering the site. Made worse only by dynamic pages and loops this can literally take down a server in some occasions. Often you can slow… Continue reading Blocking malicious crawlers or scrapers in Apache