Tag Archives: crawlers

Blocking malicious crawlers or scrapers in Apache

Occasionally we see a customer who has a popular website that often gets people trying to crawl it and copy the lot, This has the unfortunate side effect that its hammering the site. Made worse only by dynamic pages and … Continue reading

Posted in HOWTO | Tagged , , , , , , , | Comments Off on Blocking malicious crawlers or scrapers in Apache