Occasionally we see a customer who has a popular website that often gets people trying to crawl it and copy the lot, This has the unfortunate side effect that its hammering the site. Made worse only by dynamic pages and loops this can literally take down a server in some occasions. Often you can slow…