How do I slow down Site Auditor's crawl of my website?


In some rare cases, you may need to slow Site Auditor down so that it can fully crawl your website. This usually is necessary when your website has flood protection turned on or its bandwidth capabilities are impaired or otherwise minimal. For most websites, this will not be a factor — but some may find this trick useful.

If you'd like to slow down Raven's crawler, add this text to your robots.txt file:

User-agent: RavenCrawler
Crawl-delay: 5

This will tell RavenCrawler, the user-agent for Site Auditor, to wait five seconds between each page. You can increase or decrease this number as necessary, but keep in mind that Site Auditor's crawl time maxes out at three hours. Adding a Crawl-delay can prevent Site Auditor from crawling your entire website.

By default, Site Auditor delays itself for two seconds between crawling pages.

Have more questions? Submit a request