Crawling a website with Site Auditor can take up to 24 hours to complete — 20 hours maximum for the crawl, plus any time on the front-end queue and processing ends — depending on a few different variables:
Number of other queued websites
The first consideration is the number of other websites that are currently queued to be crawled. Site Auditor crawls roughly 70 websites at a time, which means that we move through queued websites fairly quickly, but there are times when there's a backlog of websites for our systems to work through.
This is particularly the case on high volume days, like the first of the month. If your crawls are set on a weekly or monthly schedule, you can change the Next Scheduled Crawl date in the Settings page to avoid these higher volume days.
Number of pages to be crawled
Site Auditor will crawl a maximum of 10,000 pages and a large website will take longer to crawl than a tiny website with just a few pages.
You can speed up crawls of large websites by changing the Maximum Number of Pages to Crawl setting in the Settings page to a lower number: 50, 100, 250, 500 or 1000. That will cap the maximum crawl, resulting in faster completion of the crawl.
How long it takes to crawl each page
The realities and settings of your web server and website will also affect Site Auditor's ability to efficiently crawl your website. A website with poor bandwidth, for example, may result in Site Auditor crawling each page more slowly.
Additionally, if you've implemented a crawl-delay setting in your robots.txt file, this will insert a delay (measured in seconds) between each page. On a large website, this can inflate the amount of time it takes for Site Auditor to crawl your website.