What is a large website?

We've had some feedback from customers asking for a version of Crawl Score to crawl their 10-30 page websites saying that our starter package with 100 page limit is too much and it got us thinking.

What is a large website? We've created sites with over 350k pages which we thought was pretty big but some of our customer sites have 4 million pages and one prospective customer has over 20 million pages.

We've just crawled a 1 million page site with no problems so we're quite pleased with that. There's a 30 million page crawl going through as we speak...

When we planned the architecture for the platform we leaned heavily on databases rather than trying to get too cute with arrays, temporary files and the like and that's paid off. We know that Google Search Appliances have page limits which is likely to be a by-product of their memory intensive principles. It's not that we disagree with their basic approach, it's just that to scale, sometimes you have to use the "power" of existing tools.

Our past experience of creating quite large web applications has helped us to ensure Crawl Score can scale to some fairly impressive levels. Microsoft get some bad press for one reason or another but for enterprise level applications, we think they take some beating.