Dan Frost's blog

We have introduced a package for smaller websites at only $19.99. Click here here for details.

This is based on feedback from customers and other SEO professionals.

As you can see, there's a spiffing new website and reporting centre.

Crawl Score is now a commercial product and there are various packages available - we've tried to keep it as affordable as possible. We would have loved to keep it free but the amount of server(s) resource and bandwidth being used is costing us a fortune!

So now the "big company changes" are all sorted we're back on the ball.

There will be some big changes to this site and the reporting site over the coming weeks.

We will be having yet another new look for this site and the reporting centre will be tidied up and carry the same overall feel.

As most of our regular users will know, we've had a few issues of late.

Because of some major changes in the company we've took our eye off the ball with crawlscore and for that, we're very sorry.

We have just spent some time housekeeping and trying to get everything back in line but it might take a while for some of the backlog to be dealt with.

It's always embarassing developing a product in plain view... We're massively and shockingly behind with the new crawler and it's put everything back.

We can see the light at the end of the tunnel though and we're hopeful of having it live in October still.

On the surface all appears quiet but below the surface our little legs are going like mad!

That's how it is for us at the moment. We're frantically trying to get a completely new version of the crawler ready so we can start to add more features to the reports.

We've got to re-design this site, spruce up the reports interface and test, test and test again.

From here on in I'm going to refer to Google, Yahoo and MSN as "GYM". As far as I'm concerned, they're the three search engines most people concern themselves with.

We've been doing some work on 304 related stuff and come to a few conclusions.

A site we're involved in has been in the supp index for a while due to some mistakes on our part and some on Google's part.

The site has around 300k pages with the vast majority in supp - they all used to be in the main index and attracted significant traffic (collectively) from long tail searches.

We seem to be getting a number of people signing up to crawlscore who have issues with Google Mini and/or Search Appliance.

It seems that people are not able to find out why the Appliances aren't crawling their sites properly.

It really depends on how much hassle is involved in creating them. Obviously if you're using crawlscore then it's no hassle but please don't think that sitemaps are a silver bullet to your indexing problems.