Collaborative search engine crawler study Posted by Dan Frost on Fri, 22/02/2008 - 13:36
We're pondering on creating an open, collaborative study into how often web crawlers visit websites. The idea is that with minor PHP/Apache mods you can turn robots.txt into an executable PHP file and basically submit the fact it has been requested to a central resource. This service would record all bot visits, updating the central list of bots dynamically, and users can opt whether to publish their individual statistics or just their stats contribute to the overall statistics summary. The big idea is that webmasters have a resource of how often sites are being visited and relate that to actions they have taken (ping, submit to directories, etc). The project and website would be community driven (using Drupal, obviously) where users can help each other if possible. If you are interested in taking part, please let us know. |
Inter