crawler

If you want to know the instant Google's bot visits your site then you're a severe algoholic. And Crawler alert will help you get your fix. Crawler alert will send you an email when any of the major search bots visit the page you specify.

My favorite sessions at the SES conference were those where Google and Yahoo appeared on the same panels. You could almost always count on some crackling tension between the two search giants. The "Meet The Crawlers" session was no exception.

This tutorial will show you how to create a local file crawler that will enable you to find a specified document type (i.e. PDF files) within a directory (and it's children directories).

I couldn't find any "meaty" questions for this issue, so I thought I'd just talk generally about what makes a site "crawler-friendly." I used to call this "search-engine-friendly" but my friend Mike Grehan convinced me that the more accurate phrase was "crawler-friendly" because it's the search engine crawlers (or spiders) that your site needs to buddy-up to, as opposed to the search engine itself.