The field of search engine optimization is a constantly evolving industry with important changes being made almost daily.
In the beginning, search engine optimizers focused on one thing only: rankings. Tracking of actual sales or conversion rates was almost unheard of.
Slowly, search engine marketers began to realize that all of the 1’s in the world won’t help if they don’t convert to traffic, and all the traffic in the world won’t help if it doesn’t convert to sales.
So, search engine experts began analyzing their log files and tracking sales. With today’s technology, Web site owners can now tell which of their marketing campaigns are truly effective and which need to be replaced by a different campaign. In other words, through log analysis, you can analyze your human traffic and their visits to your Web site.
But what about your “spider” traffic?
One part of log analysis that has remained surprisingly lacking, both in terms of content and accuracy, is spider analysis.
Though traffic analysis programs may look at spider activity, the information often isn’t detailed enough or presented in a format to do you much good. Also, spider and robot analysis is acknowledged as being a main culprit for inaccurate log analysis measurements. internetnews.com/IAR/article/0,,_908361,00.html
Therefore, the need for detailed spider analysis has begun entering the minds of search engine marketers.
So, what is “spider analysis”?
You know that when you submit a Web page to an engine for indexing, the engine sends a spider to your site to index the contents of the page. “Spider analysis” is simply analyzing the search engine spider visits to your site.
Through effective spider analysis, you can learn the following information about your site in a concise, easy-to-read format.
* Has your site been spidered? * If so, by which engines? * When did the spiders visit? * Which directories and pages did they visit? * Are certain pages getting respidered more often, signaling their importance to the search engines? * Are certain pages not getting spidered at all? * Are the spiders indexing inappropriate content? * Are the spiders getting everything they want and need, or are they receiving error messages? * Was your site spidered within the specified time agreed upon in the pay inclusion programs you’re participating in? * Is your site getting respidered on a regular basis, as agreed upon in your participating pay inclusion programs?
Another important issue in spider analysis is robots.txt files.
Though the object of search engine marketing is to help spiders find all the pages on your Web site, there will be times when you want to keep spiders out of certain pages. You can do this with a robots.txt file.
What is a robots.txt file?
A robots.txt file is a text file that is placed on your server that instructs the search engine spiders not to crawl or index certain sections or pages of your site.
But why would you want to keep the spiders out?
Let’s say that you’re creating a new page for your site, and you’ve placed the page online while you tweak and edit it. The page isn’t ready for visitors, so you wouldn’t want it indexed yet. Or, let’s say you’ve placed some employee guidelines on your site. The guidelines are of interest only to your employees, and there’s no reason for the public to view them.
Using a robots.txt file, you can keep the spiders out of those pages.
Therefore, when working with spiders or robots, you want to be able to:
*Create a robots.txt file quickly and easily; * Use a robots.txt file to present optimized pages to specific engines. For example, using a robots.txt file, you can focus English language robots onto the relevant pages and direct robots from international search engines to the localized content areas of your site; *Send e-mail harvesting programs away from your site to keep your e-mail spam down; * View highlighted pages requested in error by the spiders; and, * Direct search engine spiders to relevant areas of your Web site.
Robin Nobles conducts live SEO workshops
(http://www.searchengineworkshops.com) in locations across North
America. She also teaches online SEO training
(http://www.onlinewebtraining.com). Localized SEO training is now
being offered through the Search Engine Academy.
(http://www.searchengineacademy.com) Sign up for SEO tips of the
day at mailto:seo-tip@aweber.com.
