You are the Widget Queen. You eat, breathe, and live widgets. You sell more widgets than anyone. You want to reach more widget customers, so you have decided to sell widgets on the web. You have spared no expense in designing and building the ultimate widget website. You have widget descriptions; you have widget specifications; you even have widget movies. The only thing your widget website does not have is visitors.
Off to the search engines you go. You type in the phrase “left-handed blue widgets” and look at the results. All of your major competitors are listed. There are even competitors you have never heard of. But you, the Widget Queen, do not have a listing there.
What’s up with that? What follows is some very basic introductory material followed by some advanced technical details on dynamic sites and SEO.
What is a search engine?
First of all, you need to understand what a search engine actually searches. When a potential visitor does a search in a search engine, such as Google or AllTheWeb/FAST, she is not really searching the web; rather, she is looking at a database compiled by that search engine. This database consists of the text and links from the web pages that have been visited by the search engine’s robot.
How is a search engine database compiled?
Search engines compile these databases automatically using software programs called “robots” or “spiders”. These automatic programs visit pages on the World Wide Web, much as humans visit web pages using browsers, by starting at some arbitrary location and following links. When a website owner “submits” a page to a search engine, in most cases she is supplying the search engine’s robot with a starting point for their automatic journey. Starting in that location, the robot then follows links and thus “discovers” other pages in your website or visits other sites to which your site is linked. (This, by the way, is how search engines can find individual pages or whole sites that have never been submitted to them–if there is a link to one site from another site, chances are good that eventually a search engine robot is going to find that link and follow it.)
Even though robots visit pages like human visitors do, what they can do with what they “see” is quite different. When a human visitor uses a browser to view a web page, that visitor can read the text on the page, look at images, play movies, listen to sounds, submit information in forms, follow hyperlinks, and any number of other tasks. The human visitor really interacts with the site. The search engine robot, on the other hand, can only do a few of these things. It is this difference that can keep your dynamic page from being included in the search engine database.
What does a robot do?
Search engine robots are very simple creatures. They can “read” text, and they can follow links. That’s it. Robots cannot view a Flash movie, they cannot fill in a form, and they cannot click a “submit” button. What that means is that no matter how much great information your web page may contain, if a visitor has to select it from a list, or type a password, or submit a form full of information to get there, no robot will ever visit that page.
The origins of dynamic pages
Most dynamic web pages are generated in response to queries run against databases. Behind your widget website there is a large database of widgets. When a visitor comes to your site and looks for left-handed blue widgets, it is this database that supplies the response. The database provides that information to the visitor. Typically the visitor checks a box or selects from a list or even types text onto the page and presses a “submit” button. Once she jumps through those hoops, your visitor gets her page full of left-handed blue widgets.
I can’t see you
Unfortunately, when a search engine robot visits this page, it cannot check that box, it cannot select from that list, and it cannot click the “submit” button. Put simply, the robot cannot get to page of widgets. If the robot can’t get there, the page will not be included in the search engine database. If it’s not in the database, searchers cannot find it.
So how do you get there? So how do we attract other visitors to our dynamic page of left-handed blue widgets? There must be some way to get there without having to click on that “submit” button.
Next month we will look at several ways to get search engine robots to visit dynamic web pages. Stay tuned.
Copyright 2003 Search Innovation Marketing. All Rights Reserved.
Dale Goetsch is the Technical Consultant for Search Innovation Marketing
(http://www.searchinnovation.com), a Search Engine Promotion company serving
small businesses and non-profits. He has over twelve years experience in
software development. Along with programming in Perl, JavaScript, ASP and
VB, he is a technical writer and editor, with an emphasis on making
technical subjects accessible to non-technical readers.
