spider

For most sites, one of the first things we check is to make sure the site crawler friendly. "Crawler friendly" you say? What the heck does that mean?

A funny story is circulating in tech circles about how Googlebot inadvertently destroyed the database of a content management system (CMS) based site that took months of work to build.

Around the time Google announced "Big Daddy," there was a new Googlebot roaming the web. Since then I've heard stories from clients of websites and servers going down and previously unindexed content getting indexed.

The purpose of this article is to provide evidence and information to counteract the suggestion that Assertivenet is potentially used for malicious purposes.

With the growing use of broadband, more and more website developers are looking to Flash to create a more engaging visitor experience. The problem with Flash sites, as many have pointed out, is that they can be can be difficult optimize, especially since Google seems to be the only search engine that tries to crawl them-to a limited extent. But there's more than one way to skin a spider.

Mechanical spiders have to eat. In fact, they usually have bigger appetites than the real-life spiders you squish under your shoe.

Some web pages on your website may not be listed in the search engines, even when you have submitted your website for inclusion. This is because dynamic pages cause the spiders give up and go home.

Reader question: I think my site has been banned in Google. It used to have all of these top positions for the past three years, and it suddenly disappeared. Can you help?