robots

The robots.txt analysis tool at Google Webmaster Central received some much-needed updating, and should be more effective for webmasters today.

Two new features added to the protocol will help webmasters govern when an item should stop showing up in Google's web search, as well as providing some control over the indexing of other data types.

Yahoo's spiders will obey a new tag, called robots-nocontent, that will allow webmasters to discretely define content on a page they do not want to have indexed.

At Search Engine Strategies New York it was announced that you can now have your sitemap automatically discovered by configuring it in your Robots.txt file. It is simple and easy to do, you’ll just need to know the URL or web address of your sitemap.

The robots.txt file. Nothing can be more confusing to a website owner as the robots.txt file. Born out of technology in the programming world, the robot.txt file is nothing more than a server command for search engines. Unfortunately, while search engines understand the file, humans have a difficult time understanding machine language.

A post on official Google blog informs about Robots Exclusion Protocol. Sometime back we informed you about a previous post on Robots.txt file.

Sometimes there are parts of your website you don't want accessed by the search engines - for any number of reasons, like sensitive private data, articles that require subscriptions - whatever.

A heaping dose of Battle Bots and movies like Terminator and I, Robot have instilled a (healthy?) paranoia of centrally controlled robot futures - at least in the States. In South Korea, not so much, as the country announced its plan to have a robot in every home - very, very soon.