Webmaster life got a little easier as Yahoo Search, in the midst of some updates to the engine, added page level exclusion tags to what it can handle.
When we first heard the news about X-Robots, we thought Chris Claremont might be writing a Sentinels as superheroes title for Marvel.
Our discussion of possible pencilers was cut short by the realization these were X-Robots-Tags, which webmasters can serve up to Yahoo’s crawlers. These directives allow the site publisher improved control at the page level of how content become indexed when Yahoo’s Slurp comes calling.
Sharad Verma posted samples of these on the Yahoo Search blog:
To take advantage of this feature, simply add the following page level tags to the X-Robots-Tag directive in the HTTP Header. Here are a few examples:
X-Robots-Tag: NOINDEX — If you don’t want to show the URL in the Yahoo! Search results.
Note: We’ll still need to crawl the page to see and apply the tag, so if you don’t wish to have the page crawled, use robots disallow on robots.txt.
X-Robots-Tag: NOARCHIVE — If you don’t want to display cache link in the search results page.
X-Robots-Tag: NOSNIPPET — If you don’t want to display summary in the search results page.
X-Robots-Tag: NOFOLLOW — If you don’t want Yahoo! to crawl links in the page.
A trip to conferences like SES Chicago or PubCon would not be complete if a search site wasn’t doing some updating at the time. Here’s what Yahoo has going on:
Along with this change, we’ll be rolling out additional changes to our crawling, indexing and ranking algorithms over the next few days. We expect the update will be completed early next week, but you may see some changes in ranking as well as some shuffling of the pages in the index during this process.
