The migration of the Slurp is complete, says Yahoo. Over the past few weeks, the search engine has been transitioning its crawler, dubbed (disgustingly) “Slurp,” to a new address at crawl.yahoo.net. Adjust your server logs as necessary and join the curmudgeons who are unimpressed.
Too little (or too much, by some complaints), too late, it would seem.
The Yahoo Search Blog reads:
…all machines crawling as Slurp are now in crawl.yahoo.net. You can see this change in your web server logs, where the page accesses from inktomisearch.com are being fully replaced by crawl.yahoo.net contacts. Note that this does not cover other Yahoo! crawlers, such Yahoo! China, and other verticals, like Yahoo! Shopping, Yahoo! Travel, etc., which have their own user-agent.
Don’t fret though; there is no need to change your robots.txt file because the crawler user-agent is still Yahoo! Slurp. If you use IP based filtering, there is no need to change that either, since the IP addresses from which we crawl remain the same. However, please ensure that your network or firewall setup does not keep crawl.yahoo.net out as we won’t be able to include your content in our results.
Be sure to click that link to get more enumerated information.
Over at WebmasterWorld, the crowd is a bit mixed about it (but only a bit), the loudest complaint, from “IncrediBill,” who notes not only is it the move a year-and-a-half too late, but has gone overboard.
Why do we need to allow an army of Yahoo spiders to redundantly abuse our servers?
Is it a conceptual problem that Yahoo can’t share pages already downloaded?
When I posed that question to one of their engineers I was given a lame excuse that the various crawlers had different needs….
Funny, Google managed to make some of their crawlers share CACHE, so we know it can be done.
Negativity often rings louder and truer than other things, but there is at least one voice in that forum who thinks Yahoo’s update is “a small evolutionary improvement above” Google.
Even if in our hearts, we know that’s not true. 🙂
