When I first saw this news I thought it was some sort of news parody. Who would think that WebmasterWorld wouldn’t want traffic from search engines.
Danny Sullivan over at SearchEngineWatch describes the details here and here.
Apparently, Brett Tabke, the owner of WebmasterWorld, has taken the drastic decision to disallow spiders “in an effort to combat bandwidth loss and server sluggishness due to rogue spiders”. Anyone can ban a spider via a robots.txt file. This has had the effect of removing over 2 million indexed pages out of Google and eliminating WebmasterWorld from the Google Directory which is a mirror of DMOZ.
Why would Brett not want search engine traffic? For a high profile site with (likely) millions of pageviews a month coming from Google and other search engines this must have been a difficult decision. It’s also kind of ironic since Brett just put on an excellent and successful conference (PubCon) which was mostly about search engines. PubCon had multiple speakers from Google, Yahoo, MSN, Ask Jeeves and quite a few SEM types.
And what about GoogleGuy and MSN Dude which hang out in the WebmasterWorld forum? Will this effect their participation in the forum? By in effect delisting the search engines is WebmasterWorld making the case that Google and others are irrelevant? (Brett says no below)
WebmasterWorld is a great resource for all of us and it will be interesting to see how this affects their pageviews. It certainly is a unique experiment for a high traffic site to not allow search engine crawling.
Update: Barry Swartz of Search Engine Roundtable interviewed Brett Tabke on his decision and Brett said this was a security issue.
“It is difficult to talk about issues that brush shoulders with security related matters. Once you talk about something and your actions to counter that problem in public, you give rise to an invertible counter measure.”
Brett was also asked by Barry to respond to the claim by some “that you wanted to show the search engines that you do not need them”. Brett responded:
“Yes, a hundred thousand targeted referrals a day are just plain wrong. Lets cut to the chase; I adore search engine traffic, but my first duty as a webmaster is to the visitors and members of our site. Anything that interferers with that to the degree that rogue spiders, downloaders, offline browser, monitoring services, site rippers, or whatever you call it – I have to take action.”
Obviously, Brett felt this was the best course of action in order to protect the security of WebmasterWorld and the user experience of the site. It will be interesting to see what the PR impact will be. It is still a bit shocking to me to block search engines when you run a popular forum and a conference about them.
To access all content on WebmasterWorld you can subscribe here.
murdok is the CEO of murdok which publishes over 200 websites and email newsletters.
Rich also publishes his blog WebProBlog which focuses on internet business and marketing trends.
