RSS

Spider Suicide: WMW Out of Google, MSN

Thu, Nov 24, 2005

SEO

WebMasterWorld
And they actually wanted this to happen? Wow.

To free up bandwidth and system resources, Brette Tabke of WebMasterWorld.com decided to ban all spiders including those of the major search engines. It took Google and MSN just two days to remove WMW from their index. A site:webmasterworld.com search used to show around 2 million pages. Now, the same search returns zero pages. Check out WMW’s robot.txt file.

The funny thing about this “issue” is this:

Brett was talking about doing it about 3 years ago, but it wasn’t because of bad spiders, which won’t take any notice anyway. If I remember correctly, it was to do with getting too much unwanted traffic from the engines.

Arrogance.

Why don’t they just make it into a full paid subscription based model while they’re at it?

But then again, without a reliable alternative search facility onsite, the paid subscription model may not last long either:

  • I paid a subscription to access this site and have been doing in for 2/3 years and now i find i cant even search the site at all. Before there was site search which was okay, but you removed that in place of a Google site search, now thats removed, we can’t search for anything! Please bring it back.
  • Seems to me that the proper way to have done this would have been to implement search first, then ban the bots. Eat the traffic until you’ve got search working. Though why a site populated by SEO’ers doesn’t have a search feature is beyond me. And it’s not like we woke up this morning and realized OMG#*$!BBQ! I just realized! We don’t have search! This problem has been unresolved for ages.
    In terms of your search numbers, I don’t use search here because I can’t be bothered to figure out how to run a search. How bizarre is that?

    The fact of the matter is that the community doesn’t care about ‘why’. They just want you to make it work.

    I’m sounding like I’m attacking, let me apologize in advance for that - I don’t want to do that. I do want to suggest succinctly and pointedly that there’s a number of community members unhappy about what’s going on, and from here it tastes and feels like they’re being brushed off. Switch it around and look at it from the customer’s viewpoint not the internal workings of your org.

With honest comments like that you’d think WMW would listen and start work on a working onsite search facility already, but no… I guess everyone’s still busy highfiving and congratulating each other for this “brilliant” linkbait. Nevermind that they probably get 60% of new subscribers via search engines. Nevermind that you have hundreds of paying customers searching for forum treasures in the dark.

In the end, WMW was successful in creating two needs: WMW and WMW search. Don’t be surprised if you see a fully functional WMW search facility for paid subscribers in the not so distant feature (and if we’re lucky, a watered down version for us cheapskates).

What’s happening with the SEO forums? First, SEOChat. Now this. Can somebody start a new forum so I can feel warm and fuzzy again?

This post was written by:

Marc - who has written 605 posts on Macalua.com.


Contact the author

2 Comments For This Post

  1. vern Says:

    I don’t see how anyone but those who are subscribed to the site has any right to complain. It is their server, their choice. Plus data is stored in their database, the most efficient search would be a search through that database done in-house rather than through Google. Then again that would cause extra load on the servers themselves. It is always a tradeoff between what is ideal for the users and what is ideal for the systems administrator, or perhaps in this case, the wallet administrator.

    I have a feeling they might be a big too aggresive with limiting access to their data though. Before I switched to DSL (Cable before), my previous ISP couldn’t access their servers at all. It was a permanent Apache error and thus done through their settings and not my ISP’s fault.

  2. Andre Says:

    well great now noone can find anything on webmasterworld :)

Leave a Reply