I like to watch how many times my username shows up in Google, among other places. I noticed recently that twitter.com has recently made a change to their site that is causing them to spam Google and the other search engines by allowing duplicate content to get indexed by the search engine spiders.

First off, here is a little history. I also like to watch my username in the search engine results pages because I am curious as to which profile of mine has more link juice or which one is considered to be more popular. Since I use the same username it is pretty easy for me to identify my profiles (or at least the profiles that I want people to know about).

To see what I mean, try going to your profile at explore.twitter.com/username and then go to it at twitter.com/username You should NOT be able to get to both of those URLs. One of them should redirect, with a 301 Permanent Redirect, to the other one. By allowing the same exact, duplicate, mirror content to be indexed by the search engines you are, in effect, spamming the search engines, something that the search engines do not like. I have even seen websites get banned in the search engines for having mirrors like this.

I have been generally pretty excited about my twitter profile always showing up at the top of the Google search results for my username. And I have been impressed about how quickly it rose to the top of the results when I signed up for twitter a long time ago (when it first was launched).

Just recently, though, I noticed that my twitter profile changed URLs and I am now seeing a subdomain, explore.twitter.com show up in the search results, which is a mirror. In fact, since this subdomain appeared and twitter began allowing it to be indexed by the search engines, my twitter profile is no longer the top search result for my username.

This concerns me. Twitter, like it or not, is spamming Google by allowing your username on twitter.com and on the subdomain to be indexed.

To fix this, a simple change to the sites robots.txt file would be sufficient. There is, in fact, a robots.txt file here: http://explore.twitter.com/robots.txt and here: http://twitter.com/robots.txt which appears to be the same. Profiles on both the subdomain and the main domain do not need to be indexed. Thats clearly search engine spam, and is actually hurting my twitter profile.

Why, twitter, are you allowing a subdomain to be indexed"a mirror of profiles on twitter.com?

Del.icio.us

Digg

Reddit

Furl