June 27, 2017

Yahoo! Block Google From Delicious Bookmarks?

During the weekend, Yahoo!’s social bookmarking property del.icio.us seems to have prevented robots of other search engines including Google from spidering the website or fishing out fresh web pages, bookmarks and websites, observes Collin. It was clear that it was not a simple robots.txt exclusion, rather it was a 404 response that was being shown based on who the requesting User-Agent was.

Yahoo! Block Google From Delicious Bookmarks?
Yahoo! Block Google From Delicious Bookmarks?

An observation was made that when the User-Agent was set to Googlebot the 404 error showed up each time, and the same response was received when any page other than the homepage was attempted to be navigated.

“I took a look at del.icio.us’ robots.txt,” says Collin “and found that it was disallowing Googlebot, Slurp, Teoma, and msnbot for the following:

  • Disallow: /inbox
  • Disallow: /subscriptions
  • Disallow: /network
  • Disallow: /search
  • Disallow: /post
  • Disallow: /login
  • Disallow: /rss”

Why on earth Yahoo! may have done this, competition or what. As of now Yahoo has added delicious in its search pages, and that delicious is an important property for Yahoo to bank upon. Apparently, it has begun to make use of its powers from blocking other search engines from taking advantage from spidering the information contained therein.

Moreover, Yahoo! also has an edge over Ask, Google and MSN and that is, it’s able to notice the web pages, etc. through human bookmarking, tagging etc. on Delicious, even before other search engine can. Consequently, it leads to better search results and rankings.

In this case Yahoo seems to have made use of its discretionary powers to combat competition. While preventing others from examining your business secrets is a common strategy used by business, it is quite a bold move from the company. What are your thought about it?


Tag: , ,

Add to Del.icio.us | Digg | Reddit | Furl

Have a bookmark! –

Navneet Kaushal
About Navneet Kaushal 1137 Articles
Nav is the founder and CEO of Page Traffic, a premier search engine company known for its assured SEO service, web design and development, copywriting and full time SEO professionals.

Navneet has wide experience in natural search engine optimization, internet marketing and PPC campaigns. He is a prolific writer and his articles can be found in the "Best Articles" section of many websites and article banks. As a search engine analyst , he has over 9 years of experience and his knowledge is in application here.