Now Search Engine Roundtable reports that, over at the Google Groups, Matt Cutts and JohnMu have finally spoken up about why and how Google blocks ranking checking tools.

This is what Matt Cutts had to say:

Hi Scott. Google does use algorithms and different techniques to block
excessive automated queries and scraping, especially when a someone is
hitting Google quite hard. The reason is that scraping consumes server
resources. We don't want real users to be slowed down or affected just
because a bot is sending bunches of automated queries to Google.

We do turn off a number of tools/bots/IP addresses that scrape us too
heavily. It's a common enough phenomenon that we did a blog post on
Google's Online Security Blog about the subject.
In fact, I know that just a week or so ago our algorithms turned off
an IP belonging to one of the entities that you mentioned in your
post.

In general, I would approach the bizdev folks at Google about how to
send automated queries to Google with permission. Failing that, be
aware that if a tool sends too many queries to Google, we do reserve
the right to disable the IP address(es) of that tool. One thing I
would *not* recommend is that if a tool is blocked for bad behavior,
trying to make the tool more "sneaky" (e.g. trying to make the tool
look closer to a web browser). Attempts to fake out Google and pretend
to be more like a web browser (after you've been blocked once already)
is an example of the sort of thing that is really bad in our opinion.

JohnMu further stated:

Hi Scott and a belated welcome to the groups!

This is an interesting topic. I'm not quite sure how all those links
apply to our terms of service, which do not allow these kinds of
automated tools. I wrote a bit about tools like that a short while
ago:

http://groups.google.com/group/Google_Webmaster_Help-Requests/browse_

A tool accessing other websites should try to obey the rules set forth
by that website. In general, these rules are described in several
ways:

1. The robots.txt covers which URLs may be accessed and which ones are
   disallowed. You'll notice that in our robots.txt we explicitly
   disallow "/search", which is what most of the ranking tools generally
   try to access.


2. The server result codes give more information when a URL is
   accessed. When our network recognizes automated queries, it may return
   a result code of 500 or similar.
3. The HTTP headers returned by the server can provide information
   through the "x-robots-tag".
4. A HTML page may provide information through a "robots" (or in our
   case, "googlebot") meta tag.

As far as I am aware, there are no "SEO-tools" that have permission to
access our web-search results in an automated way. I am also not aware
of any plans to change that in the near future.

Del.icio.us

Digg

Reddit

Furl