Do you have a search site? Submit it free to the internet's best search industry directory. » Click Here Search Engines Google, Yahoo, MSN... Search Marketing Marketing, Budget, Planning... Pay Per Click Bid, Price, Quality... SEO Companies Optimization, Manage, Company... SEO Tools Track, Search, Create... Analytics Statistics, Counter... » Submit your site for FREE « More... Get Your Site Submitted for Free in the World's Largest B2B Directory! Email Address: * URL: * *Indicates Mandatory Field Terms & Conditions Titan Quest Forum Nintendo Wii Graphics Forum Halo 3 Forum Mac Software

...almost multiple websites.

In a forum thread at Webmaster World there are numerous reports of the same phenomenon that is giving Webmasters sleepless nights. Some of the excerpts from this thread are mentioned below:

I monitor several sites with more than ten thousand pages. On all of those sites the GoogleBot crawl rate has dropped by at least 75%. Is this a phenomenon that others are seeing as well.

I am observing this on my own site, and sites run by several others. The sites are not related (different ownership, development teams, link strategies, hosting facilities, etc).

The graphs in Google webmaster tools (site -> tools -> set crawl rate) show that pages and kilobytes per day have dropped drastically, almost to zero in April. The graph of time spent downloading a page has not changed.

Are other people observing this phenomenon too?

I can verify that our main site is having the exact same thing occur. The crawling stats were not updated for most of March and when they came back in April, they were almost totally flat.

Our logs verify that googlebot is now 3rd behind Y and then MSN. For the last few years it was always first by a large margin.

I see 90% less crawling and indexing all over the place, maybe they just need their horse power to crank out another penalty?

Glad to see this topicI have millions of pages spread across hundreds of sitemaps. I parse my logs daily to see what Googlebot is doing, and my totals for April are 1) way down from previous months and 2) much more volatile from day to day. I saw a lot of days in February and March in the 150,000-200,000 range of pages crawled per day. By comparison, here are the last couple weeks:

For example:
4/14/2008 139,279
4/15/2008 109,410
4/16/2008 83,809
4/17/2008 2,019
4/18/2008 79,594
4/19/2008 95,466
4/20/2008 92,461
4/21/2008 101,422
4/22/2008 8,410
4/23/2008 3,195
4/24/2008 3,667
4/25/2008 69,842
4/26/2008 65,134
4/27/2008 88,823
4/28/2008 117,792
4/29/2008 123,555
(note that these numbers include both general spidering and sitemap crawling)

Those three days starting 4/22 had me tearing my hair out. I did an early check on today, and I can see the total is going to be considerably down again from yesterday (probably sub-50,000).

A month ago I was hoping to spend some time seeing if I could speed my pages up even more to get Googlebot firing through my site even fasternow I'm afraid to do anything, since spikes in the crawl could be a result of something I screwed up or Googlebot just deciding to shut down for three days.

Back in March I introduced gzip to my site"I expected it to speed up Googlebot, but it seems to have done nothingbut then again, with spikes like these, how would I know?

Del.icio.us

Digg

Reddit

Furl