Do you have a search site? Submit it free to the internet's best search industry directory. » Click Here Search Engines Google, Yahoo, MSN... Search Marketing Marketing, Budget, Planning... Pay Per Click Bid, Price, Quality... SEO Companies Optimization, Manage, Company... SEO Tools Track, Search, Create... Analytics Statistics, Counter... » Submit your site for FREE « More... Get Your Site Submitted for Free in the World's Largest B2B Directory! Email Address: * URL: * *Indicates Mandatory Field Terms & Conditions Titan Quest Forum Nintendo Wii Graphics Forum Halo 3 Forum Mac Software

Google has added yet another handy feature to the Google Webmaster Tools. Now website owners can easily create Robots.txt files and control which web pages search engines see and "don't" see on their sites.

For those of you unfamiliar with Robots.txt files, Wikipedia explains them:

A robots.txt file on a website will function as a request that specified robots ignore specified files or directories in their search. This might be, for example, out of a preference for privacy from search engine results, or the belief that the content of the selected directories might be misleading or irrelevant to the categorization of the site as a whole, or out of a desire that an application only operate on certain data.

Now if you're not overly confident creating your robots.txt files from scratch, you can simply create one using your Google Webmaster Tools.

As the Google release post suggests:

"It's designed to give you an easy and interactive way to build a robots.txt file. It can be as simple as entering the files and directories you don't want crawled by any robots."

The tool allows users to either create broad exclusion rules which apply to all robots and crawlers or specific rules for particular robots. See the screenshots below.

Once you've created the robots.txt file using the tool, all you have to do is upload it to the top level directory (root) of your website and voila - you're all done.

As the release post highlights, not all robots obey the conventions of the Robots.txt protocol, but rest assured most of the major search engines do, which is where most of your website crawling will come from.

Del.icio.us

Digg

Reddit

Furl