Have you ever noticed a URL showing up in search results, even after youve blocked it using Robots.txt? Its a common query according to Google, so theyve created a video (below) to explain how this can occur.

In the video, Matt explains that while Google will not crawl content blocked by robots.txt, they may still display a link to this page (with an empty title, description etc.) if its the most relevant result to the query.

For example, if www.nasa.gov blocked all search engines via robots.txt, Google may still display this uncrawled URL in their results because theres a huge amount of people linking to this page with targeted anchor text. The search engine listing may look something like this:

Youll notice that this listing does not have a title and description, because Google was blocked from crawling the page by robots.txt.

If you want to completely remove your content from Google search results you have a couple of other options:

• Use a NOINDEX meta tag at the top of your page
• Use the URL removal tool within your Google webmaster account

Hopefully this clears up some confusion about robots.txt and blocked content. If you have any other questions let us know via the blog comments below!

Del.icio.us

Digg

Reddit

Furl