The Best Robots.txt File

The robots.txt can prove a useful tool if you use it correctly. Here are some pointers on to exactly what you should include to help search engines index/ crawl your site more efficiently.

Sitemap Reference

When you submit your sitemap to Google webmaster tools, your sitemap will only get indexed in Google. For your sitemap to be indexed in every search engine your website is indexed in you can add the following snippet to the top or bottom of your robots.txt file :

Sitemap: http://www.example.com/sitemap.xml (must be absolute and not relative eg not /sitemap.xml).

This will tell search engine bots who use the robots.txt file exactly where your sitemap is for them to index.

Disallowing Files / Directories

Only recently I found out exactly what blocking files / sub-directories in your robots.txt does. If you block some files / subdirectories in your robots.txt, your files could still potentially get indexed by Google, however Google will not crawl the files.

The syntax for allowing / disallowing files is :

User-agent: *
Allow: /sub-directory or Disallow: /sub-directory

The default action for a site with a website without a robots.txt is to allow all, but at a bare minimum I would tell bots to allow all and include a sitemap reference. Just to be sure!

Blocking Domain Tools

Domain Tools, quite conveniently crawls your pages and lets their own versions of your pages get indexed in Google. This can cause severe duplicate content issues and can be harmful to your rankings. However I did some digging and found by adding the following snippet to your robots.txt you can stop them from doing this.

User-agent : SurveyBot

Disallow : /

Problem solved! You can always check in your webmaster tools under site configuration -> crawler access to make sure there are no errors in your robots.txt and your sitemap reference has been read correctly.