The Best Robots.txt File
The robots.txt can prove a useful tool if you use it correctly. Here are some pointers on to exactly what you should include to help search engines index/ crawl your site more efficiently. Sitemap Reference When you submit your sitemap to Google webmaster tools, your sitemap will only get indexed in Google. For your sitemap [...]