A robots.txt file contains directives for search engines. You can use it to prevent search engines from crawling specific parts of your website and to give search engines helpful tips on how they can best crawl your website. The robots.txt file plays a big role in SEO.
When implementing robots.txt, keep the following best practices in mind:
Be careful when making changes to your robots.txt: this file has the potential to make big parts of your website inaccessible for search engines.
The robots.txt file should reside in the root of your website (e.g. http://www.example.com/robots.txt).
The robots.txt file is only valid for the full domain it resides on, including the protocol (http or https).
Different search engines interpret directives differently. By default, the first matching directive always wins. But, with Google and Bing, specificity wins.
Avoid using the crawl-delay directive for search engines as much as possible.
0 Comments