#Robots.txt

Robots.txt is a plain text file at the root of a website that tells crawlers which paths they may request. It is part of the Robots Exclusion Protocol, which is now documented as an internet standard. The file controls crawling, not indexing, so a blocked URL can still appear in search results if other pages link to it. Articles under this tag explain how to read, write and test the file for a blog or content site. They also cover the common mistakes that hide pages from search engines without any visible error.