A robots.txt file can be used to control which webpages of a website that the site owner has given a search engine permission to index.

Are You Using a Robots.Txt File?

The robots.txt file must be located in the root directory of the website, otherwise search engines will ignore it. Correctly written robots.txt files follow a very old and, thus, crude standards. The robots.txt file takes a negative approach. It lists only what should be excluded or ignored by search engines.

Of course, whether or not a bot chooses to follow your robots.txt file instructions is purely voluntary on their part.