Robots Gone Wild
Published on: Jun 12, 2008
Every serious website should be using a robots.txt file. WordPress blogs can especially benefit by blocking all their duplicate content from being indexed. Just recently, all three major search engines are now supporting wildcards for robots.txt files.
The Robots Exclusion Protocol, or the robots.txt file, takes a negative approach to what search engines are allowed to index. Originally, website operators used it to keep certain information on their websites private. Today, the robots.txt file can, also, be used to prevent indexing of certain webpages that could penalize your site for duplicate content in Google.
The protocol has always been very crude and archaic to work with. But, Google has given it new vitality with their introduction of wildcards support. Fairly recently, both Yahoo and MSN have started supporting Google’s Wildcard extensions. Now that all three major search engines work the same way, using a robots.txt file has become a whole lot easier.
You can read more complete information on how the major search engines support the Robots Exclusion Protocol, respectively on Google, Yahoo, and Microsoft.
You should study the robots.txt file that I am using on this blog for how to best prevent duplicate content issues in WordPress, provided that you also are using Google Friendly URLs.