Other Sites
Many people end up on this site because they have questions about specific search engine robots and search engines. For such questions the best place is the relevant's site's own help pages:
Extensions to the Robots Exclusion Protocol
Recently three major search engines have collaborated to support extensions to the /robotst.txt directives and related mechanisms. See the join announcements on:
Sites about Search Engines
Other sites that are useful to webmasters: