Google has informed me that a link can be removed from Google
if the webmaster of the site uses a robots.txt file. However,
Google can not confirm if other engines can do the same-MSN,
Yahoo etc. Should a webmaster be able to remove a link from
other search engines by using the robots.txt file described by Google?
First, let’s be clear – it won’t remove anything. I assume
that someday the link will disappear off of Google as it
loses relevance, or maybe Google does something more related
to cleaning dead links.
But robots.txt is still important for several reasons.
Become a Patron of Ask Leo! and go ad-free!
Robots.txt allows you to tell Google not to look at the
pages in the first place. Google doesn’t look, and the
contents don’t get added to its index.
All search engines are supposed to pay attention to
robots.txt. And most well behaved major engines do. MSN,
Yahoo, and so on are all major engines and do the right
However do know that there are other search engines
and crawlers that don’t
pay attention and will spider your site anyway.
Search Engine World has a good tutorial