Search Robots, Crawlers, Spiders

Three mechanisms to instruct robots that visit your site:

1. robots.txt file
2. robots meta tag
3. rel="nofollow" for a elements