robots.txt and Examples

Two directives:
Note: robots.txt must be at the root level of the server.

Check out some real robots.txt files!