How can I prevent search engines from indexing my site?
From the The Web Robots FAQ
What is a WWW robot?
A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.
Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them.
How do I prevent robots scanning my site?
The quick way to prevent robots from visiting your site is put the following two lines into a file called