How can I prevent search engines from indexing my site?From the The Web Robots FAQ What is a WWW robot?A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced. Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them. How do I prevent robots scanning my site?The quick way to prevent robots from visiting your site is put the following two lines into a file called
User-agent: *
|
