Webguide Home | Search
New York University

FAQs

How can I prevent search engines from indexing my site?

From the The Web Robots FAQ

What is a WWW robot?

A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced.

Web robots are sometimes referred to as Web Wanderers, Web Crawlers, or Spiders. These names are a bit misleading as they give the impression the software itself moves between sites like a virus; this not the case, a robot simply visits sites by requesting documents from them.

How do I prevent robots scanning my site?

The quick way to prevent robots from visiting your site is put the following two lines into a file called robots.txt in the root directory of your site:

User-agent: *
Disallow: /