Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell Web robots (crawlers) like Google bot which folders or homepages you don't want indexed.

robots.txt on mars.tekkom.dk

User-agent: *
Disallow: /cisco/
Disallow: /sw/
Disallow: /TCExam/
Disallow: /fs/

Links