Difference between revisions of "Robots.txt"
From Teknologisk videncenter
m |
|||
| Line 1: | Line 1: | ||
| − | Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell | + | Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell Web robots (crawlers) like Google bot which folders or homepages you don't want indexed. |
robots.txt on [http://mars.tekkom.dk mars.tekkom.dk] | robots.txt on [http://mars.tekkom.dk mars.tekkom.dk] | ||
Latest revision as of 10:28, 11 January 2012
Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell Web robots (crawlers) like Google bot which folders or homepages you don't want indexed.
robots.txt on mars.tekkom.dk
User-agent: * Disallow: /cisco/ Disallow: /sw/ Disallow: /TCExam/ Disallow: /fs/