Difference between revisions of "Robots.txt"
From Teknologisk videncenter
m (New page: Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell WEB robots (crawlers) like Goggle bot which folders or homepages you done want indexed. ro...) |
(No difference)
|
Revision as of 07:06, 11 May 2009
Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell WEB robots (crawlers) like Goggle bot which folders or homepages you done want indexed.
robots.txt on mars.tekkom.dk
User-agent: * Disallow: /cisco/ Disallow: /sw/ Disallow: /TCExam/ Disallow: /fs/