Difference between revisions of "Robots.txt"
From Teknologisk videncenter
m (New page: Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell WEB robots (crawlers) like Goggle bot which folders or homepages you done want indexed. ro...) |
m |
||
| Line 13: | Line 13: | ||
*[http://www.google.com/support/webmasters/bin/answer.py?answer=40360 Google - Creating a robots.txt] | *[http://www.google.com/support/webmasters/bin/answer.py?answer=40360 Google - Creating a robots.txt] | ||
*[http://www.thomas-rosenstand.dk/sadan-bruger-du-robotstxt/ sådan bruger du robots.txt] | *[http://www.thomas-rosenstand.dk/sadan-bruger-du-robotstxt/ sådan bruger du robots.txt] | ||
| + | [[Category:Programming]] | ||
Revision as of 07:09, 11 May 2009
Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell WEB robots (crawlers) like Goggle bot which folders or homepages you done want indexed.
robots.txt on mars.tekkom.dk
User-agent: * Disallow: /cisco/ Disallow: /sw/ Disallow: /TCExam/ Disallow: /fs/