Difference between revisions of "Robots.txt"
From Teknologisk videncenter
m (New page: Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell WEB robots (crawlers) like Goggle bot which folders or homepages you done want indexed. ro...) |
|||
| (One intermediate revision by one other user not shown) | |||
| Line 1: | Line 1: | ||
| − | Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell | + | Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell Web robots (crawlers) like Google bot which folders or homepages you don't want indexed. |
robots.txt on [http://mars.tekkom.dk mars.tekkom.dk] | robots.txt on [http://mars.tekkom.dk mars.tekkom.dk] | ||
| Line 13: | Line 13: | ||
*[http://www.google.com/support/webmasters/bin/answer.py?answer=40360 Google - Creating a robots.txt] | *[http://www.google.com/support/webmasters/bin/answer.py?answer=40360 Google - Creating a robots.txt] | ||
*[http://www.thomas-rosenstand.dk/sadan-bruger-du-robotstxt/ sådan bruger du robots.txt] | *[http://www.thomas-rosenstand.dk/sadan-bruger-du-robotstxt/ sådan bruger du robots.txt] | ||
| + | [[Category:Programming]] | ||
Latest revision as of 10:28, 11 January 2012
Robots.txt is a file you can put in the root folder of your homepage. In this file you can tell Web robots (crawlers) like Google bot which folders or homepages you don't want indexed.
robots.txt on mars.tekkom.dk
User-agent: * Disallow: /cisco/ Disallow: /sw/ Disallow: /TCExam/ Disallow: /fs/