| Author |
Message |
georgie
Member
Joined: 6 Aug 2009
Posts: 16
|
# Posted: 18 Aug 2009 20:02
Reply
What's the importance of robots.txt?
How does it work?
__________________
|
Dr_Boo
Member
Joined: 12 May 2009
Posts: 35
|
# Posted: 19 Aug 2009 00:15
Reply
The robots.txt file is used to tell search engines how to handle your site when their bots/spiders crawl your website. You can specify pages or directories on your site that should be ignored by the bots and not indexed. This comes in handy when you have subfolders for pictures or scripts that you don't want showing up on search results.
You can also specify the path to your sitemap file (if you have one) in the robots.txt file. This will help the bot find all the pages in your site.
I'm pretty new at this too, so if my info is incorrect, hopefully one of the more experienced members *cough*Newbie Shield*cough* will come along and make corrections. 
Good luck to you!
__________________
|
Vishal P. Rao
Joined: 23 Jun 2005
Posts: 978
|
# Posted: 19 Aug 2009 01:49
Reply
Right on Dr_Boo!
Here's the content of a sample robots.txt file:
User-agent: *
Disallow: /images/
Disallow: /cgi-bin/
Disallow: privacy.html
User-agent: * means all search engine bots.
Disallow: /images/
Disallow: /cgi-bin/
Disallow: privacy.html
means the search engine bots should not index sub-directories images and cgi-bin and file privacy.html
__________________
|
mynetworktoday
Member
Joined: 19 Aug 2009
Posts: 1
|
# Posted: 19 Aug 2009 06:31
Reply
Thanks for the information. I'm working with DotNetNuke. I know nothing about robots. Could you please direct me to a good location to learn other than google?
__________________
|
Vishal P. Rao
Joined: 23 Jun 2005
Posts: 978
|
# Posted: 20 Aug 2009 00:41
Reply
Here's where you can learn more about robots.txt:
http://www.robotstxt.org/robotstxt.html
__________________
|
(?)
