robots.txt

What's the importance of robots.txt?
How does it work?

The robots.txt file is used to tell search engines how to handle your site when their bots/spiders crawl your website. You can specify pages or directories on your site that should be ignored by the bots and not indexed. This comes in handy when you have subfolders for pictures or scripts that you don't want showing up on search results.

You can also specify the path to your sitemap file (if you have one) in the robots.txt file. This will help the bot find all the pages in your site.

I'm pretty new at this too, so if my info is incorrect, hopefully one of the more experienced members *cough*Newbie Shield*cough* will come along and make corrections.

Good luck to you!

Thanks for the information. I'm working with DotNetNuke. I know nothing about robots. Could you please direct me to a good location to learn other than google?

Use robots.txt to hide landing pages that you don;t want competitors to see.....

Robots.txt is just a regular text file saved on a website. On request, the specified robots will ignore specified files or directories in their search. It is an algorithm. If it is not there on your site, Google will not crawl it.

I use robots.txt to hide the plugins folder on some of my sites so
that google doesn't know that they are *auto*--=blogs. lol

I mean the sites look great so there is no reason to suspect
anything

Robots.txt is exactly a normal text file saved on a website. For the asking, the conditioned robots will ignore defined files or directories in their search.

refer to http://www.robotstxt.org/robotstxt.html

Ron S: refer to http://www.robotstxt.org/robotstxt.html
Unfortunately the link doesn't work "Internal Server Error"...

The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip.