robots.txt

Discussion in 'Search Engine Optimization (SEO)' started by georgie, Aug 18, 2009.

  1. georgie

    georgie Guest

    What's the importance of robots.txt?
    How does it work?
     
  2. Dr_Boo

    Dr_Boo New Member

    The robots.txt file is used to tell search engines how to handle your site when their bots/spiders crawl your website. You can specify pages or directories on your site that should be ignored by the bots and not indexed. This comes in handy when you have subfolders for pictures or scripts that you don't want showing up on search results.

    You can also specify the path to your sitemap file (if you have one) in the robots.txt file. This will help the bot find all the pages in your site.

    I'm pretty new at this too, so if my info is incorrect, hopefully one of the more experienced members *cough*Newbie Shield*cough* will come along and make corrections. [​IMG]

    Good luck to you!
     
  3. Vishal P. Rao

    Vishal P. Rao Administrator Staff Member

    Right on Dr_Boo!

    Here's the content of a sample robots.txt file:

    User-agent: *
    Disallow: /images/
    Disallow: /cgi-bin/
    Disallow: privacy.html

    User-agent: * means all search engine bots.

    Disallow: /images/
    Disallow: /cgi-bin/
    Disallow: privacy.html

    means the search engine bots should not index sub-directories images and cgi-bin and file privacy.html
     
  4. mynetworktoday

    mynetworktoday New Member

    Thanks for the information. I'm working with DotNetNuke. I know nothing about robots. Could you please direct me to a good location to learn other than google?
     
  5. Vishal P. Rao

    Vishal P. Rao Administrator Staff Member

  6. Maurice

    Maurice Guest

    Use robots.txt to hide landing pages that you don;t want competitors to see..... [​IMG]
     
  7. josaphlewis

    josaphlewis Guest

    Robots.txt is just a regular text file saved on a website. On request, the specified robots will ignore specified files or directories in their search. It is an algorithm. If it is not there on your site, Google will not crawl it.
     
  8. newbiz02

    newbiz02 New Member

    I use robots.txt to hide the plugins folder on some of my sites so
    that google doesn't know that they are *auto*--=blogs. lol

    I mean the sites look great so there is no reason to suspect
    anything [​IMG]
     
  9. shannonpaul2

    shannonpaul2 New Member

    Robots.txt is exactly a normal text file saved on a website. For the asking, the conditioned robots will ignore defined files or directories in their search.
     
  10. Ron S

    Ron S New Member

  11. Justine

    Justine New Member

  12. jamess

    jamess Member

    The robots.txt file is a simple text file (no html) that is placed in your website’s root directory in order to tell the search engines which pages to index and which to skip.
     
  13. Sanjay Sharma

    Sanjay Sharma New Member

    Robots.txt" is a regular text file that through its name, has special meaning to the majority of "honorable" robots on the web. By defining a few rules in this text file, you can instruct robots to not crawl and index certain files, directories within your site, or at all. For example, you may not want Google to crawl the /images directory of your site, as it's both meaningless to you and a waste of your site's bandwidth. "Robots.txt" lets you tell Google just that.
     
  14. Akash Agarwal

    Akash Agarwal New Member

    Robots.txt is frequent name of a text file that is uploaded to a Web site's root directory and linked in the html code of the Web site. The robots.txt file is used to have the funds for directions about the Web site to Web robots and spiders.
     

Share This Page