Robots.txt Installation

Robots.txt Installation

 

Robot.txt is what tells the spiders where they can and cannot go on your website; this is particularly effect when keeping the search engines out of your shopping carts area because you will find that the search engines will attempt to index many of the items you have in there. By keeping the crawlers out of these types of pages you will be able to save your bandwidth and use it in areas that will be far more beneficial to you.