Robots.txt files can be compared to security guards, they basically tell the search engines which part of a website that they can enter and which parts they are not allowed to.

Many people think that this has a negative affect on a website, the reason for this being that to get a good page ranking they believe that search engines should go through every page. This is very true for most websites, but at the same time, a lot of websites use a robots.txt file to keep sensitive information private.

These files are very useful, for example, if a website stores customers personal details, they can create a robots.txt file to tell all the search engines not to go through and index those pages. Another way of looking at it would be that of they didn’t exist then search engines could index pages that contained sensitive information of a lot of people.

It is possible to combine a robots.txt file with password protection. The reason that it is considered good to do this is so that you can protect these private pages from computer hackers who would go straight to the files that are marked as do not enter to the search engine spiders.


“We have seen our business grow by over 100% over the last year“

Amanda Nixon, New York - USA

“Indazo have managed to drive more quality traffic to our website“

Jonathan Stewart, Nottingham - UK

“Our rankings have finally started going in the right direction“

James Lieu, Singapore - Asia

“Our rankings have finally started going in the right direction“

Jackie Brown - London, UK