Robots.txt & Websites

mattyh

mattyh

Messages
7,021
Name
Matt
Edit My Images
Yes
Can I just run something past you guys....

Robots.txt, obviously controls all the web crawlers visiting a particular website, I've noticed that I've been losing a lot of bandwidth to these calling my images, so I've just altered my robots.txt to the following:

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/

Now, I also have several password protected sections (nothing overly secure, just changed the settings using Cpanel... Am I correct in saying that crawlers can't index those pages, or do I need to add them to my Robots file as well?

I'm pretty sure I know the answer, but I'd like to be sure :D
 
are you from mars? you seem to speak a language i havent heard before...:thinking:
 
the original poster makes pefect sense ;)

I believe the bots cant read password protected directories. They dont know the password, they just read plain html.

However, just because you block the bots from crawling the images folder using robots.txt, this will stop the good bots, like google, yahoo etc.. The spammers out there will ignore your robots.txt file regardless.

:)
 
surrealillusions said:
the original poster makes pefect sense ;)

I believe the bots cant read password protected directories. They dont know the password, they just read plain html.

However, just because you block the bots from crawling the images folder using robots.txt, this will stop the good bots, like google, yahoo etc.. The spammers out there will ignore your robots.txt file regardless.

:)
Click to expand...

Thats what I thought, thanks :D
 
none of that made any sence to me and im not sure it will help if u try to explain, what am i doing here i have no idea :exit:
 
You must log in or register to reply here.
Back
Top