.:: Is robots.txt necessary? ::.

[tuXian]

I dont want any restriction on indexing so dont require robots.txt file.

But is it any harm or good if you put one? coz a blank file will stop giving the 404 message.

What do u all say? Pros and Cons plz let me know

Thanks

 

[enoonmai]

If you don't want any restriction on crawling, just put in

User-agent: *
Disallow:

and leave it at that. This way, if you want to disallow your cgi-bin or some other private folder later, you can easily add it. Let the file exist though. No harm done if you leave it, and if you leave it in, chances are that it will get cross-indexed faster.

 

[tuXian]

No harm done if you leave it, and if you leave it in, chances are that it will get cross-indexed faster.

By this you mean to say that If I dont put the file on the server at all then it will cross index faster?

Am I right? Plz reply

 

[enoonmai]

No, if you dont put the file, then *some* spiderbots will usually register a 404 in your logs if they "specifically" request for the file. If you usually dont have a robots.txt file they assume unilimited access, but if a robot does try to crawl the site and explicitly requests a robots.txt, then you're faced with the obvious 404 error. Even a blank file will do, but like I said, its best if you just put in those two lines and forget about it. It will not slow down or speed up your cross-indexing but its compliant with search engines and won't go around generating errors at least.

 

[tuXian]

thanks a zillion

BTW another question:

My website uses a shared IP. How can I know the other sites on the server using the same IP?

 

[Deep]

i would sugggest keeping robots.txt file just for search engines and allowing all directories for scannig..

in anyways search engins are not gonna scann dir with password protection (.htaccess)

so if you keep some directory names to disallow then it may open doors for hackers..

coz people always try to look into the closed doors

so they might try to play around with the directories u dont want SE's to access....


about ur 2nd question..

using sites like

www.whois.sc/(SiteName.com here)
*whois.webhosting.info/(SiteName.com here)

reaplace (SiteName.com here) with the site name you want to search for..

those sites will show no. of sites hosted on the same IP and even list of those sites...

cheers
Deep

 

[tuXian]

thanks