robot.txt

[Guest]

what is robot.txt file?
waht is its content
and can i include in my site?

 

[cvvikram]

Check this site

*www.searchengineworld.com/robots/robots_tutorial.htm

 

[Guest]

OK BUT IS IT HARMFUL

 

[Deep]

Sometimes yes, specially when you deny some directories which you do not want other ppl to see...

for example...

u have denied directory name admin

so google wont scan but hacker or anything will check robots.txt and see what is denied and then try to play with it...

so if you are denying any directory then password protect that directory using .htaccess rule so that no one can play with it much..

Deep

 

[Guest]

and how to add a log file on the server to find out stats of site

 

[Guest]

and how to add a log file on the server to find out stats of site

 

[Deep]

can you be specific? I didnt get your question

 

[crashuniverse]

robots.txt also helps in bots to remove ur website totally from their indexing so dat ur website doesn't come in search results.

 

[sting]

and how to add a log file on the server to find out stats of site

U mean the ur site statistics about your visitors ? if yes try this *allcount.com/

 

[Guest]

i mean i heard many people talking that googlebot came to my site from seeing the log file . so i m talking about that log file..

 

[Deep]

hmm no google bot cannot scan any file unless you have linked it anywhere on the page...

Deep

 

[Guest]

ok .. and allcount.com is fine . i used it.. is ther any other way i can record all stats on my site in text file

 

[debiprasad_sahoo]

Check this site

*www.searchengineworld.com/robots/robots_tutorial.htm

The page does not exist. Please check.

 

[RCuber]

WTH!!!! you bumped a two year old thread

reported...