.:: About Some Bots ::.

[tuXian]

I have some doubts can anyone clarify:

1. Google bot is visting my site every 4 hours but is not going ahead than the home page. Does this mean its got some prob in indexing the rest of the pages. My homepage is already indexed by google.

2. I can see a new type of spider in my logs, is it from Google? Heres the entry

Agent: Mozilla/4.0 (MobilePhone SCP-5500/US/1.0) NetFront/3.0 MMP/2.0 (compatible; Googlebot/2.1; +*www.google.com/bot.html)

 

[NikhilVerma]

Are you using any kind of robots.txt file ??

 

[enoonmai]

Hmm, as for your first question, make sure you have a sitemap link that the bot can crawl through to find all your pages. If Googlebot runs into an error while crawling through your site, especially the index page, this can happen. Just create a sitemap.htm file and link it in the index page and dump all your page links there and the bot will do the rest.

As for the second one, which IP address did the request come from? A simple nslookup on the IP address will confirm if its from the Googlebot.com domain. It looks to be as if the Googlebot was crawling through your page for mobile phone/WAP content, or else its part of their mobile phone/PDA service testing. The SCP-5500 is a Sanyo mobile/wireless device. The cellphone browser is called NetFront and MMP could possibly even be Mozilla MobilePhone, which may in turn be part of their Minimo project. What is also weird is that this Googlebot will look only at your index page, thats what I've heard so far. Take a look at Google's cache here.

It looks like its emulating/masquerading as the Sanyo SCP-5500, from the user-agents listed there. Maybe its a new test these guys are doing. We'll just have to wait and see.