All About Wordpress - Themes, Plugins, Tips and Tweaks

[narangz]

Copy & paste this in the robots.txt file:

User-Agent: *
Allow: /

 

[Faun]

see the permission bits for ur robots.txt, may be its not allowing to read access
compare the permission bits to other files that someone can access from your site (some pics on your site etc)

 

[adi007]

Okie dokie..

Site Analysis

1.Meta tags :OK....tested both the homepage as well as some posts meta tags and everything is nice

2.Robots.txt: hmmm...i have never seen such a wierd robots.txt...
the thing is there is just the sitemap specification in it...not the usual

User-Agent: *
Allow: /

Though there is no explicit disallow for search engine bots but there is no Allow also...
So manually edit the file and make sure the above 2 lines are included at the top

Now u need to wait at least 1 or 2 weeks to check whether the problem is rectified or not..I recommend u to use all the seo wordpress related plugins

Problem Analysis

Even if the same problem continues then only one word is coming to my mind
Sandbox effect
It is still a controversial and unauthenticated term in SEO world..
Find more about it at *en.wikipedia.org/wiki/Sandbox_Effect

I just hope it isn't sandbox effect coz if it is then AFAIK no one can help...You just need to wait and wait...

Some interesting facts

I ran some SERP checker on some of the post title
Here is the screen shot which clearly shows the results

*img157.imageshack.us/img157/5903/serpbm9.png
The terms for whch SERP was checked are
"College Professor [Volume 2]" and "Accessing Ext3, NTFS, HFS+ Via Windows, Ubuntu & OS X"

The later one produced second rank in Yahoo and 1st rank in MSN

Actually this is an indication that search engines are able to crawl ur site and it can also indirectly show that this problem might not be due to robots.txt...if this fact is true then there is 90% chance that ur site has been sandboxed by google

But i am not sure..so first change that robots.txt and wait...

Thing that i need to know
Analysis of ur meta tags shows that u have verified ur site in G webmasters...
I need "Googlebot last successfully accessed your home page on ________" this detail

BTW why did u choose *www.beingmanan.com/wp/ not just *www.beingmanan.com/ ..?

Some more facts

Check this
*www.google.com/search?hl=en&q=site:aditech.info&btnG=Search (my site search results)
and
*www.google.com/search?hl=en&q=site:beingmanan.com&btnG=Search
(yours)

It seems that only one page is able to be indexed by google..i checked the cached link and found this

This is G o o g l e's cache of [url]*beingmanan.com/[/url] as retrieved on 12 Jul 2008 08:37:33 GMT.

So google is able to index and crawl *beingmanan.com/ not ur blog...

Last thing i want to conclude that i am not an SEO expert(still n00b)...These are just my possible answers that i have made from my till now SEO knowledge

First change that robots.txt...use seo helpful plugins(check my blog..)...use some seo firefox plugins to frequently analysis ur SEO position...rebuild the sitemap...submit it in G webmaster tools....and keep us updated regarding the problem

 

[iMav]

Okie dokie..

Site Analysis

1.Meta tags :OK....tested both the homepage as well as some posts meta tags and everything is nice

2.Robots.txt: hmmm...i have never seen such a wierd robots.txt...
the thing is there is just the sitemap specification in it...not the usual

User-Agent: *
Allow: /

Though there is no explicit disallow for search engine bots but there is no Allow also...
So manually edit the file and make sure the above 2 lines are included at the top

Now u need to wait at least 1 or 2 weeks to check whether the problem is rectified or not..I recommend u to use all the seo wordpress related plugins

Problem Analysis

Even if the same problem continues then only one word is coming to my mind
Sandbox effect
It is still a controversial and unauthenticated term in SEO world..
Find more about it at *en.wikipedia.org/wiki/Sandbox_Effect

I just hope it isn't sandbox effect coz if it is then AFAIK no one can help...You just need to wait and wait...

Some interesting facts

I ran some SERP checker on some of the post title
Here is the screen shot which clearly shows the results

*img157.imageshack.us/img157/5903/serpbm9.png
The terms for whch SERP was checked are
"College Professor [Volume 2]" and "Accessing Ext3, NTFS, HFS+ Via Windows, Ubuntu & OS X"

The later one produced second rank in Yahoo and 1st rank in MSN

Actually this is an indication that search engines are able to crawl ur site and it can also indirectly show that this problem might not be due to robots.txt...if this fact is true then there is 90% chance that ur site has been sandboxed by google

But i am not sure..so first change that robots.txt and wait...

Thing that i need to know
Analysis of ur meta tags shows that u have verified ur site in G webmasters...
I need "Googlebot last successfully accessed your home page on ________" this detail

BTW why did u choose *www.beingmanan.com/wp/ not just *www.beingmanan.com/ ..?

Some more facts

Check this
*www.google.com/search?hl=en&q=site:aditech.info&btnG=Search (my site search results)
and
*www.google.com/search?hl=en&q=site:beingmanan.com&btnG=Search
(yours)

It seems that only one page is able to be indexed by google..i checked the cached link and found this

This is G o o g l e's cache of [URL]*beingmanan.com/[/URL] as retrieved on 12 Jul 2008 08:37:33 GMT.

So google is able to index and crawl *beingmanan.com/ not ur blog...

Last thing i want to conclude that i am not an SEO expert(still n00b)...These are just my possible answers that i have made from my till now SEO knowledge

First change that robots.txt...use seo helpful plugins(check my blog..)...use some seo firefox plugins to frequently analysis ur SEO position...rebuild the sitemap...submit it in G webmaster tools....and keep us updated regarding the problem

Thank you for the detailed reply. Much appreciated

G Webmasters is not able to access my robots.txt It says it found the file but can't download it.

Also, in GW I added www.beingmanan.com & www.beingmanan.com/wp but it gives the same results for both. The one above.

Also, what are the permissions for your robots.txt file. Mine are 644. I am using All in 1 SEO, Google XML Sitemap geneator & DragonDesign Sitemap plugins. All of these seemed to be working fine the last time I checked but not any more.

Please let me know the permissions for robots.txt & sitemap.xml

 

[adi007]

Both should be 644 only...

I use KB Robots.txt plugin
*adambrown.info/b/widgets/kb-robots-txt/

And as a result there is no robots.txt file visible but it is created dynamically by the plugin
I recommend u to use it...

 

[Faun]

mine robots.txt content are
*visio159.com/robots.txt

User-agent: *
Disallow:

when I access *beingmanan.com/robots.txt or *www.beingmanan.com/robots.txt

I get

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request. Please contact the server administrator, webmaster@beingmanan.com and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.

Additionally, a 500 Internal Server Error error was encountered while trying to use an ErrorDocument to handle the request.
Apache/1.3.41 Server at beingmanan.com Port 80

now chek your error logs, it must be logged

 

[iMav]

My robots.txt is under *beingmanan.com/wp/robots.txt

 

[adi007]

My robots.txt is under *beingmanan.com/wp/robots.txt

U have changed the robots.txt file that's nice
wait for one day and check ur Google webmaster tools to confirm that the change is reflected in the G webmaster tools
if yes then wait for one or more week and see the results

 

[iMav]

Guys, google can't seem to get access to sitemap.xml file and I am pretty sure that is the problem. Google Webmasters says that the file is found but could not be downloaded. But the sitemap file is shown perfectly when viewed as a link.

 

[Faun]

^^just delete it and create another, see if that changes a thing or two

 

[iMav]

I deleted the .gz & xml sitemap files. Now what?

 

[Faun]

regenrate the file and then
check if you sitemap file is accessible by google webmaster tools or not ?

 

[iMav]

I think I got the problem sorted out. Google Webmasters now shows that it has 94 links for beingmanan.com/wp. The problem was with the robots.txt. As stated before it had the link to the zipped sitemap and not to the xml format. So, I changed that .gz to only .xml and Google Webmasters now shows 94 links submitted. Let's see.

 

[Faun]

lolz...so the problem is solved.

was the conflict due to plugins ?

 

[iMav]

lolz...so the problem is solved.

Hopefully. Let's see. It says that 94 URLs have been submitted.

was the conflict due to plugins ?

Not a conflict. The XML plugin has the option to generate sitemaps in 2 formats - sitemap.xml.gz & sitemap.xml. The robots.txt file had link to the sitemap.xml.gz file so I guess Webmaters wasn't able to download (access the sitemap) thereofre the error. Now I changed my robots.txt to direct crawlwers to sitemap.xml and luckily Webmasters shows 94 URLs submitted.

 

[Gigacore]

*s.wordpress.com/wp-content/themes/vip/wpforiphone/i/gallerytease.png

Wordpress for iPhone

Some info:

Introducing the first Open Source app that lets you write posts, upload photos, and edit your WordPress blog from your iPhone or iPod Touch. With support for both WordPress.com and self-hosted WordPress (2.5.1 or higher), users of all experience levels can get going in seconds. Download it now!

Check it out here: *iphone.wordpress.org/


​

 

[slugger]

@iMav
not submitting a Sitemap to Webmaster should actually not prevent Google from indexing your site

AFAIK the sitemap submission is only for thorough indexing

BTW has ur blog been verified by Webmaster now?

 

[iMav]

@iMav
not submitting a Sitemap to Webmaster should actually not prevent Google from indexing your site

Well, I have no diea about how these crawlers work.

BTW has ur blog been verified by Webmaster now?

Yeah.

 

[Faun]

Crawlers see your site as Lynx will

 

[adi007]

@iMav
not submitting a Sitemap to Webmaster should actually not prevent Google from indexing your site

AFAIK the sitemap submission is only for thorough indexing

You are absolutely right....