Susan Moskwa, a member of the Google Webmaster Central team, has said that afraid.org, a free DNS provider, has been known to block GoogleBot.
This can be an issue with other DNS providers. So be careful when picking a DNS provider for your web server, it may make the difference between getting indexed by Google and other search engines or not getting indexed.
---
In a related story seems GoogleBot itself has been having some DNS caching issues recently.
A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages.
First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools and see the error "We can't currently access your home page because of a timeout." Then they drill deeper to see "URL timeout: DNS lookup timeout."
This is happening for several webmasters, all on different hosts and servers.
The question is, is this a Google issue or a host issue outside of Google. Does Google need to clear their DNS cache in order to be able to recrawl these sites?
Here is one concerned Webmaster's story:
Quote:
This is exactly the same problem I have!
Long time indexed, no problems, then 9/5 suddenly we start getting the 'hompage timeout' errors. Same with the site map-- DNS timeout. But Yahoo! gets the sitemap just fine, and we didn't change anything on our end-- neither in DNS zone files, nor in server setup, nor the sitemap files themselves.
Googlebot is still pulling down other files from our server, just not the sitemap and index (and robots). Indexing is down *significantly*-- so this is not just a funny coincidence.
The last download of robots.txt was: September 5, 2007 3:49:14 PM PDT. Does this match yours?
|
Google representative, Susan Moskwa, said she is now looking into it:
Quote:
|
Just wanted to let you know that we're looking around to see if this is being caused by something on our end. Also, as webado noted, it helps if you post the URL in question.
|