googlebot and DNS issues - BlackHatCrew

poto · 10-21-2007, 01:33 PM

Susan Moskwa, a member of the Google Webmaster Central team, has said that afraid.org, a free DNS provider, has been known to block GoogleBot.

This can be an issue with other DNS providers. So be careful when picking a DNS provider for your web server, it may make the difference between getting indexed by Google and other search engines or not getting indexed.

---

In a related story seems GoogleBot itself has been having some DNS caching issues recently.

A detailed Google Groups thread is reporting various reports of webmasters claiming GoogleBot is timing out before reaching their pages.

First, these webmasters are noticing a drop in GoogleBot activity on their server. So they login to Google Webmaster Tools and see the error "We can't currently access your home page because of a timeout." Then they drill deeper to see "URL timeout: DNS lookup timeout."

This is happening for several webmasters, all on different hosts and servers.

The question is, is this a Google issue or a host issue outside of Google. Does Google need to clear their DNS cache in order to be able to recrawl these sites?

Here is one concerned Webmaster's story:

Quote:

This is exactly the same problem I have!

Long time indexed, no problems, then 9/5 suddenly we start getting the 'hompage timeout' errors. Same with the site map-- DNS timeout. But Yahoo! gets the sitemap just fine, and we didn't change anything on our end-- neither in DNS zone files, nor in server setup, nor the sitemap files themselves.

Googlebot is still pulling down other files from our server, just not the sitemap and index (and robots). Indexing is down *significantly*-- so this is not just a funny coincidence.

The last download of robots.txt was: September 5, 2007 3:49:14 PM PDT. Does this match yours?

Google representative, Susan Moskwa, said she is now looking into it:

Quote:

Just wanted to let you know that we're looking around to see if this is being caused by something on our end. Also, as webado noted, it helps if you post the URL in question.

pussyluver · 10-21-2007, 06:51 PM

Using Webair for name serving. No problems with googlebot. Just a few timeouts on external links. 403s a bigger issue for my TGPs.

If you have your own box you can have your own name server dedicated just to your sites. This makes sense to me should you have a farm of servers rather than just one or two. Would also make sense if there were continuous problems with the name server provided by your host.

Before my move to webair I was supporting my own name serving. Yep I fucked up a time or two, so glad to push that off on someone else.

Since most hosts use name servers that are recursive, they are subject to attack or over use.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

10-21-2007, 06:51 PM	#2 (permalink)
pussyluver member Join Date: Oct 2007 Posts: 417	Using Webair for name serving. No problems with googlebot. Just a few timeouts on external links. 403s a bigger issue for my TGPs. If you have your own box you can have your own name server dedicated just to your sites. This makes sense to me should you have a farm of servers rather than just one or two. Would also make sense if there were continuous problems with the name server provided by your host. Before my move to webair I was supporting my own name serving. Yep I fucked up a time or two, so glad to push that off on someone else. Since most hosts use name servers that are recursive, they are subject to attack or over use.