SEO News

Bots and robotsCurrently Google's crawler (known as Googlebot) sends HTTP requests with English setting as default for Accept-Language in the request header. This was spotted and reported by Pierre Far, a former Googler, who claims that this contradicts Google's own documentation.

On Google+ Fare writes about the spotted behaviour of the Google crawler which according to him is inconsistent with the information in the correspondent documentation. On the respective page Google describes how different crawl settings are used for those sites that do not have different URLs for each locale:

"In addition, the crawler sends HTTP requests without setting Accept-Language in the request header."

However, later in the document Google writes that for language-dependent crawling the Accept-Language field in the HTTP header is actually set. Googlebot tests if a site serves locale-specific content by checking the following signals:

  • Is the site serving different content on the same URL based on the user's perceived country (meaning geolocation)?
  • Is the site serving different content on the same URL based on the Accept-Language field in the HTTP header?
  • Is the site blocked completely for requests from specific countries?

So it is a bit unclear what caused the confusion. As far as it seems Googlebot's described behaviour correlates with Google's documentation for crawling locale-aware websites.


Image © studiostoks -



SEO Südwest deutsche Webseite

Hear me speak



SEO Südwest: Platz 5 bei den SEO-Wahlen 2014 zum besten deutschen SEO-Blog Tipp 12/15

iBusiness Top-100-Liste SEO-Dienstleister

Popular from

Internet World Business

SEO United


Connect now



kr3m - Die Nährstoffexperten border=
Flintec IT GmbH