SEO News

Bots and robotsCurrently Google's crawler (known as Googlebot) sends HTTP requests with English setting as default for Accept-Language in the request header. This was spotted and reported by Pierre Far, a former Googler, who claims that this contradicts Google's own documentation.

On Google+ Fare writes about the spotted behaviour of the Google crawler which according to him is inconsistent with the information in the correspondent documentation. On the respective page Google describes how different crawl settings are used for those sites that do not have different URLs for each locale:

"In addition, the crawler sends HTTP requests without setting Accept-Language in the request header."

However, later in the document Google writes that for language-dependent crawling the Accept-Language field in the HTTP header is actually set. Googlebot tests if a site serves locale-specific content by checking the following signals:

  • Is the site serving different content on the same URL based on the user's perceived country (meaning geolocation)?
  • Is the site serving different content on the same URL based on the Accept-Language field in the HTTP header?
  • Is the site blocked completely for requests from specific countries?

So it is a bit unclear what caused the confusion. As far as it seems Googlebot's described behaviour correlates with Google's documentation for crawling locale-aware websites.

 

Image © studiostoks - Fotolia.com

 


Advertisement

SEO Südwest deutsche Webseite

Hear me speak

Partner

Awards

SEO Südwest: Platz 5 bei den SEO-Wahlen 2014 zum besten deutschen SEO-Blog

 

SEO-united.de Tipp 12/15

iBusiness Top-100-Liste SEO-Dienstleister

Popular from

Internet World Business


SEO United


Webselling

Connect now

Advertisement

 References

kr3m
feel-perfect.eu - Die Nährstoffexperten border=
Flintec IT GmbH
England-Insolvenz