SEO implications of a multi lingual site with detection of system culture

Question

I have developed a multi-lingual site in ASP.NET, which detects the user's system culture, and displays content in the appropriate language.

This is all working nicely, but my client has since had an SEO audit. The SEO agency has expressed a concern that this is not good SEO practice, as there are not unique URLs for each language.

They have suggested that the site may be accused of cloaking, and that google may not index the site correctly for each different language.

Any ideas on whether these are valid concerns, and if there is an advantage to having unique URLs for each language version of the site?

Needless

Do you declare the document’s language appropriately? – Octane

Answer 1

Although you have done a beautiful job switching Language automatically, the SEO Agency is correct!

That google may not index the site correctly for each diffferent language.

This is true! Google doesn't send the accept-language header last time I checked. This means that Google will only index the default language.

They have suggested that the site may be accused of cloaking,

This differs from your Excact implementation, but it is possible your site will receive a penalty!

There IS advantage having unique URLs (for each language version) on the site!
First of all, for your users: they can link to the language they prefer. Secondary for the Search Engines as they can index your site correctly.

I advice most of the time to redirect the user only on the home page for a language switch using a 302 redirect to the correct URL (and so the correct language). (edit: You can review the post by Matt Cutts "SEO Advice: Discussing 302 redirects")

To verify my advice: install fiddler and surf to http://www.ibm.com. As shown below, i received a 302 redirect to the appropriate language, arriving at www.ibm.com/be/en.

    Result  Protocol    Host    URL Body    Caching Content-Type    
4   302 HTTP    www.ibm.com /   209     text/html
5   200 HTTP    www.ibm.com /be/en/ 5.073   no-cache  text/html;charset=UTF-8

There are a few solutions you can solve this:

• Start Rewriting Urls (adding e.g. a directory using the language)
• If you don't want to go through the hassle of adding directories (or rewriting url's) adding a QueryString would be the easiest solution (although try limiting them to maximum 2 parameters)
• Another option is using different sub-domains! www.website.com for the default language, es.website.com, fr.website.com

Just make sure you supply every time the same content for the same URL.

Good luck with it!

Bone

Thanks for the info. We came to the same conclusion after looking the other big sites, like the ones you and @yar mentioned. We are now implementing folders for each locale, with a 302 redirect on the homepage. Fortunately we have quite a powerful rewriting engine implemented in the site framework, used for aliasing all databse driven pages to friendly URLs, so it won't be a big deal. Something I will keep in mind for new projects though. – Needless

Answer 2

Hopefully we will see some people answer who know about the internals of Google (anyone?). But most suppositions about how Google and others' crawlers are... suppositions, and subject to change.

My guess is that you should use separate URLs for languages, even if they just have a ?language= difference (although better would be a truly different URL). I believe this because when you go to google.it it says, google.com in English and that link goes to... google.com. In other words, Google itself uses different URLs for different languages.

Also, another big site, Microsoft (they probably know about SEO) uses

http://www.microsoft.com/en/us/default.aspx

for US-English and

http://www.microsoft.com/it/it/default.aspx

for Italy-Italian so it's probably best practice to differentiate based on language (and country).

In any case, I am totally annoyed when I'm on an English language computer and I can't see a site in Italian or Spanish, and vice-versa. As a usability, not SEO strategy, the user should be able to override the language suggestion. This is how most big sites handle languages, too.

Intrigue

Just to clarify, the user can override the defuault language, by selecting it in a drop down list. This adds lang={culture code} to the wuerystring, and sets a cookie so that the language selection is retained. – Needless

@Matt, personally I don't think that's bad, and I doubt that Google (for instance) just ignores the query params. That said, MANY PEOPLE disagree with this and think that the URL needs to be different. However, making "friendly URLs" is a trivial regular expression translation and all web frameworks have some kind of provision for this. – Intrigue