What encoding should I use for HTTP Basic Authentication?

Question

The RFC2617 says to encode the username and password to base64 but don't say what character encoding to use when creating the octets for input into the base64 algorithm.

Should I assume US-ASCII or UTF8? Or has someone settled this question somewhere already?

Pluviometer

related: HTTP header should use what character encoding? – Etherify

Answer 1

Original spec - RFC 2617

RFC 2617 can be read as "ISO-8859-1" or "undefined". Your choice. It's known that many servers use ISO-8859-1 (like it or not) and will fail when you send something else. So probably the only safe choice is to stick to ASCII.

For more information and a proposal to fix the situation, see the draft "An Encoding Parameter for HTTP Basic Authentication" (which formed the basis for RFC 7617).

New - RFC 7617

Since 2015 there is RFC 7617, which obsoletes RFC 2617. In contrast to the old RFC, the new RFC explicitly defines the character encoding to be used for username and password.

• The default encoding is still undefined. Is is only required to be compatible with US-ASCII (meaning it maps ASCII bytes to ASCII bytes, like UTF-8 does).
• The server can optionally send an additional authentication parameter charset="UTF-8" in its challenge, like this:
  WWW-Authenticate: Basic realm="myChosenRealm", charset="UTF-8"
  This announces that the server will accept non-ASCII characters in username / password, and that it expects them to be encoded in UTF-8 (specifically Normalization Form C). Note that only UTF-8 is allowed.

Complete version:

Read the spec. It contains additional details, such as the exact encoding procedure, and the list of Unicode codepoints that should be supported.

Browser support

As of 2018, modern browsers will usually default to UTF-8 if a user enters non-ASCII characters for username or password (even if the server does not use the charset parameter).

• Chrome also appears to use UTF-8
• Internet Explorer does not use UTF-8 (issue #11879588 )
• Firefox is experimenting with a change currently planned for v59 (bug 1419658)

Realm

The realm parameter still only supports ASCII characters even in RFC 7617.

Campaign

Thanks Julian. I had run into that proposal but seems to have expired and not gone anywhere further. Too bad :-(. – Pluviometer

Your answer must be the best. I can paraphrase it as ASCII for sure, maybe ISO-8859-1 if you are lucky. – Pluviometer

It looks like the latest version 04 of the proposal (which coincidentally seems to be published today) expires on august 1, 2012. – Paleo

The answer was obsolete, as it did not mention RFC 7617. I edited to include this. Julian: Hope you don't mind. – Superdominant

Oops - I just realized you are actually the author of RFC 7617. Now I really hope I did not mis-edit something. – Superdominant

RFC 7617 says: "The 'realm' parameter carries data that can be considered textual; however, [RFC7235] does not define a way to reliably transport non- US-ASCII characters. This is a known issue that would need to be addressed in a revision to that specification." But the realm is a quoted-string, and the definition of quoted-string in RFC 7230 used by 7235 allows for octets up to 0xFF, so one would think UTF-8 can be used. – Fortalice

@RemyLebeau - nope, non-ASCII characters in quoted-string are discouraged and have no agreed upon character encoding – Campaign

@JulianReschke yes I realize that – Fortalice

I'm reading a contradiction in this answer w.r.t RFC 7617: it says "the new RFC explicitly defines the character encoding to be used for username and password" but then it says "The default encoding is still undefined" - which means it isn't explicitly defined... – Morningglory

Updated by RFC 9110. The values of auth-param, including realm, are still within the ASCII range. – Binni

Answer 2

Short answer: iso-8859-1 unless encoded-words are used in accordance with RFC2047 (MIME).

Longer explanation:

RFC2617, section 2 (HTTP Authentication) defines basic-credentials:

basic-credentials = base64-user-pass
base64-user-pass  = <base64 encoding of user-pass, 
                     except not limited to 76 char/line>
user-pass         = userid ":" password
userid            = *<TEXT excluding ":">
password          = *TEXT

The spec should not be read without referring to RFC2616 (HTTP 1.1) for definitions in BNF (like the one above):

This specification is a companion to the HTTP/1.1 specification 2. It uses the augmented BNF section 2.1 of that document, and relies on both the non-terminals defined in that document and other aspects of the HTTP/1.1 specification.

RFC2616, section 2.1 defines TEXT (emphasis mine):

The TEXT rule is only used for descriptive field contents and values that are not intended to be interpreted by the message parser. Words of *TEXT MAY contain characters from character sets other than ISO-8859-1 only when encoded according to the rules of RFC 2047.

TEXT           = <any OCTET except CTLs, but including LWS>

So it's definitely iso-8859-1 unless you detect some other encoding according to RFC2047 (MIME pt. 3) rules:

// Username: Mike
// Password T€ST
Mike:=?iso-8859-15?q?T€ST?=

In this case the euro sign in the word would be encoded as 0xA4 according to iso-8859-15. It is my understanding that you should check for these encoded word delimiters, and then decode the words inside based on the specified encoding. If you don't, you will think the password is =?iso-8859-15?q?T¤ST?= (notice that 0xA4 would be decoded to ¤ when interpreted as iso-8859-1).

This is my understanding, I can't find more explicit confirmation than these RFCs. And some of it seems contradictory. For example, one of the 4 stated goals of RFC2047 (MIME, pt. 3) is to redefine:

the format of messages to allow for ... textual header information in character sets other than US-ASCII.

But then RFC2616 (HTTP 1.1) defines a header using the TEXT rule which defaults to iso-8859-1. Does that mean that every word in this header should be an encoded-word (i.e. the =?...?= form)?

Also relevant, no current browser does this. They use utf-8 (Chrome, Opera), iso-8859-1 (Safari), the system code page (IE) or something else (like only the most significant bit from utf-8 in the case of Firefox).

Edit: I just realized this answer looks at the issue more from the server-side perspective.

Paleo

RFC 2047 encoding doesn't apply in this case. – Campaign

@JulianReschke Well, the spec clearly states "only when encoded according to the rules of RFC 2047". I understand the rules in RFC2047 may not be applicable to HTTP headers, but the spec is pretty clear in referring to it. I have added the fact that no browser actually does this. – Paleo

the HTTPbis specs will not mention RFC 2047 anymore. – Campaign

Very detailed write-up, thanks @MichielvanOosterhout! – Lemniscate

RFC 7617 updated the definitions of the user-id and password. It does not allow LWS (linear whitespace) in them anymore. All control characters are forbidden in them. “The user-id and password MUST NOT contain any control characters (see ‘CTL’ in Appendix B.1 of [RFC5234]).” – Binni

Answer 3

If you are interested in what browsers do when you enter non-ascii characters at the login prompt, I just tried with Firefox.

It seems to lazily convert everithing to ISO-8859-1 by taking the least significant byte of each unicode value, e.g.:

User: 豚 (\u8c5a)
Password: 虎 (\u864e)

Are encoded the same as:

User: Z (\u005a)
Password: N (\u004e)

0x5a 0x3a 0x4e base64-> WjpO

Leet

Yes, that is the old behavior in Firefox. It was changed (in V57, it seems) and now uses UTF-8 instead. – Superdominant

V59, not V57. Currently in beta test. – Campaign

I tried with V59 & V60, didn't worked for me. I also added "charset=utf-8". – Eladiaelaeoptene

Answer 4

RFCs aside, in Spring framework, the BasicAuthenticationFilter class, the default is UTF-8.

The reason for this choice I believe is that UTF-8 is capable of encoding all possible characters, while ISO-8859-1 (or ASCII) is not. Trying to use username/password with characters not supported in the system can lead to broken behaviour or (maybe worse) degraded security.

Millenarianism

Well, using UTF-8 doesn't help if the other side doesn't know about it. So it would be good if the Spring framework implemented the charset parameter described in <greenbytes.de/tech/webdav/rfc7617.html#rfc.section.2.1> – Campaign

@JulianReschke I informed how it is implemented in one of the most common frameworks and a likely reason for it. Don't shoot the messenger! – Millenarianism