Custom HTTP headers : naming conventions

Question

Several of our users have asked us to include data relative to their account in the HTTP headers of requests we send them, or even responses they get from our API. What is the general convention to add custom HTTP headers, in terms of naming, format... etc.

Also, feel free to post any smart usage of these that you stumbled upon on the web; We're trying to implement this using what's best out there as a target :)

Konya

Be aware that firewalls can remove response header fields. Some remove everything that isn't mentioned in RFC 2616 (June 1999, HTTP 1.1). The client side should still be usable without the new fields. – Lail

Note that the comment by @Lail doesn't apply when using HTTPS. – Melainemelamed

Note that the comment by @Melainemelamed is an urban legend. Firewalls can filter HTTPS content. See howtoforge.com/filtering-https-traffic-with-squid and watchguard.com/help/docs/wsm/xtm_11/en-us/content/en-us/… – Lail

@Lail Given that your article basically turns the proxy into something similar to a MiTM (it takes encrypted client connection and then makes a new one) then sure, you can do almost anything, but that fact negates the encryption from the proxy's PoV b/c it's decrypting the client's content itself. In that case, from the proxy's PoV, it's basically as if you weren't using HTTPS in the 1st place... – Melainemelamed

If anyone is in the academic market EzProxy will strip out custom headers. You need to edit the EzProxy config to allow them. Afterwards you have to hope that every member institution updates their EzProxy configuration. EzProxy is most commonly used for off campus access. Which has been pretty popular the last couple years (for some reason /s) – Bereave

NGINX ignores headers with underscores by default: nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/… – Heterosexuality

Answer 1

The recommendation is was to start their name with "X-". E.g. X-Forwarded-For, X-Requested-With. This is also mentioned in a.o. section 5 of RFC 2047.

---

Update 1: On June 2011, the first IETF draft was posted to deprecate the recommendation of using the "X-" prefix for non-standard headers. The reason is that when non-standard headers prefixed with "X-" become standard, removing the "X-" prefix breaks backwards compatibility, forcing application protocols to support both names (E.g, x-gzip & gzip are now equivalent). So, the official recommendation is to just name them sensibly without the "X-" prefix.

---

Update 2: On June 2012, the deprecation of recommendation to use the "X-" prefix has become official as RFC 6648. Below are cites of relevance:

3. Recommendations for Creators of New Parameters

...

3. SHOULD NOT prefix their parameter names with "X-" or similar constructs.

4. Recommendations for Protocol Designers

...

4. SHOULD NOT prohibit parameters with an "X-" prefix or similar constructs from being registered.

5. MUST NOT stipulate that a parameter with an "X-" prefix or similar constructs needs to be understood as unstandardized.

6. MUST NOT stipulate that a parameter without an "X-" prefix or similar constructs needs to be understood as standardized.

Note that "SHOULD NOT" ("discouraged") is not the same as "MUST NOT" ("forbidden"), see also RFC 2119 for another spec on those keywords. In other words, you can keep using "X-" prefixed headers, but it's not officially recommended anymore and you may definitely not document them as if they are public standard.

---

Summary:

• the official recommendation is to just name them sensibly without the "X-" prefix
• you can keep using "X-" prefixed headers, but it's not officially recommended anymore and you may definitely not document them as if they are public standard

Kailey

Just as there are many kids that will never end up as professional athletes, many custom headers will never end up as standards. I'm inclined to keep the "X-" on those. – Skepful

@Skepful Agreed. There are so many custom headers that will never end up standardized. The few that do, it's easy to just edit your code from if (header == "x-gzip") to if (header == "x-gzip" || header == "gzip"). As for your analogy, here's another: it's like the military saying "Oh, it's troublesome to change someone from Private to General. So, from now on, you're all Generals. Now we don't need to do so much work" – Verisimilitude

@ColeJohnson Not sure if that analogy works. The problem here is that there is no central point you can change the name. Every single snippet of code that expects x-gzip now has to be changed, or the old header needs to continue to be used in addition to the new one. It's preferable to go with RFC 6648. – Pretension

@Vinod yes. It makes sense to, but there's so many proposed standards that will never see the light of day. For file types, sure; drop the X- prefix. I'm against it, but go ahead and do it. For headers OTOH, don't drop it. It makes it easy to look at and go, "oh, it's non-standard; I can ignore it" vs "there's those non-standard X- headers, and then there's this one I don't recognize; can I ignore it safely?" – Verisimilitude

So I'm unclear on whether this deprecation applies to MIME types as well? Are we still expected to use e.g. application/x-foo-bar until the MIME type is registered, or does this new deprecation override the MIME type registration process? – Monoxide

Although the tone of cweekly's answer is unnecessarily defensive, I believe he is right, and his point solves the problem illustrated in this comment thread. In short, don't try to identify whether a header will "graduate" or not; instead determine if it's a private or public header (application-specific or "generic"/"global"). For private headers, optionally use X- to ensure no clash with public headers (thanks to RFC6648, which deals with public headers), and in addition definitely use an arbitrary private prefix. For public headers, don't use X- under any circumstances. – Mahoney

@Cole Johnson to your first comment: are you sure you want to change products you worked on 5 years ago or earlier? It's not always easy to change old projects and test them after the changes – Excrescent

@Mr.Cat Realistically, there aren't many products I'll still be using 5 to 10 years from now. Or at least I won't be using the same versions of them. If you're not up for maintaining a piece of code, then that's fine, but the code will cease to be relevant. Personally, I think the most sensible thing for nonstandardized headers would be to go with a convention of X-<RAND_NUM>-<NAME>, since, if two people want to implement the same functionality, or different functionality that share the same "sensible" name, they are disambiguated while they're standards-track and one can become standard. – Ayers

Use X-Data-____ similar to HTML5 "data-____" attribute, it should be perfectly fine and backward compatible, it will not change the response body size and you can provide almost any type of data, if it's "weird looking" encode it (either base64, or urlcomponent or even escape) :) – Shotgun

@Elad: I know. The answer boils down that you're free to pick whatever you want as long as you understand there's no standard nor recommendation for it and that you should in case of a public API not document it as part of public standard. – Kailey

@G-Mac, You missed the point. For such headers, it doesn't matter what they are, so they can start with X- or don't and it doesn't matter. – Sari

@ColeJohnson, Any http reader needs to be able to tell a standard header from a nonstandard header. Try these tests on any http reader you know, including yourself: Test-1) Foobar doesn't start with X-; is it a standard header or not? Test-2) Xkcd doesn't start with X-; is it a standard header or not? – Sari

@Sari Why does an HTTP reader need to know whiat is a standard vs non-standard header? It should treat them both the same. – Micrometeorite

web is freakin mess xD – Pathogenesis

Answer 2

The question bears re-reading. The actual question asked is not similar to vendor prefixes in CSS properties, where future-proofing and thinking about vendor support and official standards is appropriate. The actual question asked is more akin to choosing URL query parameter names. Nobody should care what they are. But name-spacing the custom ones is a perfectly valid -- and common, and correct -- thing to do.

Rationale:
It is about conventions among developers for custom, application-specific headers -- "data relevant to their account" -- which have nothing to do with vendors, standards bodies, or protocols to be implemented by third parties, except that the developer in question simply needs to avoid header names that may have other intended use by servers, proxies or clients. For this reason, the "X-Gzip/Gzip" and "X-Forwarded-For/Forwarded-For" examples given are moot. The question posed is about conventions in the context of a private API, akin to URL query parameter naming conventions. It's a matter of preference and name-spacing; concerns about "X-ClientDataFoo" being supported by any proxy or vendor without the "X" are clearly misplaced.

There's nothing special or magical about the "X-" prefix, but it helps to make it clear that it is a custom header. In fact, RFC-6648 et al help bolster the case for use of an "X-" prefix, because -- as vendors of HTTP clients and servers abandon the prefix -- your app-specific, private-API, personal-data-passing-mechanism is becoming even better-insulated against name-space collisions with the small number of official reserved header names. That said, my personal preference and recommendation is to go a step further and do e.g. "X-ACME-ClientDataFoo" (if your widget company is "ACME").

IMHO the IETF spec is insufficiently specific to answer the OP's question, because it fails to distinguish between completely different use cases: (A) vendors introducing new globally-applicable features like "Forwarded-For" on the one hand, vs. (B) app developers passing app-specific strings to/from client and server. The spec only concerns itself with the former, (A). The question here is whether there are conventions for (B). There are. They involve grouping the parameters together alphabetically, and separating them from the many standards-relevant headers of type (A). Using the "X-" or "X-ACME-" prefix is convenient and legitimate for (B), and does not conflict with (A). The more vendors stop using "X-" for (A), the more cleanly-distinct the (B) ones will become.

Example:
Google (who carry a bit of weight in the various standards bodies) are -- as of today, 20141102 in this slight edit to my answer -- currently using "X-Mod-Pagespeed" to indicate the version of their Apache module involved in transforming a given response. Is anyone really suggesting that Google should use "Mod-Pagespeed", without the "X-", and/or ask the IETF to bless its use?

Summary:
If you're using custom HTTP Headers (as a sometimes-appropriate alternative to cookies) within your app to pass data to/from your server, and these headers are, explicitly, NOT intended ever to be used outside the context of your application, name-spacing them with an "X-" or "X-FOO-" prefix is a reasonable, and common, convention.

Polytechnic

I'd appreciate it if any downvoters of my comment could explain what part of my answer they find objectionable. I don't care that much about my reputation score, but I'm genuinely curious. Where does the disagreement lie? Thanks. – Polytechnic

I completely agree with your answer and it is the only answer here that answers the actual question asked. We ARE talking about custom, application-specific headers here, never to be standardized in the HTTP standards. Is there a common convention for these that people tend to use? (such as prefixing them with "_" perhaps? ie: ("_ClientDataFoo") – Obliquity

Thanks Marchy, yeah, the accepted answer doesn't address the question asked. IETF deprecation of "X-" prefix for non-standard (but generic) headers is irrelevant to custom app-specific headers that will never be standardized. To answer your question, in my opinion and experience (16 years of webdev), the best convention is to use the aforementioned "X-ACME-ClientData". "X-" bc it's not standard (nor will it ever be, which is why the IETF deprecation is moot here), "ACME-" to namespace it to your "ACME" company or specific app, and "ClientData" can be whatever semantic name you like. :) – Polytechnic

Further to your recommendations, I'd suggest using a similar convention to package or namespace names in many languages, that is something similar to a reverse domain: com.mycompany.headers.custom:somevalue. I can't find anything anywhere that says you can't use dots in header names. You won't get collisions that way. – Extractor

Prefix the headers with your company/product name. The x- adds zero value and by using it you are demonstrating a disregard/ignorance of IETF the spec which clearly states that it was a mistake to use the x- prefix and we have since learned from that mistake. – Idioplasm

@Extractor - I've always used just letters, numbers and hyphens (which map cleanly to Apache mod_setenvif underscores for corresponding env vars), since pairing headers to env vars in Apache is my primary use for custom headers. YMMV. – Polytechnic

@Darrel Miller, thanks for your comment. You're probably right that a X- prefix doesn't add much value per se, IF you're clearly namespacing them with something else which will never collide with any IETF reserved/official names. (Which goal is effectively guaranteed by "X-" prefix, thanks to the spec.) Now I am repeating myself again. Sigh. Have a nice day. – Polytechnic

@Polytechnic I think clashing with IETF official names is the least of the problem. Clashing with everyone else's extension names is the real danger. The idea of X- is that it never made it out on to the big bad internet and therefore clashes were not possible. – Idioplasm

@DarrelMiller ... hence the recommendation to use X-ACMECO-WIDGET-FOO. I insist that, for the OP's question as asked, use of X- is simply not contra-indicated by RFC-6648 and the like. If you're a vendor providing a framework, library or module for use in other peoples' projects, that's a different story, and by all means follow that RFC to a T. But it's just moot for individual one-off applications, where custom app-specific header naming conventions are effectively completely private APIs. How would they clash with "everyone else's" names? Whose would those be? – Polytechnic

As you are complaining: actually your question does not (completely) answer the question, too, as you are only speaking about naming but not format what was asked and definitely corresponds to the HTTP spec. You are - even worse - negating that fact. See Tom Anderson answer below: https://mcmap.net/q/18578/-custom-http-headers-naming-conventions. – Pantheas

1. X-headers that never intend to drop the X- (assuming the intended meaning for "X" is Xtra or eXtension or the like) are just as valid as so-called standard headers. 2. Given little requirements on naming, X-, Y-, or Z- prefixes should all be acceptable. 3. Headers are often sorted alphabetically in "devtools" of well-known browsers, allowing devops to see a group of all X-headers in one place which yields an abstraction layer of application-specific headers above the standard web server headers for cache-control et. al. – Absurdity

@Polytechnic Funny enough, the accepted answer has received more downvotes than yours, both in absolute numbers (10 vs. 2) as in percentages (2.3% vs. 2%) :-D – Rape

I like the idea that we could distinguish between global-but-experimental and application-specific. However RFC6648 seems to apply to both; for example, it claims it 'Does not recommend against the practice of private, local, preliminary, experimental, or implementation-specific parameters, only against the use of "X-" and similar constructs in the names of such parameters.' One may decide this is a mistake, but it is uncomfortable to go against IETF "Best Current Practice". For that reason I think I'd go with ACMECO-WIDGET-FOO without the X-. – Chrotoem

I'm honestly having a little trouble understanding the RFC reasoning. Granted that, if and when the parameter is standardized, there will be both x- and non-x- versions. That is only a problem if the behavior of the x- and non-x- versions is identical. I stumbled over here because I am looking at adding an "on behalf of" header to my API. It could become public some day (as it's a common kind of use case). If I used "On-Behalf-Of" and someday they add that as a standard header, what are the odds that my semantics will be identical to the standardized one? – Maenad

"The more vendors stop using X- for (A), the more cleanly-distinct the (B) ones will become." ... standing ovation – Busra

Answer 3

The format for HTTP headers is defined in the HTTP specification. I'm going to talk about HTTP 1.1, for which the specification is RFC 2616. In section 4.2, 'Message Headers', the general structure of a header is defined:

   message-header = field-name ":" [ field-value ]
   field-name     = token
   field-value    = *( field-content | LWS )
   field-content  = <the OCTETs making up the field-value
                    and consisting of either *TEXT or combinations
                    of token, separators, and quoted-string>

This definition rests on two main pillars, token and TEXT. Both are defined in section 2.2, 'Basic Rules'. Token is:

   token          = 1*<any CHAR except CTLs or separators>

In turn resting on CHAR, CTL and separators:

   CHAR           = <any US-ASCII character (octets 0 - 127)>

   CTL            = <any US-ASCII control character
                    (octets 0 - 31) and DEL (127)>

   separators     = "(" | ")" | "<" | ">" | "@"
                  | "," | ";" | ":" | "\" | <">
                  | "/" | "[" | "]" | "?" | "="
                  | "{" | "}" | SP | HT

TEXT is:

   TEXT           = <any OCTET except CTLs,
                    but including LWS>

Where LWS is linear white space, whose definition i won't reproduce, and OCTET is:

   OCTET          = <any 8-bit sequence of data>

There is a note accompanying the definition:

The TEXT rule is only used for descriptive field contents and values
that are not intended to be interpreted by the message parser. Words
of *TEXT MAY contain characters from character sets other than ISO-
8859-1 [22] only when encoded according to the rules of RFC 2047
[14].

So, two conclusions. Firstly, it's clear that the header name must be composed from a subset of ASCII characters - alphanumerics, some punctuation, not a lot else. Secondly, there is nothing in the definition of a header value that restricts it to ASCII or excludes 8-bit characters: it's explicitly composed of octets, with only control characters barred (note that CR and LF are considered controls). Furthermore, the comment on the TEXT production implies that the octets are to be interpreted as being in ISO-8859-1, and that there is an encoding mechanism (which is horrible, incidentally) for representing characters outside that encoding.

So, to respond to @BalusC in particular, it's quite clear that according to the specification, header values are in ISO-8859-1. I've sent high-8859-1 characters (specifically, some accented vowels as used in French) in a header out of Tomcat, and had them interpreted correctly by Firefox, so to some extent, this works in practice as well as in theory (although this was a Location header, which contains a URL, and these characters are not legal in URLs, so this was actually illegal, but under a different rule!).

That said, i wouldn't rely on ISO-8859-1 working across all servers, proxies, and clients, so i would stick to ASCII as a matter of defensive programming.

Lallation

The newer HTTP spec RFC7230 says "Newly defined header fields SHOULD limit their field values to US-ASCII octets." – Chrotoem

Thank you for the specification this is really important. Also, can you give an example or point to a common use name out there? – Gherardo

Answer 4

RFC6648 recommends that you assume that your custom header "might become standardized, public, commonly deployed, or usable across multiple implementations." Therefore, it recommends not to prefix it with "X-" or similar constructs.

However, there is an exception "when it is extremely unlikely that [your header] will ever be standardized." For such "implementation-specific and private-use" headers, the RFC says a namespace such as a vendor prefix is justified.

Auteur

"RFC6648 recommends that you assume that your custom header "might become standardized, public, commonly deployed, or usable across multiple implementations." I think this gives a reason to use X- prefix because it's more likely something without any prefix might become standarized. – Film

@Film If you assume someone else's similar header (not your header) might become standardized, you could avoid a conflict with X-, but that's a different assumption than RFC6648 primarily takes. The RFC's exception accounts for potential conflicts between a future standard header and a header from another vendor whose technology may become integrated with yours through company merger, etc. That's why the exception calls for a vendor prefix. – Auteur

@EdwardBrey With all due respect, how often has this (http header conflict after company merger etc.) actually happened in the wild? After all, using a vendor prefix would not be bulletproof either: Company #1, uses prefix XY for header H1 in their product P1, sells a developer license to company #2, company #1 goes bankrupt, their IP including brand name XY is purchased by company #3 who builds their product P3 that uses XY prefix for header H1, dev-licenses that product to company #2 that now must integrate 2 products with incompatible uses of the same headers. – Kickback

@Kickback RFCs get their name because they are Requests For your kind of Comment. You can examine the history of technical consideration to see whether a complexity-compatibility tradeoff was considered. – Auteur

Answer 5

Modifying, or more correctly, adding additional HTTP headers is a great code debugging tool if nothing else.

When a URL request returns a redirect or an image there is no html "page" to temporarily write the results of debug code to - at least not one that is visible in a browser.

One approach is to write the data to a local log file and view that file later. Another is to temporarily add HTTP headers reflecting the data and variables being debugged.

I regularly add extra HTTP headers like X-fubar-somevar: or X-testing-someresult: to test things out - and have found a lot of bugs that would have otherwise been very difficult to trace.

Gorman

Why should he use this "standard"? The headers work the same. Even with a "WHO_EVER_READS_THIS_IS_DUMB_" prefix... – Trapezohedron

Answer 6

The header field name registry is defined in RFC3864, and there's nothing special with "X-".

As far as I can tell, there are no guidelines for private headers; in doubt, avoid them. Or have a look at the HTTP Extension Framework (RFC 2774).

It would be interesting to understand more of the use case; why can't the information be added to the message body?

Gully

The main reason I am considering some custom headers is so that I can make routing decisions without having to parse out the body... – Disaffection