nodeJS max header size in http.request

Question

With nodeJS v0.10.28, is there a limit in the size/length of the header content in an http request?

Let me explain:

I need to consume rest services provided by a 3rd party provider. The data returned to me is in the header of the request, the body is mostly empty (120 or so chars). The amount of data in the header varies from a few chars to several 100kb.

var https = require('https');

var httpHeaders = {
    Authorization: 'Basic ' + new Buffer(user + ':' + psw).toString('base64'),
    accept: '*/*',
    'Content-Type': 'text/plain; charset=utf-8'
};
var options = {
    host: "www.website.com",
    port: 8080,            
    path: "/" ,   
    method: 'GET',
    headers: httpHeaders,
    rejectUnauthorized: false,
    requestCert: true,
    agent: false
};

https.request(options, function(res) {
    res.setEncoding('utf8');
    if (res.statusCode == 200) {
        var json = res.headers["someHeaderInfo"];
        callback(null,{ "result" : JSON.parse(json) });
    } else {
        callback({ "error" : res.statusCode  });                            
    }
}).on('data', function (chunk) {
    console.log('BODY: ' + chunk);
}).on('error', function(e, res) {
    console.log("  Got error: " + e.message);
    callback({ "error" : e.message });
}).end();

The code above works fine for smaller size headers but fails on the on('error', with "Parse Error" message on larger headers.

Removing the on error clause throws this exception:

Error: Parse Error
    at CleartextStream.socketOnData (http.js:1583:20)
    at CleartextStream.read [as _read] (tls.js:511:12)
    at CleartextStream.Readable.read (_stream_readable.js:320:10)
    at EncryptedStream.write [as _write] (tls.js:366:25)
    at doWrite (_stream_writable.js:226:10)
    at writeOrBuffer (_stream_writable.js:216:5)
    at EncryptedStream.Writable.write (_stream_writable.js:183:11)
    at write (_stream_readable.js:582:24)
    at flow (_stream_readable.js:591:7)
    at Socket.pipeOnReadable (_stream_readable.js:623:5)

Is there a limit on the header size, can it me changed? what solution do I have?

Thanks

Accumbent

Answer 1

The HTTP protocol parser that Node uses appears to be hard-coded with a maximum header size of 8KB. Relevant constant. Since that is a compile-time constant, you would have to use a custom-compiled version of Node to set that constant larger.

It really sounds like the service you are using has made a mistake by putting that much data in a header though. Headers are meant for metadata about the request body. If they have that much data to return, they should probably be including it in the request body.

You could explore using an alternate HTTP parser like http-parser-js, since it doesn't appear to have a limit.

Sherillsherilyn

Important: recently the default maximum header size was changed from 80K to 8K. See the discussion here github.com/nodejs/node/issues/24692 To configure that there is an argument available --max-http-header-size – Arturo

Update: this has changed, see this answer for a detailed explanation of how and why. – Mutism

Answer 2

Use --max-http-header-size on the node command-line to accept larger headers. If you get "node: bad option: --max-http-header-size", upgrade to node v10.15.0 or newer. Relevant changelog

node --max-http-header-size 15000 client.js

Credit to @murad.

Jenniejennifer

Just noting --max-http-header-size is also supported in node v8 but not v9 ( nodejs.org/docs/latest-v8.x/api/… ) – Multifid

@DavidThomas so what do we do to increase maximum header size for latest node version? – Agone

Answer 3

In a recent update to Node.js the default allowed maximum header size has recently changed from 80KB to 8KB:

The total size of HTTP headers received by Node.js now must not exceed 8192 bytes.

In our case, we updated versions of Node and all of a sudden started getting 400 Bad Request's from our express server, which was confusing to say the least.

There's an issue open to change this response to a 431 Request Header Fields Too Large which will be more descriptive and helpful.

It has also been made configurable because it was a breaking change for some:

--max-http-header-size=size#
Added in: v11.6.0
Specify the maximum size, in bytes, of HTTP headers. Defaults to 8KB.

Mutism

Yes, exactly, recently I meet the same issue, and find the max size is 8K, not 80K; and there is description in: node_modules/@types/node/http.d.ts: * Defaults to 8KB. Configurable using the [`--max-http-header-size`][] CLI option. – Deferral

The header size has been updated to 16KB in v13.13.0 – Kansu

Answer 4

Headers should contain meta data - if they pass 80KB that means that you doing something wrong.

Node.js allows 80KB of maximum size in header data.

You can read more about http headers in here: https://nodejs.org/en/docs/guides/anatomy-of-an-http-transaction/

Gaul

Do headers include the query string? ie if i have a query string with a text value that is 15200 bytes, is that added to the header size? – Syracuse

I didn't check that, but you probably right. Anyway, try to id's, some short values and not full objects and actual data in headers and you shouldn't take the 80kb in any consideration. – Gaul