NodeJS Express Request Entity Too Large

Question

I've tried many solutions listed here (increasing of memory limit, adding parameterlimit, adding type as 'application/json') to fix this 'Request Entity too large' error (it also returns http code 413). But none of them seem to make this error go away.

The current size of the json can range from 200k up to 400k entities.

Here is the current configuration:

app.use( bodyParser.json({limit: "15360mb", type:'application/json'}) );      
app.use(bodyParser.urlencoded({
  limit: "15360mb",
  extended: true,
  parameterLimit:5000000,
  type:'application/json'
})); 
app.use(bodyParser())

Any ideas on how to increase the limit?

More information

This is the full error message if it helps:

    { Error: Request Entity Too Large
        at respond (/home/nodejs/server/node_modules/elasticsearch/src/lib/transport.js:307:15)
        at checkRespForFailure (/home/nodejs/server/node_modules/elasticsearch/src/lib/transport.js:266:7)
        at HttpConnector.<anonymous> (/home/nodejs/server/node_modules/elasticsearch/src/lib/connectors/http.js:159:7)
        at IncomingMessage.bound (/home/nodejs/server/node_modules/lodash/dist/lodash.js:729:21)
        at emitNone (events.js:111:20)
        at IncomingMessage.emit (events.js:208:7)
        at endReadableNT (_stream_readable.js:1056:12)
        at _combinedTickCallback (internal/process/next_tick.js:138:11)
        at process._tickCallback (internal/process/next_tick.js:180:9)
      status: 413,
      displayName: 'RequestEntityTooLarge',
      message: 'Request Entity Too Large',
      path: '/_bulk',
      query: {},
      body: '<large body here>'
}

Solution

The error was indeed due to wrong configuration on elasticsearch and not nodejs. Manage to fix this by following https://github.com/elastic/elasticsearch-js/issues/241 and setting http.max_content_length: 500mb in elasticsearch.yml.

@ryanlutgen also provided a link for more information about this error here. https://github.com/elastic/elasticsearch/issues/2902

This issue has been fixed. Thanks for all the input!

Camisole

Try removing this line app.use(bodyParser()) – Othella

You should never send that much data. You should send your data in chunks to avoid timeouts, like when you upload a video through YouTube's API. See developers.google.com/youtube/v3/guides/… In our app, we upload in 50Mb chunks – Luht

without that last line, the same error still occurs. I added that hoping that the error will be solved but it didn't sadly – Camisole

i would send it in chunks if i could. but due to limitations set, i can't do it unfortunately. The node can process it if its at 200k-300k~ but once it hits 400k, the error will occur. Or could nodejs split the json into chunks? if it could, it might work. – Camisole

Am I reading wrong? or you just want to send 15360 MB data in one request? – Hourigan

@Camisole Start by looking at a package like npmjs.com/package/chunk-upload – Luht

@gokcand, i set it at that number due to 'range error: invalid string length error' i had previously. But the max data size should be <1GB – Camisole

@Camisole Is your app deployed to some server like nginx? or just testing in your own localhost? – Hourigan

@gokcand, no, its not deployed on nginx. It is running on hyperv ubuntu server with 16gb ram allocated to it. The node was started using nodejs --max_old_space_size=15360 index.js – Camisole

i've included the error message. Now that i look at it, it might be a limit on elasticsearch node library? – Camisole

What version of ES are you using? See here: github.com/elastic/elasticsearch/issues/2902 for older ES versions SIde note, you are gimping your performance by not optimizing bulk requests. – Flew

@ryanlutgen, that was one of the solution i found a few hours ago that seem to be working. just doing some stress test on the system now to ensure that nothing breaks before marking/indicating this as solved. Thanks for answering though! :) – Camisole

Hi I am using node js and I cant find elasticsearch.yml where do I need to set http.max_content_length: 500mb i can see few property inside export interface ConfigOptions like host, hosts, requestTimeout, ssl, deadTimeout etc – Lumberjack

Please post the self-answer as an answer, not as an edit to the question. Thanks. – Demantoid

Answer 1

If You doesn't solve your issue, bodyParser also has a limit. You must use

app.use(bodyParser({limit: '4MB'}))

Verbid

Answer 2

I'm not using bodyParser so I've fixed my issue with the following:

app.use(express.json({
  limit: '2MB',
}));

Sufferable