express body-parser utf-8 error in test

Question

Super stumped by this. I have some server code that for some reason throws a UTF-8 error in my tests but works fine when running the server normally:

code:

export default ({ projectId = PROJECT_ID, esHost = ES_HOST } = {}) => {
  let app = express();
  app.use(cors());
  app.use(bodyParser.json({ limit: '50mb' }));

  let http = Server(app);
  let io = socketIO(http);

  let server = {
    app,
    io,
    http,
    status: 'off',
    listen(
      port = PORT,
      cb = () => {
        rainbow(`⚡️ Listening on port ${port} ⚡️`);
      },
    ) {
      this.http.listen(port, () => {
        main({ io, app, projectId, esHost, port });
        this.status = 'on';
        cb();
      });
    },
    close(cb = () => {}) {
      if (this.http) {
        this.http.close(() => {
          this.status = 'off';
          cb();
        });
      } else {
        throw '❗️ cannot close server that has not been started ❗️';
      }
    },
  };

  return server;
};

usage (exactly the same, but in jest test body-parser isn't working properly):

import createServer from '../server'

let server = createServer()
server.listen(5050);

I'm using postman, post response outside of test:

{
    "projects": [
        {
            "id": "test",
            "active": true,
            "timestamp": "2018-02-25T21:33:08.006Z"
        },
        {
            "id": "TEST-PROJECT",
            "active": true,
            "timestamp": "2018-03-05T21:34:34.604Z"
        },
        {
            "id": "asd",
            "active": true,
            "timestamp": "2018-03-06T23:29:55.348Z"
        }
    ],
    "total": 3
}

unexpected post response inside jest test server:

Error

UnsupportedMediaTypeError: unsupported charset "UTF-8"
    at /Users/awilmer/Projects/arranger/node_modules/body-parser/lib/read.js:83:18
    at invokeCallback (/Users/awilmer/Projects/arranger/node_modules/raw-body/index.js:224:16)
    at _combinedTickCallback (internal/process/next_tick.js:131:7)
    at process._tickCallback (internal/process/next_tick.js:180:9)

Amersfoort

installed everything, tests all pass, did you try to delete node_modules, npm cache clean and fresh install everything again? – Stubstad

Which OS are you using to test this? – Krugersdorp

I'm running a osx 10.12.6..nodejs 8.4. I will try a completely fresh install on another machine – Amersfoort

@azium, I am able to reproduce your issue, the issue is jest only here. It won't happen in other frameworks. I am looking for possible solutions now – Krugersdorp

@azium, posted the solution. Have a look – Krugersdorp

Answer 1

So I was able to reproduce the issue and find the source of the issue and the workaround to make it work. The issue is caused by jest framework.

Before you jump on reading the rest of the thread, I would suggest you read another Jest thread I answer long back. This would help get some context internals about the require method in jest

Specify code to run before any Jest setup happens

Cause

The issue happens only in test and not in production. This is because of jest require method.

When you run your tests, it starts a express server, which calls the node_modules/raw-body/index.js as shown in below image

As you can see the encodings is null. This is because the iconv-lite module does a lazy loading of encodings. The encodings are only loaded when getCodec method gets executed.

Now when your test has fired the API, the server needs to read the body so the getCodec gets called

This then goes through the jest-runtime/build/index.js custom require method (which is overloaded if you read the previous link).

The execModule has a check for this._environment.global, which is blank in this case and hence a null value is returned and the module never gets executed

Now when you look at the exports of the encodings module, it just is a blank object

So the issue is purely a jest. A feature jest lacks or a bug mostly?

Related Issues

Related issues have already been discussed on below threads

https://github.com/facebook/jest/issues/2605

https://github.com/RubenVerborgh/N3.js/issues/120

https://github.com/sidorares/node-mysql2/issues/489#issuecomment-313374683

https://github.com/ashtuchkin/iconv-lite/issues/118

https://github.com/Jason-Rev/vscode-spell-checker/issues/159

Fix

The fix to the problem is that we load the module during our test itself and force a early loading instead of lazy loading. This can be done by adding a line to your index.test.js at the top

import encodings from '../../node_modules/iconv-lite/encodings';
import createServer from '@arranger/server';

After the change all the test pass, though you have a error in the url of the test so you get Cannot POST /

Krugersdorp

Absolutely brilliant. Best karma I've ever spent. – Amersfoort

I have been trying to create a small reproducible test for the same and figure out whats wrong. But seems this is an edge case that you have unearthed. The express and body parser work fine even across project with lazy loading. You should open a issue with Jest and give them this repo and the SO link to reproduce and fix the issue – Krugersdorp

Great answer. I just added require('iconv-lite/encodings'); to the top of my file and it magically works. – Undervalue

hi there, after importing '../../node_modules/iconv-lite/encodings', where to use assigned variable then – Breaststroke

You don't need to use it, just importing does the job – Krugersdorp

Answer 2

I'm adding a slightly different solution inspired from @Tarun Lalwani

Add the following lines at the top of your test file.

const encodings = require('./node_modules/iconv-lite/encodings');                                                                                                                                                                       
const iconvLite = require('./node_modules/iconv-lite/lib');                                                                                                                                                                             
iconvLite.getCodec('UTF-8');

Epiblast

What is your version of iconvlite ? I only see utf-7 and utf-16 version in iconvlite – Calycle

Answer 3

I spent many hours trying to figure out why Jest would report a 415 error code when testing the Node.js server. Node.js is configured to use app.use(bodyParser.json(...)); on our system, too. That didn't solve the issue.

Solution

When using res.status(...), you MUST either chain on .json() or use res.json(), too. That means if you respond with a 500 error or otherwise and you don't return any JSON data, you still need to use res.json(). No idea why, as that defeats the whole purpose of app.use(bodyParser.json(...)); in the first place.

Example

const express = require('express');
const router = express.Router();

router.post("/register", (req, res) => {
  // ...
  res.status(500).json();
  // ...
});

Hepplewhite