How to collect code coverage with Istanbul when executing HTTP endpoints via Postman or Karate

Question

I have a JS project that provides a set of endpoints leveraging Express with a typical express/router pattern.

const express = require('express');
const router = new express.Router();

router.post('/', async (req, res, next) => { });
router.get('/:abc', async (req, res, next) => { });

module.exports = router;

I can successfully start the server with npm start which calls node ./src/index.js and makes the endpoints available at https://localhost:8080

I can also successfully test these endpoints utilizing a tool like Postman or automation like Karate.

The problem i'm having is that I can't seem to collect code coverage using Istanbul when exercising the product source JS through http://localhost:8080.

I've tried npm start followed by nyc --all src/**/*.js gradlew test. The latter being automation that tests the endpoints. This results in 0% coverage which i'm assuming was due to not running nyc with npm start.

Next I tried nyc --all src/**/*.js npm start and noticed some coverage, but this was just coverage from starting the Express server.

Next I tried nyc --all src/**/*.js npm start followed by gradlew test and noticed the code coverage results were the same as when no endpoint tests were run.

Next I tried putting the prior two commands into a single JS script(myscript.js) running each asynchronously wherein the Express server was started before the gradle tests started running and ran nyc --all src/**/*.js myscript.js. The results from this were the same as my previous trial wherein only npm start received code coverage.

Next I tried nyc --all src/**/*.js npm start followed by nyc --all src/**/*.js -no-clean gradlew test and noticed the code coverage results were the same as when no endpoint tests were run.

Next I tried all of the attempts above by wrapping them into package.json scripts and running npm run <scriptName> getting the same exact behavior.

Finally I tried nyc instrument src instrumented/src --compact=false followed by npm run start:coverage wherein this start:coverage script calls the instrumented index.js at node ./instrumented/src/index.js followed by gradlew test followed by nyc report --reporter=lcov. This attempt also failing to produce any additional code coverage from the gradlew endpoint tests.

Doing some research online I came across this post How do I setup code coverage on my Express based API?

And thought this looks eerily similar to my problems. Eg Istanbul doesn't know how to cover code when exercising the code through executing endpoints.

I decided to still post this as the above post is quite a bit old and wanted to get opinions and see if there is a better solution than https://github.com/gotwarlost/istanbul-middleware

EDIT

Adding more specifics about how we start the Express server and run automation without Istanbul today. Just to clarify what we're working with and automation tools we're invested in. (Mainly Karate and Java)

/*
  calls --> node -r dotenv/config src/index.js
*/
npm start

/*
  calls --> gradlew clean test
  this effectively calls a tool called Karate
  Karate's base url is pointed to: https://locahost:8080
  Karate tests execute endpoints on that base url
  This would be akin to using Postman however Karate has quite a bit of configuration options
  https://github.com/intuit/karate
*/
npm test

Outofdoors

I'm a bit off with what you are trying to do... we start the server but we don't start the tests when starting the server... we either run the tests (to run the tests in the CI/CD tool), or start the server (stage and prod)... and code coverage report should be only created upon runing the tests (unit tests + interations tests + e2e tests).... what are you really trying to accomplish? – Adrian

Our use case is to 1) start the express server with 'npm start' which makes testable endpoints available at localhost:8080 2) test the endpoints at the noted URI 3) produce a code coverage report showing what JS code was touched based on which endpoints were called. – Outofdoors

you do not start the server for testing, the tests start it by itself... check chai-http - I've also created a simple repo so you can give a look, checkout and see how everything plays together, I hope it helps. – Adrian

Answer 1

Through many hours of investigation we've managed to solve this. Prior project posted by @balexandre has been updated to illustrate how to do this.

https://github.com/kirksl/karate-istanbul

Outofdoors

What do you do if the project you are testing is a Java back end api? In that case, scanning a NPM project doesn't make sense. – Tjirebon

Answer 2

As said on the comments, you never start your server to run the tests... the tests will point to your server when you require the server file.

in my example, I'm running mocha with chai and the chai-http package helps to call the server

server.js

const app = require("express")();

// everything else ...

exports.server = app;

in your end-to-end tests, you can easily have:

const chai = require('chai');
const chaiHttp = require('chai-http');
chai.use(chaiHttp);

const server = require("./server.js").server;

...

it("should calculate the circumference", done => {
    chai
       .request(server) // <-- attach your server here
       .get('/v1/circumference/10')
       .end((err, res) => {
         expect(res.status).to.be.eql(200);
         expect(res.type).to.be.eql('application/json');
         expect(res.body.result).to.eql(62.83185307179586);
         done();
       });
    });
});

I've made a very simple project and pushed to GitHub so you can checkout and run everything, in order to see how all work together

GitHub Project

---

Added

I've added a route so it can show the coverage report (I used the html report) and created a static route to it ...

when you run the coverage npm run coverage it will generate the report inside ./report folder and a simple express route pointed to that folder, will enable one to see it as an endpoint.

commit info for such change

Adrian

Thank you for the solution. Are you implying that Postman would not produce coverage? Also i've updated my initial post with an "EDIT" stating what tools we're invested in. Unfortunately we're not using Chai, we are using Karate. I know Karate has quite a bit of horsepower and configuration options as shown on their readme. I'm happy to work with you to test out anything you think might be suggestive to resolve this github.com/intuit/karate – Outofdoors

I still have no idea what do you really want to accomplish... you mean, you want to create a route to access the result of the coverage done when deploying? is that it? you run coverage in the continuous deployment process (a part of locally) and that's it, the code does not change, so will never coverage... – Adrian

@CaptainKirk added route to point to generated coverage report, is that what you wanted? – Adrian

I think I may have caused some confusion when I say "produce coverage". I know exactly how to create an Istanbul report as you're showing in your latest commit info with "npm coverage". What I mean when I say "produce coverage" is to cause the instrumented JS code to be executed when I test it via localhost:8080. This post really speaks to my issue. Very similar. Also I cannot use Mocha/Chai as you're using so that won't be a solution I can use unfortunately. I can only use Karate. github.com/istanbuljs/nyc/issues/1168 – Outofdoors

so you want to run a bash command from an endpoint? but that's easy no? but I still think you have your architecture wrong, as you should never run such thing, coverage points to a test suite, and that never changes after you deploy... why do you need to run it from an endpoint? – Adrian

I don't want to run a bash command. Just start the Express server pointing to all instrumented code behind it, call endpoints bound to that server, tell Istanbul to create my report showing what JS code was touched from running the endpoints. It sounds so easy and we do this all the time with success in the Java world. As shown in this project with Jacoco(Java Code Coverage). github.com/intuit/karate/tree/master/karate-demo – Outofdoors

no matter what I read, it ends up as "...is a Behavior Driven Development (BDD) testing framework for Java" ... it's not for nodeJs... so I'll rest my case here and need to focus on other things, good luck! – Adrian