Angular + Cypress code coverage reporting not working

Question

I have been trying to get Cypress code coverage working with my Angular production project to no avail.

To try and help diagnose it, I have created a minimal implementation project to make sure I wasn't introducing anything weird in the production version, which I don't think I am as the same issue is still happening. It's starting to drive me mad!

I have used a few references and as far as I can see I have the things in place I need to:

• Cypress Coverage Documentation
• Article 1
• Code Example (older dependencies)

As far as I can tell the Angular and Cypress side is all hooked up and am getting output in the .nyc_output folder and a coverage report. However the report is not indicating typescript line coverage or including those stats.

I have seen this but didn't seem to help.

Code Instrumentation (webpack extension + angular.json):

module.exports = {
  module: {
    rules: [
      {
        test: /\.(js|ts)$/,
        loader: "istanbul-instrumenter-loader",
        options: { esModules: true },
        enforce: "post",
        include: require("path").join(__dirname, "..", "src"),
        exclude: [
          /\.(e2e|spec)\.ts$/,
          /node_modules/,
          /(ngfactory|ngstyle)\.js/,
        ],
      },
    ],
  },
};

"serve": {
  "builder": "ngx-build-plus:dev-server",
  "options": {
    "browserTarget": "architecture-testing:build",
    "extraWebpackConfig": "./cypress/coverage.webpack.js",
    "sourceMap": true
  },
  "configurations": {
    "production": {
      "browserTarget": "architecture-testing:build:production"
    }
  }
}

Cypress appears to be recording and saving coverage:

const registerCodeCoverageTasks = require("@cypress/code-coverage/task");

module.exports = (on, config) => {
  registerCodeCoverageTasks(on, config);

  return config;
};

out.json appears to have the correct file and code mapping:

:

package.json (nyc config + deps):

{
  "name": "architecture-testing",
  "version": "0.0.0",
  "scripts": {
    "postinstall": "ngcc",
    "ng": "ng",
    "start": "ng serve",
    "build": "ng build",
    "test": "ng test",
    "lint": "ng lint",
    "e2e": "ng e2e",
    "precypress": "rimraf .nyc_output coverage",
    "cypress": "ng run architecture-testing:cypress-run",
    "cypress:open": "cypress open",
    "cypress:run": "cypress run"
  },
  "nyc": {
    "extends": "@istanbuljs/nyc-config-typescript",
    "all": true,
    "exclude": [
      "coverage/**",
      "cypress/**",
      "**/*.spec.ts"
    ]
  },
  "private": true,
  "dependencies": {
    "@angular/animations": "~9.1.9",
    "@angular/common": "~9.1.9",
    "@angular/compiler": "~9.1.9",
    "@angular/core": "~9.1.9",
    "@angular/forms": "~9.1.9",
    "@angular/platform-browser": "~9.1.9",
    "@angular/platform-browser-dynamic": "~9.1.9",
    "@angular/router": "~9.1.9",
    "rxjs": "~6.5.4",
    "tslib": "^1.10.0",
    "zone.js": "~0.10.2"
  },
  "devDependencies": {
    "@angular-devkit/build-angular": "~0.901.7",
    "@angular/cli": "~9.1.7",
    "@angular/compiler-cli": "~9.1.9",
    "@briebug/cypress-schematic": "^3.3.0",
    "@cypress/code-coverage": "^3.8.1",
    "@cypress/webpack-preprocessor": "5.4.1",
    "@istanbuljs/nyc-config-typescript": "^1.0.1",
    "@types/node": "^12.11.1",
    "codelyzer": "^5.1.2",
    "cypress": "^4.8.0",
    "istanbul-instrumenter-loader": "^3.0.1",
    "istanbul-lib-coverage": "^3.0.0",
    "ngx-build-plus": "^9.0.6",
    "nyc": "^15.1.0",
    "rimraf": "^3.0.2",
    "source-map-support": "^0.5.19",
    "ts-loader": "7.0.5",
    "ts-node": "^8.3.0",
    "tslint": "~6.1.0",
    "typescript": "~3.8.3"
  }
}

Spec file:

it('does something', () => {
  cy.visit('http://localhost:4200');
  cy.get('[data-cy=button-one]').click();
  cy.get('[data-cy=button-output]').should('have.text', 'you clicked button 1');
});

Sorry it's so long but I am stuck as to where to go next. Many thanks if you can point me in any direction.

Update based on answer investigation:

Looking at the past versions of the @cypress/code-coverage it appears the issue for me was introduced in v3.3.0 of the plugin. All the versions for v3.2.* were working for me when downgrading my minimal project. After looking at the documentation changes for v3.3.0 the key bit of information in the readme was:

**Note:** if you have `all: true` NYC option set, this plugin will check the produced `.nyc_output/out.json` before generating the final report. If the `out.json` file does not have information for some files that should be there according to `include` list, then an empty placeholder will be included, see [PR 208](https://github.com/cypress-io/code-coverage/pull/208).

My original nyc config was:

"nyc": {
  "extends": "@istanbuljs/nyc-config-typescript",
  "all": true,
  "exclude": [
    "coverage/**",
    "cypress/**",
    "**/*.spec.ts"
  ]
}

So for some reason even though I have metrics for the files that were being tested in out.json a second "empty placeholder" node was being created and overriding the subsequent report generation. I am guessing that this is potentially a bug or a problem with my setup again so will ask the creators.

I can now see coverage if I change my nyc config to:

"nyc": {
  "extends": "@istanbuljs/nyc-config-typescript",
  "all": true,
  "include": [
    "src/**/*.ts"
  ],
  "exclude": [
    "coverage/**",
    "cypress/**",
    "**/*.spec.ts"
  ]
}

This does mean that if I don't hit a file with testing it won't be included as an empty placeholder as "all": true is no longer present.

Looking at @briebug/[email protected] this doesn't appear to be causing any issues (same happens without using their builder) but has been raised as one here and here.

Fuzz

Excellent work finding the root cause. If you look at the examples for main.ts below, the 1st node with metrics has escaped back-slashes in the path, whereas the second has forward-slashes, so they are technically different keys. This explains why the extra nodes appear by the feature added in v3.3.0 [PR 208], but I guess only on Windows since Linux/Mac has forward slashes in the path from the start. – Sexless

This issue Windows specific? Running with nyc all=true causes all files to display as 0% coverage seems to be the same one. – Sexless

Ah yes I didn't notice that, good spot! It does seem like the same issue (I'm Windows 10) so will keep a keen eye on that. – Fuzz

BTW debug statements in code-coverage can be turned on in the script: "cy:open": "set DEBUG=code-coverage & cypress open" – Sexless

Take a look at my answer here it may be the same issue – Rizzi

Answer 1

Comparing ang-cy-cov-example to your package.json, a major difference is that he uses @cypress/[email protected] where-as you have the latest v3.8.1.

Changing back to this v1.14.0 works ok with your setup. Since your info indicates data is appearing in .nyc_output/out.json, I tested with the command line ./node_modules/.bin/nyc report which gives a quick view in the console.

Comparing .nyc_output/out.json between the two versions, the individual nodes are structurally the same, i.e have the correct sections (path, statementMap, inputSourceMap etc).

There are two types of additional nodes

• additional files such as karma.conf.js, coverage.webpack.js, cy-ts-preprocessor.js, integration/spec.ts, support/commands.ts - which we are not interested in.

• the files we are interested in are duplicated at the end of the file but the duplicates have no coverage metrics.

e.g

First copy of main.ts with metrics

  "path-to\\src\\main.ts": {
    "path": "path-to\\src\\main.ts",
    "statementMap": {
         ...
      },
      "1": {
         ...
      },
      "2": {
         ...
      }
    },
    "fnMap": {},
    "branchMap": {
         ...
    },
    "s": {
      "0": 1,   // indicates one visit to this statement
      "1": 0,
      "2": 1
    },
    "f": {},
    "b": {
      "0": [
        0,
        1
      ]
    },
    "inputSourceMap": {
         ...
    },
    "_coverageSchema": "332fd63041d2c1bcb487cc26dd0d5f7d97098a6c",
    "hash": "5959c383a9744c99a600a28ff82b12f0a540c5e6"
  },

Second copy of main.ts with no metrics

  "path-to/src/main.ts": {
    "path": "path-to/src/main.ts",
    "statementMap": {},
    "fnMap": {},
    "branchMap": {},
    "s": {},          // no metrics recorded here
    "f": {},
    "b": {}
  },

So, conclusion is the NYC report is replacing the first nodes metrics with the empty metrics of the second node.

I skipped back through the versions, v3.2.0 was the latest I found working.

Also note this warning when adding node modules, but can't say if it is a contributing factor.

warning " > @briebug/[email protected]" has incorrect peer dependency "cypress@^3.6.1"

---

Point of failure

The basic problem is in task-utils.js.

ref getting allFiles

const allFiles = globby.sync(patterns, { absolute: true })

where globby returns paths with forward-slash even on Windows

and ref getting coveredPaths

const coveredPaths = coverageKeys.map(key => nycCoverage[key].path)

where the keys have be saved in out.json with back-slash in Windows.

A quick resolve would be to normalize the paths at this point

const coveredPaths = coverageKeys.map(key => nycCoverage[key].path)
  .map(path => path.replace(/\\/g, '/'))  // Compare "normalized" paths 

Patching '/node_modules/@cypress/code-coverage/task-utils.js' fixes the problem.

Sexless

Thank you so much! This helped find the route cause of this. I will update the question with my findings based on your answer. – Fuzz

Answer 2

This patch has now been applied in pull request, looks like v3.8.6

Ref Handle backslashes in coverage file path (Dec 2020).

This is (partly) what the link says:

🎉 This PR is included in version 3.8.6 🎉

The release is available on:

npm package (@latest dist-tag)
GitHub release

Aureaaureate

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review – Surfboat

The essential parts have indeed been posted here - This patch has now been applied in pull request. The link is just to save users scouting through the package pull requests, in the event they want to check it. – Aureaaureate