Rust coverage using kcov does not appear correct

Question

When I record code coverage of my Rust project using codecov.io, the coverage does not appear correct.

1. The unwrap() function and the end bracket are not covered

2. The function declaration is not covered

This is very strange.

---

I cannot provide the full project for reproducing.

I'm using the standard TravisCI configuration for Rust. Here is my .travis.yml:

language: rust
cache: cargo
dist: trusty
sudo: required

rust:
  - stable
  - beta
  - nightly

matrix:
  allow_failures:
    - rust: nightly

script:
  - cargo build --verbose --all
  - cargo test --verbose --all

after_success: |
  wget https://github.com/SimonKagstrom/kcov/archive/master.tar.gz &&
  tar xzf master.tar.gz &&
  cd kcov-master &&
  mkdir build &&
  cd build &&
  cmake .. &&
  make &&
  make install DESTDIR=../../kcov-build &&
  cd ../.. &&
  rm -rf kcov-master &&
  for file in target/debug/myproject-*[^\.d]; do mkdir -p "target/cov/$(basename $file)"; ./kcov-build/usr/local/bin/kcov --exclude-pattern=/.cargo,/usr/lib --verify "target/cov/$(basename $file)" "$file"; done &&
  bash <(curl -s https://codecov.io/bash)
  echo "Uploaded code coverage"

Jambeau

How do you measure coverage? (tools/commands/configs used) – Coblenz

For unwrap: it is quite possible that unwrap is inlined and you never cover the branch that panics. Therefore kcov believe there is a non-covered branch on the unwrap call. – Caudex

@kazemakase added config file and description to original question. – Jambeau

@Caudex If this is so, then this is the bug kcov – Jambeau

IMHO, it's not a bug, <strike>it's a feature</strike>, but instead intended. Kconv analyses the assembly and sees a not-taken branch, so it will mark it as not covered – Pentane

@Pentane It's not always true. I analyzed my results, and sometimes line not marked as covered or not-covered for unwrap. And how it's possbile not-teken branch if unwrap invoked? – Jambeau

@Jambeau Thanks for the update. Coverage tools rely on the line number table which maps source lines to machine instruction locations. This is not a 1:1 mapping. A source line often relates to many instructions. Sometimes a line does not have instructions associated with it (e.g. function arguments, or if the compiler was smart with removing unneccessary code). I am surprised by the not-covered }s, though, because I would expect them to be related to RET instructions at least. Could be that your functions were inlined so no code for returning had to be generated... – Coblenz

Please review how to create a minimal reproducible example and then edit your question to include it. No one wants your "full project" to start with, we want you to create a brand-new project that has only this problem. There are Rust-specific MCVE tips if you need them. Saying "the standard TravisCI configuration" is useless because that can change over time. – Gammy

@Gammy I put complete tavis-ci config. I will try to reproduce it via minimal Rust project and put here if success. Thanks. – Jambeau

Sounds great! I bet you don't even need to involve travis or codecov.io and can just run it locally (assuming kcov creates the UI) – Gammy

Another guess for the uncovered `}ˋ: those might be destructor calls that have already been called somewhere else and a drop-glue check is done – Caudex

Answer 1

Assuming Cargo's and Travis' behavior hasn't changed significantly since this question was posted, there are a couple of things at play here.

• Whenever a build configuration changes, your build's fingerprint changes, resulting in a full or partial rebuild and a new filename for the resulting binary in target. Admittedly I'm not aware of the intricacies of exactly when or why this happens, I just know that it happens. In fact, for the project I'm working on, Cargo seems so confused about one of the dependencies that it forces a rebuild almost every single time.
• Travis' cache: cargo default is pretty dumb; it caches all of $CARGO_HOME and target without exceptions. Note that this in combination with the former also means that these caches grow without bound, so you need to throw them away once in a while or use a smarter caching scheme.
• for file in target/debug/myproject-*[^\.d] runs kcov for all builds of myproject, regardless of whether it's newly built or from Travis' build cache. Older builds may of course have different line numbers since they were built from different (older) sources, and coverage may be different.
• coverage.io merges coverage results by making a line red if it is included in any report, unless it's covered by any (other) report. It doesn't show any indication whatsoever if the line numbers from different reports don't match up, or even if one of the reports contains line numbers beyond EOF. In fact, as far as I could find, it doesn't even show which binaries covered/did not cover a line number even though it has this information. You have to download the XML reports and interpret them manually to see that.

Therefore, those uncovered lines might not (all?) be due to the way Rust compiles its binaries as the comments to the question suggest, but might in fact be referring to a different (older) source file entirely. This became pretty obvious in our project after a while...

If it's not this obvious, the easiest way to verify that this is what's going on is to just throw away Travis' build cache and force a rebuild.

Since incremental builds don't really work for our project anyway, the solution we used was to just not have Travis cache the target directory, as suggested here. Depending on how much your CI build time depends on incremental builds you may be forced to do something smarter.

Steward