rust-covfix

Rustc is known to report an incorrect coverage for some lines https://stackoverflow.com/questions/32521800/why-does-kcov-calculate-incorrect-code-coverage-statistics-for-rust-programs. rust-covfix will read coverage from the file generated by grcov, fix it, then outputs the correct coverage.

Though only lcov format is supported at current, Another formats are going to be supported in future releases.

Features

• Compatible with the latest stable/beta/nightly Rust compiler
• Windows/OSX/Linux are all supported
• Lightweight (small dependencies)
• Fast and safe (implemented in Rust language)
• rust-covfix is also available with Rust API (Documentation)
• Show summary of coverage difference.

Optional features

Optional features are available with cargo's --features option. You can specify the features like:

$ cargo install --no-default-features --features "cli lcov"

Feature name	Description	Default?
cli	Command Line Interface. This feature is required to build rust-covfix executable.	yes
lcov	Make LcovParser available	yes
noinline	Avoid adding #cfg[inline] attribute on function. (deprecated)	no
backtrace	Dump backtrace information on every time the error has occured.	no

Install

Download the latest release from GitHub Release Page.

You can also install via cargo command.

$ cargo install rust-covfix

How to generate correct code coverage from Rust program?

1. Avoid inlining the functions (optional)

It seems that the current version of rustc (1.42) will automatically inline the function that is only called from one place. This behaviour causes incorrect coverage for your tests.

To avoid this, you have to add #[inline(never)] attributes for their functions manually. I recommend defining a new feature flag coverage in your crate. In Cargo.toml, append the following lines.

[features]
coverage = []

And then, add the attribute like #[cfg(feature = "coverage", inline(never))] to the functions which is called from just one location.

#[cfg(feature = "coverage", inline(never))]
fn foo() {
  // ...
}

This will avoid inlining the functions only when you enable coverage feature flag.

2. Compile your crate with -Zprofile option

In order to generate code coverage with rustc, you must specify -Zprofile option. This option is currently (1.42) unstable and only available from nightly toolchain.

Also, some other flags will be required to generate correct coverage. Here is my recommend.

$ export CARGO_INCREMENTAL=0
$ export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"

Then, compile your crate and run tests.

$ cargo test --features coverage

3. aggregate code coverage data

Now there are code coverage data in target/debug/deps directory. The next step is aggregating them and convert the format so that rust-covfix can read coverage data.

I highly recommend to use grcov to aggregate them. This project is developed by mozilla team and supports the latest Rust toolchains.

Install latest grcov and run the following commands from your project root directory.

zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info

where YOUR_PROJECT_NAME is your crate name specified in Cargo.toml.

4. Fix coverage data using rust-covfix

Now rust-covfix can read coverage from lcov.info.

$ rust-covfix -o lcov_correct.info lcov.info

This command will write a correct coverage into lcov_correct.info. You can upload them into codecov.io, or generate HTML summary using genhtml.

Use rust-covfix on Travis CI

Here is an example script to use rust-covfix on Travis CI environment.

#!/bin/bash
# ci/script.sh

set -ex

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # Setup grcov
  wget https://github.com/mozilla/grcov/releases/download/v0.5.7/grcov-linux-x86_64.tar.bz2
  tar xvf grcov-linux-x86_64.tar.bz2

  # Setup environmental variables
  export CARGO_INCREMENTAL=0
  export RUSTFLAGS="-Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort"
fi

# Compile and run tests
cargo test --all-features

if [ "$TRAVIS_RUST_VERSION" = "nightly" ] && [ -z "$TRAVIS_TAG" ]; then
  # collect coverages
  zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
  ./grcov ccov.zip -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov.info

  # fix coverage using rust-covfix
  rust-covfix lcov.info -o lcov.info

  # upload coverage to codecov
  bash <(curl -s https://codecov.io/bash) -f lcov.info
fi

Then, call this script from travis.yml

language: rust

matrix:
  include:
    - os: linux
      rust: nightly

script:
  - bash ci/script.sh

Use rust-covfix on Gitlab CI

codecov:
  stage:                           coverage
  artifacts:
    name:                          "${CI_JOB_NAME}_${CI_COMMIT_REF_NAME}"
    paths:
      - artifacts/
  variables:
    # Variables partly came from https://github.com/mozilla/grcov/blob/master/README.md
    CARGO_INCREMENTAL:             0
    RUSTFLAGS:                     "-Zprofile -Zmir-opt-level=0 -Ccodegen-units=1
                                      -Copt-level=0 -Clink-dead-code -Coverflow-checks=off"
    # We removed the `-Cinline-threshold=0` flag from the above `RUSTFLAGS` since it was bugged
    # at the time and lead to inlining of functions that shouldn't be inlined for the coverage
    # report. (More information here: https://github.com/Kogia-sima/rust-covfix/issues/2)
  script:
    # RUSTFLAGS are the cause target cache can't be used here
    - unset "CARGO_TARGET_DIR" # only needed if it's not default
    - cargo clean
    - cargo test --verbose --all-features --no-fail-fast --workspace
    - cargo build --verbose --all-features --workspace
    # coverage with branches
    - grcov ./target -s . -t lcov --llvm --branch --ignore-not-existing --ignore "/*" --ignore "tests/*" -o lcov-w-branch.info
    - rust-covfix -o lcov-w-branch_correct.info lcov-w-branch.info
    # We'd like to not use a remote bash script for uploading the coverage reports,
    # however this job seems to be more tricky than we hoped.
    - bash <(curl -s https://codecov.io/bash) -t "$CODECOV_P_TOKEN" -f lcov-w-branch_correct.info
    - cp *.info /artifacts # this is to output the report files, for the debug purposes

Use rust-covfix on GitHub actions

name: Coverage

on: [push, pull_request]

jobs:
  coverage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Install toolchain
        uses: actions-rs/toolchain@v1
        with:
          profile: minimal
          target: x86_64-unknown-linux-gnu
          toolchain: nightly
          override: true
      - name: Install grcov
        run: curl -L https://github.com/mozilla/grcov/releases/download/v0.6.1/grcov-linux-x86_64.tar.bz2 | tar jxf -
      - name: Install rust-covfix
        run: |
          curl -L https://github.com/Kogia-sima/rust-covfix/releases/download/v0.2.2/rust-covfix-linux-x86_64.tar.xz |tar Jxf -
          mv rust-covfix-linux-x86_64/rust-covfix ./
      - name: Test all crates
        env:
          CARGO_INCREMENTAL: 0
          RUSTFLAGS: -Zprofile -Ccodegen-units=1 -Copt-level=0 -Clink-dead-code -Coverflow-checks=off -Zpanic_abort_tests -C panic=abort
          RUSTDOCFLAGS: -C panic=abort
        run: |
          cargo build --all-features --workspace
          cargo test --all-features --workspace
      - name: collect coverages
        run: |
          zip -0 ccov.zip `find . \( -name "YOUR_PROJECT_NAME*.gc*" -o -name "test-*.gc*" \) -print`
          ./grcov ccov.zip --llvm --branch -t lcov -o lcov.info --ignore "/*" --ignore "sailfish-tests/*"
      - name: fix coverages
        run: ./rust-covfix -o lcov.info lcov.info
      - name: upload coverage
        uses: codecov/codecov-action@v1
        with:
          file: ./lcov.info

Why is this project developed as a standalone package?

There are 3 reasons for this.

1. A grcov collaborator is aware of rustc reporting incorrect coverage mozilla/grcov#249 (comment). I'm trying to compare original Rust code and transformed LLVM IR, but still don't fully understand why Rustc report incorrect coverage. The rust-covfix was originally developed as a temporal workaround for this problem until I understand when and where rustc reports incorrect coverage. It is not completely unravelled yet.

2. grcov is just a coverage collection tool, not a coverage generator. It only aggregates the raw coverage data, and convert the format. It does not read the source code, nor manipulate any coverage data. Also, although grcov is targetting many programming languages such as C/C++, Nim and Rust, rust-covfix only supports Rust.

3. rust-covfix is targetting not only grcov, but also cargo-kcov and Tarpaulin in the future. Actually, Tarpaulin can now generate the lcov.info. It means rust-covfix is already able to fix the coverage data generated by Tarpaulin. I'm working on feature/cobertura branch to support cargo-kcov too. The final objective is to support these 3 tools.

How is the incorrect line coverage detected?

rust_covfix fixes the coverage information using some rules. You can pass --rules option to specify which rules are used to fix coverages.

Rules

close

closing brackets, line of else block will be ignored.

if a > 0 {
    b = a
} else {  // <-- marked as "not executable"
    b = -a
};  // <-- marked as "not executable"

test

module block named test or tests which has attribute cfg(test) will be ignored.

all functions with #[test] attribute will be also ignored.

#[cfg(test)]
mod tests {  // <-- removed from coverage
    fn util() { ... }  // <-- removed from coverage

    #[test]
    fn test_hoge() { ... }  // <-- removed from coverage
}

loop

Fix rust internal bugs that loop branches are not correctly passed.

for i in 0..10 {  // <-- fix branch coverage information
    println!("{}", i);
}

derive

structs with derive(...) attribute will be ignored

#[derive(Clone, Debug)]  // <-- removed from coverage
struct Point {  // <-- removed from coverage
  x: f64,  // <-- removed from coverage
  y: f64  // <-- removed from coverage
}  // <-- removed from coverage

comment

ignore coverage based on comment marker

fn main() {
    let a = 1 + 2; // cov:ignore-line

    // cov:begin-ignore-branch
    println!("Hello");
    println!("world!");
    // cov:end-ignore-branch

    // cov: begin-ignore-line
    if a > 2 {
        println!("a is large!");
    } else if a == 0 {
        println!("a is small!");
    }
    // cov: end-ignore-line

    // cov:begin-ignore
    println!("a = {}", a);
    // cov:end-ignore

    println!("finish."); // cov:ignore-branch

    return (); // cov:ignore
}

Roadmap

• Support cobertura.xml file. (WIP)
• Add option for uploading the correct coverages to coveralls.
• Use syntax tree generated using syn crate.
• Performance improvement

Author

👤 Kogia-sima

• Twitter: @Kogia_sima
• Github: @Kogia-sima

🤝 Contributing

Contributions, issues and feature requests are welcome!

Feel free to check issues page.

Show your support

Give a ⭐️ if this project helped you!

📝 License

Copyright © 2019 Kogia-sima.

This project is MIT licensed.

---

This README was generated with ❤️ by readme-md-generator