Mutation Testing with cargo-mutants

Your tests pass. But are they actually checking anything? Mutation testing answers that question by systematically modifying your code and checking whether your tests catch each change.

What Is Mutation Testing?

A mutation testing tool:

1. Parses your source code
2. Creates mutants — small, targeted changes (replacing + with -, swapping return values, etc.)
3. Runs your test suite against each mutant
4. Reports which mutants survived (tests still passed despite the change)

Surviving mutants reveal weak spots in your test suite — code paths where your tests don't actually verify behavior.

Installation

cargo install cargo-mutants

Basic Usage

Run against your crate:

cargo mutants

This generates a mutants.out/ directory with results:

Setting a Timeout

Some mutants create infinite loops. Set a per-test timeout:

cargo mutants --timeout 30

Reading Results

The missed.txt file is where the action is:

src/frequency.rs:163:5: replace mhz_to_thz -> f64 with 1.0
src/frequency.rs:223:5: replace khz_to_thz -> f64 with 1.0
src/frequency.rs:283:5: replace hz_to_thz -> f64 with 1.0

This tells you: if someone replaced mhz_to_thz with a function that always returns 1.0, your tests would still pass. That's a real gap.

The "Unity Test Value" Trap

A common pattern that mutation testing catches:

#[test]
fn frequency_conversions() {
    let thz = 1.0;
    let mhz = 1_000_000.0;
    assert_eq!(mhz_to_thz(mhz), thz); // thz is 1.0!
}

The mutant replace mhz_to_thz -> f64 with 1.0 survives because the expected output is 1.0. Fix: add a non-unity test point:

#[test]
fn frequency_conversions_non_unity() {
    let thz = 2.5;
    let mhz = 2_500_000.0;
    assert_eq!(mhz_to_thz(mhz), thz); // Now 1.0 would fail
}

Filtering Mutants

Test a specific file:

cargo mutants -f src/frequency.rs

Exclude files (e.g., skip main.rs which often has trivial mutants):

cargo mutants -e src/main.rs

CI Integration

Add to your CI pipeline for periodic checks (mutation testing is slow, so run nightly rather than on every push):

# GitHub Actions example
- name: Mutation testing
  run: |
    cargo install cargo-mutants
    cargo mutants --timeout 60
    if [ -s mutants.out/missed.txt ]; then
      echo "::warning::Survived mutants found"
      cat mutants.out/missed.txt
    fi

.gitignore

Always ignore the output directory:

mutants.out/

Practical Tips

• Start small: Run on one module at a time with -f to keep iteration fast
• Focus on missed: caught.txt growing is good but missed.txt shrinking is the goal
• Ignore trivial misses: replace main with () is expected and harmless
• Use with criterion: Mutation testing validates correctness; benchmarks validate performance. Together they give high confidence in both
• Run periodically: Full mutation runs take minutes to hours. Weekly or after major changes is a good cadence

Real-World Results

Running cargo mutants on rfconversions (a unit conversion library with 70 unit tests and 53 doc tests):

• 128 caught — tests killed the mutant ✅
• 3 missed — THz conversion functions only tested with unity values
• 0 timeouts, 0 unviable

The 3 missed mutants were fixed by adding a single test with non-unity values. That's the power of mutation testing: one run, one test, measurably better coverage.