mutation-testing

Expert knowledge for mutation testing - validating that your tests actually catch bugs by introducing deliberate code mutations.

When to Use This Skill

| Use this skill when... | Use another skill instead when... | |------------------------|----------------------------------| | Validating test effectiveness | Writing unit tests (use vitest-testing) | | Finding weak/insufficient tests | Analyzing test smells (use test-quality-analysis) | | Setting up Stryker or mutmut | Writing E2E tests (use playwright-testing) | | Improving mutation score | Generating test data (use property-based-testing) | | Checking if tests catch real bugs | Setting up code coverage only |

Core Expertise

Mutation Testing Concept

• Mutants: Small code changes (mutations) introduced automatically
• Killed: Test fails with mutation (good - test caught the bug)
• Survived: Test passes with mutation (bad - weak test)
• Coverage: Tests execute mutated code but don't catch it
• Score: Percentage of mutants killed (aim for 80%+)

What Mutation Testing Reveals

• Tests that don't actually verify behavior
• Missing assertions or edge cases
• Overly permissive assertions
• Dead code or unnecessary logic
• Areas needing stronger tests

TypeScript/JavaScript (Stryker)

Installation

# Using Bun
bun add -d @stryker-mutator/core

# For Vitest
bun add -d @stryker-mutator/vitest-runner

# For Jest
bun add -d @stryker-mutator/jest-runner

Running Stryker

npx stryker run                                    # Run mutation testing
npx stryker run --incremental                      # Only changed files
npx stryker run --mutate "src/utils/**/*.ts"       # Specific files
npx stryker run --reporters html,clear-text        # HTML report
open reports/mutation/html/index.html              # View report

Understanding Results

Mutation score: 82.5%
- Killed: 66 (tests caught the mutation)
- Survived: 14 (tests passed despite mutation - weak tests!)
- No Coverage: 0 (mutated code not executed)
- Timeout: 0 (tests took too long)

Example: Weak vs Strong Test

// Source code
function calculateDiscount(price: number, percentage: number): number {
  return price - (price * percentage / 100)
}

// WEAK: Test passes even if we mutate the calculation
test('applies discount', () => {
  const result = calculateDiscount(100, 10)
  expect(result).toBeDefined() // Too weak!
})

// STRONG: Test catches mutation
test('applies discount correctly', () => {
  expect(calculateDiscount(100, 10)).toBe(90)
  expect(calculateDiscount(100, 20)).toBe(80)
  expect(calculateDiscount(50, 10)).toBe(45)
})

Python (mutmut)

Installation

uv add --dev mutmut                    # Using uv
pip install mutmut                     # Using pip

Running mutmut

uv run mutmut run                                          # Run mutation testing
uv run mutmut run --paths-to-mutate=src/calculator.py      # Specific files
uv run mutmut results                                      # Show results
uv run mutmut summary                                      # Summary
uv run mutmut show 1                                       # Show specific mutant
uv run mutmut apply 1                                      # Apply mutant manually
uv run mutmut html                                         # HTML report

Understanding Results

Status: 45/50 mutants killed (90%)
- Killed: 45 (tests caught the mutation)
- Survived: 5 (tests passed despite mutation)

Hand-rolled harnesses report LESS than Stryker and mutmut do

Everything above assumes a framework. Plenty of real mutation testing is a hand-rolled loop instead — apply a mutation, run one assertion, catch the failure — typically because the thing under test is a build-time check in a generator or builder rather than a unit test suite.

That loop is worth writing. But it drops the one piece of bookkeeping the frameworks give you for free: Stryker and mutmut tell you which test killed each mutant. A hand-rolled harness usually reports only that something failed, and "something failed" is indistinguishable from "the check I am testing failed". Four ways that goes wrong — the first three observed in one session, the fourth in another:

1. An earlier check masks the one under test

run(mutate_frame_count, "check P: off-grid length")
  -> CAUGHT: "beat 'x' asks for 20 words in 5.42 s (3.69 words/s, ceiling 3.0)"

Reported as caught; the message is from check N, a words-per-second rule that fires before the grid check ever runs. Check P was never exercised. The mutation tripped a different assertion on the way past.

Always print and read the failure message, never just the pass/fail. If the message does not name the check you are testing, the mutation did not reach it.

2. The mutation has to be one ONLY the target check can see

Fixing the above is not "mutate harder" — it is choosing a mutation that no earlier check can intercept:

| Testing | Bad mutation | Works | |---|---|---| | an off-grid frame count | any beat (a talky one trips the words/sec check first) | a wordless beat | | a cast-shrink rule | a beat whose prose also names the removed character (trips the alias check) | a beat where only the count changes |

This is the same discipline as isolating a variable in an A/B: the mutation is the independent variable, and anything else it perturbs is a confound.

3. Mutating a table leaves import-time derived state stale

The subtlest one, and it caused two of the three maskings. Modules commonly build lookup dicts from a table at import:

SEGMENTS = (...)
_SEG_OF = {beat: name for name, beats, _ in SEGMENTS for beat in beats}

Monkeypatching SEGMENTS in the harness leaves _SEG_OF describing the old table, so the first check that consults it fails with a stale-lookup error — masking everything downstream:

mod.SEGMENTS = new_table
mod._SEG_OF = {b: n for n, ids, _ in mod.SEGMENTS for b in ids}   # REQUIRED

Rebuild every derived structure you can find, or reload the module. Grep for comprehensions over the table you mutated.

4. The mutated file was never imported

The mirror of the three above. Those are all false CAUGHT — a mutation reported killed by an assertion other than the intended one. This one is false MISSED: the harness edits a file the run never loads, and reports a coverage hole that does not exist.

A 25-row harness over a builder + loader pair staged six named files into a temp directory, wrote the mutated copy over one of them, and put the real source directory on PYTHONPATH so the remaining imports would resolve. First run: 25 mutations, 15 mismatches. Twelve of the fifteen were every row mutating one of the two files, each expect=CAUGHT got=MISSED 0 red. The natural reading — "those twelve assertions are vacuous, go strengthen the tests" — is wrong. They were running the pristine source.

The tell is the control row. A META reject-all mutation inserts a hard-wired err.add() at the top of the function under test, and it reported MISSED with 0 red. A suite that does not go red against a hard-wired failure is not a weak suite — it is proof the harness is not running the file it edited.

The mechanism was an ordinary, otherwise harmless idiom in a sibling module, staged from the real directory:

sys.path.insert(0, str(Path(__file__).resolve().parent))

__file__ there is the real directory, so importing that sibling re-inserts the real directory at sys.path[0], ahead of the temp directory. The builder imports the sibling before it imports the loader, so the loader — the mutated file — resolved to the unmutated copy for every later import. Printing resolved paths inside the run confirms it:

PATH0: ['/tmp/tmp.GcJ4RTwCg7', '/tmp/tmp.GcJ4RTwCg7', '/mnt/.../lab/scripts', ...]
B: /tmp/tmp.GcJ4RTwCg7/build_...py        <- staged copy, mutated rows worked
C-in-modules: /mnt/.../lab/scripts/dataputki_content.py   <- REAL file

The four rows mutating the other file worked correctly, because that file was staged and imported directly. That mix is what made the report look plausible rather than broken.

Stage the whole directory and pass no search path at all. With no second copy anywhere on the path there is nothing for an import to bind to:

shutil.copytree(SRC, td, dirs_exist_ok=True,
                ignore=shutil.ignore_patterns("__pycache__", "*.pyc", ".pytest_cache"))
env = {k: v for k, v in os.environ.items() if k != "PYTHONPATH"}

After that change: 25 mutations, 0 mismatches, every row caught by its intended test and the CONTROL correctly missed. A per-file copy list also encodes an import graph that nothing checks — it stops being correct the moment someone adds an import.

This is not Python-specific. Any runtime that resolves by search path has the same shape — a second copy of the unmutated code reachable ahead of the one you edited:

| Runtime | The second copy binds via | |---|---| | Python | PYTHONPATH, or a sys.path.insert inside any imported module | | Node | NODE_PATH, or node_modules resolution walking up from the real file | | Go | GOPATH | | Ruby | RUBYLIB | | Perl | PERL5LIB | | A binary under test | PATH — a stub shadowed by a real command of the same name |

The PATH row is issue #2451 in this repo: the bash-antipatterns probe for sg matched shadow-utils' sg instead of ast-grep.

The consequence for a green table

A harness that prints CAUGHT for every mutation is often quoted as proof the suite is sound. It proves something weaker:

An all-CAUGHT table proves each mutation was caught by some assertion. It never proves the assertion you meant was the one that caught it — nor that any individual assertion is capable of failing.

Two cheap additions close most of the gap:

• A deliberate no-op mutation the harness should miss. A table where everything is CAUGHT is indistinguishable from a broken harness; one expected MISS tells them apart. The symmetry holds and the control does not cover it: an all-MISSED table is equally indistinguishable from a broken harness, and a no-op reporting MISSED as designed looks identical beside real mutations reporting MISSED because nothing loaded them. Read the META/accept-all row first — a hard-wired raise or accept-all that fails to turn the suite red is not a weak assertion, it is proof the harness is not running the file it edited (§4).
• Assert on the message, not just the exception. Match the mutation to an expected substring of the failure, so a masked result is a harness failure rather than a silent pass.

Mutation Score Targets

| Score | Quality | Action | |-------|---------|--------| | 90%+ | Excellent | Maintain quality | | 80-89% | Good | Small improvements | | 70-79% | Acceptable | Focus on weak areas | | 60-69% | Needs work | Add missing tests | | < 60% | Poor | Major test improvements needed |

Agentic Optimizations

| Context | Command | |---------|---------| | Quick TS mutation | npx stryker run --incremental --reporters clear-text | | Targeted TS mutation | npx stryker run --mutate "src/core/**/*.ts" | | Quick Python mutation | uv run mutmut run --paths-to-mutate=src/core/ | | View survived | uv run mutmut results \| grep Survived | | CI mode | npx stryker run --reporters json |

For detailed examples, advanced patterns, and best practices, see REFERENCE.md.

See Also

• vitest-testing - Unit testing framework
• python-testing - Python pytest testing
• test-quality-analysis - Detecting test smells
• api-testing - HTTP API testing
• agent-patterns-plugin:tool-result-traps - Control-testing any negative that gates an action (§4's hard-wired raise is exactly that control)

References

• Stryker: https://stryker-mutator.io/
• mutmut: https://github.com/boxed/mutmut
• Mutation Testing Intro: https://en.wikipedia.org/wiki/Mutation_testing