Test Coverage vs Test Quality: Why 80% Coverage Can Mean Zero Protection

Coverage is a metric. Metrics can be gamed. And nowhere in software testing is gaming more common than with code coverage.

A test can execute every line of your code and assert nothing. It'll show 100% coverage. It'll catch zero bugs.

What Code Coverage Actually Measures

Code coverage measures which lines were executed during tests — not whether those lines were verified to behave correctly.

fun calculateDiscount(price: Double, percentage: Int): Double {
    return price * (percentage / 100.0)
}

// This test achieves 100% coverage
@Test
fun `calculateDiscount runs without error`() {
    calculateDiscount(100.0, 20) // No assertion!
}

The function is "covered." It could return

0.0

The Coverage Metric Game

When teams set coverage thresholds ("we need 80% before merge"), engineers respond rationally to the incentive:

1. Write tests that execute code paths without asserting on results
2. Test trivial code (getters, simple constructors) to boost the number
3. Skip asserting on complex edge cases because "the line is already covered"

This produces a metric that looks good and protects nothing.

What Actually Matters: Mutation Testing

Mutation testing inserts small bugs into your code and checks if your tests catch them. If they don't, your test coverage is hollow.

A mutation tester might change:

• code Copy

  >

  to
  code Copy

  >=

• code Copy

  +

  to
  code Copy

  -

• code Copy

  true

  to
  code Copy

  false

• Return
  code Copy

  null

  instead of a real value

If your tests pass after these mutations, those tests aren't asserting on the right things.

// Original
fun isEligibleForDiscount(age: Int): Boolean = age >= 65

// Mutation: >= changed to >
fun isEligibleForDiscount(age: Int): Boolean = age > 65

// Test that would catch this:
@Test
fun `65-year-old is eligible for discount`() {
    assertTrue(isEligibleForDiscount(65))
}

// Test that wouldn't catch this:
@Test
fun `80-year-old is eligible for discount`() {
    assertTrue(isEligibleForDiscount(80)) // Passes even with the mutation
}

Tools like Pitest for Java/Kotlin automate this analysis.

Coverage Types — What Each Actually Tells You

Line coverage is the default metric. It's also the easiest to game and the least meaningful.

Branch coverage is better — it requires that both the

true

false

Mutation score is the most meaningful but requires dedicated tooling.

The Right Coverage Goals

Coverage thresholds make sense for some code, not all:

Good candidates for high coverage requirements:

• Financial calculations (pricing, billing, tax)
• Authentication and authorization logic
• Data validation rules
• Core business rules

Poor candidates:

• UI rendering code (use visual testing instead)
• Configuration loading
• Simple getters/setters
• Generated code

[!NOTE] A 70% coverage score on your business logic, with meaningful assertions, protects you more than 90% coverage on the whole codebase including getters and config parsing.

Writing Tests That Actually Protect You

The question to ask before every assertion: What bug would this catch?

If you can't answer that, the assertion is likely weak.

// Weak assertion — checks that *something* returns
val result = orderService.calculateTotal(items)
assertNotNull(result) // What bug does this catch? Almost nothing.

// Strong assertion — checks the actual business rule
val items = listOf(
    Item(price = 10.0, quantity = 2),
    Item(price = 5.0, quantity = 3)
)
val result = orderService.calculateTotal(items)
assertEquals(35.0, result, 0.001) // Catches wrong price, wrong quantity, wrong formula

How to Evaluate Test Suite Quality

Beyond coverage numbers:

1. Read your tests. Can you understand what behavior they document? Good tests are readable specifications.

2. Kill mutations. Run Pitest or a similar tool. If your mutation score is below 50%, your tests are largely decorative.

3. Review assertion quality. Check the last 20 tests added to the repo. Count tests with no assertions or trivial assertions.

4. Check edge cases. For each function, are the boundary conditions tested? Empty collections? Zero values? Maximum values? Null inputs?

5. Check negative cases. Are error paths tested? What happens with invalid input? What happens when a dependency fails?

The Better Metric: Defect Escape Rate

The real measure of test suite quality is defects that reached production that should have been caught by tests. This is your escape rate.

If you have 90% coverage and bugs still escape to production regularly, your tests aren't protecting you. If you have 60% coverage and your release is consistently clean, your tests are well-targeted.

Track bugs that made it to production. For each one, ask: should a test have caught this? If yes: write that test. This is more valuable than chasing a coverage number.

Takeaways

• Coverage measures execution, not verification — it's easy to fake
• A test with no assertions is worse than no test — it creates false confidence
• Mutation testing reveals whether your assertions actually protect your code
• Focus high coverage requirements on business logic, not the whole codebase
• Track defect escape rate — that's the real measure of test suite effectiveness