Regression Testing Strategy: How to Stop Shipping Old Bugs

A bug that comes back after it was fixed is called a regression. It means the fix didn't stick — either the root cause wasn't fully addressed or a later change reintroduced it.

Regressions are uniquely demoralizing. You fixed it once. You're fixing it again. And users lose trust.

Here's how to prevent them.

Why Regressions Happen

The fix was incomplete. The symptom was fixed but not the root cause. The bug returns in a slightly different form.

The fix was specific, not general. A hardcoded value was corrected but the validation that should prevent bad values wasn't added.

Another change broke the fix. Someone refactored code near the fix without knowing it was load-bearing. The regression isn't intentional — it's accidental.

Test coverage didn't cover the scenario. The bug was fixed but no test was added to catch it if it regressed.

The last one is the most preventable.

The Golden Rule: Bug Fix + Test = Regression Prevention

Every time you fix a bug, write a test that would have caught it.

Not "write a test for the function you changed." Write a test that reproduces the exact bug you fixed, then verify the fix makes it pass.

// Bug: task with empty title was saved, causing crash when displayed
// Fix: added validation to reject empty titles
// Regression test:

@Test
fun `creating task with empty title returns validation error`() = runTest {
    val result = taskService.createTask(title = "")
    
    assertTrue(result.isFailure)
    assertEquals(
        "Title cannot be empty",
        (result.exceptionOrNull() as ValidationException).message
    )
}

This test is now permanent documentation of the bug. If the validation is ever accidentally removed, the test fails immediately.

Building a Regression Test Suite

A dedicated regression test suite contains:

1. Tests for every reported production bug (with the issue/ticket ID as a comment)
2. Tests for edge cases discovered in QA before they became bugs
3. Tests for scenarios that historically recur across releases

// GH-1234: Empty task title caused NullPointerException in TaskAdapter
@Test
fun `gh1234 empty title handled gracefully`() { ... }

// GH-1456: Completed tasks reappeared after app restart
@Test
fun `gh1456 completed tasks persist across sessions`() { ... }

The ticket reference serves as a history — when this test fails, you know exactly which scenario you're dealing with.

What to Include in Manual Regression Passes

Not every regression can be caught by automated tests. A manual regression pass should cover:

High-risk areas for this release:

• Any code that changed in this release
• Any code that depends on code that changed
• Core flows adjacent to the changed areas

Historical problem areas:

• Any feature that has regressed more than once
• Any feature that was recently refactored
• Integration points with third-party services

The top 10 user flows:

• Login/logout
• Primary feature flow (create, read, update, delete)
• Payment/purchase flow (if applicable)
• Notification handling
• Settings changes that affect behavior

[!TIP] Keep a "watch list" of features that have regressed multiple times. Before each release, always test these manually, regardless of code changes nearby.

Regression Triage: Severity vs Coverage

You can't regression-test everything before every release. Prioritize by risk:

Focus your manual time on P1 and P2. Let automation cover P3 and P4.

Regression Testing Cadence

Before every PR merge: Automated unit + integration tests run. Any test failure blocks the merge.

Before every release (QA pass): Manual regression of top flows + automated test suite.

After every hotfix: Targeted regression of the fixed area only. A hotfix that fixes one thing shouldn't require a full regression pass.

Scheduled regression (monthly or quarterly): Full manual regression of the entire app. Catches things that automated tests and targeted passes miss.

When Regressions Still Slip Through

Even with good regression coverage, some bugs return. When they do:

1. Write the test first — reproduce the bug in a failing test
2. Fix the bug — make the test pass
3. Add the test permanently — to the regression suite
4. Review why the regression happened — was there a test gap? A process gap?

The post-mortem isn't about blame. It's about understanding whether your regression testing strategy needs adjustment.

Common Mistakes

Only automating the happy path. Most regressions happen in edge cases and error paths. Those need tests too.

Not writing regression tests for every bug fix. "I'll add it later" means never. Write the test immediately when the bug is fresh.

Marking a regression pass as "done" without running it. Checkbox QA (marking tests passed without executing them) is worse than no QA — it creates false confidence.

Ignoring automated test failures as "known flaky tests." If a test is flaky, fix or remove it. A team that ignores test failures will ignore real regressions.

Takeaways

• Every bug fix should include a regression test — this is non-negotiable
• Reference ticket IDs in regression tests for traceability
• Manual regression focuses on high-risk areas for this specific release
• Maintain a "watch list" of historically problematic features
• When a regression slips through, the immediate action is writing the test, then fixing the bug