Should You Automate This Test? A Decision Framework

"Should we automate this?" gets asked dozens of times a week on active QA teams. Without a framework, it's answered by gut feel, which means it's answered inconsistently. Some things get automated that shouldn't. Some things stay manual that should have been automated years ago.

Here's a concrete framework that makes the decision explicit and consistent.

---

The Five Questions

Ask these in order. The first "no" stops the analysis — manual testing is the answer.

Question 1: Is the test objective and deterministic?

A test is objective if it has a clear pass/fail condition that doesn't require human judgment.

Objective: "The cart total displays $29.99 after adding the test item" Requires judgment: "The checkout experience feels smooth and professional"

If the answer requires judgment → Manual.

Question 2: Will this test run frequently?

Automation ROI comes from repeated execution. A test that runs once a week has much better ROI than one that runs once a month.

High frequency: Every commit, every PR, nightly — automate Low frequency: Monthly, quarterly, one-time — likely manual is more cost-effective

If the test runs fewer than 10 times per year → Manual (unless it's very fast to automate).

Question 3: Is the feature stable?

Automating a feature that changes every sprint means constant test maintenance. You spend more time maintaining tests than the tests save.

Stable: Design locked, API contract finalized, no changes planned Unstable: Active development, spec still evolving, frequent design changes

If the feature is actively changing → Manual for now, automate when stable.

Question 4: Can you write a reliable, maintainable test?

Some tests can be automated in theory but are painful in practice — dynamic content that changes legitimately, flows that require real-world credentials, tests that depend on timing-sensitive external services.

If reliability requires significant hacks → Manual.

Question 5: Does the value justify the cost?

Estimate: how long to build the test? How long to maintain it annually? How much time does it save per year?

If maintenance cost > time saved → Manual.

---

The Decision Matrix

Objective?	Frequent?	Stable?	Reliable?	Value > Cost?	Decision
✅	✅	✅	✅	✅	Automate
✅	✅	✅	❌	—	Manual
✅	✅	❌	—	—	Manual (for now)
✅	❌	—	—	—	Manual
❌	—	—	—	—	Manual

---

Common Test Types and Their Usual Answer

Test Type	Usually
API contract tests	Automate
Unit tests	Automate
Critical path regression	Automate
Visual design review	Manual
Accessibility experience	Manual
New feature exploration	Manual (first sprint)
Stable feature regression	Automate
Performance benchmarks	Automate
UX evaluation	Manual
Security penetration	Mixed
Multi-device smoke (stable feature)	Automate
Ad-hoc investigation	Manual

---

Applying the Framework

The framework doesn't make the decision — you do. It makes the reasoning explicit, so when someone asks "why is this still manual?" or "why did we automate this?" there's a concrete answer.

[!TIP] Apply this framework to your existing test suite as well as new tests. Many manual tests in mature products should be automated by now. Many automated tests in active products should be manual until stability is reached.

The framework also changes over time. A test that was manual because the feature was unstable should be revisited for automation once the feature stabilizes. Build a habit of reviewing the manual/automated split quarterly.