Stop Automating Everything: How to Spot a Test Worth Automating | Automation Foundations #2

Most automation suites don’t fail because of the tool. They fail because the team automated the wrong tests.

You have probably seen this. A team gets a new framework, sets a target such as “automate 80% of regression by Q3,” and starts converting manual test cases into scripts one by one. Six months later, the suite has 1,200 tests, takes four hours to run, fails randomly every night, and nobody trusts the results. The team spends more time fixing scripts than finding bugs.

The framework was fine. The problem was the selection.

Choosing what to automate is an engineering decision, not a clerical one. Here’s how to make it well.

Start With the Right Question

The usual question is:

“Can this test be automated?”

That question is almost useless. With enough effort, nearly everything can be automated. A better question is:

“Will automating this test give us more value than it costs to build and maintain?”

Every automated test is a small piece of software. It needs to be written, reviewed, debugged, updated when the UI changes, and investigated when it fails. If a test runs once a quarter and breaks every sprint, it costs more than it returns.

A good automation candidate is a test whose long-term value is clearly greater than its long-term cost.

The Seven Traits of a Good Candidate

1. It runs often

Automation pays off through repetition. A test executed in every build, every pull request, or every regression cycle returns its investment quickly. A test executed twice a year rarely does.

Ask: How many times will this test run in the next six months?

2. The expected result is deterministic

A good automated test produces a clear pass or fail. “The order total should be ₹1,180 after 18% GST” is easy to assert. “The page should feel responsive” or “the layout should look professional” is not.

If a human must interpret the result, it’s not yet a good automation candidate.

3. The feature is stable

Automating a screen that is redesigned every sprint means rewriting the test every sprint. Wait until the behaviour and the interface settle, especially for UI automation. (We’ll go deeper into this in Article 4.)

4. It protects something important

A broken login, payment, or checkout flow hurts the business far more than a misaligned footer. Tests protecting revenue, compliance, security, and core user journeys deserve priority, even when they are harder to automate.

5. It’s tedious or error-prone for humans

Some tests are boring precisely because they are valuable to repeat:

• Validating 40 combinations of user role and permission

• Checking 200 rows of calculation logic

• Submitting the same form with dozens of boundary values

• Comparing API responses against a schema

Humans get careless by the twentieth repetition. Scripts don’t.

6. The test data and environment are controllable

If a test depends on a third-party sandbox that goes down twice a week, a shared database that other teams modify, or data that someone creates manually, the test will be flaky regardless of how well it’s written.

Ask: Can the test create, control, and clean up its own data?

7. It can be tested at a lower layer

This is the trait most teams ignore. Many “UI tests” are really business-logic tests that happen to go through the browser.

Consider a discount rule: orders above ₹5,000 get 10% off. You can verify it with:

The best candidate is often not “automate this UI test” but “automate this check at the API or unit layer, and keep one UI test for the end-to-end flow.”

Signs of a Poor Candidate

Be cautious when you see these:

• One-time checks such as data migration verification that will never run again

• Exploratory and usability testing, where human judgement is the point

• Features under active redesign

• Visual judgement without a proper visual-comparison tool

• Tests requiring physical interaction, such as scanning a real card or confirming an SMS on a physical device

• Tests with no reliable oracle, where nobody can state the expected result precisely

• Extremely complex setup for a low-risk scenario

“Poor candidate” doesn’t mean never automate. It means not first.

A Quick Example

Suppose you’re testing a food-delivery app and have these five tests:

Notice that the coupon rules go to the API level, the redesign waits, and the usability question stays with a human. That’s selection at work.

The Five-Question Filter

Before automating any test, run it through these five questions:

1. Frequency: Will it run often enough to justify the build cost?

2. Risk: What breaks for the business if this fails in production?

3. Stability: Will the feature stay unchanged long enough to repay the effort?

4. Determinism: Can I write a clear, unambiguous assertion?

5. Layer: What is the lowest layer where this can be tested reliably?

If the answers are mostly positive, automate it. If two or more are negative, park it and revisit later.

The Mindset Shift

Junior automation engineers measure success by how many tests they automated.

Senior engineers measure success by how much risk the suite covers, how quickly it gives feedback, and how much the team trusts it.

A suite of 150 carefully selected, stable tests that run in 12 minutes is worth more than 1,500 brittle tests that take four hours and fail randomly.

Automation is not about replacing manual testing. It’s about giving the repetitive, predictable, high-value checks to machines, so humans can spend their time on the testing that truly needs human thinking.

Interview Questions to Prepare

1. How do you decide which test cases to automate first?

2. What makes a test case a poor automation candidate?

3. Should you automate every regression test? Why or why not?

4. How would you handle a request to achieve 100% automation coverage?

5. When would you automate a check at the API layer instead of the UI layer?

6. How do you measure whether your automation suite is delivering value?