Maintenance is the Key to Success with Test Suites

End-to-End (E2E) software testing suites can fail for various reasons. But there are two primary reasons a test suite fails its test. Either:

  • The test suite becomes unmanageable and unstable, prone to false positives
  • The test suite cannot be or was not updated in conjunction with the rest of the system due to redistribution of resources elsewhere
person using a test suite on a desktop computer
Photo by Charles Deluvio

These two reasons for test failure encompass other related reasons for the test failure of E2E software. But what do they mean for your test suite?

Test Suite is Unwieldy and Its Feedback Disregarded

When the information generated by test suites gets disregarded, it typically points towards a test failure resulting from multiple false positives.

Concurrently, the tests fail at each attempt to run them, and the test suite’s runtime is so sufficiently excessive that investigation into all failures is no longer tenable.

This happens because of the pace at which software development moves. As workers strive to meet software release dates, there is an increasing sense that the test suite is optional, especially when it isn’t operating at full functionality.

And since the software world is frequently rushing towards deadlines, test suites get left behind, generating still more false positives and longer run times than ever.

The Test Suite Isn’t Updated Because Resources Aren’t Immediately Available

Another reason that a test suite in software testing might fail is that the test suite is outdated. Typically E2E testing simulates what a user will be doing with the software. But if the test suite can’t be regularly updated, then the test fails because the pathways developed to take the user through the system have changed too much for the test suite to reflect them at the tester.

That means that as the test suite continues in use for E2E testing, the number of bugs and errors users encounter increases. Therefore the software can no longer accurately depict the user experience. If it continues in use, the number of bugs in features escalates, not because the application is faulty, but because the test suite interacts with the application in a way that is outmoded or irrelevant.

two women standing beside white board discussing test suites
Photo by Christina @ wocintechchat

And while test failure and test suite failure aren’t uncommon, they don’t have to be normal. A well-maintained test suite that receives regular updates and its share of resources can save developers time and money in the long run.

But what do you need to do for that to happen?

Create a Well-Designed Test Suite

Nobody has an endless budget, and E2E testing is expensive. That’s why in trying to maintain your test suite and avoid test failure, the place to start is at the beginning.

A well-constructed test suite prioritizes your tests. By narrowing the focus, it enables the builder to:

  • Reduce manual work building the test suite system
  • Specify the parameters of the test suite in a manageable way

It also eliminates the need to run unnecessary E2E tests, a problem almost all test suite engineers fall prey to. Applications can run hundreds, or even thousands of E2E tests, despite the application not being nearly big enough to justify the testing output. It’s no wonder they become unwieldy and fail tests.

But with guidance, the designers behind test suites can prioritize which tests to run. Not everything needs to be an E2E test and helping production understand this and distinguish what should be E2E and what shouldn’t go a long way to maintaining a test suite.

two men using a test suite to validate their code
Photo by Annie Spratt

Instead, testers can replace E2E, where applicable, with low-level and integrational tests. By using low-level tests where it makes sense, you not only avoid test failure and false positives but are left with a test suite that is easier and faster to update than one that runs everything as E2E tests.

Avoid Complex Scenarios

Another key to avoiding failed tests is to simplify what scripts your test suite runs. While it’s entirely possible to run complicated scenarios, it isn’t always necessary. Distinguishing when elaborate tests are called for and when they aren’t can help reduce the failed tests you encounter.

Don’t String Tests Together

Another common culprit of the “fail.test” is developers’ propensity to run multiple tests together instead of keeping them separate. A good automated test requires specificity, which means avoiding the temptation to blend tests into longer-running, more complicated scenarios.

Instead, keeping tests independent and running their data can reduce the chances of the test suite generating a “fail.test” and has the added advantage of ensuring the system is manageable and updatable.

Reframe Maintenance as Multiple Processes

Finally, the best way to maintain your test suite and avoid errors like “fail.test” is to reconsider how you think of maintenance. Instead of one ongoing process, think of it as three interwoven ones. These processes are:

  • Build-to-build
  • Stability
  • Maintenance over time

Build-to-Build

Build-to-Build maintenance allows you to change the user interface or UI as the opportunity arises. This matters because when the UI changes, it has ramifications for the system and the tests running. Any tests running when the UI changes are doomed to failure because they can’t navigate the altered interface.

When done well, build-to-build maintenance anticipates those changes and allows new, altered tests to start running after the UI changes. You’ll see fewer failed tests, and it’s one less thing to keep on top of across the test suite.

Stability

The best way to guarantee stability in a testing environment is to run those tests repeatedly. Instead of waiting for tests to fail before retesting, have the test repeatedly run against the application.

Using this method allows you to catch failed tests and prioritize the instability they generate. Instead of updating the test suite system, you can focus on the part of the application or test suite generating the failed test. Leaving this until there are multiple failures to rectify contributes to the unmanageability of test suite maintenance.

Maintenance Over Time

What maintenance over time aims to do is stop the test suite from becoming overburdened. There will always be new tests to run when you add new UI features. But it helps to remember that, like people, test suites can’t do everything at once.

And since no one will use all application features at once, there’s no reason to test everything simultaneously. Trying to do so only leads to failed tests. Instead, as you add new features, reprioritize what gets tested.

Prioritizing in this way reduces the need for your test suite to expand endlessly and saves it from becoming unwieldy, sluggish, and hard to update.

Conclusion

Maintaining a test suite to prevent test failure takes time, effort, and careful prioritization. But good communication with the development team and a clear understanding of objectives can help everyone reassess what to test.

By adhering to the practices outlined here, maintaining your test suite becomes simpler and failed tests fewer, resulting in a more successful application and manageable test suite.


Author

Erik Fogg

Bio

Erik is the MIT-educated COO and Co-Founder of ProdPerfect. He loves unleashing the potential of the great folks in the world and loves helping make decisions with facts. A couple of years ago he co-founded ProdPerfect, where he helps his team grow personally and improve their ability to help people solve QA problems. He also helps customers use actual live data–instead of educated guesses–when deciding what tests to write and maintain. In his spare time, he podcasts and writes books about making fact-based decisions in business and politics.

Leave a Reply