Common Continuous Testing Pitfalls In DevOps And How To Avoid Them
Continuous testing as part of a CI/CD (Continuous Integration/Continuous Development) pipeline is a pillar of all Agile approaches to software development but especially DevOps. Agile development is cornered on short, regular iteration cycles that build on and improve software systems, often those already in production, based on user experience data and feedback.
The tight feedback and development loops of the Agile approach have been demonstrated to improve software quality, in terms of both its technical performance and better serving user needs. But regularly iterating on software in production also injects a heightened risk of downtime.
Every time software is updated by a new iteration being pushed into production, there is a danger of errors and bugs being introduced and it not running as anticipated as a result. The larger the scale of a new iteration, the more opportunity there is for bugs and errors being missed in testing.
Agile’s smaller, regular iterations help reduce the chance of defects and make fixing any that are introduced far easier. The DevOps extension of Agile software development introduces automated testing into CI/CD pipelines, further reducing that risk and making testing a more efficient process.
Automated testing adds to and takes much of the pressure off of manual testing, which is time-consuming and expensive as well as intrinsically unreliable because humans aren’t good at repetitive tasks. It is also limited in the extent to which it can predict the impact of changes to complex software systems.
Automated testing doesn’t completely replace the need for manual testing, which remains a necessary component of any software delivery cycle testing phase. But it significantly reduces how much needs to be done and compliments its weaknesses.
Google Cloud recommends that software development teams “continually run both automated and manual tests throughout the delivery process to validate the functionality and architecture of the system under development”.>
But to realise the full value of continuous testing, both manual and automated, development teams have to optimally execute it. Doing so involves avoiding common pitfalls.
Common pitfalls to avoid when introducing continuous testing to an Agile DevOps software development cycle
Google Cloud also pinpoints what it sees as several common points of failure in continuous testing as part of a DevOps CI/CD pipeline.
1. Separating Development & Testing Functions
A well-evolved DevOps approach introduces collective accountability and cross-discipline software development teams that unify development, testing, and operations functions.
Achieving that can be compromised if developers do not continuously test their own code as they are writing it and simply pass it on to dedicated testers, along with responsibility for its in-production quality.
Involving developers in testing by making them responsible for the creation and maintenance of automated testing pipelines makes it easier for them to fix failures. Taking testing responsibility away from developers means they are more likely to write code that is harder to test.
That, in turn, often leads to code designed to an inferior level and high-maintenance, expensive test suites. And tests that need to be fixed by a separate testing team if updates are required to handle changes to the codebase.
2. Not Pairing Testers & QA Experts With Developers
Dedicated testers and QA teams still have a crucial role especially larger DevOps teams because they contribute a different perspective on how the software works – one that has a deeper understanding of how users interact with it.
Pairing developers with testers allows them to learn from each other, something else that is key to a well-evolved DevOps culture, and discover and deal with any problems in real-time. Specialist testers also work on UI/UX testing and help developers curate test suites.
3. Failure To Curate Test Suites
Failure to curate test suites, continuously reviewing and improving them for greater effectiveness, is another pitfall development teams can often trip upon. Acceptance test suites should be designed to not only automate collections of acceptance criteria but mimic real user journeys through a software system.
To remain optimally effective, these scenarios must evolve alongside new interactions of the software system.
Unit tests will inevitably have to be updated as the codebase evolves but if several have to be changed with every code update, there is probably an over-reliance on mocking.
If test suites are difficult and expensive to maintain, this suggests potential problems with the underlying software architecture. Incorporating refactoring into daily work is an investment but should ultimately lead to software that is easier to test, saving significant resources across the lifetime of a software system.
4. Wrong Ratio Between Unit & Acceptance Tests
An automated test suite’s goal is to highlight flaws and errors as early in the process as possible. That’s the reasoning behind the flow of different categories of testing from fast-running unit tests to slower-running acceptance tests, followed by manual testing.
The aim should be that the unit tests find errors. If they are not discovered before acceptance or even manual exploratory testing, a new unit test should be added to catch such errors earlier in the testing cycle next time. The sooner they are found, the easier and cheaper they are to fix.
5. Toleration Of Unreliable Tests
DevOps teams should show no tolerance for unreliable tests. Software that passes through the testing pyramid should always perform to a high quality on release. Presuming an error making it through the testing pyramid is a one-off, rather than improving the test suite to make sure it never happens again is one of the most commonly encountered flaws of poorly evolved, and so poorly performing, DevOps teams.
Don’t Just Pay Lip Service To Continuous Testing
The reality is a majority of DevOps teams are not highly evolved DevOps teams. In contemporary software development, it is almost obligatory for a software development team to nail their flag to the mast of a particular, usually Agile, methodology.
However, there is a huge difference between the long-term performance metrics of teams that do so on a superficial level, paying lip service to a methodology, and genuinely integrating it at every level of the development process and within the team’s culture.
Strictly insisting on and optimizing continuous testing is integral to any highly evolved DevOps team or organisational approach. Avoiding the common continuous testing pitfalls highlighted here will see a team taking significant strides toward a more evolved, and ultimately more efficient, DevOps culture. And that should, ultimately, lead to software systems that are both higher quality and cheaper to build and maintain.
