In my DevOps consulting experience, with enterprise and manufacturing organizations, I have seen many value streams in which test activities such as finding and setting up test environments, test configurations, test execution and test results interpretation are primarily manual tasks, and are a primary source of bottlenecks. Success with DevOps requires a mature approach to continuous automated testing.
“Continuous Test Engineering is a quality assessment strategy in which most tests are automated and integrated as a core and essential part of DevOps. Continuous testing is more than simply “automating tests.” Book Reference “Engineering DevOps” by Marc Hornbeek
Why Is Continuous Test Automation Maturity Important to DevOps?
Test acceleration provided by automation of test activities is critical to assure adequate quality coverage of products and services while reducing bottlenecks needed to meet the increasing pressure for fast releases. More mature continuous test automation strategies are better able to meet the competing needs of quality and speed.
How Is Continuous Automated Testing Engineered for DevOps?
The figure below—Continuous Test Engineering Blueprint shows how mature continuous test automation enables, as many relevant tests as possible to be executed as early as possible in the pipeline.
Tests are performed “continuously and automatically” on incremental product changes using an automatically orchestrated, production equivalent test environment (Physical and/or Virtual). Tests are selected automatically at each stage and triggered to run automatically by the toolchain prior to exiting each stage of the pipeline. Results of tests, which are critical to deciding promotions of changes to the next stage in the pipeline, are automatically reported and analyzed. The aggregate set of test results accumulated over the pipeline are analyzed automatically to decide whether a change is acceptable for release to production. The following are example activities in each stage of the pipeline that support mature continuous test automation:
- Plan Stage Continuous Test Automation:
- Test plans tasks are in backlog for each user story.
- Test resources are identified for each test task (physical and virtual).
- Dev Stage Continuous Test Automation:
- Unit and static analysis test verdicts for each coding task.
- Functional, not functional, and regression test verdicts for each user story.
- Integration Stage Continuous Test Automation:
- Dev test results are verified for each code commit.
- 100% of Integration tests are automated, automated regression tests are integrated together with the code changes.
- Pre-Prod Stage Continuous Test Automation:Â 85% of System, User Acceptance verdict analysis is automated, 100% of deployment tests are automated.
- Post-Prod Stage Continuous Testing Automation:
- A/B application comparison test statistics analysis are automated.
- Test escapes—production failures for which there is no defined test case are automatically reported and analyzed.
Many organizations have not achieved the level of continuous test automation maturity described above. For an organization to mature its continuous test automation capabilities they need to know their current state of maturity and target future state. There are three dimensions to consider at each level: People, Process and Technology. Technology such as better test tools will not accomplish a higher level of maturity unless the People and Processes use the tools effectively.
Continuous Test Automation Maturity Levels
There is no standard continuous test automation maturity model. As a practical matter, I have found it useful to apply a five-level maturity model adapted from the “standard” software Capability Maturity Model. I define five levels of continuous test automation maturity as described in the following paragraphs.
Continuous Test Automation Maturity Level 1: Chaos
The figure below shows key characteristics of People, Process and Technology evident at this level of maturity. At this level there is little evidence of test automation skills, test automation processes are not apparent and test automation technology has major gaps.
Continuous Test Automation Maturity Level 2: Continuous Integration
The figure below shows key characteristics of People, Process and Technology evident at this level of maturity. At this level there is some knowledge of automated testing. The integration and build processes are well supported by processes and technology for automated testing. End-to-end testing, especially delivery ad deployment stages are not well supported with automated testing.
Continuous Test Automation Maturity Level 3: Continuous Flow
The figure below shows key characteristics of People, Process and Technology evident at this level of maturity. At this level test automation is used from end-to-end across the pipeline. Dev and QA teams cooperate to ensure a good level of test coverage is automated. Release standards are using automated testing metrics.
Continuous Test Automation Maturity Level 4: Continuous Feedback
The figure below shows key characteristics of People, Process and Technology evident at this level of maturity. At this level more advanced knowledge of continuous test automation is apparent.  Goals and metrics are set for test automation. The culture includes training and mentoring for test automation. There is a focus on end-to-end performance trends rather than spot results. Automation is applied to test environment orchestration and analytics.
Continuous Test Automation Maturity Level 5: Continuous Improvement
The figure below shows key characteristics of People, Process and Technology evident at this level of maturity. At this level there is a high level or knowledge and confidence regarding continuous test automation. Dev and QA teams are tightly integrated to optimize knowledge and efficiency. End-to-end tests focus on the end customer experience and more sophisticated risk-based strategies. Test creation strategies leverage automation.
Continuous Test Automation Maturity Assessment
While the above five levels of continuous test automation maturity provide a practical guide for defining maturity against characteristics of People, Process and Technology, they are not an absolute measure of maturity. Organizations, or specific applications within an organization, may match some of the characteristics for different levels. The figure below of Continuous Test Automation Maturity Model is a useful tool to determine the “best fit” for the maturity of an organization or application within an organization. By marketing the characteristics that best match, gives a visual picture of the dominant level of maturity. This also is a quick way to determine areas to address to improve the level of maturity.
Summary
In this article, I presented a blueprint for mature continuous test automation. I explained why continuous test automation is important to improve both agility and quality at the same time, as expected with DevOps transformations. Five levels of continuous test automation maturity were described. It was explained how to use the continuous test automation maturity model to assess the maturity of an organization and to identify improvements that will improve maturity. For more information readers are encouraged to review the material in my book “Engineering DevOps” and on my website www.engineeringdevops.com.