This article was written by Enrique De Coss  and published on the TestProject Blog

User acceptance testing is something that some Agile teams perform. However, only a few of them are taking advantage of tools to automate it. The general trend for most Agile teams is to rely on functional testing of new features before automating any flow.

Based on what I’ve seen in different Agile teams, working with code-based solutions is what turns UAT into a headache for most of them. Applications can be 100% functional from a technical standpoint and still fail because requirements are not clearly defined or effectively communicated to developers

In other cases, new code that appeared efficient in every virtual deployment model may have been inadequately tested for specific customer scenarios.

A User Acceptance Testing Brief

User acceptance testing can be an expensive exercise for some companies, but the cost of not having one can be even higher. For example, Microsoft rushed to release a flawed games console to stay ahead of the competition. It released the Xbox 360 games console in November 2005, just ahead of Nintendo and PlayStation.

However, it quickly became evident that the Xbox was subject to several technical troubles and failures- a series of glowing red lights that flashed on the front of the console, known as the ‘Red Ring of Death’.

Until July of 2007, Microsoft published an open letter recognizing the console’s problems and revealing a three-year warranty for every Xbox 360 console that experienced a general hardware failure. The design problems were alleged to result from management decisions and inadequate testing resources before the console’s release.

Source: reddit image Red Ring of Death

The objective of UAT is to ensure the application does what it’s set out to do, and that it meets the acceptance criteria the business has for it. Here you can find the definition of the term UAT taken from the ISTQB: 

“Formal testing concerning user needs, requirements, and business processes, conducted to determine whether or not a system satisfies the acceptance criteria and enables the user, customers or other authorized entity to determine whether or not to accept the system.”

Best Practices for Writing User Acceptance Testing

Acceptance criteria defines what must be done to complete a user story. They specify the limits of the story and are used to confirm when it is working as intended As QA engineers, we must understand the acceptance criteria and work with the producto owners to deliver a valuable product.

Let’s have a bit of context related to acceptance criteria and user stories. In 2001, Ron Jeffries wrote about the three C’s of the user story:

1. Card– Stories are traditionally written on notecards, and these cards can be annotated with extra details.
2. Conversation– Details behind the story come out through discussions with the product owner.
3. Confirmation– Acceptance tests confirm the story is finished and working as intended.

While we are testing, we should be clear about the difference between the acceptance criteria and the definition of done. The main difference is that the definition of done applies to all the work during the sprint, and the acceptance criteria is specific to individual stories.

Keep in mind that QA is a moving target. Sometimes, the acceptance criteria is the minimum applicable to complete a user story. But we should pay special attention to customer scenarios to provide high-quality products. Benefits of good acceptance criteria are:

1. Understanding how a feature or a piece of functionality will work from the customer’s perspective (special focus on customer scenarios).
2. Remove ambiguity from requirements.
3. Identifying tests that will confirm that a feature or part of the functionality is working and complete.

Automating User Acceptance Testing

Doing functional checks for user acceptance testing sounds like a big challenge during the time constraint in the sprints, with potential long-term consequences. However, the proper approach to UAT doesn’t come down to choosing between functional validation or automated user acceptance tests. Automation is possible in ~97% of cases, but the question is how to combine automation and functional validations for your projects

Knowing that coded tests are unproductive in UAT (The coded approach can work for unit testing and API testing), it might make more sense to go with a no-code/low-code testing tool. Using these types of tools to automate user acceptance testing can help QA operations be more productive.

We have seen teams where a part of the QA tasks falls on the shoulders of the product owners and certain product managers. So it makes a lot of sense with UAT, where the tester focuses on customer-centric scenarios.

In addition, with no-code/low-code automation of user acceptance tests, non-technical people gain more control over creating, running, and managing the test suites. In teams following this approach, testers run user acceptance tests. Still, the people outside of QA can also execute them, as quality is everyone’s responsibility!

AUAT and No-code/Low-code Testing Tools 

Keeping user acceptance tests in sync with user stories is challenging in Agile projects, where changes happen often. Due to its heavy focus on end-user testing, UAT also overlaps with UI testing, where things tend to be less change-proof.

Needing to work your way around brittle selectors constantly is one major factor that slows you down when testing the ever-changing UI. One of the vital advantages of no-code/low-code platforms is the speed you can create and edit tests ‍♀️

The speed boost you get from not having to code your tests makes it possible to cover customer scenarios quickly. In addition, some no-code/low-code tools address the problem of test flakiness with AI solutions. These make the maintenance of your tests become a lot easier.

Final Thoughts

As we discussed above, in cases where there is no way to identify clear acceptance criteria, we should protect ourselves and identify some minimum parameters to evaluate correctly-

1. Stability of the application– We must ensure a stable application and have confidence our application provides business value to the customers.
2. Usability of the application– We must provide an excellent guide to the application’s usability as perceived during our UAT.
3. Coverage of the Testing– Considering the testing we have done through the UAT and the other testing phases to measure range is enough, low, or acceptable. If we can’t measure it, we can’t improve it.

My recommendation is to move from a traditional UAT approach to an automation one. It’s easy to cover only the happy paths of the user stories, but we need to provide value to our customers by covering more scenarios.

Happy Bug Hunting!