Skip to main content

Automation, particularly UI Automation, is fun, and useful. However, automated UI test performance can be slow. They have to spin up a browser session, load front-end components, and so on. It doesn’t take much for a UI test suite to start taking tens of minutes to run.

I like Charity Majors’ recommendation of a 15-minute deploy time (15 minutes or bust!). Fast deploys are a key factor in building a quality SAAS product. But, this is impossible to achieve if you have an automated test suite that’s taking twenty or thirty minutes or even hours ⌚️ So, how can we keep the time of our automated UI tests down? What are some simple ways of keeping your automated tests speedy? 

Test the important things, not everything

It can be really tempting to want to write UI tests that cover every possible feature and flow. Every nook and cranny in the product. But, some of those tests might not be useful, in context. Does everything really need to be tested? 

I have a rule of thumb I like to use when adding an automated test. It’s this: if the feature I’m testing stopped working, what would happen? Would we wake someone up to fix it?

Context matters a lot here. Does the feature we’re working on make the company money? Or, does it affect our customers’ ability to perform their main function? If the answer is yes, then it’s probably something you want to add an automated test for. A shopping cart or login page might be a good example.

But many products have features that aren’t as important – “nice to haves”, that don’t perform critical functions. An example might be a ‘wish list’ on an eCommerce site. It’s reasonable to say it’s not vital to the business. We want it to work, for sure. But if it’s:

  • Already being tested at the unit level
  • Not being changed very often
  • Very easy to deploy a fix if it breaks

Then the cost of adding and maintaining an automated UI test may not be worth it. If we want to keep the duration of our tests low, it can pay to be a bit selective about what gets tested this way. 

Lean on your APIs for better test performance

As a rule of thumb, testing functionality through an API is faster than testing through the UI. There are a couple of ways we can make UI testing faster by leaning on APIs 🚀

Firstly, by testing most functionality through API tests

If there are good APIs that can exercise the functionality, there’s not always the need to test the same functionality in the UI too. In some cases, we just end up duplicating tests. In one product I worked on, there were many paths a user could take to perform the same function.

The underlying APIs were different, but the UI was largely the same. We could have added UI tests for each different path. But, it made more sense to test the different paths at the API layer, and just test one of them through the UI. This gave enough coverage of the different APIs and still exercised the UI enough to give confidence.

Secondly, by using APIs to do data setup for UI tests

Some functionality only comes into play when there is a particular data setup. An easy mistake to make is to drive the UI to set up all the data you need when there are other, faster ways.

As an example, let’s say we’re testing that a discount appears when there are five or more items in a shopping cart. It can be tempting to lean on UI tests to do this. Browse for an item, add it to the cart, and repeat five times. But – we aren’t testing the add-to-cart functionality, this is just the setup!

So, assuming there’s an API to do this, it makes more sense to do that instead. Use APIs to add the items to the cart, which will be much faster. Then the UI testing framework only needs to check the one page – the shopping cart – to assert the right discount appears.

Build with parallelization in mind

Finally, parallelization is an option 💡 Breaking the tests up and allowing them to run concurrently. In most products I’ve worked on, this hasn’t been necessary – getting to this point is a sign that there are too many tests. But, there are definitely cases where running tests in parallel might be desirable, and it’s obviously  faster.

So, it’s important to write tests with this in mind from the start. Write tests that set up and tear down the data they need, in a way that is completely isolated from other tests.

One trap I have fallen into is having every different test login as the same user. This is fine when running tests in series. But in parallel, they start to bump into each other. Assertions would fail because another test has changed the data it was expecting because it ran at the same time.

A better approach would be to have each test create its own user account, so the tests would never clash. If we start by creating tests with this in mind, we don’t run into problems when it comes time to parallelize later.

Start thinking about test performance early

In conclusion – if we want to increase our automated test suite performance, it can pay to ask some questions ahead of time:

  • How important is the feature we’re writing a UI test for? Do we really need this?
  • Could we test this at other layers instead, such as through an API?
  • Can we do the data setup for this through an API, to reduce the time the UI testing takes?
  • Is this test isolated enough that we could run it safely in parallel with other tests?

Conclusion

Fast deploys are good for a lot of reasons, and we want to keep deploy times low. So, it’s important to consider how to improve our test performance, and make our automation as lean as possible. Thinking about some of these things can help us achieve that! 💠

 

aThis article was originally written by James Espie for TestProject]

Automation, particularly UI Automation, is fun, and useful. However, automated UI test performance can be slow. They have to spin up a browser session, load front-end components, and so on. It doesn’t take much for a UI test suite to start taking tens of minutes to run.

I like Charity Majors’ recommendation of a 15-minute deploy time (15 minutes or bust!). Fast deploys are a key factor in building a quality SAAS product. But, this is impossible to achieve if you have an automated test suite that’s taking twenty or thirty minutes or even hours ⌚️ So, how can we keep the time of our automated UI tests down? What are some simple ways of keeping your automated tests speedy? 

Test the important things, not everything

It can be really tempting to want to write UI tests that cover every possible feature and flow. Every nook and cranny in the product. But, some of those tests might not be useful, in context. Does everything really need to be tested? 

I have a rule of thumb I like to use when adding an automated test. It’s this: if the feature I’m testing stopped working, what would happen? Would we wake someone up to fix it?

Context matters a lot here. Does the feature we’re working on make the company money? Or, does it affect our customers’ ability to perform their main function? If the answer is yes, then it’s probably something you want to add an automated test for. A shopping cart or login page might be a good example.

But many products have features that aren’t as important – “nice to haves”, that don’t perform critical functions. An example might be a ‘wish list’ on an eCommerce site. It’s reasonable to say it’s not vital to the business. We want it to work, for sure. But if it’s:

  • Already being tested at the unit level
  • Not being changed very often
  • Very easy to deploy a fix if it breaks

Then the cost of adding and maintaining an automated UI test may not be worth it. If we want to keep the duration of our tests low, it can pay to be a bit selective about what gets tested this way. 

Lean on your APIs for better test performance

As a rule of thumb, testing functionality through an API is faster than testing through the UI. There are a couple of ways we can make UI testing faster by leaning on APIs 🚀

Firstly, by testing most functionality through API tests

If there are good APIs that can exercise the functionality, there’s not always the need to test the same functionality in the UI too. In some cases, we just end up duplicating tests. In one product I worked on, there were many paths a user could take to perform the same function.

The underlying APIs were different, but the UI was largely the same. We could have added UI tests for each different path. But, it made more sense to test the different paths at the API layer, and just test one of them through the UI. This gave enough coverage of the different APIs and still exercised the UI enough to give confidence.

Secondly, by using APIs to do data setup for UI tests

Some functionality only comes into play when there is a particular data setup. An easy mistake to make is to drive the UI to set up all the data you need when there are other, faster ways.

As an example, let’s say we’re testing that a discount appears when there are five or more items in a shopping cart. It can be tempting to lean on UI tests to do this. Browse for an item, add it to the cart, and repeat five times. But – we aren’t testing the add-to-cart functionality, this is just the setup!

So, assuming there’s an API to do this, it makes more sense to do that instead. Use APIs to add the items to the cart, which will be much faster. Then the UI testing framework only needs to check the one page – the shopping cart – to assert the right discount appears.

Build with parallelization in mind

Finally, parallelization is an option 💡 Breaking the tests up and allowing them to run concurrently. In most products I’ve worked on, this hasn’t been necessary – getting to this point is a sign that there are too many tests. But, there are definitely cases where running tests in parallel might be desirable, and it’s obviously  faster.

So, it’s important to write tests with this in mind from the start. Write tests that set up and tear down the data they need, in a way that is completely isolated from other tests.

One trap I have fallen into is having every different test login as the same user. This is fine when running tests in series. But in parallel, they start to bump into each other. Assertions would fail because another test has changed the data it was expecting because it ran at the same time.

A better approach would be to have each test create its own user account, so the tests would never clash. If we start by creating tests with this in mind, we don’t run into problems when it comes time to parallelize later.

Start thinking about test performance early

In conclusion – if we want to increase our automated test suite performance, it can pay to ask some questions ahead of time:

  • How important is the feature we’re writing a UI test for? Do we really need this?
  • Could we test this at other layers instead, such as through an API?
  • Can we do the data setup for this through an API, to reduce the time the UI testing takes?
  • Is this test isolated enough that we could run it safely in parallel with other tests?

Conclusion

Fast deploys are good for a lot of reasons, and we want to keep deploy times low. So, it’s important to consider how to improve our test performance, and make our automation as lean as possible. Thinking about some of these things can help us achieve that! 💠

 

iThis article was originally written by James Espie for TestProject]

Great points on how to optimize UI automation testing! The key takeaway here is being strategic about what we test and how we approach it. It's easy to get carried away trying to test every feature, but focusing on the critical ones—those that directly impact users or business outcomes—can save a lot of time and effort. Your rule of thumb on testing only what would "wake someone up" is a fantastic guideline to prioritize tests that truly matter.

Leaning on APIs to handle the functionality and data setup is another brilliant idea. It’s often much faster to test through the API layer rather than the UI, and it can help reduce redundant tests. I particularly like the example of using APIs to set up test data, such as adding items to the cart. This keeps the UI test focused on the actual functionality you're testing, rather than spending time on the setup.

The idea of parallelization is also a game-changer for scaling test suites. Designing tests to be isolated and independent from each other early on will save so much time when you eventually need to run tests in parallel. Plus, by keeping the tests focused on critical paths and optimizing their performance, you’re aligning perfectly with the goal of achieving fast deploy times and keeping your testing suite efficient and effective.

Thinking about test performance upfront is crucial, and I agree that being selective, using APIs wisely, and planning for parallelization are great strategies to keep your automated UI tests fast and maintainable. Excellent insights!

Reply


ShiftSync Terms and Conditions