SLAs, SLOs, and SLIs: How to Decide Whether a Performance Test Passed in NeoLoad
Adapted from the first session of Tricentis’s Performance Engineering Fundamentals webinar series, presented by Bryan Cole, Director of Customer Engineering at Tricentis.
TL;DR: Performance tests rarely pass or fail cleanly, so you need defined targets to judge them. SLAs set business-facing goals (process, user load, response time), SLOs set internal engineering targets, and SLIs are the raw metrics underneath. When the business can’t supply numbers, use production telemetry or your first test results as a starting point and refine from there.
Most software testing gives a yes-or-no answer. A tester clicks “calculate tax” and checks the result. The tax is either correct or it isn’t. When a test like that fails, it is usually clear what went wrong and what needs fixing.
Performance testing doesn’t work that way. A test simulates many users at once, each doing different things: some searching, some adding items to a cart, a few updating an address. The result is rarely all good or all bad. Some transactions are fast, some are slow, and a few return errors. Even when everything looks fine, the question remains whether it is fine enough.
That’s why service level agreements (SLAs) and service level objectives (SLOs) need to be defined inside your performance testing tool. Without them, the tool has no basis for calling a test a success or a failure.
SLAs: targets the business understands
In contract terms, an SLA is a formal commitment between a provider and a customer, often with penalties attached. SaaS contracts with vendors like Workday or ServiceNow include them. Internal performance engineering usually has no legal contract, but the core idea still applies: a clear statement of how a specific part of the system should perform.
A good performance requirement has three parts: a business process, a user load, and a response time.
“Login shall complete within three seconds”
isn’t testable, because a developer can meet it on a laptop with no one else in the system.
“Login shall complete within three seconds with 250 concurrent users”
is a requirement you can test.
Not every feature needs the same attention. On a retail site, search and checkout must be fast. People move about once every seven years, so a slow address-change page has little impact.
SLAs should also use the language of the business. Executives don’t care about transport layers or encryption. They care how long it takes to log in or run a report.
SLOs and SLIs: the engineering view
SLOs are internal targets for developers and operations teams. If the business wants login in three seconds, engineers know login calls several subsystems, and each needs its own target. An SLO might say a particular API must respond within 500 milliseconds.
Service level indicators (SLIs) are the raw measurements underneath, such as CPU usage on a pod. They are narrow, technical, and numerous, and they usually come from APM tools like Dynatrace, Datadog, or New Relic.
Together these form a hierarchy. SLIs feed SLOs, and SLOs support SLAs. When login is slow, you first find which part is slow, then why: which code it calls, which resources it consumes. As a general guide, SLAs are for the business and SLOs are for engineering.
Defining these targets also protects the performance engineer. Deciding what is fast enough is a business decision. Ask a stakeholder without documented targets and you’ll likely hear “it should be fast,” which can’t be tested.
Setting up SLAs in your testing tool
Nearly every performance testing tool supports SLAs. In NeoLoad, you create SLA profiles and apply them to user paths or individual requests, with as many profiles as you need. That flexibility matters because targets vary. A two-minute report might delight mortgage specialists if it used to take ninety minutes, while two minutes would be unacceptable for a page load.
NeoLoad evaluates SLAs either at the end of a run or at intervals during it, and each SLA can have warning and failure thresholds. For example, you might warn at more than 50 errors and fail at 200. The numbers should reflect the business’s tolerance for risk.
Percentiles are often more useful than averages, because averages hide the slowest experiences. Stakes matter too. A live streaming event with millions of viewers needs far stricter thresholds than an internal expense system.
Skipping this step has consequences. In one example run with no SLAs, NeoLoad recorded 106 request errors and still marked the test as passed, because it had no definition of failure.
When three seconds becomes 3.01
Suppose the SLA says three seconds and the test returns 3.01. Technically, it failed. In Cole’s experience, the business almost always responds: “That’s great. Ship it.”
That tells you the SLA was set wrong. If 3.01 is acceptable, the threshold should have been 3.1 or 3.5. Treat moments like this as a chance to refine targets, not a reason to ignore them.
Where the numbers come from
In theory, the business already knows its performance requirements. In practice, they’re rarely documented.
What can you do? Put performance requirements in the backlog. Write down the process, the load, and the time target. Developers take performance seriously once it’s part of their assigned work.
When the business can’t supply numbers, use production telemetry. Most projects upgrade or replace an existing system, so operations teams already have data on how long current processes take with real users. That data also shows the real workload mix, which helps you build realistic test scenarios.
If telemetry isn’t available, run a test, set initial SLAs from the results, and show them to the business. People who can’t say what they want are quick to say what they don’t like. When they say “that’s too slow,” ask why.
SLOs can follow a similar path. In NeoLoad, you can set them as fixed values, compare runs against a baseline with a tolerance, or use standard deviation to flag outliers. A run can meet every SLA and miss its SLOs, and that’s acceptable. If the business is satisfied and the system is stable under load, the job is done.
Name transactions after what users do
The most valuable habit for a new performance engineer is inserting named transaction markers while recording scripts, such as “Open main page” or “Add to cart.” The recorder groups every request under the step it belongs to, and the names match how stakeholders describe the application.
Don’t accept scripts without these markers. Have the author re-record them. Re-recording takes minutes; working out afterward which of hundreds of resources belongs to which step takes hours.
Report to your audience
Business users want to know how long tasks take, not how long a style sheet took to load. Be ready for budget conversations too. If login takes one second and the business wants half a second, operations might say that’s possible with another million dollars in cloud spending. At that point, one second often turns out to be fine.
Developers want data on the components they own. Operations teams want to see how the application uses system resources. In decades of testing, Cole has seen a misconfigured network cause poor performance exactly once. Far more often, the application uses the network inefficiently.
Data also helps with perception. After one bank rolled out a teller application that tests showed was 10 percent faster than its predecessor, tellers complained it was slow on Friday at lunchtime. Production data showed a check deposit took two seconds every day of the week. The difference was the line of customers out the door. Showing users before-and-after comparisons ahead of launch heads off complaints like these.
Bringing it together
Integrate your APM tools with your performance testing tool so you can see not just that there’s a problem, but where it is and what caused it. Send SLA results to your test management system alongside other quality metrics; NeoLoad, for instance, can publish them to qTest. Finally, connect performance tests to your CI pipeline. Even without SLAs, a trend history lets a developer spot an endpoint jumping from 30 to 390 milliseconds and fix it before the change goes further.
And when a stakeholder asks, “what’s an SLA?”, take the opportunity to explain. Clear, documented performance requirements make every later step easier.
