Teaching the agent to hunt: why execution alone will never produce exploratory testing
Author: Ujjwal Kumar Singh, Senior QA Engineer · 8+ years in enterprise QA · beinghumantester.github.io
Key takeaways:
- Why a test run can pass every check and still hide a real defect between connected systems
- How experienced testers turn an unexpected observation into a hypothesis and a deliberate experiment
- Why searching for explanations differs from searching for actions, and why more automated actions don't replace investigation
- How the six-step investigative loop lets testing change direction as evidence builds up
Imagine an order flow that spans two systems: a web storefront that takes the order, and an ERP that owns the stock. Two customers buy the last available unit at almost the same moment. Both orders are confirmed. The storefront's checks pass, and so do the ERP's. No exception is raised. From the perspective of an execution-oriented automation system, the test has succeeded and there may be nothing left to investigate.
An experienced tester may see something different. There are now two confirmed orders for one unit, and the ERP shows stock at minus one. Neither system did anything it considers wrong, and no single test suite checks what happens between them. The behaviour may only occur under a particular timing condition, or only when both orders overlap in a specific way. Nothing has technically failed according to the existing checks, but the observation is suspicious.
The difference is not simply how many tests are executed. It is what the tester is searching for.
An execution-oriented system searches for the next action to perform. An exploratory tester searches for an explanation of what has been observed. Both may click, type, submit, repeat and manipulate the same system. What differs is the question guiding those actions.
That distinction matters if autonomous testing systems are expected to do more than execute generated sequences. Executing more actions can increase the number of situations a system encounters. It does not necessarily give the system a mechanism for deciding what an unexpected observation means or which explanation should be investigated next.
What experienced testers actually do
When experienced testers encounter unexpected behaviour, they do not necessarily treat the observation as the end of an execution path. The observation can become the beginning of an investigation.
Consider the inventory example again. Both systems have completed their transactions and produced an unexpected state. A tester may start considering whether the behaviour is related to transaction ordering, concurrency, stale state, timing, a boundary condition, or an interaction between components.
The tester might reproduce the behaviour under slightly different conditions. They might change one variable and observe what happens. They might deliberately try to make the behaviour disappear. They might consider another explanation that could produce the same result.
The important change is that the tester is no longer simply asking what action should be executed next. They are trying to learn something about the system.
This is consistent with the tradition of exploratory testing, where learning and making new choices occur alongside test execution rather than after a fully predetermined procedure. The tester is not simply following a fixed sequence and evaluating the final result. What is learned during the investigation can influence what happens next.
That makes the tester's observation an active part of the testing process rather than merely an output produced at the end of execution. For an execution-oriented agent, however, an observation is typically an output to consume or record; the question of what that observation might mean is a different problem.
From observation to investigation
An observation by itself is not an explanation.
Suppose a tester notices that inventory becomes inconsistent after two concurrent transactions. That observation tells us something about the behaviour of the system, but it does not tell us why the behaviour occurred.
The tester may ask whether both transactions could have read the same inventory state before either transaction committed its update. That question leads to a hypothesis: perhaps the inconsistency occurs because both transactions are operating on stale information.
Now the next experiment has a purpose.
Instead of simply trying another quantity or repeating the same sequence, the tester can deliberately manipulate transaction timing to determine whether the suspected interaction is actually relevant. If changing the timing changes the outcome, that evidence affects the explanation. If it does not, the tester has reason to reconsider the hypothesis.
This is the transition from observation to investigation. The tester has moved from noticing that something happened to developing an explanation that can be examined.
The distinction becomes important because the same product action can have very different meanings depending on what the tester is trying to learn from it. Changing transaction timing may simply be another action in an execution sequence. It becomes an experiment when it is deliberately chosen to provide evidence about a particular hypothesis.
For an autonomous execution system, the difficult step is not performing the same action. It is representing why that action is worth performing and what explanation the resulting evidence is meant to distinguish.
The investigative loop
This investigative behaviour can be expressed as a loop. The tester observes behaviour, turns that observation into a question, develops a hypothesis about what might explain it, designs an experiment, evaluates the resulting evidence, and updates their understanding before deciding what to investigate next.
The important part is not the six labels themselves. It is the feedback between them.
An experiment produces an observation. That observation can strengthen a hypothesis, weaken it, or reveal that the tester was investigating the wrong explanation. The updated understanding then influences the next investigation.
Exploration therefore does not proceed as a fixed sequence of actions. It changes direction as evidence accumulates.
Exploratory testing can be understood as an iterative investigative loop in which observations generate questions, hypotheses guide experiments, evidence is evaluated, and understanding is updated before the next cycle begins.
This feedback is what allows an investigation to evolve rather than simply continue. For an autonomous system, making that feedback explicit is the challenge: execution can produce observations, but investigation requires those observations to influence what the system believes and what it chooses to investigate next.
Hypothesis, experiment, and observation
The distinction between an action and an experiment becomes clearer when we separate the hypothesis, the experiment and the observation.
A hypothesis is a claim about what might explain something that has been observed. It does not have to be correct. It needs to be specific enough that an experiment can produce evidence that changes what we believe about it.
An experiment is an action deliberately selected because it can provide useful evidence about the hypothesis. The action is therefore connected to a question. It is not simply another step generated because the system has reached the end of its previous action.
The observation is what the experiment gives us in return. The resulting evidence may support the current explanation, make it less plausible, or reveal something that requires a different line of investigation.
Return to the inventory example. “Try another quantity” is an action, but by itself it tells us little about what we are trying to learn.
Now consider an experiment designed to determine whether the inventory inconsistency appears only when the requested quantity crosses a suspected boundary while concurrent transactions are active. The actions involved may look similar to those generated by an execution system, but their role is different.
The tester is deliberately seeking evidence that can distinguish between possible explanations.
This is why the number of actions performed tells us very little about whether a system is actually conducting an investigation.
Exploration is not randomness
It is tempting to interpret exploratory testing as simply trying things that were not tried before. Modern testing systems can already generate large numbers of inputs, sequences and combinations. A high-volume stress run, for example, may surface an inventory failure because it happens to hit an unusual combination of operations under an unusual timing condition.
That discovery can be extremely valuable. Unexpected combinations can expose behaviour that a carefully designed test never encounters.
But discovering something by chance is different from investigating it.
Suppose a stress run surfaces the inventory inconsistency. An investigator can take that observation and deliberately vary the conditions around it. They might change the transaction timing while keeping the inputs constant, then change the inputs while keeping the timing constant. If one change makes the problem disappear while another does not, the resulting evidence can eliminate some explanations and make others more plausible.
The investigation is therefore not defined by whether the original discovery was random. It is defined by what happens after the discovery.
Random search can produce an interesting observation. Investigation uses that observation to determine where to look next.
This distinction becomes increasingly important as the number of possible actions grows. A system can execute enormous numbers of combinations without necessarily becoming better at explaining the behaviour those combinations reveal.
There's a practical side to this, too. An investigation that isn't recorded can't be reviewed, repeated, or handed to the next person. On a team testing a dozen connected applications, the hypotheses you ruled out are nearly as valuable as the one you confirmed.
The search space is different
This leads to a deeper distinction between execution and exploration.
An execution-oriented system primarily searches through a space of possible actions. It can decide whether to click, type, submit, navigate, change an input or repeat a sequence. The central question is what action should be performed next.
An exploratory tester is searching through a different space. After observing unexpected behaviour, the tester considers possible explanations for what happened. One explanation may involve state. Another may involve timing. Another may involve an interaction between two components. The tester is trying to determine which explanation is worth investigating next.
The question has changed from “What should I do next?” to “What should I understand next?”
This is why exploratory testing cannot be reduced to simply increasing the number of actions performed by an autonomous system. A larger action space may increase the number of behaviours encountered, but it does not automatically create a space of explanations or a mechanism for reasoning through competing explanations.
An experienced tester is effectively following clues through that explanation space. An unexpected observation becomes a clue. A hypothesis provides a possible explanation. An experiment provides evidence. What is learned determines where the investigation goes next.
That is the sense in which the tester is hunting.
Not hunting randomly for a failure, but following evidence toward an explanation.
Most autonomous testing agents search a space of actions. Exploratory testers search a space of explanations. These are different cognitive activities producing different outputs.
The harder question: can testing be automated?
At this point, an important question needs to be addressed directly. If exploratory testing involves investigation, learning and experimentation, what exactly can an autonomous system take over?
Bach and Bolton's refined formulation, most recently updated in 2024, defines testing as evaluating a product by learning about it through experiencing, exploring and experimenting. They define checking as the mechanistic process of verifying propositions about the product.
They further distinguish the two by arguing that checking can, in principle, be performed by a tool, whereas testing can only be supported by tools. They describe testing as an open-ended investigation and make an important distinction between the two: a check is describable, while a test might not be, because testing involves tacit knowledge.
At the same time, their position does not make tools irrelevant to testing. They explicitly argue for a robust role for tools and note that tools can support testing in ways that extend beyond the automation of checks.
This creates a more interesting question than whether testing can simply be automated.
If exploratory testing involves investigation, experimentation, learning and interpretation, could an autonomous system perform those investigative activities itself rather than merely automate the execution and checking around them?
This article does not assume that the answer is yes. It treats that question as an engineering and empirical problem.
To investigate it, the investigative process first needs to be made explicit.
From investigation to architecture
The distinction can now be stated simply.
Execution asks: What should I do next?
Investigation asks: What should I understand next?
If an autonomous system is expected to investigate rather than merely execute, it needs the investigative loop itself, including the ability to weigh competing explanations and change course as evidence accumulates.
That does not yet tell us what such a system should look like. It tells us what problem the architecture must solve.
If the investigative loop were made explicit, what would an architecture for it look like?
An investigation is only as useful as the trail it leaves
The hunt happens in the tester's head, but the evidence doesn't have to stay there. On a large testing estate, the hard part often comes afterwards: showing what you investigated, what you ruled out and which findings deserve to become regression checks.
Tricentis qTest manages exploratory, manual and automated testing in one place, and qTest Explorer captures exploratory sessions as you run them, so a finding can become the starting point for an automated test. The thinking stays yours. The record doesn't have to live in a notebook.
→ See how other testers run and record exploratory sessions in the qTest space
