Teaching the Agent to Hunt: What We Tested and What Didn't Change [Part 2]

Author: Ujjwal Kumar Singh, QA Engineer · 4+ years in enterprise QA  · beinghumantester.com

In Part 1, I argued that an agent that executes steps successfully is not doing exploratory testing. Exploration is an investigation, not a script. A test can pass every assertion on the happy path and still miss the duplicate order created by a double-click.

So I tried the obvious fix: add explicit investigative structure to the agent's instructions, and measure what changes. I wrote down my prediction first. It did not hold.

I wanted to test one version of that idea instead of asserting it. The question for this article is narrow:

Does adding hypothesis-and-prediction logging, followed by a required "signals" step, change the number and types of seeded defects an AI agent finds?

I was not measuring whether an agent had learned to think like an exploratory tester. I measured something narrower: whether two additions to the instructions of a general-purpose agent, working through a browser tool, changed which seeded defects it found. The short answer: neither addition met its pre-registered success criterion. The more interesting finding is a distinction the result forced on me: a log of an investigation, the investigation itself, the defects found and the risks left untested are four different things, and a tidy log does not, on its own, guarantee the other three.

The experiment

A pilot first. Round 1 was one run per condition on a small shop app with seeded defects and decoys, with a budget of about 40 tool calls. Both conditions found 2 of 5 seeded defects. Both missed all three timing and concurrency defects, and neither report mentions the order latency that surrounded them. One run each cannot separate a scaffold from luck, and the call cap may have decided where each run stopped. So I treated Round 1 as a pilot and built a new app for Round 2, instead of tuning a test to a system I had already seen.

The app. Mini Tickets is a small ticket-ordering web app with six seeded defects and four decoys, which are behaviours that look suspicious but are correct. The defects:

• A 60-ticket order is clamped to 59.
• A stale quote overwrites a newer one.
• Two buyers can both get the last ticket (a check-then-act race).
• A double submit creates two orders.
• A non-digit quantity returns HTTP 200 with NaN amounts.
• A rejected order shows nothing in the UI.

The middle three are timing and concurrency defects. In the tables below I call them "timing defects" for short.

Three conditions.

• A: a default charter. Explore the ordering flow and report anything incorrect or suspicious, with reproduction steps.
• B: A plus a log. For every hypothesis, write a prediction and a falsifier before running the experiment.
• C: B plus a required signals step. After every observation, and before choosing the next hypothesis, note anything surprising, such as latency, ordering or status codes. This prompt names the kinds of signals to look for, so a gain in C could have come from that hint and not from the step itself.

The design. Five runs per condition, 15 in total, in five waves of three concurrent sessions. Each wave held one run of each condition, in random slots, and the waves ran in random order. A seeded shuffle, with the seed recorded, generated the slots and the wave order at the start of the run, before any session began. Every session had its own app instance, port, seed and browser profile. All used the same model (Claude Sonnet 5.5) with a 20-minute limit. I committed predictions, metrics, scoring rules and decision rules before the app was built or any run started. Two independent AI scorers then read only the reports, with the condition hidden, and scored each against an answer key. They disagreed on 4 of 90 verdicts, and I settled those before the conditions were revealed.

What nearly went wrong. Before the real run I caught several problems that would each have undermined the result. The app's public JavaScript contained comments describing three of the defects, readable from any browser. Read permissions left the defect list and the other conditions' prompts reachable. Sessions could reach each other's app ports. A hung process could have blocked saving results. After the run, my harness flagged all 15 sessions as infrastructure failures, because its check matched a routine status message that appears in every transcript. I fixed the first four before running and recognised the last as a false alarm. I list them because evaluation setups leak in boring ways, and a leaked answer would have made the result meaningless.

The results did not follow my predictions

I had written down what I expected before any run: the scaffolds would help, and the signals step would help most on timing defects. They did not.

Under my pre-registered rules, the scaffold does not help if B's median is no higher than A's, and the signals step helps only if C's median timing count beats B's. B matched A, and C matched B, so neither condition for "helps" was met. That does not prove the scaffolds have no effect. It means they did not meet the criteria I set in advance.

A note on scoring. In short:

• The medians above are my primary analysis, using the verdicts I settled after comparing the two scorers and before the conditions were revealed.
• Each scorer alone gives different total medians, 2 / 2 / 3 for scorer 1 and 3 / 3 / 3 for scorer 2, because of borderline verdicts on two defects.
• Neither scorer changes the two pre-registered decisions: B's median is no higher than A's, and C's median timing count is no higher than B's.
• C's higher total under scorer 1 is exploratory. It was not a pre-registered success criterion, and I do not read it as evidence that the signals step worked.

The per-defect counts are more revealing than the medians. Each cell is the number of runs, out of five, that found the defect with a reproducible trigger and evidence.

The last row is the only one with an apparent pattern across conditions. With five runs each, I would not call it an effect, and it is not a timing defect.

Sessions finished in a median of 155 seconds for A, 194 for B and 172 for C, well inside the 20-minute limit, at a median cost of about $0.16, $0.23 and $0.22 per run. I report these for completeness. They do not show that any condition is more efficient.

What the detection counts show, and what they cannot

The result I trust most is not the medians. It is the split between two kinds of defect.

In this app, defects exposed by a single input or screen were found almost every time: the cap boundary in 15 of 15 runs, and the NaN amounts in 13 of 15. Defects that needed a specific interaction sequence or timing were found rarely. These defects differ in more than difficulty, including how visible they are and how easily the browser tool can trigger them, so the pattern describes this app and does not isolate one cause. The last-ticket race was found in 1 of 15 runs, and double submit in none. Both scorers agreed on those two counts, so they do not depend on a borderline call. The stale-quote defect, which needs responses to arrive out of order, sits in between at 9 of 15, and one of those verdicts is borderline.

The prompt changes did not eliminate the gap for this general-purpose agent. The last-ticket race was found once under C, and double submit stayed undetected in all 15 runs.

What the counts cannot tell us. Two explanations fit the data, and the experiment does not separate them:

1. A capability limit. The agent cannot easily set up or reason about concurrent interactions through a browser tool.
2. A policy choice. The prompt told the agent to stop when further effort was not worth it, and said it had a time limit it could not see. The sessions ended after two to five minutes of a 20-minute limit.

I also have not tested whether the browser tooling makes concurrent interactions harder to trigger. Those are open questions, and the next experiment should pull them apart.

A low detection rate tells us where the outcome fell short. It does not tell us where in the investigation the shortfall began. The agent may never have considered the interaction, may have considered it and not spent effort on it, may have failed to trigger it, or may have triggered it without recognising the evidence. Each of those calls for a different fix.

Stopping also deserves three separate names. The available budget is the 20 minutes the harness allowed. The agent's stopping behaviour is when and why it ended. The prompted stopping policy is the instruction to stop when further effort is not worth it. These sessions show that the budget was not reached. They do not show that the agent's sense of effort played no part.

A defect count tells us what was found. It does not tell us where the investigation failed. To see that, keep four kinds of evidence apart. I offer them as a framework for interpreting results, not as something this experiment validated. They are easy to blur into one score: process (did the agent record a hypothesis, a prediction and a falsifier?), investigation (did it run an experiment that could separate the possibilities?), outcome (did it find and reproduce the defect?) and coverage (which important risks were left untested?). A tidy investigation log is evidence of process, not proof of adequate coverage. In this experiment, adding that structure did not improve the measured defect-detection outcomes under my pre-registered criteria. This is my proposed framework, not a validated model. The four also point to different fixes. If the agent never considered concurrency, the fix is in prioritisation. If it considered it but could not trigger the race, look at the tooling or the execution strategy. If it triggered the race and did not recognise it, look at how it interprets evidence. I did not measure all four dimensions here.

A bounded proposal: a human-defined investigation charter

In Part 1, I described a six-step investigative loop and asked what an architecture for it would need. This experiment did not test such an architecture. It tested two prompt-level changes, and I do not claim any architecture is validated.

The result points to a narrower question. Who decides what must be investigated before an agent is allowed to stop? The need to answer that does not depend on this result. What the result adds is a concrete case: structure in the prompt did not, by itself, deliver coverage.

My proposal, which is untested, is a charter that a human sets before the session and reviews after it:

• Minimum risk coverage. A short list of risk classes, such as concurrent orders and repeated submits, that the agent must either test or explicitly justify skipping. The charter also defines what evidence counts as covering a risk, and a person judges whether it meets that standard. A convincing explanation is not a substitute for an investigation.
• Stopping criteria. A condition for stopping that is not just the agent's own judgement that more effort is not worth it.
• Review of abandoned hypotheses. A record of what was dropped and why, read by a person.

This is a design proposal, not a finding. Round 2 does not show that a charter would help. A charter that tells the agent to test concurrency is also a prompt change. If it works, that suggests the gap was mostly a prompting and stopping choice, and the decision about what to cover stays with a human. If it fails, the gap is deeper than the prompt. Either result would tell us something the current data cannot.

There is a distinction inside that proposal. A human can own the decisions about whether concurrency is in scope and what counts as enough investigation. That does not make the agent able to build the right concurrent interaction, run it reliably or recognise a race when it sees one. The experiment cannot say which mechanism would enforce the charter best.

The practical takeaway does not depend on the agent. A test charter should name more than the feature or journey to explore. It should name the risk classes that matter, what evidence counts as a discovery and which risks may not be skipped. That holds whether a human, a browser agent or a team of both does the testing.

Limits and the next experiment

What this experiment cannot support:

• Size. Five runs per condition, one model and one purpose-built app. I report descriptive numbers only and make no claim of statistical significance.
• Scoring. The scorers were AI models from the same family as the testers.
• Prediction order. Conditions B and C were told to log a prediction and a falsifier before each experiment. In the transcripts, 25 of 77 logged predictions were saved before a browser action ran. The other 52 first appear in the same log write as the observation, so the log cannot show that the prediction came first. I did not check whether they were stated earlier in the conversation. This is one more reason I treat the log as process evidence only.
• The hint in Condition C. Its prompt names latency, ordering and status codes. Three sessions also shared one machine with limited memory, which the wave design balances across conditions but does not remove.
• The prompt itself. It required a "Not investigated" section, told agents to stop when further effort was not worth it, and mentioned a time limit they could not see.
• Generality. Nothing here says how other models, other apps or longer sessions would behave.

The next experiment should pull apart the two explanations above, changing one thing at a time.

First, hold the model, the app and the scoring criteria constant. Use Condition B as the baseline and compare it with a prompt that changes only one thing: it explicitly requires concurrency and repeated-submit testing. Keep the stopping policy constant too. Record whether the agent considered each risk, attempted the interaction, triggered the defect and interpreted the evidence. Only then change the model or the stopping policy, separately, and add more runs. If the gap closes, that would support the idea that explicit investigation requirements matter. If it does not, I would need to look at the tooling, the execution strategy and the agent's ability to act on those requirements before drawing a conclusion. Even a clear result would show only that an explicit requirement matters in this setup, with this model and this app. It would not show that prompting and stopping policy cause such gaps in general.

The experiment did not test an architecture for the investigative loop I described in Part 1. It tested two smaller changes to a general-purpose agent’s instructions, without changing its tooling, context or oversight, and neither met its pre-registered success criterion. What remains is a question I cannot yet answer: whether the gap comes from what these agents can do, or from what we have not required them to do. The practical implication does not wait for the answer. Teaching an agent to hunt is not only a question of which instructions it follows. It is also a question of what we require it to investigate, what evidence we accept, and what remains unknown when it stops.
 

Read more on ShiftSync:

• Teaching the agent to hunt: why execution alone will never produce exploratory testing | ShiftSync Community

• Tosca can't find your control? First ask who is looking | ShiftSync Community

• What Changes When Your Interviewer Is an AI | ShiftSync Community