Introduction
Over the last couple of years, I’ve found myself in a situation I never quite anticipated.
Test cases are no longer always written by testers.
They are increasingly being generated by AI.
What started as experiments inside delivery teams quickly became standard practice. Teams began using AI to generate test scenarios from user stories, derive edge cases from requirements, suggest regression suites, and even recommend automation coverage.
The promise was extremely attractive.
Faster coverage. Reduced manual effort. Improved productivity. The ability to scale testing without scaling teams at the same rate.
And honestly, the early results looked impressive.
I’ve seen AI generate meaningful test cases in seconds that would normally take hours manually. I’ve seen it identify edge conditions that even experienced testers initially overlooked. In one of our enterprise programs, introducing AI into the test design phase improved our initial test coverage significantly within just a few sprints.
But somewhere in the middle of all this progress, a question started bothering me.
If AI is writing the test cases, who is testing the AI?
That question stayed with me because the deeper we went into AI-assisted testing, the more complicated the answer became.
The Problem We Didn’t See Coming
This concern did not come from theory or conference discussions.
It came from real project experience.
In one of our Salesforce-heavy implementations, we started integrating AI into our Jira-to-testing workflow. The system could read user stories, interpret acceptance criteria, analyze linked documentation, and generate structured test cases automatically in the test management platform/workflow.
On the surface, it felt like a breakthrough.
The generated test cases looked polished. Coverage increased rapidly. Dashboards looked healthier than before. Teams felt more productive because repetitive test design effort had reduced drastically.
Initially, everyone was excited. But within a few sprints, subtle gaps started appearing. The AI-generated tests were logically correct, but they were not contextually aware.
That distinction became extremely important.
When Logic Wasn’t Enough
I still remember one specific issue in a pricing and taxation module.
The AI generated detailed test cases validating discount calculations, approval flows, tax rules, and pricing combinations. Everything appeared comprehensive during review. But after release, a customer escalation exposed a serious issue. The application behaved incorrectly when multiple regional discount rules overlapped under a very specific customer configuration.
The AI had followed the documented requirements perfectly. But it had completely missed the business risk hidden between the requirements. That was the moment I realized something important. AI understands patterns. It does not automatically understand consequences.
And testing, at its core, is about understanding consequences. We often assume that if enough test cases are generated and executed, coverage has been achieved.
But coverage without relevance is just noise.
More Testing Did Not Mean Better Testing
In another project involving a large CPQ transformation, we used AI to accelerate regression suite creation across hundreds of pricing combinations and dependencies.
Maintaining regression coverage manually had already become exhausting. AI looked like the perfect answer.
And initially, it absolutely felt that way.
The AI generated hundreds of test cases quickly. The reporting looked fantastic. The team felt confident that they can achieve their goals.
But when we reviewed the generated suite more carefully, another problem surfaced.
The AI was producing duplication at scale.
Very similar test scenarios kept appearing repeatedly across different flows. The AI optimized for completeness, not efficiency. It could not understand that many of those tests carried identical business risk.
We had more test cases. But not necessarily more intelligence. And that distinction matters more than most teams realize.
The Bigger Risk Nobody Talks About
There was another risk hiding underneath all of this.
Overdependence slowly weakens critical thinking.
I’ve seen this happen gradually across teams. When AI-generated outputs consistently look polished and structured, people naturally begin trusting them more.
- Reviews become lighter.
- Questions become fewer.
- Eventually, ownership of thinking quietly shifts from humans to the system.
That is dangerous.
At one point, we deliberately experimented with incomplete and slightly ambiguous requirements. We wanted to observe how the AI behaved when clarity was missing.
The results were fascinating.
The AI still generated detailed and confident-looking test cases. Structured. Readable. Well-formatted. But underneath, those test cases were built on assumptions that nobody had validated.
That is one of the biggest differences between humans and AI.
Experienced testers pause when something feels incomplete.
AI often fills gaps based on patterns or assumptions.
And in testing, filling gaps without questioning them can become extremely risky.
The Role of Modern Testing Platforms
What made this even more interesting was watching how modern testing platforms themselves are evolving around this challenge.
Companies like Tricentis are increasingly introducing AI-assisted capabilities into test generation, risk-based testing, and automation optimization. The goal is not to eliminate testers from the process, but to reduce repetitive effort and allow teams to focus more on business validation and intelligent risk analysis.
And honestly, I believe that is the right direction.
The real value of AI in testing is not generating more scripts.
It is enabling testers to spend less time writing repetitive validations and more time thinking critically about failures, customer behavior, edge conditions, and business impact.
That shift matters.
Because AI can accelerate testing.
But only humans can decide what is truly worth testing.
So Who Actually Tests the AI?
This brings us back to the original question. Who tests the AI that writes test cases?
AI is useful, but it needs the right platform, governance, and human oversight. Hence our role continues to evolve. We are no longer just creators of test cases. Increasingly, we are becoming evaluators of AI-generated quality itself. That requires a completely different mindset.
Instead of asking:
“Are these test cases correct?”
We now need to ask:
“What could still be missing?”
Instead of validating outputs alone, we need to validate assumptions, intent, and business relevance.
What We Changed Inside Our Teams
This realization forced us to change how our teams operated.
First, we stopped treating AI-generated test cases as finished deliverables. They became starting points instead.
We introduced a simple internal practice.
Every AI-generated test case had to be challenged by at least one human question:
- What assumption is being made here?
- What customer behavior is missing?
- What happens if this fails in production?
- Is this testing logic or actual business risk?
That small shift changed the quality of conversations dramatically.
Second, we deliberately brought business context back into testing discussions.
AI performs extremely well with structured inputs. But customer behavior, production history, regional nuances, and emotional impact rarely exist cleanly inside requirement documents.
In one of our payment systems, this became critical.
The AI generated strong transaction validation scenarios but completely missed timeout handling under unstable network conditions. One of our testers had seen similar failures years earlier in another system and manually introduced those scenarios.
That single intervention prevented what could have become a major production issue.
Third, we changed how we measured success.
We stopped celebrating only execution numbers and coverage percentages.
Instead, we started asking better questions.
- How many AI-generated tests required modification?
- How often did production issues map back to AI-generated gaps?
- How much redundancy existed?
- Did the generated suite actually improve confidence or just increase volume?
Those metrics told us a far more honest story.
The Role of Testers Is Becoming More Important
One of the biggest lessons from all of this is that testers are not becoming less important.
Their role is becoming more critical.
But differently critical.
The future tester is not someone who simply writes more manual test cases.
The future tester is someone who understands systems, questions assumptions, evaluates business risk, interprets customer impact, and challenges AI-generated confidence.
AI can process information faster.
It can generate at scale.
It can identify patterns rapidly.
But it does not remember past outages.
It does not understand customer frustration.
It does not feel the pressure of a failed release.
It does not carry intuition.
Humans do.
And that still remains our greatest advantage.
Closing Thought
AI can write test cases. But it cannot take responsibility for what it misses. That responsibility still belongs to us.
The real shift happening in QA is not about whether AI can do our work. It is about whether we are ready to evolve our role alongside it. Because testing is no longer just about validating applications. It is increasingly about validating the systems that validate the applications themselves.
And that is not a responsibility we can automate away.
