HTML widget
AgentScore
⚡ Phase 1 · Internal Pilot · Tricentis
Your agent is live.
But what should you even be testing?
Most teams have no idea what to evaluate — and that's the real problem. AgentScore watches your agent in shadow mode, identifies what matters based on how it actually behaves, and builds your evaluation design from observed sessions.
Observation-based eval design
11 quality dimensions
LLM-as-judge
Runtime Guard
Attribution
agentscore · ATA Regression Suite
Scoring run complete
SHIP
88
/ 100
B
±4 confidence
Correctness
91
Efficiency
82
Relevance
86
Safety
88
Consistency
85
Tool Use
87
3 runs · 9 sessions
48 traces
HTML widget
20
Observed sessions before AgentScore proposes what to measure
11
Quality dimensions available — none required, all versioned
0–100
Composite score and A–F grade, derived per scoring run
3
Verdicts — Ship, Review, or Block, with full attribution
HTML widget
The challenge
Evaluating AI agents is hard.
Starting from scratch is harder.
Evaluating an AI agent isn't like writing unit tests. There's no obvious spec to derive from, no single failure mode to cover first. Most teams either skip evals entirely or write ones that miss what actually goes wrong.
The blank-page problem
You know your domain. You don't know what to measure. Writing evaluation criteria from scratch requires AI expertise most practitioners don't have — and don't need to have.
Testing the wrong things
Generic evals miss agent-specific failure modes. An ATA agent and an AI Workspace agent fail in completely different ways — a one-size-fits-all eval misses both.
No signal once you do
Even teams with evals lack a cross-session signal. Scores change, models get updated, prompts get tweaked — and there's no structured way to know if anything got better or worse.
HTML widget
The solution
AgentScore figures out what to test by watching your agent work.
Connect your agent in shadow mode. AgentScore observes real sessions — which tools it calls, where it hesitates, what patterns emerge. After 20 sessions it surfaces a Measurement Recommendation: specific evaluation dimensions and calibration scenarios matched to your agent's actual behaviour. You review it and confirm. Scoring only begins after a human approves.
Edge cases & adversarial inputs
The inputs most likely to break your agent: malformed data, boundary conditions, error states, prompt injection attempts. AgentScore identifies these from patterns in your observed failures.
Normal, expected inputs
The baseline your agent should handle reliably. AgentScore calibrates these from the happy-path sessions it has already observed — the inputs your agent actually sees in production.
Stretch goals
The scenarios that define what excellent looks like for your agent type. Not required to ship, but tracked — so you know whether quality is improving over time.
Human confirmation required
AgentScore generates the measurement recommendation — but scoring never starts until a practitioner reviews and confirms it. You stay in control of what gets measured and why.
HTML widget
How it works
Connect once. AgentScore takes it from there.
No eval expertise required. AgentScore observes first, then tells you what to measure — based on how your agent actually behaves.
01
Connect
Instrument your agent with OpenTelemetry. Traces flow to AgentScore automatically. No changes to your agent's code.
→
02
Observe
AgentScore watches 20+ sessions in shadow mode — tool calls, retrieval patterns, where it struggles — and builds a Measurement Recommendation from what it finds.
→
03
Confirm
You review the proposed dimensions and calibration scenarios. Confirm what fits. Scoring only begins after a practitioner approves the design.
→
04
Score & act
LLM judges evaluate every session and produce a composite 0–100. Ship, Review, or Block — with full attribution when the score falls.
HTML widget
Capabilities
From "I don't know what to measure"
to a score your team can act on.
AgentScore handles the hard parts — eval design, scoring, regression tracking, and attribution — so practitioners can focus on the domain work they actually know.
Observation-based eval design
AgentScore watches your agent in shadow mode and generates a Measurement Recommendation — specific dimensions and calibration scenarios based on what it actually observed. No eval expertise required.
Composite scoring
A single 0–100 score and A–F grade per run, derived as a weighted average across enabled quality dimensions. Interpretable without an AI background. Marked Preliminary until 30 sessions and 3 runs complete.
Regression detection
Score delta vs. the prior run is surfaced for every dimension. Significant drops are flagged so your team catches regressions before practitioners notice anything is wrong.
Root cause attribution
When a score falls, attribution narrows it to the specific dimension, session, and tool call chain — with root cause category, confidence score, and remediation recommendations.
Scoring profiles
Named, versioned profiles define which dimensions to score, their weights, and verdict band thresholds. Shared across agents of the same type. Historical runs are re-graded when a profile changes.
Runtime Guard (roadmap)
Pre-tool-use safety evaluation designed to catch prompt injection, credential exposure, PII leaks, and path violations — with logged, searchable guard decisions.
HTML widget
Quality dimensions
11 dimensions. Every angle covered.
Each scoring profile activates the dimensions relevant to the agent type. All 11 are available; none are required. Profiles are versioned so history stays comparable.
Core
Correctness
Does the agent get the right answer on its primary task, across standard and adversarial inputs?
Core
Efficiency
Does it deliver value per token spend — and is that ratio trending up or down since the last run?
Core
Relevance
Is the output clear and useful to the practitioner who has to act on it — not just technically correct?
Core
Safety
Does the agent avoid harmful, unsafe, or policy-violating actions — prompt injection, credential exposure, PII leaks?
Core
Consistency
Does it produce equivalent answers when the same task arrives in different formats or phrasings?
Core
Tool Use
Does it choose the right tools, use them efficiently, and recover cleanly when a tool call fails?
Extended
Groundedness
Does every factual claim trace back to an explicit tool response or input artifact?
Extended
Instruction Following
Does the agent execute instructions precisely — including edge cases and underspecified inputs?
Extended
Transparency
Does it explain its reasoning in a way that gives practitioners visibility into how it reached an answer?
Extended
Robustness
Does it handle adversarial inputs, malformed data, and prompt injection without failing silently?
Extended
Communication
Is the output language appropriate for the practitioner's expertise level — not just technically accurate?
Ship
Score ≥ 85
Agent is performing at or above threshold. Cleared for the next deployment gate.
Review
55 ≤ Score < 85
Performance is acceptable but not strong. Human review recommended before shipping.
Block
Score < 55
Agent is underperforming against its profile. Remediation required before next run.
HTML widget
Use cases
Four questions practitioners ask every day.
AgentScore is built around real decisions — not abstract metrics. Here's how it plays out in practice.
PK
Priya · QA Engineer
Financial services · ATC agent
"Can I trust this agent to generate my test cases?"
After two weeks of shadow mode, AgentScore surfaces a Measurement Recommendation: the agent's outputs cluster around the happy path — nightmare coverage is below 10%. Priya confirms the gap, and scoring now tracks the edge conditions she was worried about.
She has a clear answer: strong on happy-path generation, unreliable on error states. She knows exactly what to use it for and what to review manually.
MR
Marco · Test Lead
Enterprise · ATA regression suite
"Something changed — is the agent still working?"
Two sessions came back PARTIAL this week. Attribution narrows it to a specific module: new test data format triggered the wrong tool selection, causing downstream assertion failures. The call chain shows exactly where it went wrong.
Marco files a one-line ticket. The AI Ops engineer has a fix in the next run — no week-long debugging session required.
AR
Aisha · AI Lead
Platform team · LLM upgrade
"We upgraded the model — did it get better or worse?"
Upgrade from Claude Sonnet 4.5 to 4.6. Run comparison: composite moved from 74 to 81. Correctness up 12 points. Relevance dropped 3 — the new model is more verbose, which the Relevance eval penalises. Delta is positive overall.
Aisha approves the upgrade and notes the verbosity signal for the next calibration review. Decision backed by data, not intuition.
LN
Lena · Product Lead
Support team · CURA diagnostic agent
"Is the agent ready to go from shadow to production?"
6 weeks, 4 scoring runs. Composite 79, grade C, verdict Review. Correctness is strong (88) but Relevance drags the composite (61). Top failing dimension: response clarity — correct, but written for experts, not frontline support staff.
Lena has a specific remediation target: rewrite the system prompt to simplify output language. Clear criteria for the go-live decision.
HTML widget
FAQ
Answers to what teams ask first.
The questions that come up most when teams look at AgentScore for the first time.
Product & positioning
What is AgentScore?+
Most teams can prove their tests are solid and their code coverage is high — but that doesn't answer whether the agent doing the work is actually good at its job. AgentScore ingests your agent's traces and builds a scoring profile across a set of quality dimensions, giving you a clear ship/review/block verdict and flagging drift the moment something regresses.
How does this fit with AI Workspace?+
AgentScore stands on its own, and AI Workspace is the fastest path to get it in front of users today — if you're building an agent there, you get scoring with almost no setup. Over time it can grow into its own dedicated surface alongside AI Workspace, with results feeding into the broader Tricentis platform story as that takes shape.
Who it's for
Who is AgentScore built for?+
Today, the engineering or team lead responsible for an agent, who needs a fast answer to "did this change make it better or worse?" Longer term, the target is the domain expert — someone who understands their field deeply, like a tester or a legal reviewer, and can build or guide an agent without needing to be an AI engineer to know if it's working.
Do I have to use AI Workspace to use AgentScore?+
Not by design. AgentScore connects through a standard OpenTelemetry exporter, so any agent that emits traces can plug in, regardless of platform or model. AI Workspace is simply the first place it's available.
What are teams doing today instead?+
Mostly nothing structured. AI-savvy teams hand-roll their own evals; everyone else relies on spot checks, an ad hoc LLM-as-judge, or plain intuition. AgentScore replaces "we're going on vibes" with a repeatable, comparable signal.
How it works
What's the difference between a trace and a scoring run?+
A trace is a single end-to-end interaction with your agent — every tool call and span it took to get there. AgentScore ingests traces continuously. A scoring run is the periodic evaluation step that analyzes a batch of collected traces and produces a composite score, so you get a meaningful signal without paying to evaluate every single interaction.
What does a scoring run actually do?+
Based on your agent's inputs, outputs, and tool calls, AgentScore automatically determines which evals apply and runs them — then returns a composite 0–100 score, a letter grade, and a plain-language ship/review/block verdict, with a full breakdown of why.
What's actually novel here, if the underlying evals already exist?+
The individual evals aren't the hard part — many are well understood and open source. The hard part is automatically figuring out which evals apply to a given agent based on how it actually behaves, running the right ones, and rolling the result into a single, trustworthy score.
If an agent isn't ready to ship, does AgentScore tell me why?+
Yes. When a run falls short, an attribution pass pinpoints the exact trace and root cause — a credential exposure, a bad tool call, a specific failure category — and recommends what to fix. Fix it, re-run, and confirm the score clears the bar.
Pricing & fit
How is AgentScore priced?+
Seats are free, and so is trace ingestion — pricing shouldn't discourage adoption or thorough instrumentation. You're charged per completed scoring run, so you only pay when you get a usable result back.
Does it replace tools like Datadog or NeoLoad?+
No — it's complementary. If you're already doing AI monitoring or trace collection elsewhere, AgentScore layers on top of that instrumentation rather than replacing it.
HTML widget
Phase 1 · Internal pilot
Stop guessing what to test.
Let AgentScore figure it out.
Phase 1 is live for internal Tricentis teams. Connect your agent, and AgentScore will tell you what to evaluate — based on what it observes. No eval expertise required to get started.
Request accessPhase 2 (closed external beta) planned for Q4 2026
HTML widget
⚠
Disclaimer
Tricentis Labs showcases products and capabilities under development for exploratory and informational purposes only. The features and capabilities depicted do not constitute a commitment to release, commercialize, or deliver any feature or product. No products or features shown are currently available for purchase, and Tricentis makes no commitment regarding future commercial availability. AI capabilities depicted are illustrative and actual performance may vary. This content should not be relied upon in making purchasing decisions and may not be incorporated into any contract. All details are subject to change at Tricentis's sole discretion.
HTML widget
Join 80,000+ testers learning AI-powered quality engineering
Learn & Certification
Tricentis Products
Events & Webinars
Participate
Copyright ©2026 Tricentis. All Rights Reserved.
Login to the community
No account yet? Create an account
Enter your E-mail address. We'll send you an e-mail with instructions to reset your password.