AI agent testing & release assurance

Release agents with confidence

DialogAssert replays your customer journeys against each new agent version, checks the tool calls and the conversation, and shows what regressed since the baseline.

See a sample report

The problem

Another agent change. Another round of manual testing.

A prompt, model or tool change means checking your critical customer journeys again. DialogAssert makes those journeys repeatable and brings the results and the evidence behind them into one report.

Today, for every release, someone has to

  1. Run the conversations again
  2. Check which tools the agent called
  3. Investigate what failed and why
  4. Assemble the evidence for sign-off

How it works

Write it. Check it. Compare it.

  1. 01

    Write

    Business teams write the customer journey and what must happen, in plain language.

    • Agent shows the customer their cards before asking which to block.
    • Agent asks for confirmation before blocking.
  2. 02

    Check

    Rule-based controls assert tool calls. LLM-judged controls assess the conversation. Every result says which.

    • Rule-basedCalls block_card exactly once
    • LLM-judgedAsks the customer to confirm the block
  3. 03

    Compare

    Each new version is compared with its baseline against your release policy.

    Baseline
    8 of 8 passed
    Candidate
    6 of 8 passed
    Release policy
    Blocked

From a run to a decision

One report.
One release decision.

See what failed, what changed since the baseline, and the evidence behind each finding.

  • Frozen run configuration
  • 3/3 turns traced
  • Artifact integrity verified
  • Export JSON and HTML
Explore the sample reports →
Release reviewSample comparison

South River Banking Agent
demo-v1-confirmation → demo-v2-no-confirmation

Release blocked

2 new findings
Journey outcome
Passed → Failed
Controls passed
8 of 8 → 6 of 8
Resolved findings
0
  • Does not block the card until the customer confirmsRule-basedHigh
  • Asks the customer to confirm before blockingLLM-judgedHigh
Comparable · integrity verified on both runs

A little more clarity

Good questions.

How is this different from evals?

Most evals score the final answer. DialogAssert also checks what the agent did to get there: which tools it called, in what order, on which record, and what it asked first. Each finding comes with the transcript and tool evidence behind it.

What are rule-based and LLM-judged checks?

Rule-based controls are exact assertions on tool calls: which tool, how many times, in what order, before which turn. LLM-judged controls assess what was said against your acceptance criteria. Every control in a report is labelled, so you can see which kind produced a finding.

Do we need to rewrite our agent?

Not for supported interfaces. DialogAssert connects through an adapter config for the OpenAI Agents SDK, LangChain or a generic HTTP endpoint, and evaluates the agent as it is. Some checks need access to tool calls or traces, so we confirm what your environment exposes before a pilot.

Who writes the tests?

Business teams write the customer journey and acceptance criteria. Engineering provides the agent endpoint or adapter, plus optional trajectory checks. DialogAssert compiles the tests, runs them and writes the report.

Does DialogAssert certify compliance?

No. It reports evidence of what an agent did against the process you defined. It does not assert that an agent complies with any regulation.

A measured pilot

Measure the value on one journey.

Start with one non-production agent, one customer journey and three to five acceptance requirements. Run DialogAssert alongside your current testing and compare the two.

What we compare

  • Hands-on time to prepare, run and review tests
  • Time to investigate a failed requirement
  • Known regressions caught and missed
  • False alarms and evidence gaps

Integration and maintenance effort count too.