End-to-end testing for the agentic era.

01State

Our model of your systems holds state, so a read reflects a write the agent just made.

the problem

An agent writes a record on turn two and reads it back on turn five. A mock returns a fixed value regardless of what came before, and a recorded fixture returns what happened in a different run — so the second read is meaningless, and multi-step tasks are where agents actually fail.

Checks / trajectory3 calls · escaped 0

refund-retry-reads-back

The agent writes a settlement on turn two, then reads it back on turn five. Both calls resolve against the same simulated world.

TurnTool callSourceTime

2refund_create {"request":"rq_4417"}simulated5.0s

3ticket_note {"ticket":"tk_204"}simulated0.2s

5refund_lookup → {"count":1}consistent0.4s

A mock would answer turn five without ever having seen turn two.

02Non-determinism

Behavior is a distribution, and one execution is a single sample of it. Seeing that distribution before you deploy means running a large suite of scenarios and invariants, at volume and repeatedly.

We think any team shipping an agent that takes real actions should be doing this, and most still are not.

the problem

An agent given the same input twice may take a different route through it. That property is intrinsic to how these systems work, not a defect that a better prompt removes. Scoring a suite once, or gating on a threshold, measures the noise as often as the change.

Checks / variancerepeats 5 · base agent · nothing changed

order-lifecycle — base agent, five repeats

The same agent, unchanged, run five times against every scenario in the suite.

ScenarioRepeats

cancel-open-order5/5 · stable

refund-retry-reads-back5/5 · stable

order-status-summary5/5 · stable

search-desk-lamps4/5 · in variance

add-lamp-to-cart2/5 · in variance

Nothing changed between these runs. Two scenarios still disagree with themselves, and they get stabilized or retired before any threshold is set.

03The agent’s account

We keep the calls the agent actually made, what they returned, and what changed in your systems — then grade its own account of the run against all three.

That same record also shows how the work was done, so repeat calls, loops and dead ends surface on their own. What it never does is score that route against one you wrote in advance, since two agents can be equally efficient by different paths.

the problem

A transcript is the agent’s own account of what it did, and agents misreport. “I’ve processed your refund” is a claim, not a fact, and a judge reading only the conversation has no way to tell the difference.

The usual alternative is to check the path against a sequence written out in advance. That costs real work to specify, and it still fails agents that reached the right outcome by a different route.

Checks / trajectory2 calls · escaped 0

cart-add-has-no-cart-tool

The agent’s own account of the run, held against the calls it made and the state they left behind.

SourceRecordVerdict

said“I attempted to add the Aurora Desk Lamp to your cart”contradicted

didsearch_products, get_cart — no write issued2 calls

worldcart 1001 — items unchangedverified

No such call was made and the cart is untouched. The run is red on the contradiction, not on the route it took.

04Defining correct

Most of the value is in what must never be true — money never moves twice for the same request, no record changes for a customer other than the one in the request.

Your team already knows these, and they survive changes to your implementation. The rest comes free from the comparison against the previous commit: if it left one refund and yours leaves two, that is a regression whether or not anyone wrote the rule down.

the problem

Writing out the correct end state for every scenario would be an enormous amount of work, and most checks do not need it.

Library / scorersengine constraint · v1

no-double-settlement

Money never moves twice for the same request, however many times the agent retries.

Bindings

128

scenarios that bind this scorer

Upheld

127

across the last check

Violated

1

rq_8841

Written once, bound to a hundred and twenty-eight scenarios, and unchanged by any rewrite of the implementation underneath it.

05Attribution

A run can fail at the model, the instructions, a tool, the integration beneath it, the environment or the input. We diagnose which of those it was, so a red check tells you where to look.

the problem

A score tells you a run failed, not what failed. Everything after that is manual: re-running the check, adding logging, and working backwards until the cause turns up.

Fleet / findings10 open · 3 this week

Findings

Recurring failure patterns the platform grouped from this agent’s checks. Each one carries the check that triggered it.

SeverityFindingSourceTrigger

criticalover-completion — agent did not refuse where refusal was expectedinstructionschk_61652

majortask incomplete with memory inconsistencytoolchk_82180

minorconfirmation not requested before writeinstructionschk_43668

Each finding names what produced it and the check it came from, so nothing has to be inferred.

06Judgment

Deterministic checks resolve first, a model handles only what genuinely remains open, and its output is recorded as a judgment rather than as a fact.

Both kinds of answer belong in an evaluation. The error is recording one as the other.

the problem

Some questions have definite answers and some do not. Whether exactly one refund exists is a lookup. Whether an explanation was adequate is not.

Checks / scorers31 on this check

settlement-integrity

Deterministic scorers resolve first. Only what genuinely remains open is passed to a model, and it is filed differently.

VerdictScorerEngine

upheldExactly one settlement exists for rq_9207constraint

upheldNo writes outside the requested customerconstraint

violatedThe refusal was explained adequatelybehavior

Twenty-seven resolved on the constraint engine and are recorded as fact. Four reached behavior and are recorded as judgment.

Validate agent behavior, before you deploy.

We model the world your agent acts on, and grade what it did to that world.

Six problems everyone hits, and how we handle each one.

Measuring agents, not models

Alongside the tools you already run, not in place of them.

Dystopic works on agents that take actions you cannot take back.