From agent execution
to verified behavior.

def record(name: str, inputs: dict, outputs: dict, milliseconds: int) -> None:
    tools.append(Span(f"{root_id}/{len(tools) + 1}", root_id, name,
                      len(tools) + 1, inputs, outputs, milliseconds))

# ... replay the synthetic tool actions ...
root = Span(root_id, None, "support_request", 0,
            {"case_id": case.case_id, "request": case.prompt}, {"response": response},
            sum(span.synthetic_ms for span in tools))
return Run(case.case_id, version, response, claims, (root, *tools), state)

Selected lines from replay(): record() links tool spans to the root request, while Run preserves both the response and the resulting state. The full function is linked below. Timing fields in this fixture are invented, not performance measurements.

observations[run.case_id] = run
if set(expectations) != set(observations):
    raise ValueError("Observed runs must cover exactly the expected case identities")
covered = {case.requirement for case in expectations.values()}
missing = REQUIRED_COVERAGE - covered
if missing:
    raise ValueError(f"Missing requirement coverage: {sorted(missing)}")
return {key: BenchmarkEntry(case, observations[key])
        for key, case in expectations.items()}

Selected lines from build_benchmark(): join by case identity and check declared coverage. BenchmarkEntry retains the source Run and exposes its root span ID and response. The full function also rejects duplicate IDs and malformed root spans.

def grounding_score(run: Run) -> bool:
    """Exact structured-fact check; not a general natural-language judge."""
    evidence = {}
    for span in run.tools:
        evidence.update(span.outputs)
    return all(key in evidence and evidence[key] == value
               for key, value in run.claims.items())

The companion compares structured claims with tool outputs. Its separate judge_agreement() function aligns four fixed synthetic label fixtures and computes accuracy and Cohen’s kappa; it does not call an LLM or validate natural-language answers.

identities = [span.sequence for span in tools
              if span.name == "verify_identity" and span.outputs.get("verified") is True]
refunds = [span for span in tools if span.name == "issue_refund"]
ordering = all(any(verified < refund.sequence for verified in identities) for refund in refunds)
matching_refunds = [row for row in run.state["refunds"]
                    if row["order_id"] == case.order_id and row["refunded_amount"] == case.amount]
if case.requirement == "complete_refund":
    outcome = bool(matching_refunds)

Selected lines from evaluate_run(): ordering inspects earlier identity events; outcome inspects the retained refund state. Empty refund calls can pass the ordering rule, so a separate selection/outcome failure is essential.

paired = {case_id: {"baseline": baseline[case_id], "candidate": candidate[case_id]}
          for case_id in baseline}
layer_regressions = {
    case_id: [layer for layer in PROTECTED_LAYERS
              if pair["baseline"][layer] and not pair["candidate"][layer]]
    for case_id, pair in paired.items()
}
layer_regressions = {key: layers for key, layers in layer_regressions.items() if layers}
return {"cases": len(paired), "paired": paired, "regressions": list(layer_regressions),
        "layer_regressions": layer_regressions, "gate_passed": not layer_regressions,
        "baseline_outcomes": sum(row["outcome"] for row in baseline.values()),
        "candidate_outcomes": sum(row["outcome"] for row in candidate.values())}

Selected lines from compare_versions(): pair stable keys and flag any true→false change across all six protected layers. The full function rejects missing cases or malformed score vectors. Deliberately reversed candidate output order does not change the result.