Tests, AI evaluations and evidence

One end-to-end acceptance path, independent oracles and separate measures of correctness, interpretation and usefulness.

One acceptance path, three evidence layers

Layer	Oracle	Failure cannot be hidden by
Data and contract correctness	Schema, source/target identity, exact plan digest, count reconciliation, ownership and ledger assertions.	An attractive report, high AI score or large unit-test count.
Analysis quality where AI is active	Reviewed gold cases, claim-to-source entailment, uncertainty calibration, contradiction handling and representative slices.	The model judging its own output or a deterministic schema pass.
Report and operator outcome	A person can reload the result, trace a conclusion, see missing data and understand final charges.	A successful agent run, generated docs or screenshots without the underlying result.

Acceptance and failure matrix

Scenario	Required oracle and artifact	Pass boundary
Full selected-plan journey	Captured quote → approval → queue → final report → reload → export; same run and source manifest.	All selected groups accounted for, all supported results complete or honestly terminal; final success requires complete requested coverage.
Changed scope and identity	Alter market, rival binding, quantity, source digest or price after quoting.	Dispatch is refused before reserve/purchase; no old approval carries forward.
Deep-size persistence	Replay representative provider envelopes through the real queue/storage seam with measured bytes and timings.	The pre-fix failure is reproducible; repaired full-size data survives restart with no evidence loss or duplicate purchase.
Ambiguous provider success	Inject failure after provider acceptance but before receipt commit.	Uncertain cost remains held; recovery uses owned task evidence and cannot repurchase silently.
Concurrency and lost lease	Two claimants, stale checkpoint writer, closed browser and restarted driver.	One valid owner and one purchase; stale writer rejected; progress remains truthful.
Permission and privacy	Wrong account/scopes, cross-owner run/brand references, raw HTML and signed URLs.	Zero cross-owner data access or private credential leakage; explicit denied states.
Unavailable and empty data	Zero rows, sparse domain, timeout, malformed response, failed module and parked module.	Absent, zero, partial, failed and skipped remain distinct in API, report and export.
Cost and credit reconciliation	Per-request receipts, immediate balances, reservation/debit/refund ledger and archived run.	Exact arithmetic agrees; any unexplained delta blocks reconciled status.
Regression and export drift	Run against held-out representative cases and reload/export after source change.	No critical invariant regresses; affected proof is rerun against the changed source.

Current no-network journey evidence: tests/selected-plan-e2e.test.mjs drives one authenticated session through the actual Next quote route, exact approval and reservation, the fenced multi-slice queue, static public-HTML collection, the real DataForSEO technologies adapter against a controlled provider fixture, terminal receipt and one-credit ledger debit, owner-scoped report reload, brand-presence projection and ZIP export. It asserts the same plan hash at quote, stored execution and reload; one $0.012 provider purchase with an owned task receipt; matching $40.000 to $39.988 balance movement labeled as shared-account observation; all selected coverage passed; and exactly one customer spend row. Supabase, public HTML, provider response and balances are controlled fixtures, so this is coherent deterministic integration evidence, not live database, provider or funded acceptance.

Existing modules include src/eval/spec-checks.mjs, section-quality.mjs, analysis-quality.mjs, run-quality.mjs, reviewer-checks.mjs, llm-judge.mjs and check-registry.mjs. scripts/eval-suite.mjs uses gold specs and stored runs. Their existence does not establish sufficient coverage of the current selected-plan workflow.

The eval suite now selects the most recent run for a domain regardless of outcome, fails when that latest run is not passed, and only accepts passed runs as baselines. A failed or partial current run therefore cannot be hidden behind an older passing sample. The section regression threshold remains one point. Alaris has a versioned deterministic case for reviewed identity, market, domain hygiene, parked lanes and historical ceilings, but it intentionally asserts no palette, social roster or AI-quality score before a complete result exists; the suite currently reports no_run_available for Alaris.

run-quality measures wall clock separately from overlapping stage durations and requires measured speed baselines. An absent or weak baseline must remain unrated/advisory; never invent a latency target or average speed into quality.

Audit the existing no-paid-network guard for transport completeness. A global fetch denylist alone does not prove all HTTP clients, subprocesses or newly added providers are prevented from reaching the network.

AI evaluation design

First inventory the AI calls that actually execute for supported plans: input evidence, prompt/model/version, output schema, allowed inference, token and cost limits, traces and failure fallback. The engineering agents are separate from production report AI.

Create versioned, privacy-safe gold cases with source snapshots, expected facts, prohibited claims, acceptable uncertainty and human-reviewed rationale. Include incorrect entity matches, conflicting sources, sparse and stale evidence, ambiguous competitors and injected instructions in retrieved content.

Keep training/development cases separate from held-out evaluation. Record grader identity, rubric version and disagreement handling. Repeated cases should measure variability; model self-grading is not independent proof.

Set proposed thresholds before grading, justify them by risk and distinguish them from measured baselines. Zero unauthorized spend, privacy leaks, wrong-entity evidence and unsupported factual claims are hard invariants. Semantic usefulness and completeness need separate per-slice thresholds and human calibration.

Use existing redacted trace exporters when supported and approved. Store run/prompt/model/rubric/source digests and latency/token/cost evidence locally even if external trace upload is unavailable. Do not enable parked production AI merely to fill an evaluation score.

Evidence binding

Every receipt must identify exercised source hashes, environment and database/schema version, input fixture or capture digest, exact command/action, timestamps, oracle, result and verification limits.

A distinct verifier reruns the reproduction independently and signs the same manifest. Edited sources invalidate affected acceptance; passing historical receipts remain history.

Do not update baselines to turn a regression green. Preserve failed runs and explain any approved baseline revision with the previous values and its evidence.

Lead-review additions to the failure matrix

Case	Required assertion	Evidence boundary
Crash after provider success before checkpoint	No repeat purchase; unknown attempt remains held until owned-task/request evidence resolves it.	Prefer provider-supported idempotency. Do not assume arbitrary keys are supported; record request identity and refusal behavior.
Price/rate/source version changes after quote	Execution refuses the stale plan or requires a fresh confirmed quote; no hidden cost increase.	Exercise public quote-to-execution boundary with recorded price versions.
Large vs small checkpoint; slow read vs slow write	Measure byte count and elapsed time separately; distinguish lock/index/query/network causes before selecting a fix.	Isolated local database or approved staging only; no unapproved writes to live storage.
Ledger RPC fails or returns incomplete pages	Explicit error/unknown, never empty-success or zero balance.	Test positive retrieval and missing/inconsistent response shapes.
Mixed known and unknown provider costs	Report, export and reload preserve uncertainty and do not publish a false actual total.	Golden receipt oracle separates estimates, ceilings, holds and actual cost.