01 · A regression check

One feature, two versions, the same 40 cases.

Synthetic example. The tickets and the router outputs are invented; the scores, the verdict and the exit code come from the released CLI. Run it yourself →

REGRESSION CHECK · SYNTHETIC EXAMPLEticket-router · v1 → v2 · 40 cases

Did not meet the configured requirementsredcrown check · exit 2

Feature: Support-ticket router: billing, technical, account, or shipping
Compared: Reference run v1 against candidate v2, the same 40 labeled cases
Method: Exact match against the labels; policy applied by redcrown check

Requirements, as the check reported them
Requirement	Observed	Status
pass: At least 40 cases	observed 40	pass
pass: No case errored	observed 0.0	pass
pass: No case left ungraded	observed 0.0	pass
pass: Configuration declared	observed yes	pass
fail: Exact match at least 0.95	observed 0.9	fail
fail: Regression at most 0.05	observed 0.1	fail

Changed cases: 4 of 40 (0 excluded)
Case	Label	v1	v2
case-07	account	account	billing
case-18	technical	technical	account
case-25	billing	billing	technical
case-36	shipping	shipping	account

Invented tickets and outputs. No model ran. The scores and the verdict come from the released CLI.Run it yourself →

Terminal output

$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40
$ echo $?
2

02 · What changed?

Find the cases that changed, not just the average score.

Compare compatible results and inspect the checks or items that moved. Keep failures, ungraded items, and missing evidence visible so a clean-looking average does not hide the question you need to investigate.

What you can see depends on the evidence you bring. Scored outputs give per-case scores on both sides, so the changed cases and the excluded ones can be listed from the evidence file. A test harness journal gives per-check statuses, so redcrown harness compare lists what regressed, what was fixed, and what could not be measured, as in the run beside this. An aggregate import carries summary scores only, and nothing on the record pretends otherwise.

redcrown harness compare · two runs, nine days apart

corpus drift: base 661d9f3c1725 vs head d978855c0393; shared cases only
cases: regressed 3, fixed 1, inconclusive 0, newly measured 6, unchanged 15, only in base 0, only in head 0
  accurate                 regressed   4  fixed   1  inconclusive   0  newly measured   0
  held_everything          regressed   3  fixed   2  inconclusive   0  newly measured   0
  persisted                regressed   1  fixed   3  inconclusive   0  newly measured   0
  recallable               regressed   0  fixed   0  inconclusive   3  newly measured   1

regressed (8):
  case-1  accurate: ok -> fail
  case-2  accurate: ok -> fail
  case-3  accurate: ok -> fail
  case-4  accurate: ok -> fail
  case-5  held_everything: ok -> fail
  case-5  persisted: ok -> fail
  case-6  held_everything: ok -> fail
  case-7  held_everything: ok -> fail

[fixed (6), inconclusive (3) and newly measured (1) omitted here]

exit 1

From a client engagement, anonymized and abbreviated: case ids replaced by case-1 to case-7, the fixed and unmeasured lines left out. 25 cases, 4 checks, a harness the client's own test runner produced. The hand-written report said 3 to 10 passes; the tool listed 8 regressions across 7 cases.

03 · How it works

From requirements to a reviewable result.

Four steps. Each one is a command or a screen that exists today; the scope table below says which surface carries it.

STEP 1

Define the check.

Choose the cases, references, and requirements appropriate to the supported workflow. A policy is a JSON file with the thresholds written down before anyone looks at the candidate.

STEP 2

Establish the comparison.

Use a reference run, or explicitly accept an eligible baseline in the app where that is supported. Nothing is accepted for you.

STEP 3

Check the change.

Evaluate the candidate and inspect unmet requirements or missing evidence. redcrown check exits 0 on pass and 2 on fail or insufficient evidence.

STEP 4

Review the result.

Share the supported report or export, keeping its limits attached. A shared report is a frozen copy of one run with the caveats it carried.

Your application
Produces the outputsRuns the feature, or your test harness does. RedCrown does not run your app.
A model provider
Answers the callsWhen RedCrown runs an evaluation for you, the calls go to the provider on your keys and your rates. When you bring outputs, no provider is called.
RedCrown
Scores and checksScores outputs against your references, applies the policy, and keeps the record: cases, counts, exclusions, versions, caveats.
Your pipeline
Acts on the exit codeCI keeps the status. Whether a failed check blocks a release is your pipeline's rule, not RedCrown's.
A person
DecidesReads the shared report, casts a blinded review where a metric cannot decide, or accepts a baseline in the app. RedCrown records; it does not approve.

04 · Your existing tools

Keep the workflow that produces your outputs.

Score supplied outputs, compare supported harness runs, or run a supported evaluation with RedCrown. The input you bring determines which checks and evidence are available.

Bring outputs

Score supplied outputs

Inputs, correct answers, and the outputs your feature produced. Deterministic scoring, no provider key, no model call. RedCrown scored what you supplied; it did not see how the outputs were generated.

Quickstart, offline →

Bring a harness

Compare harness runs

A journal your test runner already writes: one directory per run, one line per case, named checks that are ok, fail, or unmeasured. RedCrown reports and compares; your runner executes.

Journal format and commands →

Run it here

Run an evaluation with RedCrown

Give the task, the cases and the references, and RedCrown calls the models on your keys, scores every output, and keeps the receipts. Repeat on demand against the same cases.

Native evaluation guide →

05 · The handoff

Give the decision-maker evidence they can inspect.

Make the tested cases, scoring method, results, and limitations visible. Share the available evidence instead of asking someone to rely on a screenshot or an unexplained score. Three different things can happen to a result, and the product keeps them apart.

Read the evidence

A shared reportA frozen copy of one run: every case, the score, the caveats, the versions, signed. Open one: the SEC extraction record, a public benchmark run, no login. A report shares a run; a policy result from your CI stays with your CI.

Submit a review

A blinded verdictIllustration of the reviewer page. A reviewer opens a link, sees outputs without model names, and casts a verdict that joins the record as a review. It is a review, not an approval. How review works →

Accept a baseline

A recorded decisionIn the app, the owner of an eligible completed run can accept one candidate as the baseline. The acceptance and any revocation are kept with their history. It records that you stand behind a configuration; it deploys nothing and authorizes nothing. Eligibility and controls →

06 · Scope and limits

Clear about what the check does, and what it does not.

Checks run when you run them: on demand, or in the pipeline you configure. Passing the configured checks means the candidate met the requirements you wrote, on the cases you supplied. It is not a statement about production reliability. Each row below is a path that exists in the released product, with the surface that carries it.

What each path supports
Path	You bring	You get	Where	Limits
Score supplied outputs	Cases with references and the outputs your feature produced (CSV and JSONL, or one request file)	A scored report per candidate with per-case receipts; a private evidence file for the offline check	CLI `score`, offline. App: Advanced → Upload. API.	Text in, text out, one step. Metrics: exact match, similarity, JSON field accuracy, F1. Declared configuration, not witnessed. In the app, pick a new feature or add a version to an existing one.
Offline policy check	Two evidence files for the same feature and cases, plus a policy JSON	pass, fail or indeterminate per rule; exit 0, 2 or 1; changed cases from the evidence file	CLI `check`, offline, no login	Evaluates existing evidence only. Cost and latency rules parse but stay indeterminate. Not a hosted acceptance.
Compare harness runs	A journal directory per run: `meta.json` and `verdicts.jsonl` with named checks	Per check: regressed, fixed, inconclusive, newly measured; exit 1 on a regression, 2 on corpus drift	CLI `harness report`, `compare`, `export`; pushed runs show their checks in the app	Your runner executes the cases. An unmeasured check is never a pass or a fail.
Run an evaluation	A task, cases, references or an incumbent, and your provider keys	A ranked report with receipts, cost and latency per candidate	App (Prove, Advanced), CLI `eval`, API, MCP	Calls go to the providers on your keys. Only label-referenced, single-step, single-pass text runs on the four metrics above can be accepted as a baseline.
Accept a baseline	An eligible completed run in your workspace	A stored acceptance with history; later runs of the same feature compared against it	App: Proofs → Baseline decisions. API.	Not in the CLI or MCP. Refused for runs with errors, ungraded items or a blocking caveat. A scored version needs a declared provider and model on each candidate to be compared or accepted.
Import a summary	Aggregate scores from another system	A run with summary scores and whatever receipts the payload carried	CLI `push` and `import-results`, MCP `import_results`	No evaluation identity, so no acceptance or comparison. Case-level evidence only if you supplied it.
Share a report	A completed run	A frozen public or unlisted page with the disclosure level you choose; revocable	App, CLI `push --proof-link`, MCP	Shares a run. A local policy result is not part of it. Revocation stops the link; it cannot recall a saved copy.
Proxy capture and replay	An endpoint you point at RedCrown's proxy	Captured calls you can replay through an evaluation	API and MCP setup; Growth and above	A separate advanced workflow. Nothing runs on a schedule.

Not built, and this page says soRoot-cause tracing (why an output changed), scheduled re-runs, alerting, approval workflows, and monitoring. RedCrown names the case that moved and the check it failed. It does not tell you why.

What a retired model looks like hereA model id the provider reports as not found on every item of a run is taken off the Face-Off roster for seven days, by name. A model that errored on every call stays listed, un-ticked, with the provider's error beside it. Nothing takes its place without a notice. Re-run on demand against the same cases.

A signature is a signatureA signed attestation shows a record was issued by RedCrown and was not altered since. It does not show that the supplied cases represent production, or that a declared configuration was the one that ran.

Your data, by pathThe offline CLI sends nothing anywhere. A native evaluation sends requests to the providers you configure. Uploading a run and creating a share link are two separate, deliberate actions. Data flow by path →

07 · Also supported: the model decision

Which model, at your volume, at what cost?

The same task across several models and providers, ranked on the quality bar you set, priced at your volume. The record beside this ran one extraction task across eleven frontier models and named the cheapest one that clears the bar. It is a public benchmark run, not a customer result. Open it and check the receipts.

Open the record → Compare models on your workload →

DECISION · RC-4627B127SEC 10-Q extraction · threshold 0.99

Selected

deepseek · v4-flash

✓ clears threshold at 0.9969

−98%

$613.35 → $6.30
per 10,000 filings

✓ deepseek · v4-flash

google · gemini-3.7-flash

80 fresh 2026 10-Q filings, answer key from SEC XBRL

Method

Field F1 over 8 facts, 11 models

Attested

v1 · 2026-09-10
sha256 13e6…dcce

More model-comparison records

Each tile is a real run on synthetic or public data with its sample size and caveats stated on the record. The publish gate refuses a number it cannot stand behind; it exists because our own benchmark hit five configuration mistakes that looked like model failures.

Synthetic or public data, small samples, caveats on every record.
All example records →

TranscriptionNormalized WER

whisper · large-v3-turbo

✓ accuracy 0.94, level with the incumbent

−49%

✓ large-v3-turbo0.94

large-v3 (incumbent)0.94

8 synthetic clinical clipsreceipts →

SummarizationLLM judge

qwen3 · 235b

✓ judge score 0.78, above the incumbent

−99%