Check your first AI change
Check whether a changed AI feature still meets agreed requirements, and see the evidence behind the result. This page runs a synthetic regression example on your own machine: one feature, four versions, the same 40 cases, one written policy. Every command runs offline. It needs no account and no provider key. Pick CLI above. The UI and MCP surfaces say what they do and do not carry.
Looking for model comparison? Running several models on one task and ranking them is a separate guide: Compare models on your workload.
What the example is. The tickets and the router outputs are invented. No model ran. The scores, the verdicts and the exit codes come from the released CLI, and the site build re-runs every command on this page before it can publish. A scored-output fixture is evidence RedCrown scored, not a model execution RedCrown witnessed. The same is true of any outputs you supply from your own system.
1. Install the CLI
You need Python 3.11 or newer and a shell. The released version is 0.1.23. Pin it, so the output on this page matches the output on your machine.
pip install redcrown==0.1.23The package installs the redcrown command. It does not install the transcription stack, and this page does not need it.
Nothing to install. Steps 1 to 8 run in a terminal. The hosted path in step 9 uses app.redcrown.ai.
Not available via MCP. The MCP tools do not score supplied outputs and do not run redcrown check.
2. Download the example files
Three files. All three are inputs.
mkdir regression-check && cd regression-check
curl -fsSLO https://redcrown.ai/docs/quickstart/regression-check/score-request.json
curl -fsSLO https://redcrown.ai/docs/quickstart/regression-check/other-feature.json
curl -fsSLO https://redcrown.ai/docs/quickstart/regression-check/policy.jsonThe same files are in the repository under examples/regression-check/, with build_fixture.py, the script that wrote them, and items.csv, the cases in CSV form. The files here are byte-identical copies.
Not needed for the app. You can open the files in a browser: score-request.json, other-feature.json, policy.json, items.csv.
Not available via MCP.
3. What is in the files
score-request.json describes one AI feature, a support-ticket router, under the stable id experiment_id: "ticket-router". It carries 40 cases, case-01 to case-40. Each case has an input (one ticket) and a reference, the correct label. There are four labels: billing, technical, account, and shipping, ten cases each. The metric is exact_match: a case scores 1 when the output equals the reference after trimming and lower-casing, and 0 otherwise.
The file holds four candidates. Each candidate has an id, a label, a declared configuration, and one outputs row per case.
reference-v1: the reference run. Every label is correct.candidate-v2: the proposed change. Four labels are wrong.candidate-v3: the repaired change. One label is wrong.candidate-v2-partial: the same four wrong labels, plus one case,case-12, that carries anerrorinstead of an output.
The configuration block is what the supplier declares about how the outputs were produced: provider (your-app), model (ticket-router-v1, -v2, -v3), the prompt, params, and a source_revision (v1, v2, v3). RedCrown records the declaration and hashes it. It cannot verify it, because it did not run the feature. The policy in step 6 requires a declared configuration to be present.
other-feature.json is a different feature, refund-summarizer, with two cases. It exists to show what happens when two evidence files do not describe the same feature. policy.json is the written requirements, shown in step 6.
python3 -c "import json; d = json.load(open('score-request.json')); print(len(d['items']), 'cases;', [c['id'] for c in d['candidates']])"Prints 40 cases; ['reference-v1', 'candidate-v2', 'candidate-v3', 'candidate-v2-partial'].
The app's Score supplied outputs section (Advanced → Upload) accepts the CSV and JSONL form of the same data. It does not accept this request file. See step 9 for what the hosted form can and cannot do.
Not available via MCP.
4. Score the outputs
One command scores every candidate against the references and writes the evidence file. No network call is made. The mean score per candidate prints as a table; cost and latency read unknown because the request declared neither.
$ redcrown score score-request.json --evidence-json evidence.json
Experiment ticket-router objective=quality bar=0.95 baseline=labels
Step outputs:
candidate quality $/min $/mo cost latency flags
reference-v1 1.000 n/a n/a unknown unknown baseline, WINNER
candidate-v3 0.975 n/a n/a unknown unknown
candidate-v2 0.900 n/a n/a unknown unknown
candidate-v2-partial 0.875 n/a n/a unknown unknown 1 errors
wrote private raw evidence; it may contain prompts, outputs, references, and other customer data. Keep it private.reference-v1 scores 1.000 (40 of 40). candidate-v2 scores 0.900 (36 of 40). candidate-v3 scores 0.975 (39 of 40). candidate-v2-partial scores 0.875 and shows 1 errors: the errored case counts in the denominator and scores nothing, so 35 of 40. The WINNER flag is the ranking view of the same numbers; the check in step 7 is what applies your requirements.
Hosted scoring lives at Advanced → Upload → Score supplied outputs and takes CSV plus JSONL. Pick New feature to use a plan slot. Or pick an existing feature in Add to feature to add a version and use no new slot. The select lists every active feature, but you can add a version only to one you created with Score supplied outputs. It cannot write the local evidence file this page uses.
Not available via MCP. import_results records aggregate scores another system computed; it does not score outputs.
5. Two output files, two purposes
redcrown score can write two different files. Keep them apart.
--evidence-json evidence.jsonwrites the private raw evidence: every input, reference, output, per-case score, the declared configuration, the scorer manifest and content hashes. This is the fileredcrown checkreads. It can contain your data. Keep it on your machine or in your private CI artifacts. Never publish it.--report-json out.jsonwrites the upload format forredcrown push: the ranked report with receipts, in the shape the hosted import accepts. Step 9 uses it.redcrown checkrefuses it (step 8 shows the refusal).
python3 -c "import json; d = json.load(open('evidence.json')); print(d['experiment_id'], list(d['per_step']))"Prints ticket-router ['outputs']: one feature id, one pipeline step. Two evidence files can be checked against each other only when they share this id, the same cases and references, and the same scorer.
The app never receives evidence.json. The report view shows a pushed out.json.
Not available via MCP.
6. The policy
A policy is a JSON file. It is written down before anyone looks at the candidate. This is policy.json, verbatim:
{
"schema_version": 1,
"requirements": {
"minimum_items": 40,
"maximum_error_rate": 0.0,
"maximum_ungraded_rate": 0.0,
"require_known_config": true,
"quality": {
"minimum": 0.95,
"maximum_regression": 0.05
}
}
}The app does not read policy files. The hosted comparison on Proofs → Baseline decisions shows paired wins, ties and losses without thresholds; see Accepted baselines and regression checks.
Not available via MCP.
Each rule, and how the check reports it:
minimum_items: the candidate must carry at least 40 cases. Observed is the case count.maximum_error_rate: at most this fraction of cases may carry an error.0.0means none.maximum_ungraded_rate: at most this fraction of cases may be ungraded, for example a case with no reference.0.0means none.require_known_config: the candidate must declare a configuration. Observed isTrueorFalse.quality.minimum: the candidate's mean score must be at least 0.95.quality.maximum_regression: the baseline's mean score minus the candidate's mean score, over the cases scored on both sides, must be at most 0.05. This rule needs a baseline, and it needs every case scored on both sides.
A rule is pass, fail, or indeterminate, each with a reason. Indeterminate means the evidence could not answer the rule. It is never counted as a pass and never shown as 0. The whole check is pass only when every rule passes. A demonstrated breach always decides the status: when one rule fails and another is indeterminate, the check is fail. The schema also accepts cost and latency rules. They parse, and they are always indeterminate today, because supplied outputs carry declared costs, not measured ones.
7. Check the change
First the control: the reference run checked against itself. It must pass. If it does not, the policy or the evidence is wrong, and no candidate result would mean anything.
$ redcrown check evidence.json --candidate-id reference-v1 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: pass (exit 0)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: pass [requirement_met]; observed=1.0, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: pass [requirement_met]; observed=-0.0, threshold=0.05, count=40, denominator=40
$ echo $?
0Every rule passes. Exit 0. quality.maximum_regression reads -0.0 because the two sides are the same file.
Not available in the app. The check is a local command over local files.
Not available via MCP.
Now the candidate. Same policy, same baseline, candidate-v2 on the candidate side.
$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40
$ echo $?
2The check is fail, exit 2. Two rules breached, each with the reason threshold_breached: quality.minimum observed 0.9 against a threshold of 0.95, and quality.maximum_regression observed 0.1 against a threshold of 0.05. The other four rules pass. count=40, denominator=40 on the quality rules says every case was scored on both sides, so nothing was excluded.
The check output lists rules, counts, and any excluded case ids. It does not list the cases that changed. Add --changed to the same command to list them. --changed prints outputs. Do not use it in a CI job whose logs other people can read.
$ redcrown check evidence.json --candidate-id candidate-v2 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json --changed
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: fail [threshold_breached]; observed=0.9, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: fail [threshold_breached]; observed=0.1, threshold=0.05, count=40, denominator=40
changed cases: 4 regressed, 0 fixed, 0 excluded, 40 in the set
- case-07: regressed; baseline 'account', candidate 'billing'
- case-18: regressed; baseline 'technical', candidate 'account'
- case-25: regressed; baseline 'billing', candidate 'technical'
- case-36: regressed; baseline 'shipping', candidate 'account'
$ echo $?
2The rules, the status, and the exit code do not change. Below them, four cases regressed. Each line shows what v1 returned and what v2 returned. --changed computes nothing new: every score it compares was written by the scorer in step 4. Add --format json or --format markdown to redcrown check for the same result in those forms; the JSON carries exit_code, the rules, the population counts, and the hashes of both sides.
For hosted runs that carry an evaluation identity, Proofs → Baseline decisions pairs a later run with the accepted baseline and lists the items that lost, with each receipt. That path does not apply to this local example; see step 9 for why.
Not available via MCP.
8. The repaired candidate, and three failure shapes
candidate-v3 fixed three of the four labels. One is still wrong.
$ redcrown check evidence.json --candidate-id candidate-v3 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: pass (exit 0)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: pass [requirement_met]; observed=0.975, threshold=0.95, count=40, denominator=40
- quality.maximum_regression: pass [requirement_met]; observed=0.025, threshold=0.05, count=40, denominator=40
$ echo $?
0Pass, exit 0. Quality 0.975 clears the 0.95 minimum, and the regression of 0.025 is within 0.05. One wrong label out of 40 is inside what this policy allows. A stricter policy would refuse it; the policy is yours to write.
Not available in the app.
Not available via MCP.
Shape 1: one case errored
candidate-v2-partial carries the four wrong labels and one case that timed out.
$ redcrown check evidence.json --candidate-id candidate-v2-partial --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: pass [requirement_met]; observed=40, threshold=40, count=40, denominator=40
- maximum_error_rate: fail [threshold_breached]; observed=0.025, threshold=0.0, count=1, denominator=40
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=40
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: indeterminate [incomplete_quality_coverage]; threshold=0.95, count=39, denominator=40
- quality.maximum_regression: indeterminate [incomplete_quality_coverage]; threshold=0.05, count=39, denominator=40
$ echo $?
2Fail, exit 2, on maximum_error_rate: 1 of 40 cases errored, 0.025 against a threshold of 0.0. Both quality rules are indeterminate with the reason incomplete_quality_coverage: 39 of 40 cases were scored, and a quality rule needs every case scored on both sides. The check does not score the missing case as 0 and does not average over 39. The four wrong labels are still in the evidence; run redcrown check evidence.json --candidate-id candidate-v2-partial --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json --changed to see them listed with the errored case marked excluded (timeout after 30 s).
A hosted acceptance is refused for a run with any errored or ungraded item; the refusal names the reason.
Not available via MCP.
Shape 2: a different feature
Score the second feature, then check it against the ticket router's reference run.
$ redcrown score other-feature.json --evidence-json other-evidence.json
Experiment refund-summarizer objective=quality bar=0.95 baseline=labels
Step outputs:
candidate quality $/min $/mo cost latency flags
summarizer-v1 1.000 n/a n/a unknown unknown baseline, WINNER
wrote private raw evidence; it may contain prompts, outputs, references, and other customer data. Keep it private.$ redcrown check other-evidence.json --candidate-id summarizer-v1 --baseline evidence.json --baseline-candidate-id reference-v1 --policy policy.json
redcrown check: fail (exit 2)
- minimum_items: fail [threshold_breached]; observed=2, threshold=40, count=2, denominator=2
- maximum_error_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=2
- maximum_ungraded_rate: pass [requirement_met]; observed=0.0, threshold=0.0, count=0, denominator=2
- require_known_config: pass [requirement_met]; observed=True, threshold=True, count=1, denominator=1
- quality.minimum: pass [requirement_met]; observed=1.0, threshold=0.95, count=2, denominator=2
- quality.maximum_regression: indeterminate [incompatible_evidence]; threshold=0.05
- evidence: workload_mismatch
- evidence: content_hash_mismatch
$ echo $?
2The comparison is refused: quality.maximum_regression is indeterminate with the reason incompatible_evidence, and the two evidence: lines say why, workload_mismatch (a different experiment_id) and content_hash_mismatch (different cases and references). The overall status is fail, not indeterminate, because minimum_items was breached on its own: 2 cases against a minimum of 40. A demonstrated breach decides the status even when the comparison could not be made. Two evidence files that share the feature id but were scored with a different scorer, or on a different scoring runtime, are refused the same way with evaluator_hash_mismatch or platform_changed.
The hosted comparison shows the same mismatch codes when two runs do not share an evaluation identity.
Not available via MCP.
Shape 3: a malformed candidate file
Point the check at a file that is not raw evidence. Here the policy file stands in for the candidate.
$ redcrown check policy.json --policy policy.json
redcrown check: error (exit 1)
- evidence: candidate_raw_report_required
$ echo $?
1Status error, exit 1, reason candidate_raw_report_required. Exit 1 is a command, file or policy error, and it is different from exit 2. An upload-format out.json from --report-json gets the same refusal, because it is not the raw evidence. A missing file, a policy in YAML, or a candidate id that is not in the file also exit 1. The three exit codes are the contract a CI job reads: Run the check in CI.
Not available in the app.
Not available via MCP.
9. Optional: keep the record in the app
Everything above stayed on your machine. The hosted path stores a run report, lets you share it, and, for eligible runs, lets a workspace member accept a baseline. It needs an account. The Free plan allows one active AI feature and one hosted proof for the life of the account, with no card. The commands below were not run for this page, because they need an account; the wording of each step comes from the CLI's own messages and from the code.
redcrown login
redcrown score score-request.json --report-json out.json
redcrown push out.jsonredcrown login opens a device-code page and stores a token on your machine. redcrown push out.json uploads the ranked report with its receipts and creates a new active AI feature in your workspace; the Free plan allows one. Add --no-receipts to strip the per-case inputs and outputs before upload, and --proof-link to create a share link at the same time. Without a stored token, push prints error: not logged in. Run `redcrown login`, or set REDCROWN_TOKEN, or set REDCROWN_EMAIL + REDCROWN_PASSWORD (machine accounts). and exits 1.
redcrown push out.json --experiment <id> appends the run to an experiment that already exists instead of creating a new one.
Open app.redcrown.ai/check. The page offers the three supported starts: this synthetic example, bring outputs you already have, or bring a test harness journal. Model comparison is linked from it as the secondary path. Advanced → Upload → Score supplied outputs scores a CSV plus JSONL pair in the app. Proofs lists every run; Proofs → Baseline decisions is where a baseline is accepted.
import_results records aggregate scores and only the receipts the payload supplies. There is no MCP tool for scoring supplied outputs, for redcrown check, or for accepting a baseline.
What can be accepted as a baseline, and what cannot
A hosted acceptance is a stored decision on one candidate of one completed hosted run: Accepted baselines and regression checks. It is eligible when the run has an evaluation identity (label references, one step, one pass, metric exact_match, similarity, json_field_accuracy or f1), no errored item, no ungraded item, and no blocking caveat. Two things matter for the example on this page:
- Scoring supplied outputs, whether through Score supplied outputs or
POST /experiments/score, can add a version to an existing feature instead of creating a new one, but only to a feature you created by scoring supplied outputs. Comparing or accepting a version needs a declared provider and model on each candidate. RedCrown refuses the upload withcontent_hash_mismatchwhen the cases or references differ. Confirm the case change to add it anyway, with a warning. RedCrown refuses withevaluator_hash_mismatchwhen the scorer differs. redcrown push --experiment <id>appends a run to a standing experiment, but a pushed run carries no evaluation identity, so it cannot be accepted.
The offline check on this page stays a working alternative for supplied outputs with no account: one stable experiment_id in the request file, one evidence file per version, and redcrown check between them.
Sharing
A share link publishes a frozen copy of one run report: the ranked table, the receipts you did not strip, the caveats, and a signed attestation. It does not carry a local redcrown check result. Free allows one hosted proof for the life of the account, and archiving or deleting the feature does not return it. See Share a proof for what becomes public and what a revocation does.
Last verified . Every command in steps 1 to 8 was run on the example files before it was written here, and the site build re-runs them. Next: Run the check in CI, or Concepts.