RedCrown.aiRedCrown.ai Docs Open the app

Accepted baselines and regression checks

Score outputs produced anywhere, check a later version against a reference run offline with a written policy, and, for eligible hosted runs, accept one configuration as the baseline in the app.

1. Which runs qualify

A run can become a baseline, be compared, or be diagnosed only when RedCrown can give it an evaluation identity: a private, deterministic fingerprint of the exact items, references and scorer. Today an identity is written for a run that meets all of these:

  • One pipeline step, text inputs and text outputs.
  • Label references (your own answer key), not an incumbent model.
  • Quality metric exact_match, similarity, json_field_accuracy, or f1.
  • A single pass. A run with repeat_n above 1, or a rejudged report, fails the compatibility check and cannot be accepted.

Judge and rubric runs, word-error-rate (wer) transcription runs, and multi-step pipelines get no identity. They still rank, publish and attest as before, but the baseline controls below are not offered for them. The report says evaluation_identity: null rather than pretending.

f1 is the extraction scorer: it compares numbers by value, never treats a boolean as equal to a number, and marks unparseable output as unscorable rather than zero. json_field_accuracy is the older exact-value scorer and treats true and 1 as equal.

2. Score outputs produced elsewhere

You do not need RedCrown to generate the outputs. Scoring supplied outputs needs no provider key and makes no model call. The result carries a declared configuration, not a witnessed one: RedCrown scored what you supplied and does not claim to know how it was generated.

There are two input forms, and only one of them can be checked against a reference run.

  • A scoring request JSON (the form the quickstart uses): one file with a stable experiment_id, the items with their references, and one or more candidates, each with a declared configuration and its outputs. Its evidence file carries the feature id and the configuration, so redcrown check can compare two versions of the same feature and can answer require_known_config.
  • CSV plus JSONL (--items-csv and --outputs-jsonl): the quickest way to score, and scorable in the app. This form has no place to declare a configuration and gets a fresh random experiment_id on every scoring. Its evidence file can be read, pushed and shared, but a redcrown check on it returns indeterminate with missing_or_invalid_config on every rule (verified on 0.1.22). Use it to score; use the request JSON to check.
UI

Open Advanced → Upload and use the Score supplied outputs section. Click Choose CSV or JSONL, map the columns, pick the metric, then click Score outputs. Choosing a file only previews it. Nothing is sent until you click. Use the Add to feature select to choose the destination. New feature, the default, creates a feature and uses a slot against your plan's cap. Choose an existing feature instead to add a version to it. That uses no new slot. The select lists every active feature, but you can add a version only to a feature you created with Score supplied outputs. RedCrown refuses any other feature. Declare the provider and model for each candidate. Without a declared configuration, versions cannot be compared or accepted. The app does not accept the request JSON form and does not write a local evidence file.

CLI

The checkable form. score-request.json is the quickstart's file; the shape is experiment_id, name, quality_metric, quality_bar, objective, items (each with item_id, input, reference), and candidates (each with id, label, configuration, and outputs rows with item_id and output or error).

redcrown score score-request.json --evidence-json evidence.json

The scorable-only form:

redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --evidence-json scored-evidence.json

items.csv needs item_id, input and reference columns. outputs.jsonl needs candidate_id, item_id and output (or error). --metric accepts the four metrics above. f1 against references that are not JSON objects is refused before scoring. In both forms, --evidence-json writes the private raw report that redcrown check reads; it can contain your data, so do not publish it. --report-json writes the separate upload format for redcrown push.

MCP

Not available via MCP. import_results records aggregate results, with receipts only when the payload supplies them, and does not produce an evaluation identity. Use the CLI or the UI above.

3. Accept a baseline

An acceptance records that one candidate of one completed hosted run is the configuration you stand behind. It is a decision record, not a deployment authorization. It does not deploy anything and never picks a winner for you.

  • Who accepts: a member of the workspace that owns the experiment, in the app. There is no CLI command and no MCP tool for it.
  • What is stored: an acceptance record on the experiment, naming the run and the candidate, with the history of every acceptance and revocation. The record stays private to the workspace; the evaluation identity behind it never appears on a public page or in an attestation.
  • What revocation does: clears the current baseline. The history keeps the revoked record. Nothing is deleted.
  • What is refused: a run with no evaluation identity (section 1), a run with any errored or ungraded item, or a report with a blocking publish caveat. The refusal names the reason.
  • Why an incompatible candidate cannot be compared: a later run is paired with the baseline only when the two identities match. A mismatch is reported by code: workload_mismatch (a different experiment), content_hash_mismatch (different items or references), evaluator_hash_mismatch (a different scorer or metric), or platform_changed (a different scoring runtime, such as another Python version). In the last two cases, re-run the baseline on the current runtime to compare.

A hosted acceptance is not a local reference file. The offline redcrown check in section 4 compares two evidence files on your machine and never reads a hosted acceptance. The hosted acceptance compares two runs stored in the app and never reads a policy file. The two paths do not meet today.

Same-feature continuity. The app and POST /experiments/score?experiment_id=<id> can add a version to a feature created by scoring supplied outputs. A missing or deleted feature id returns 404. RedCrown refuses with feature_archived for an archived feature. RedCrown refuses with not_a_supplied_output_feature when the feature's latest run was not a scoring of supplied outputs. Neither of those two refusals can be overridden. RedCrown refuses with evaluator_hash_mismatch when the scorer or metric differs from the feature's latest version. This cannot be overridden either. RedCrown refuses with continuity_unverified when it cannot compare the two versions' identities. Pass allow_case_change=true to add it anyway, with a warning. A different scoring runtime alone adds the version with a platform_changed warning, not a refusal. RedCrown refuses with content_hash_mismatch when the cases or references differ from the feature's latest version. Pass allow_case_change=true to add it anyway, with a warning. Once added, the new version gets its own revision and its own evaluation identity. Comparing or accepting a version still needs a declared provider and model on each candidate, as in section 2. redcrown push --experiment <id> still appends a pushed run with no evaluation identity. A pushed run cannot be accepted. The offline check in section 4 remains the documented alternative. It compares two evidence files on your machine with one stable experiment_id in the request file, independent of the hosted feature. Hosted acceptance today applies to runs RedCrown executed itself (Compare models on your workload or Run and rank) or to a scored version, each of which must meet section 1.

UI

Open Proofs → Baseline decisions, select the experiment, then under Decision baseline choose a candidate from a completed run and click Confirm baseline. No candidate is selected automatically. Revoke clears the current baseline; the history of every acceptance and revocation stays.

CLI

Not available via the CLI. Acceptance is a hosted decision on a stored run. Use the UI, or the API route POST /experiments/{id}/acceptances.

MCP

Not available via MCP.

4. Check a later run against it

A check is a fixed policy, written down before you look at the candidate, applied to two runs that share an identity. Errors and ungraded items are excluded from the pairing, never counted as zero, and a quality rule needs every item scored on both sides.

Cost and latency rules exist in the policy schema. They parse, and they are always indeterminate today, because supplied outputs carry declared costs, not measured ones. Put them in a policy only if you want that indeterminate result recorded; an indeterminate required rule makes the whole check exit 2.

UI

On the same Decision baseline panel, pick a later run and candidate to see the paired result: items compared, wins, ties, losses, mean difference, and everything excluded with the reason. Diagnose receipts opens the per-item losses with each private receipt. An incompatible pair shows the mismatch code from section 3. This panel applies no policy thresholds; it shows the pairing.

CLI 
redcrown check candidate-evidence.json --candidate-id candidate-b \
  --baseline baseline-evidence.json --baseline-candidate-id candidate-a \
  --policy policy.json --format json

Exit 0 is pass. Exit 2 is fail or indeterminate. Exit 1 is a command, file or policy error. Keep that code in CI (Run the check in CI). The check runs with no network, no provider and no login. Both files must be raw evidence from --evidence-json, not the upload format from --report-json. A policy is JSON only:

{
  "schema_version": 1,
  "requirements": {
    "minimum_items": 30,
    "maximum_error_rate": 0.02,
    "maximum_ungraded_rate": 0.0,
    "require_known_config": true,
    "quality": {"minimum": 0.85, "maximum_regression": 0.01}
  }
}

The check output lists each rule with its status and reason, the counts, and any excluded item ids. The cases that changed are not in it by default. Add --changed to list them from the evidence file. --changed prints outputs. Do not use it in a CI job whose logs other people can read.

MCP

Not available via MCP. Use the CLI above.

5. Share with a disclosure level

When you publish a proof link for a run that carries variant manifests, you choose how much configuration the public page shows. Each link freezes its own copy of the report; a later link does not rewrite an earlier one.

  • Hashes only (the default): shows the shared instruction, and hashes each variant's system prompt and parameters.
  • Full configuration: shows every prompt, system prompt and parameter that was recorded.
  • Private configuration: hides the instruction, model names and variant labels, including in caveat text, error text and the page title. Inputs, outputs and references stay visible, because they are your content, not configuration.

The preview in the share dialog is exactly what the public page will serve. The private evaluation identity never appears on a public page or in a signed attestation under any level.

Last verified .