# Accepted baselines and regression checks

Source: https://redcrown.ai/docs/guides/accepted-baselines/

Score outputs produced anywhere, check a later version against a reference run offline with a written policy, and, for eligible hosted runs, accept one configuration as the baseline in the app.

## 1. Which runs qualify

A run can become a baseline, be compared, or be diagnosed only when RedCrown can give it an **evaluation identity**: a private, deterministic fingerprint of the exact items, references and scorer. Today an identity is written for a run that meets all of these:

- One pipeline step, text inputs and text outputs.
- Label references (your own answer key), not an incumbent model.
- Quality metric `exact_match`, `similarity`, `json_field_accuracy`, or `f1`.
- A single pass. A run with `repeat_n` above 1, or a rejudged report, fails the compatibility check and cannot be accepted.

Judge and rubric runs, word-error-rate (`wer`) transcription runs, and multi-step pipelines get no identity. They still rank, publish and attest as before, but the baseline controls below are not offered for them. The report says `evaluation_identity: null` rather than pretending.

`f1` is the extraction scorer: it compares numbers by value, never treats a boolean as equal to a number, and marks unparseable output as unscorable rather than zero. `json_field_accuracy` is the older exact-value scorer and treats `true` and `1` as equal.

## 2. Score outputs produced elsewhere

You do not need RedCrown to generate the outputs. Scoring supplied outputs needs no provider key and makes no model call. The result carries a declared configuration, not a witnessed one: RedCrown scored what you supplied and does not claim to know how it was generated.

There are two input forms, and only one of them can be checked against a reference run.

- **A scoring request JSON** (the form the [quickstart](../../quickstart/) uses): one file with a stable `experiment_id`, the items with their references, and one or more candidates, each with a declared `configuration` and its outputs. Its evidence file carries the feature id and the configuration, so `redcrown check` can compare two versions of the same feature and can answer `require_known_config`.
- **CSV plus JSONL** (`--items-csv` and `--outputs-jsonl`): the quickest way to score, and scorable in the app. This form has no place to declare a configuration and gets a fresh random `experiment_id` on every scoring. Its evidence file can be read, pushed and shared, but a `redcrown check` on it returns `indeterminate` with `missing_or_invalid_config` on every rule (verified on 0.1.22). Use it to score; use the request JSON to check.

**UI**

Open **Advanced → Upload** and use the **Score supplied outputs** section. Click **Choose CSV or JSONL**, map the columns, pick the metric, then click **Score outputs**. Choosing a file only previews it. Nothing is sent until you click. Use the **Add to feature** select to choose the destination. **New feature**, the default, creates a feature and uses a slot against your plan's cap. Choose an existing feature instead to add a version to it. That uses no new slot. The select lists every active feature, but you can add a version only to a feature you created with **Score supplied outputs**. RedCrown refuses any other feature. Declare the provider and model for each candidate. Without a declared configuration, versions cannot be compared or accepted. The app does not accept the request JSON form and does not write a local evidence file.

**CLI**

The checkable form. `score-request.json` is the quickstart's file; the shape is `experiment_id`, `name`, `quality_metric`, `quality_bar`, `objective`, `items` (each with `item_id`, `input`, `reference`), and `candidates` (each with `id`, `label`, `configuration`, and `outputs` rows with `item_id` and `output` or `error`).

```
redcrown score score-request.json --evidence-json evidence.json
```

The scorable-only form:

```
redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --evidence-json scored-evidence.json
```

`items.csv` needs `item_id`, `input` and `reference` columns. `outputs.jsonl` needs `candidate_id`, `item_id` and `output` (or `error`). `--metric` accepts the four metrics above. `f1` against references that are not JSON objects is refused before scoring. In both forms, `--evidence-json` writes the private raw report that `redcrown check` reads; it can contain your data, so do not publish it. `--report-json` writes the separate upload format for `redcrown push`.

**MCP**

Not available via MCP. `import_results` records aggregate results, with receipts only when the payload supplies them, and does not produce an evaluation identity. Use the CLI or the UI above.

## 3. Accept a baseline

An acceptance records that one candidate of one completed hosted run is the configuration you stand behind. It is a decision record, not a deployment authorization. It does not deploy anything and never picks a winner for you.

- **Who accepts**: a member of the workspace that owns the experiment, in the app. There is no CLI command and no MCP tool for it.
- **What is stored**: an acceptance record on the experiment, naming the run and the candidate, with the history of every acceptance and revocation. The record stays private to the workspace; the evaluation identity behind it never appears on a public page or in an attestation.
- **What revocation does**: clears the current baseline. The history keeps the revoked record. Nothing is deleted.
- **What is refused**: a run with no evaluation identity (section 1), a run with any errored or ungraded item, or a report with a blocking publish caveat. The refusal names the reason.
- **Why an incompatible candidate cannot be compared**: a later run is paired with the baseline only when the two identities match. A mismatch is reported by code: `workload_mismatch` (a different experiment), `content_hash_mismatch` (different items or references), `evaluator_hash_mismatch` (a different scorer or metric), or `platform_changed` (a different scoring runtime, such as another Python version). In the last two cases, re-run the baseline on the current runtime to compare.

**A hosted acceptance is not a local reference file.** The offline `redcrown check` in section 4 compares two evidence files on your machine and never reads a hosted acceptance. The hosted acceptance compares two runs stored in the app and never reads a policy file. The two paths do not meet today.

**Same-feature continuity.** The app and `POST /experiments/score?experiment_id=<id>` can add a version to a feature created by scoring supplied outputs. A missing or deleted feature id returns 404. RedCrown refuses with `feature_archived` for an archived feature. RedCrown refuses with `not_a_supplied_output_feature` when the feature's latest run was not a scoring of supplied outputs. Neither of those two refusals can be overridden. RedCrown refuses with `evaluator_hash_mismatch` when the scorer or metric differs from the feature's latest version. This cannot be overridden either. RedCrown refuses with `continuity_unverified` when it cannot compare the two versions' identities. Pass `allow_case_change=true` to add it anyway, with a warning. A different scoring runtime alone adds the version with a `platform_changed` warning, not a refusal. RedCrown refuses with `content_hash_mismatch` when the cases or references differ from the feature's latest version. Pass `allow_case_change=true` to add it anyway, with a warning. Once added, the new version gets its own revision and its own evaluation identity. Comparing or accepting a version still needs a declared provider and model on each candidate, as in section 2. `redcrown push --experiment <id>` still appends a pushed run with no evaluation identity. A pushed run cannot be accepted. The offline check in section 4 remains the documented alternative. It compares two evidence files on your machine with one stable `experiment_id` in the request file, independent of the hosted feature. Hosted acceptance today applies to runs RedCrown executed itself ([Compare models on your workload](../prove-a-task/) or [Run and rank](../run-and-rank/)) or to a scored version, each of which must meet section 1.

**UI**

Open **Proofs → Baseline decisions**, select the experiment, then under **Decision baseline** choose a candidate from a completed run and click **Confirm baseline**. No candidate is selected automatically. **Revoke** clears the current baseline; the history of every acceptance and revocation stays.

**CLI**

Not available via the CLI. Acceptance is a hosted decision on a stored run. Use the UI, or the API route `POST /experiments/{id}/acceptances`.

**MCP**

Not available via MCP.

## 4. Check a later run against it

A check is a fixed policy, written down before you look at the candidate, applied to two runs that share an identity. Errors and ungraded items are excluded from the pairing, never counted as zero, and a quality rule needs every item scored on both sides.

Cost and latency rules exist in the policy schema. They parse, and they are always `indeterminate` today, because supplied outputs carry declared costs, not measured ones. Put them in a policy only if you want that indeterminate result recorded; an indeterminate required rule makes the whole check exit 2.

**UI**

On the same **Decision baseline** panel, pick a later run and candidate to see the paired result: items compared, wins, ties, losses, mean difference, and everything excluded with the reason. **Diagnose receipts** opens the per-item losses with each private receipt. An incompatible pair shows the mismatch code from section 3. This panel applies no policy thresholds; it shows the pairing.

**CLI**

```
redcrown check candidate-evidence.json --candidate-id candidate-b \
  --baseline baseline-evidence.json --baseline-candidate-id candidate-a \
  --policy policy.json --format json
```

Exit `0` is pass. Exit `2` is fail or indeterminate. Exit `1` is a command, file or policy error. Keep that code in CI ([Run the check in CI](../ci-gate/)). The check runs with no network, no provider and no login. Both files must be raw evidence from `--evidence-json`, not the upload format from `--report-json`. A policy is JSON only:

```
{
  "schema_version": 1,
  "requirements": {
    "minimum_items": 30,
    "maximum_error_rate": 0.02,
    "maximum_ungraded_rate": 0.0,
    "require_known_config": true,
    "quality": {"minimum": 0.85, "maximum_regression": 0.01}
  }
}
```

The check output lists each rule with its status and reason, the counts, and any excluded item ids. The cases that changed are not in it by default. Add `--changed` to list them from the evidence file. `--changed` prints outputs. Do not use it in a CI job whose logs other people can read.

**MCP**

Not available via MCP. Use the CLI above.

## 5. Share with a disclosure level

When you publish a proof link for a run that carries variant manifests, you choose how much configuration the public page shows. Each link freezes its own copy of the report; a later link does not rewrite an earlier one.

- **Hashes only** (the default): shows the shared instruction, and hashes each variant's system prompt and parameters.
- **Full configuration**: shows every prompt, system prompt and parameter that was recorded.
- **Private configuration**: hides the instruction, model names and variant labels, including in caveat text, error text and the page title. Inputs, outputs and references stay visible, because they are your content, not configuration.

The preview in the share dialog is exactly what the public page will serve. The private evaluation identity never appears on a public page or in a signed attestation under any level.

Last verified 2026-09-22.