Compare all three

Start from what you already have

You do not need a new experiment to get a first record. Three things you probably have today are enough: a journal your harness already writes, a set of production traces, or a rules list your client wrote. Each section below ends in one command. Every command on this page runs offline, with no account and no provider key. Install the CLI with pip install redcrown (Python 3.11 or newer).

1. A journal you already keep

A journal is one directory per run. Each run directory holds a meta.json and one or more verdicts*.jsonl files. Each line of a verdicts file is one case. A case carries a case_id, an optional written status, an optional elapsed_ms, and a list of checks. Each check has a name, a status of ok, fail, or unmeasured, and an optional one-line evidence. A journal that wrote a legacy dimensions object (name to true, false, or null) is read the same way, without a rewrite.

This is a three-row journal written by hand, in runs/2026-09-16-a/:

runs/2026-09-16-a/meta.json
{
  "corpus_version": "corpus-v3",
  "harness_sha": "unknown",
  "target": {"commit": "a1b2c3d", "prompt_hash": "p9"},
  "opened_at": 0
}

runs/2026-09-16-a/verdicts.jsonl
{"case_id": "pass-case", "status": "PASS", "elapsed_ms": 1, "checks": [{"name": "persisted", "status": "ok"}]}
{"case_id": "fail-case", "status": "FAIL", "elapsed_ms": 1, "checks": [{"name": "persisted", "status": "fail"}]}
{"case_id": "blocked-case", "status": "BLOCKED", "elapsed_ms": 1, "checks": [{"name": "persisted", "status": "unmeasured"}]}

Not available in the app. A journal is read on the machine that wrote it. The record reaches the app in step 4, with redcrown push.

CLI

redcrown harness report 2026-09-16-a --runs runs

The real output:

PASS 1 of 2 graded (1 of 2 counting partial), 1 not graded, 3 cases
  (no group)   PASS   1  FAIL   1  PARTIAL   0  BLOCKED   1  ERROR   0  (2 of 3 graded)

checks, worst first:
  persisted                 50.0% (1 of 2), 1 unmeasured

--runs is the journal directory, and the first argument is the run subdirectory. The summary prints two denominators: strict (PASS plus FAIL) and with partial. The blocked case shows as not graded. It is never a pass and never a fail. The command exits 0. It exits 1 when the run directory does not exist.

MCP

Not available via MCP. Use the CLI above.

2. Production traces

A trace is what your product already produced: an input, the output the model returned, and the answer that was correct. Put the inputs and correct answers in a CSV and the outputs in a JSONL file. RedCrown scores the outputs with a deterministic metric. This path makes no model call and needs no provider key. The result carries a declared configuration, not a witnessed one: RedCrown scored what you supplied and does not claim to know how it was generated.

items.csv needs an item_id column (or id), an input column, and a reference column. outputs.jsonl needs one line per output with candidate_id, item_id, and output (or error). candidate_label, cost_usd, and latency_ms are optional. These are the two files used below, two rows each:

items.csv
item_id,input,reference
t-1001,"Ticket: printer on floor 2 is offline again",hardware
t-1002,"Ticket: cannot log in after the password reset",access

outputs.jsonl
{"candidate_id": "prod", "candidate_label": "production, week 37", "item_id": "t-1001", "output": "hardware"}
{"candidate_id": "prod", "candidate_label": "production, week 37", "item_id": "t-1002", "output": "billing"}

Open Advanced → Upload and use the Score supplied outputs section. Click Choose CSV or JSONL, map the columns, pick the metric, then click Score outputs. Pick New feature to create an active AI feature. That uses a slot on your plan. Pick an existing feature in Add to feature to add a version instead. The existing feature must be one you created with Score supplied outputs. Adding a version uses no new slot.

CLI

redcrown score --items-csv items.csv --outputs-jsonl outputs.jsonl \
  --metric exact_match --evidence-json evidence.json --report-json scored.json

The real output:

Experiment f534e02e-f873-4a6e-9a1a-ca7c8cbd9c8d  objective=quality  bar=0.0  baseline=labels

Step outputs:
  candidate                 quality     $/min         $/mo       cost   latency  flags
  prod                        0.500       n/a          n/a    unknown   unknown  baseline, WINNER

wrote private raw evidence; it may contain prompts, outputs, references, and other customer data. Keep it private.

wrote scored results to scored.json; push to the cloud with: redcrown push scored.json

One of two outputs matched its reference, so quality is 0.500. Cost and latency read unknown because the traces carried neither. --metric accepts exact_match, similarity (the default), json_field_accuracy, and f1. f1 against references that are not JSON objects is refused before scoring. --evidence-json writes the private raw report that redcrown check reads; it can contain your data, so do not publish it. --report-json writes the upload format for redcrown push.

MCP

Not available via MCP. import_results records aggregate results, with receipts only when the payload supplies them, and does not produce an evaluation identity. Use the CLI or the UI above.

3. A rules list your client wrote

A client states rules in plain words. Each rule becomes one named check. The harness grades every case on every check and writes one line of evidence a reader can verify. Three rules and the check names they become:

"The order lands in the database with the right customer." becomes order_persisted.
"The reply names the pickup date." becomes pickup_date_named.
"No price is quoted when the rate is unknown." becomes no_price_when_rate_unknown.

A rule that did not apply to a case is unmeasured on that case, with the reason in the evidence line. Here the third rule did not apply to the third order because the rate table had a price. This is the journal, one line per case, with no written status:

runs/2026-09-17-rules/verdicts.jsonl
{"case_id": "order-017", "group": "intake", "elapsed_ms": 2140, "checks": [{"name": "order_persisted", "status": "ok", "evidence": "orders row 8812, customer 4410"}, {"name": "pickup_date_named", "status": "ok", "evidence": "reply: pickup Thursday, September 18"}, {"name": "no_price_when_rate_unknown", "status": "ok", "evidence": "no dollar amount in the reply"}]}
{"case_id": "order-018", "group": "intake", "elapsed_ms": 1980, "checks": [{"name": "order_persisted", "status": "ok", "evidence": "orders row 8813, customer 4410"}, {"name": "pickup_date_named", "status": "fail", "evidence": "reply: we will confirm the pickup soon"}, {"name": "no_price_when_rate_unknown", "status": "ok", "evidence": "no dollar amount in the reply"}]}
{"case_id": "order-019", "group": "intake", "elapsed_ms": 2305, "checks": [{"name": "order_persisted", "status": "ok", "evidence": "orders row 8814, customer 2207"}, {"name": "pickup_date_named", "status": "ok", "evidence": "reply: pickup Friday, September 19"}, {"name": "no_price_when_rate_unknown", "status": "unmeasured", "evidence": "rate table had a price, so the rule did not apply"}]}

When a row carries no written status, RedCrown derives one from its checks, in this order: an error is ERROR; any fail is FAIL; nothing measured is BLOCKED; something measured and the rest unmeasured is PARTIAL; otherwise PASS. An unmeasured check is never a pass or a fail. A status the harness wrote is authoritative, because the harness knew things the row cannot show, such as a case for a feature the product has not built yet.

A grading override is a rulings file, passed with --rulings. It flips one named check on one case, records who ruled and why on that check, and keeps the raw status on the row. The journal on disk is never rewritten.

Not available in the app. After redcrown push, the report view shows every check on every case, worst first.

CLI

redcrown harness report 2026-09-17-rules --runs runs

The real output:

PASS 1 of 2 graded (1 of 3 counting partial), 1 not graded, 3 cases
  intake       PASS   1  FAIL   1  PARTIAL   1  BLOCKED   0  ERROR   0  (2 of 3 graded)

checks, worst first:
  pickup_date_named         66.7% (2 of 3)
  no_price_when_rate_unknown 100.0% (2 of 2), 1 unmeasured
  order_persisted          100.0% (3 of 3)

The three rows became PASS, FAIL, and PARTIAL. The group field gives the per-group line. A check's pass rate counts measured checks only, and the unmeasured count is printed beside it.

To check the run against the client's written requirements, put them in a policy file and pass it with --policy. Needs redcrown 0.1.23 or later. This policy.json asks for at least 3 cases and an ok on order_persisted and pickup_date_named on every case:

{
  "schema_version": 1,
  "kind": "harness",
  "requirements": {
    "minimum_cases": 3,
    "required_checks": [{"names": ["order_persisted", "pickup_date_named"]}]
  }
}

redcrown harness report 2026-09-17-rules --runs runs --policy policy.json

The command prints the same summary, then this block, and exits 2 because order-018 failed a required check:

redcrown harness policy: fail (exit 2)
  minimum_cases: pass; observed 3; threshold 3
  required_checks[0]: fail; reason required_check_failed; 1 of 3; cases order-018

MCP

Not available via MCP. Use the CLI above.

4. Compare, export, push

A second run of the same corpus goes in a second run directory. Here runs/2026-09-17-b/ holds the same three cases as step 1, after a change that made pass-case fail its persisted check. The compare is check by check on the shared case ids. A measured check that becomes unmeasured is inconclusive, never a regression. A case on one side only is listed, never counted.

After both runs are pushed to one experiment, the report view of the later run shows Compared with an earlier run, with a base-run selector and the same per-check table.

CLI

redcrown harness compare 2026-09-16-a 2026-09-17-b --runs runs

The real output:

cases: regressed 1, fixed 0, inconclusive 0, newly measured 0, unchanged 2, only in base 0, only in head 0
  persisted                regressed   1  fixed   0  inconclusive   0  newly measured   0

regressed (1):
  pass-case  persisted: ok -> fail

exit 1

The command exits 1 on any regression, so a CI step stops there. It exits 0 when nothing regressed. Two runs on different corpus versions are refused with exit 2, because a pass rate is only comparable within one corpus; --allow-corpus-drift compares the shared case ids anyway and prints the drift. --format markdown and --format json give the same result in those forms.

MCP

Not available via MCP. Use the CLI above.

Export turns one run into the upload payload. --name is the experiment name shown on the record and --label names the system under test at one version.

Not available in the app. The export is a file on your machine.

CLI

redcrown harness export 2026-09-17-b --runs runs \
  --name "Order intake regression" --label "order-intake@d4e5f6a" --out upload.json

The real output:

wrote upload.json: 3 cases

The payload has quality_metric "checks", one candidate, and one item per case. Each item carries its checks and its harness_status. The candidate's source_revision reads d4e5f6a@corpus-v3, the target commit and the corpus version from meta.json. Quality is 1.0 for PASS, 0.0 for FAIL, and null for anything else, because an unmeasured case is not a score. Cost is null when any case's cost is unknown.

MCP

Not available via MCP. Use the CLI above.

Push sends the payload to RedCrown as one run. This is the one command on this page that needs an account. Run redcrown login once first. Push the first run with no --experiment; it prints pushed run <run_id> (experiment <experiment_id>). Push every later run with --experiment <experiment_id> so the runs share one experiment and the app can compare them.

The run lands in History. Its report view lists every case with its checks, worst first, and compares the run with the earlier run of the same experiment.

CLI

redcrown push upload.json --experiment <experiment_id>

On success the command prints pushed run <run_id> (experiment <experiment_id>). Without a login it prints error: not logged in and names the three ways to authenticate. --no-receipts strips the per-item receipts before upload, so outputs and evidence lines never leave your machine. --proof-link also mints a public proof link and prints it as proof link: https://app.redcrown.ai/proof/<token>. A run with an ERROR case is pushed and visible in the app, but the proof link is refused with the report's own caveat, quoted in full, until every case ran; --publish-anyway overrides that and the override is recorded. A BLOCKED or PARTIAL case does not refuse the link: the page discloses the unmeasured count beside the pass rate, because no re-run scores a check the product has not built.

MCP

Not available via MCP. Use the CLI above.

A coding agent that runs this loop for you can read the Harness guide for coding agents. Every command and exit code on that page is the same as here.

Last verified 2026-09-23.