Harness guide for coding agents

Install

Run pip install redcrown. It needs Python 3.11 or newer. Confirm the install with redcrown harness --help. Every harness command, score, and check run offline with no account and no provider key. Only push needs an account.

The three harness commands

Each command reads a journal directory: one subdirectory per run, holding meta.json and one or more verdicts*.jsonl files. Pass the journal directory with --runs. The run id is the subdirectory name. These are the required flags:

redcrown harness report <run_id> --runs <dir>
redcrown harness compare <base_run_id> <head_run_id> --runs <dir>
redcrown harness export <run_id> --runs <dir> --name "<experiment name>" \
  --label "<system>@<commit>" --out <file>

Optional flags: --rulings <file> on all three; --format text|markdown|json and --allow-corpus-drift on compare; --quality-bar and --task-summary on export. On report, --policy <file>, --baseline <run_id>, and --format text|json check the run against a written policy. Needs redcrown 0.1.23 or later. Ruling notes from compare go to stderr, so --format json stays parseable on stdout.

Exit codes

• report without --policy: 0. It exits 1 when the run is not found under --runs.
• report --policy <policy.json>: 0 when every rule in the policy passed. 2 when a rule failed or a rule is indeterminate. 1 on an input or policy error: the run or the baseline run is not found, the policy file cannot be read, the policy is YAML or not valid, or the policy has a no_regression rule and no --baseline. --baseline or --format json without --policy also exits 1.
• compare: 0 when nothing regressed. 1 on any regression. 2 when the compare is refused because the two runs carry different corpus versions; pass --allow-corpus-drift to compare the shared case ids anyway.
• export: 0. It exits 1 when the run is not found.
• check (redcrown check <evidence.json> --policy <policy.json>): 0 on pass. 2 on fail or indeterminate. 1 on a command, file, or policy error. It reads the private evidence file from redcrown score --evidence-json or redcrown eval --evidence-json, never a journal or an export.
• push: 0 when the run was pushed. 1 when not logged in, when the request failed, on a plan limit, or when the proof link was refused by the publish gate.

Read the process exit code. Do not parse the last line of the output for it.

A policy result and a compare result are different results. compare says whether anything regressed against an earlier run. report --policy says whether the run meets the requirements the owner wrote. A rule with no evidence is indeterminate, never a pass. This policy file asks for no ERROR case and an ok rate of at least 50% on the persisted check over at least 3 measurements:

{
  "schema_version": 1,
  "kind": "harness",
  "requirements": {
    "maximum_status_rate": {"ERROR": 0},
    "check_minimum_rate": [{"name": "persisted", "threshold": 0.5, "minimum_measured": 3}]
  }
}

On a run with one PASS, one FAIL, and one BLOCKED case, persisted has 2 measurements, so the second rule has too little evidence. The command prints the report summary, then this block, and exits 2:

redcrown harness policy: indeterminate (exit 2)
  maximum_status_rate.ERROR: pass; observed 0; threshold 0; 0 of 3
  check_minimum_rate[0]: indeterminate; reason insufficient_measurements; threshold 0.5; 1 of 2

Add --format json to get one JSON document with the status, the exit code, and every rule.

Statuses

A case is a list of named checks. Each check is ok, fail, or unmeasured. The case status is derived in this order: an error is ERROR; any fail is FAIL; nothing measured is BLOCKED; something measured and the rest unmeasured is PARTIAL; otherwise PASS. An unmeasured check is never a pass or a fail. A status the harness wrote into the journal is kept as written.

The export payload

• quality_metric is "checks".
• candidates holds one candidate: the system under test at one version, named by --label.
• Each candidate item is one case and carries its checks, each with name, status, evidence, and ruled.
• Each item carries harness_status: PASS, FAIL, PARTIAL, BLOCKED, or ERROR. Quality is 1.0 for PASS, 0.0 for FAIL, and null for the rest.
• The candidate's source_revision is "<commit>@<corpus_version>", read from meta.json.

Push

redcrown login
redcrown push <file> [--experiment <experiment_id>] [--no-receipts] [--proof-link]

Push the first run with no --experiment. The output line is pushed run <run_id> (experiment <experiment_id>). Pass that experiment id with --experiment on every later push, so the runs share one experiment.

Where the record lands: the run appears in the app under History. Its report view lists every case with its checks, worst first, and compares the run with the earlier run of the same experiment, check by check. --proof-link mints a public page and prints proof link: https://app.redcrown.ai/proof/<token>. A run with an ERROR case is pushed, but the proof link is refused by the publish gate and the output quotes the report's own caveat. A BLOCKED or PARTIAL case is disclosed on the page, not refused. Do not pass --publish-anyway unless the owner asked for it.

Outputs and evidence lines reach the record. Keep private content out of evidence, or push with --no-receipts.

Never claim

When you report a result, in a commit message, a pull request, a summary, or a message to the owner:

• Do not call a result verified. Say "graded against the rules the owner stated".
• Do not describe RedCrown as a monitor or as a watcher of the product. It reads the journal when you run a command.
• Do not say RedCrown knows the reason for a failure. It records which check failed and the evidence line.
• Do not say an expert checked the result. A reviewer verdict exists only when a reviewer cast one through the reviewer portal.
• Say "compared with the last run" for a compare, and quote the regressed, fixed, inconclusive, and newly measured counts as printed.
• Do not turn an unmeasured check or a BLOCKED case into a pass rate. Quote both denominators as the report prints them.