Concepts

The objects RedCrown works with, in the order a check moves through them, then the older object model and the harness vocabulary.

From a feature to retained evidence

A check starts with a feature and a set of requirements, compares a candidate version with a reference, produces a result, and ends in evidence a person can inspect. Each step is a distinct object, and one never stands in for another.

• AI feature: the thing under test, for example a ticket router or a document extractor. In the product it is an experiment: the task, the cases, the references, the metric, the quality bar and the candidates. The billing entity is an active AI feature, an experiment that is neither archived nor deleted (see Workspaces and billing).
• Requirements and cases: the cases are the fixed set of inputs with their references (labels, or an incumbent's outputs). The requirements are the rules a version must meet, written as a policy JSON before anyone looks at the candidate: minimum items, maximum error and ungraded rates, a declared configuration, a quality minimum and a maximum regression. A rule can pass, fail, or be indeterminate; indeterminate is never a pass.
• Reference run: the version you compare against, held as a local evidence file written by redcrown score --evidence-json or redcrown eval --evidence-json. It is private, it can contain your data, and it is not a hosted acceptance.
• Accepted baseline: the hosted acceptance object, a workspace member's record that one candidate of one completed hosted run is the configuration they stand behind, with the history of every acceptance and revocation. Created in the app or through POST /experiments/{id}/acceptances, never from the CLI or MCP. Not a deployment authorization (see Accepted baselines and regression checks).
• Candidate: the proposed change, whether a new prompt, a new model, or a new revision of the application. On the record it is one candidate id with a declared or recorded configuration and one output per case.
• Check result: the local result of applying a policy to a candidate and a reference run with redcrown check: pass, fail, or indeterminate per rule and overall, with counts and excluded case ids. Exit 0, 2 or 1. It lives on your machine and in your CI artifacts. It is not part of a shared report and not stored in the app.
• Human review: verdicts cast by a person through the reviewer portal on cases a metric could not decide, blinded to model names. A verdict reaches the shared report of the run it was cast on. It does not feed back into re-scoring (see Reviewer portal).
• Retained evidence: what outlives the session. A shared report (a proof link) is a frozen copy of one run report with its receipts, caveats and any reviewer verdicts, public or unlisted, revocable. A signed attestation is a downloadable, signed copy of that record that shows it was issued by RedCrown and not altered since. Neither carries a local check result or an acceptance (see Share a proof).

Compare, and the ranked record

The model-comparison workflow runs one task across several models and configurations, scores the outputs against your references on cost, quality and latency, and names the cheapest configuration that clears your quality bar. It is the secondary workflow; the primary one is the check above. See Compare models on your workload.

The object model

• Experiment: a task definition: the objective, the quality metric and bar, the candidate models, and the dataset. The product's name for an AI feature.
• Run: one execution of an experiment that produces a ranked report.
• Report: the ranked result: per-candidate cost, quality, latency percentiles, and the winner.
• Receipts: per-item evidence (the input, each candidate's output, the score) behind the ranking.
• Provenance: who ran it, when, the dataset hash, the model versions, and the methodology version.
• Proof link: a no-login page that shares a run's report and receipts. These pages call it a shared report.
• Attestation: a downloadable, cryptographically signed, tamper-evident version of the proof (see Signed attestation).
• Evaluation identity: a private fingerprint of a run's exact items, references and scorer. Only single-step, single-pass runs on label references with exact_match, similarity, json_field_accuracy or f1 get one; judge, rubric, wer and multi-step runs do not. A pushed run and an aggregate import carry none.
• Acceptance: the owner's record that one candidate of one run is the baseline. Later runs with the same identity are checked against it (see Accepted baselines and regression checks).

Harness, checks and cases

A harness is a regression suite for one deployed AI feature. It runs the feature on a fixed set of cases and grades what the feature did against rules you state. RedCrown records each run, compares it with an earlier run check by check, and exports it as a report with receipts. The redcrown harness report, compare and export commands are in the CLI reference.

• Case: one situation the product is run on, identified by a case id.
• Check: one named rule graded on that case. Each check is ok, fail, or unmeasured. An unmeasured check is never a pass or a fail.
• Case status: derived from the case's checks, in this order. ERROR: the run of the case raised an error. FAIL: at least one check failed. BLOCKED: no check was measured. PARTIAL: at least one check was measured, none failed, and at least one was unmeasured. PASS: every check was measured and none failed.
• Written status: a status the harness wrote into its journal is authoritative. The harness knew things the row cannot show, such as a case for a feature the product has not built yet. RedCrown derives the status from the checks only when the harness wrote none of the five.
• Ruling: a grading override kept as a file. A ruling flips one named check on one case to ok, records who ruled and why on the check, and keeps the raw status on the row. The journal on disk is never rewritten.
• Two denominators: the report prints both: strict (PASS plus FAIL) and with partial (PASS plus FAIL plus PARTIAL). The summary line reads PASS n of m graded (n of k counting partial), so neither reader has to convert. A bare percentage is refused.
• Compare: each check on each shared case is one of regressed, fixed, inconclusive, newly measured, or unchanged. A measured check that becomes unmeasured is inconclusive, never a regression. A case present on one side only is listed, never counted. The compare exits 1 on any regression. Two runs on different corpus versions are refused with exit code 2 unless the caller allows the drift, and then the drift is printed.

How the surfaces relate

The same objects are reachable from the app UI, the redcrown CLI, and the MCP tools. See the CLI, MCP, and UI reference for the full mapping.

Last verified 2026-09-17.