How a check is made, how it is scored, and what it does not prove.

A record is only worth what its method is worth. This page states how RedCrown structures a check or a comparison, scores it, counts what it could not score, brings in human judgment, gates what gets published, and versions the whole thing, along with the places where the answer is weaker than it looks. If you are going to hand our output to a client or an auditor, read this first and decide whether you believe it.

1. What a check is made of

Every record starts from a feature: your task and your cases, not a public benchmark. You define:

• The cases. Your inputs, each with a reference where you have one. A reference is a label, a correct answer, or the output of the configuration you run today.
• The candidates. The prompt, model, or app revision you are considering. A model, a speech-to-text service, and an extraction pipeline are scored on the same footing.
• The reference run. The earlier run a candidate is checked against: a private evidence file for redcrown check, or a baseline you accepted in the app.
• The requirements. A policy file with the thresholds written down before anyone looks at the candidate: a minimum number of cases, a maximum error rate, a maximum ungraded rate, a minimum quality, and a maximum regression against the reference run. Each rule reports pass, fail, or indeterminate.

For a model comparison, the same parts apply with one addition: a quality bar that a candidate must clear before cost is allowed to matter. RedCrown does not rank on quality alone or cost alone. It finds the candidates that clear the bar you set, then reports the economics among those. Change the bar and the answer can change, which is why the bar is recorded on the report.

2. How outputs are scored

Scoring depends on the task, and the method used is recorded on the result with its version.

If you have no labels

You can still get a defensible cost answer. Score the candidates against your current configuration's output instead of a labeled reference. That does not tell you the current output is correct. It tells you which candidates match what you already ship, which is often the question when the goal is to reduce spend without a regression.

3. What is counted, and what is left out

The number a report shows is only as honest as its denominator, so the report states it. These are the rules the code applies.

• Unscorable is not zero. An output the scorer cannot parse, or a judge panel with no usable verdict, gets a quality of None. n_scored counts only scored outputs, so an unscorable candidate leaves the ranking rather than losing points, and the report says so.
• Errors are counted separately. A call that failed is an error, not a low score. The report carries the error count, and a policy can cap the error rate. In the offline check, a missing quality value stays missing; it is never zeroed.
• Ungraded cases are named. A case that scored on no candidate is ungraded. A policy can cap the ungraded rate. A candidate cannot pass by omitting the cases it did worst on: every rule uses the complete case population as its denominator, not the scored subset.
• Exclusions are listed. When two runs are compared, only the cases present in both count, and the record lists what was left out. A harness comparison reports corpus drift, with a non-zero exit code, when the two runs carry different corpus versions; a case present on one side only is listed and not counted.
• Substitutions are disclosed. When a run could not use the model you named and another was used in its place, the report carries a substitution caveat and the substituted model is named. Nothing takes a model's place silently.
• Repeated sampling is measured per case. When a run repeats each case, the report states how many cases had comparable scores across passes and how many of those moved by more than 0.05. That population is its own count, not n_scored. A run that never repeated shows stability as not measured, never as perfect.
• Judge disagreement is surfaced. With a panel, the case score is the median of the parseable verdicts. When the verdicts spread by more than one point on the underlying five-point scale, the result carries a disagreement caveat instead of a confident-looking median.

4. On using a model to judge a model

Grading LLM output with an LLM is circular if you leave it there, so it is not left there.

• Judges are pinned and versioned. The judge model and its version are recorded on the result. A judge that changes underneath you would otherwise silently change your history.
• Fixed settings and recorded versions help make evaluations more comparable. Model-generated outputs and judge scores can still vary between runs. The judge is asked for its most deterministic setting where the provider supports one, and nothing here guarantees that two runs score the same.
• Panels, not a single grader. A panel of judges from different labs can score the same output. The case score is the median of the parseable verdicts, not the mean, so one outlier grader cannot drag the result.
• Failed judges are excluded, never invented. If a judge errors or returns something unparseable, it is dropped. If no judge on the panel returns a usable verdict, the case scores None and is excluded from the aggregate rather than filled in with a guess.
• Conflicts are disclosed. Judges are mapped to their lab family, so when a judge and a candidate come from the same lab, the report says so. You then decide how much weight that verdict deserves.

When the question is a matter of expert judgment rather than a metric, do not use a judge at all. Use section 5.

5. Blinded human review

Some quality cannot be reduced to a score. Whether a clinical note is safe to sign, whether an extraction missed the operative clause, whether a reply is in the right register: those need a person who knows the domain.

• Reviewers get a link. No account, no install, no eval framework to learn.
• Outputs are blinded. The reviewer sees the work, not which candidate produced it, so nobody is voting for a brand.
• Reviewers vote pairwise on the cases you selected, and can record why.
• Their verdicts join the record beside the metrics as a review. A review is not an approval, and the record never presents it as one.

6. How cost is modeled

Cost is reported three ways, and they are not interchangeable:

• Actual: what the run itself consumed.
• Projected API: the run extrapolated to your stated volume at each provider's published list price.
• Self-hosted: an estimate of GPU cost per cloud, for candidates you would run yourself.

7. The publish gate

A report diagnoses its own defects and writes them in as caveats. Before a share link is created, the gate reads them.

• Blocking caveats refuse the link. Cases dropped from the quality mean so that candidates rest on different subsets; an output truncated by a token budget and scored 0; a candidate that did not answer every case; a gate decided on part of the data; placeholder output from the demo provider; and candidates with unequal scored counts. Each of these is a defect a re-run can fix, and the refusal quotes the caveat verbatim.
• Advisory caveats are disclosed, not blocked. A small sample, an undeclared incumbent, a substituted model, an unpriced candidate, or judge disagreement stay on the page. A re-run does not fix them, and blocking them would refuse a customer's first report.
• The gate is not absolute. An owner can publish anyway. The override is written to the audit log as an event on that run before the link is created. No customer-facing page reads that log today, so the override is recorded, not displayed; the caveat itself stays on the published page.

8. Versioning and the signature

Every result records the methodology version that produced it, so a change in method is visible rather than silent, and an old record can still be read in the terms it was made under.

A record can also carry a signed attestation. It is signed with a key whose public half is published at a well-known URL, so a third party can verify a report offline, without asking us and without trusting our website at the time they check it. The signature shows the record was issued by RedCrown and not altered since. It does not show that every supplied field is true.

9. Distinctions the record keeps

10. Limits, stated plainly

11. Reproducing a result

Everything needed to re-derive a record is on it: dataset version, candidate list, model versions, thresholds, methodology version, and per-case evidence. Re-running the same feature produces a comparable record you can put next to the original, and redcrown check compares the two under your policy. Changing a threshold does not require a re-run, because the threshold governs only which candidates clear it, so it can be re-derived from the scores you already have.

If you want to see the shape of one before you run anything, the example records include a synthetic regression check and five model comparisons with their receipts attached.