How a decision gets made, and what it does not prove.

A decision record is only worth what its method is worth. This page states how RedCrown structures a comparison, scores it, brings in human judgement, and versions the whole thing, along with the places where the answer is weaker than it looks. If you are going to hand our output to a client or an auditor, you should be able to read this first and decide whether you believe it.

1. What a comparison is made of

Every decision starts from a workload: your real task, your real inputs. Not a public benchmark. You define:

• The incumbent. What you run today, so the comparison has a baseline rather than an abstract ranking.
• The candidates. The models and configurations you are actually considering, LLM or not. Speech-to-text and document extraction are scored on the same footing as language models.
• The dataset. Your inputs, with labels if you have them.
• The threshold. The quality bar a candidate must clear before cost is allowed to matter.

The threshold is the part people skip and it is the part that decides the outcome. RedCrown does not pick a winner on quality alone or cost alone. It finds the candidates that clear the bar you set, and then reports the economics among those. Change the bar and the answer can change, which is why the bar is recorded on the decision.

2. How candidates are scored

Scoring depends on what the task is, and the method used is recorded alongside the result.

If you have no labels

You can still get a defensible cost answer. Score the challengers against your current model's output instead of a labeled answer key. That does not tell you the incumbent is good; it tells you which candidates match what you already ship, which is often the actual question when the goal is to reduce spend without regressing.

3. On using a model to judge a model

Grading LLM output with an LLM is circular if you leave it there, so it is not left there.

• Judges are pinned and versioned. The judge model and its version are recorded on the result. A judge that changes underneath you would otherwise silently change your history.
• Judging runs at temperature 0 so the same input scores the same way on a re-run.
• Panels, not a single grader. A panel of judges from different labs can score the same output. The item score is the median of the parseable verdicts, not the mean, so one outlier grader cannot drag the result.
• Failed judges are excluded, never invented. If a judge errors or returns something unparseable, it is dropped. If no judge on the panel returns a usable verdict, the item scores None and is excluded from the aggregate rather than being filled in with a guess.
• Disagreement is surfaced. When panel verdicts spread by more than one point on the underlying five-point scale, the result carries a disagreement caveat instead of presenting a confident-looking median.
• Conflicts are disclosed. Judges are mapped to their lab family, so when a judge and a candidate come from the same lab, the decision says so. You can then decide how much weight that verdict deserves.

And when the question is genuinely a matter of expert judgement rather than a metric, do not use a judge at all. Use section 4.

4. Blinded expert review

Some quality cannot be reduced to a score. Whether a clinical note is safe to sign, whether an extraction missed the operative clause, whether a reply is in the right register for your customers: those need a person who knows the domain.

• Reviewers get a magic link. No account, no install, no eval framework to learn.
• Outputs are blinded. The reviewer sees the work, not which model produced it, so nobody is voting for a brand.
• Reviewers vote pairwise on the examples you selected, and can record why.
• Their verdicts become part of the decision record alongside the metrics, rather than living in a separate email thread that nobody can find in six months.

5. How cost is modeled

Cost is reported three ways, and they are not interchangeable:

• Actual: what the run itself consumed.
• Projected API: the run extrapolated to your stated volume at each provider's published list price.
• Self-hosted: an estimate of GPU cost per cloud, for candidates you would run yourself.

6. Versioning and reproducibility

Every result records the methodology version that produced it, so a change in method is visible rather than silent, and an old decision can still be read in the terms it was made under.

A decision can also carry a signed attestation. It is signed with a key whose public half is published at a well-known URL, so a third party can verify a proof offline, without asking us and without trusting our website at the time they check it.

7. Limits, stated plainly

8. Reproducing a result

Everything needed to re-derive a decision is recorded on it: dataset version, candidate list, model versions, threshold, methodology version, and per-item evidence. Re-running the same workload produces a comparable decision you can put next to the original. Changing the threshold does not require a re-run at all, because the bar governs only which candidates clear it, so it can be re-derived from the scores you already have.

If you want to see the shape of one before you run anything, the examples gallery has real decisions with their receipts attached.