Run and rank
Run an experiment spec and read the ranked report.
1. Run
Execute a saved experiment spec against all candidate models. Raise concurrency for large token benchmarks to finish faster.
Open Experiments under the Advanced menu. Select a saved experiment and click Run. The ranked report appears when the run completes.
redcrown eval exp.json --concurrency 10Raising --concurrency runs candidates in parallel. Useful for many-candidate token benchmarks where sequential execution would take too long.
Create then run an experiment in two calls:
{
"experiment_id": "..."
}First call create_experiment with your spec, then call run_experiment with the returned experiment_id.
2. Reproducibility
Run a candidate multiple times to measure run-to-run quality variance. The report includes a mean and standard deviation confidence band.
The Prove composer at the app's home page cannot set repeat_n. This is a known, deliberate gap. Use Face-Off, the CLI, or the MCP tools for repeat runs.
redcrown eval exp.json --repeat 5Each candidate runs 5 times. The report shows the mean quality score and run-to-run standard deviation. Note that --repeat 5 multiplies spend by 5.
Set repeat_n in the experiment definition when calling create_experiment. The resulting report includes the mean and standard deviation for each candidate.
The standard deviation and per-item stability measure different things. quality_stdev is computed from each pass's dataset mean. It cannot see two items flip in opposite directions: the mean stays the same on every pass, so quality_stdev reads 0.000 and the run looks perfectly reproducible while every answer changed underneath it. Per-item stability answers the question quality_stdev cannot. It compares each item's own scores across the passes, not the dataset mean.
Each candidate in the report carries three stability fields:
stability, the fraction of items whose score held steady across passes. An item counts as unstable when its scores across passes differ by more than 0.05. A value of 1.0 means every scored item held steady; it does not mean every item was measured.unstable_items, the count of items that crossed that 0.05 threshold.stability_scored_n, the count of items that entered the stability measurement, that is, items with at least two comparable scores across passes.
Read stability_scored_n, not n_scored, as the population a stability claim is made over. The two counts answer different questions. n_scored counts every item that produced a usable score on its first pass, even one that later errored on a repeat pass. stability_scored_n counts only items with two or more comparable scores, because a single score has no spread to measure. Stating unstable_items "of n_scored" would divide by the wrong population and over- or understate the true rate depending on where the errors land.
Any of these three fields can be absent (None). An absent value means stability was never measured, most often because repeat_n is 1 or unset. An absent value is not the same as a perfectly stable 1.0, and must never be read as one.
On a run graded by an LLM judge, the report's reproducibility note says so explicitly: the per-item variation shown is the model and the grader together, not the model alone.
3. Gate a decision on stability
Add a gate on the stability metric to bar a candidate whose per-item answers flip too often. A gate can bar a candidate on quality or cost the same way.
Not available via the UI. Build the experiment spec with the CLI or the MCP create_experiment tool.
Add a gate with metric "stability" to the experiment spec's gates list, and set --repeat (or repeat_n in the spec) to 2 or more.
Add {"metric": "stability", ...} to the experiment's gates list when calling create_experiment, with repeat_n set to 2 or more.
A stability gate needs repeat_n of 2 or more. Stability is only ever measured across repeated passes. A spec that declares a stability gate with repeat_n below 2 is refused before any spend. Stability would never be measured, so every candidate would fail the gate for a reason the report cannot explain. Set repeat_n to 2 or more, or remove the gate.
A stability gate is opt-in. Ranking is unchanged unless a spec declares one.
4. Read the winner
The ranked report identifies the cheapest candidate that clears your quality bar. Every other candidate is ranked below it with its score and cost.
The ranked report highlights the winner row in green. The table shows quality score, cost, latency, and savings vs. the incumbent for each candidate.
The ranked report prints to the terminal at the end of the run. Add --report-json to save it:
redcrown eval exp.json --report-json out.jsonCall get_report with the run id to fetch the full ranked report:
{
"run_id": "..."
}5. Score with a judge panel
For the free-text judge quality metric, use more than one independent grader model instead of one, so a single judge's bias or blind spot does not decide the winner alone.
Not available via the UI yet. Build the experiment spec with the CLI or the MCP create_experiment tool, then run or push it; the ranked report renders the panel result on every surface.
redcrown eval exp.json --judge openrouter:openai/gpt-5.5 --judge openrouter:anthropic/claude-sonnet-4.6 --judge openrouter:google/gemini-3-proRepeat --judge provider:model for each panel member (the experiment's quality_metric must be "judge"). Every judge scores each output independently at temperature 0, and an item's score is the MEDIAN of the panel's verdicts, not the mean, so one outlier judge cannot swing the result on its own. A judge that errors or returns unparseable output is dropped from that item's median rather than counted as a low score. Omit --judge and RedCrown falls back to a single default judge, same as before.
When a panel scores an item, the report flags it when the judges disagree (their verdicts spread more than one point apart on the underlying 1 to 5 scale), and it discloses two things worth weighing before trusting the verdict: when a panel judge shares a lab family with the winning model (for example, a Qwen judge scoring a Qwen challenger), and when the winner would flip under a single judge alone instead of the full panel. Judge-call spend is totaled and reported separately from candidate cost; it is never folded into cost or savings.
Set judges (a list of {"provider", "model"} objects) and quality_metric: "judge" in the experiment definition passed to create_experiment. There is no dedicated MCP tool for choosing a panel yet.
6. Rejudge a run without re-running generation
Re-score an already-run report's stored outputs with a different judge panel. Generation is never re-run, so a rejudge costs one judge pass per panel member, not a full re-run of every candidate.
Not available via the UI. Use the CLI.
redcrown rejudge report.json --judge openrouter:anthropic/claude-sonnet-4.6 --judge openrouter:openai/gpt-5.5 --out rejudged.json
redcrown push rejudged.json --proof-linkrejudge needs a report produced with eval --report-json that still carries its per-item receipts (not a run pushed with --no-receipts), and it only applies to a report whose quality_metric is "judge". Add --keep-existing to fold the original run's judge score into the new panel's median instead of replacing it outright. Push the result to get a new run and, with --proof-link, a new shareable proof. Rejudge only runs when you invoke it; RedCrown does not re-judge runs automatically or on a schedule.
Rejudge sends the stored outputs and their references to each new judge's provider, a third party, for scoring, so only rejudge with a panel you are willing to show that data to.
Example: a pushed run was judged by a single in-house model. Rejudge its receipts with a cross-lab panel, say one OpenAI model and one Anthropic model, to check whether the original judge's lab family biased the result, for the cost of a handful of judge calls instead of paying to re-run every candidate again.
Not on MCP. Use the CLI.
Last verified .