Run and rank
Run an experiment spec and read the ranked report.
1. Run
Execute a saved experiment spec against all candidate models. Raise concurrency for large token benchmarks to finish faster.
Open Experiments under the Advanced menu. Select a saved experiment and click Run. The ranked report appears when the run completes.
redcrown eval exp.json --concurrency 10Raising --concurrency runs candidates in parallel. Useful for many-candidate token benchmarks where sequential execution would take too long.
Create then run an experiment in two calls:
{
"experiment_id": "..."
}First call create_experiment with your spec, then call run_experiment with the returned experiment_id.
2. Reproducibility
Run a candidate multiple times to measure run-to-run quality variance. The report includes a mean and standard deviation confidence band.
Not available via the UI. Use the CLI or MCP for repeat runs.
redcrown eval exp.json --repeat 5Each candidate runs 5 times. The report shows the mean quality score and run-to-run standard deviation. Note that --repeat 5 multiplies spend by 5.
Set repeat_n in the experiment definition when calling create_experiment. The resulting report includes the mean and standard deviation for each candidate.
3. Read the winner
The ranked report identifies the cheapest candidate that clears your quality bar. Every other candidate is ranked below it with its score and cost.
The ranked report highlights the winner row in green. The table shows quality score, cost, latency, and savings vs. the incumbent for each candidate.
The ranked report prints to the terminal at the end of the run. Add --report-json to save it:
redcrown eval exp.json --report-json out.jsonCall get_report with the run id to fetch the full ranked report:
{
"run_id": "..."
}4. Score with a judge panel
For the free-text judge quality metric, use more than one independent grader model instead of one, so a single judge's bias or blind spot does not decide the winner alone.
Not available via the UI yet. Build the experiment spec with the CLI or the MCP create_experiment tool, then run or push it; the ranked report renders the panel result on every surface.
redcrown eval exp.json --judge openrouter:openai/gpt-5.5 --judge openrouter:anthropic/claude-sonnet-4.6 --judge openrouter:google/gemini-3-proRepeat --judge provider:model for each panel member (the experiment's quality_metric must be "judge"). Every judge scores each output independently at temperature 0, and an item's score is the MEDIAN of the panel's verdicts, not the mean, so one outlier judge cannot swing the result on its own. A judge that errors or returns unparseable output is dropped from that item's median rather than counted as a low score. Omit --judge and RedCrown falls back to a single default judge, same as before.
When a panel scores an item, the report flags it when the judges disagree (their verdicts spread more than one point apart on the underlying 1 to 5 scale), and it discloses two things worth weighing before trusting the verdict: when a panel judge shares a lab family with the winning model (for example, a Qwen judge scoring a Qwen challenger), and when the winner would flip under a single judge alone instead of the full panel. Judge-call spend is totaled and reported separately from candidate cost; it is never folded into cost or savings.
Set judges (a list of {"provider", "model"} objects) and quality_metric: "judge" in the experiment definition passed to create_experiment. There is no dedicated MCP tool for choosing a panel yet.
5. Rejudge a run without re-running generation
Re-score an already-run report's stored outputs with a different judge panel. Generation is never re-run, so a rejudge costs one judge pass per panel member, not a full re-run of every candidate.
Not available via the UI. Use the CLI.
redcrown rejudge report.json --judge openrouter:anthropic/claude-sonnet-4.6 --judge openrouter:openai/gpt-5.5 --out rejudged.json
redcrown push rejudged.json --proof-linkrejudge needs a report produced with eval --report-json that still carries its per-item receipts (not a run pushed with --no-receipts), and it only applies to a report whose quality_metric is "judge". Add --keep-existing to fold the original run's judge score into the new panel's median instead of replacing it outright. Push the result to get a new run and, with --proof-link, a new shareable proof. Rejudge only runs when you invoke it; RedCrown does not re-judge runs automatically or on a schedule.
Rejudge sends the stored outputs and their references to each new judge's provider, a third party, for scoring, so only rejudge with a panel you are willing to show that data to.
Example: a pushed run was judged by a single in-house model. Rejudge its receipts with a cross-lab panel, say one OpenAI model and one Anthropic model, to check whether the original judge's lab family biased the result, for the cost of a handful of judge calls instead of paying to re-run every candidate again.
Not on MCP. Use the CLI.
Last verified .