Run and rank · RedCrown Docs

1. Run

Execute a saved experiment spec against all candidate models. Raise concurrency for large token benchmarks to finish faster.

UI

Open Experiments under the Advanced menu. Select a saved experiment and click Run. The ranked report appears when the run completes.

CLI

redcrown eval exp.json --concurrency 10

Raising --concurrency runs candidates in parallel. Useful for many-candidate token benchmarks where sequential execution would take too long.

MCP

Create then run an experiment in two calls:

{
  "experiment_id": "..."
}

First call create_experiment with your spec, then call run_experiment with the returned experiment_id.

2. Reproducibility

Run a candidate multiple times to measure run-to-run quality variance. The report includes a mean and standard deviation confidence band.

UI

Not available via the UI. Use the CLI or MCP for repeat runs.

CLI

redcrown eval exp.json --repeat 5

Each candidate runs 5 times. The report shows the mean quality score and run-to-run standard deviation. Note that --repeat 5 multiplies spend by 5.

MCP

Set repeat_n in the experiment definition when calling create_experiment. The resulting report includes the mean and standard deviation for each candidate.

3. Read the winner

The ranked report identifies the cheapest candidate that clears your quality bar. Every other candidate is ranked below it with its score and cost.

UI

The ranked report highlights the winner row in green. The table shows quality score, cost, latency, and savings vs. the incumbent for each candidate.

CLI

The ranked report prints to the terminal at the end of the run. Add --report-json to save it:

redcrown eval exp.json --report-json out.json

MCP

Call get_report with the run id to fetch the full ranked report:

{
  "run_id": "..."
}