# Benchmarks

Source: https://redcrown.ai/docs/guides/benchmarks/

Run RedCrown's public weekly benchmarks, or a pinned subset of your own.

## 1. Build a pinned subset

Pin exactly the items you want to evaluate by passing an IDs file. This ensures every re-run uses the same inputs so results are comparable over time.

**UI**

Benchmarks run from the CLI. View public benchmark results at [redcrown.ai/benchmarks/](https://redcrown.ai/benchmarks/).

**CLI**

Build a pinned experiment from the PriMock57 medical-transcription corpus. The `--ids-file` flag pins exactly those items:

```
redcrown build-dataset primock57 --ids-file subset.txt --out exp.json
```

`subset.txt` is a newline-delimited list of item IDs. An empty file raises an error loudly. The output `exp.json` is a validated experiment spec ready for `redcrown eval`.

**MCP**

Not available via MCP. Use the CLI above to build benchmark datasets.

## 2. Run it

Evaluate the pinned experiment across all candidates. Use concurrency to finish large benchmark runs in minutes rather than hours.

**UI**

View the results of RedCrown's public benchmarks at [redcrown.ai/benchmarks/](https://redcrown.ai/benchmarks/). The page shows the ranked table, winner, savings over the incumbent, and methodology for each benchmark.

**CLI**

```
redcrown eval exp.json --concurrency 10
```

The ranked report prints to the terminal when the run completes. Add `--report-json out.json` to write the full receipts to a file, and `redcrown push out.json --proof-link` to share the results.

**MCP**

Not available via MCP. Use the CLI above to run benchmark experiments.

Last verified 2026-07-02.