Benchmarks
Run RedCrown's public weekly benchmarks, or a pinned subset of your own.
1. Build a pinned subset
Pin exactly the items you want to evaluate by passing an IDs file. This ensures every re-run uses the same inputs so results are comparable over time.
Benchmarks run from the CLI. View public benchmark results at redcrown.ai/benchmarks/.
Build a pinned experiment from the PriMock57 medical-transcription corpus. The --ids-file flag pins exactly those items:
redcrown build-dataset primock57 --ids-file subset.txt --out exp.jsonsubset.txt is a newline-delimited list of item IDs. An empty file raises an error loudly. The output exp.json is a validated experiment spec ready for redcrown eval.
Not available via MCP. Use the CLI above to build benchmark datasets.
2. Run it
Evaluate the pinned experiment across all candidates. Use concurrency to finish large benchmark runs in minutes rather than hours.
View the results of RedCrown's public benchmarks at redcrown.ai/benchmarks/. The page shows the ranked table, winner, savings over the incumbent, and methodology for each benchmark.
redcrown eval exp.json --concurrency 10The ranked report prints to the terminal when the run completes. Add --report-json out.json to write the full receipts to a file, and redcrown push out.json --proof-link to share the results.
Not available via MCP. Use the CLI above to run benchmark experiments.
Last verified .