Compare models on your workload
Run every candidate model on your task, score each output against your examples, and rank the candidates by the cheapest one that clears your quality bar. This is the model-comparison workflow. It is a supported, secondary path: the primary path, checking a changed AI feature against a reference run and a written policy, is Check your first AI change. Come here when the decision is which model or configuration to put in the feature, not whether a change to it still meets the requirements.
This guide has three steps: describe the task and give a few examples; run every candidate and rank them; share the record. Each step is shown three ways: in the app UI, with the redcrown CLI, and with the MCP tools. Pick a surface above, or compare all three. A run here calls the providers on your own keys, so it costs what the providers charge; the sample tasks are the keyless way to see the shape of a result first.
1. Describe the task and give a few examples
A task is a plain-language instruction plus a few example inputs and, if you have them, the correct outputs to score against. RedCrown uses these to build a scored experiment. Or start from a journal you already keep: Start from what you already have grades a journal, production traces, or a rules list offline, with no account.
No labeled data? Set reference_source: "incumbent" to score every challenger against your current model's own output. You find which configuration is cheaper at equal-or-better quality without labeling anything: the incumbent's output is the reference. A run scored against an incumbent gets no evaluation identity, so it cannot be accepted as a baseline; see Accepted baselines.
RedCrown also picks the quality metric for you, from the shape of the example outputs you gave it, not just the task's kind. A short literal answer (roughly, 8 words or fewer on one line) gets a deterministic metric (exact match, field overlap, or a coarse text-similarity score); so does a reference that parses as structured data (JSON, a list, a code block) even if it runs longer than that. A genuinely free-form reference, most commonly an open-ended summary or a synthesized answer, is scored by an LLM judge instead, because a deterministic metric cannot grade meaning, and RedCrown requires a clear majority of your examples to look free-form before it makes that switch, so one long outlier among many short answers does not move the whole task onto the paid metric.
A judge call runs once per candidate, per item: 4 candidate models scored against 20 examples is 80 judge calls, not 20, and a multi-judge panel (see Score with a judge panel) multiplies that further by the panel size. Each call is billed on your own connected key for the judge's provider (openai by default), the same as any other model call in the run.
Reachability is checked at both ends, but by two different mechanisms depending on how the experiment was built. When the "Prove" plain-language flow builds the experiment (see above), if no connected key reaches the judge it falls back to a deterministic metric instead and says so on the report, rather than build a run that would score nothing. Face-Off picks its judge from what you have connected too, with its own fallback to a funded default, so it rarely reaches this case at all. Either way, at the moment you START a run, whatever experiment it is against and however it was built, RedCrown checks the judge against your CURRENT credentials and refuses the run outright if none reach it (credentials can change between when an experiment was built and when it is run, most commonly a key that has since been removed), rather than spend real model calls on a run that cannot produce a number; see What happens when a key is missing.
Open app.redcrown.ai and land on Prove (/). Type your task, expand add examples & a baseline, paste in a few example rows, then click Run on my data. If you are signed out, you are redirected to log in and then back to Prove.
Before the run spends anything, Prove can show a confirm card that names the scoring method. It shows when RedCrown's metric choice needs your attention. Reasons include an LLM judge that bills your own connected key, a judge RedCrown could not reach, or prose answers scored by text match. The card states the method and what it measures. It lists the other eligible methods you can pick instead. Picking another method re-scaffolds for free, before any spend. A judge needs a connected OpenAI key. Without one, RedCrown scores by comparing the text instead. The run starts only when you click Run, scored by <method>. If you pick a method other than the one RedCrown recommended, the report and its public proof carry a note naming both choices. The MCP prove_task tool asks the same way: it runs nothing and returns status: needs_confirmation with the eligible methods. The agent confirms by calling again with quality_metric.
For a RAG or system-prompt workload, click Add instructions & context above the rows to add, per example, the instructions the model should follow and the retrieved context it should use, kept separate from the question itself. RedCrown folds all three into the one input the model sees, in that order; a plain row with only a question runs exactly as before. Instructions and context you add here can appear on the report you share, the same as the task text.
Build a dataset from a CSV with your inputs and reference outputs:
redcrown build-dataset --from-csv examples.csv --task summarize \
--input-col text --reference-col summary --out task.jsonCall scaffold_experiment with a plain-language task and examples to get a validated experiment spec, or pass them directly to prove_task and let it handle scaffolding automatically:
prove_task({
"task": "Summarize each support ticket in one sentence.",
"examples": [{"input": "...", "output": "..."}]
})2. Run every candidate and rank them
RedCrown fans the task across every candidate model, scores each output against your references, records cost and latency per candidate, and ranks by the cheapest model that clears the quality bar. Every call goes to the provider on your keys.
After clicking Run on my data, Prove runs the experiment and shows the ranked report inline, with cost, quality, and receipts. The winner row is highlighted in green.
redcrown eval task.json --report-json out.jsonThe ranked report prints to the terminal. --report-json also writes the full receipts to out.json for sharing or inspection. Add --evidence-json evidence.json to write the private raw evidence that redcrown check reads, if you want to check a later run of the same experiment offline.
Call prove_task with your task and examples. It ranks several models on your examples and returns the ranking with the report's caveats. When the scoring method is not clear, it runs nothing and returns status: needs_confirmation with the eligible methods. Call it again with quality_metric set to one of them, for example "quality_metric": "similarity". Pass publish: true for a share link. For a stored example report, use try_sample.
{
"task": "Summarize a customer support ticket in one sentence.",
"examples": [
{"input": "Ticket text here...", "output": "One-sentence summary."}
]
}Last verified . To check a later change to the feature you chose, go back to Check your first AI change.