Reviewer portal
Send a run to a non-technical domain expert for a blinded verdict.
1. Create a session and invite
A review session selects the contested cases from a run and sends a magic-link invite to an expert. Model names are hidden from the reviewer.
Every Prove result now offers a review, not only ones you go looking for: click Send for review to invite an expert, or Review it yourself to judge the same blinded cases under your own name, still through the same /review/:token flow. When the machine could not decide on its own (no model cleared the bar, a model returned nothing, the top two are a near-tie, too few items, or nothing beat what you already use) the result screen says so and points at review instead of leaving you to notice. RedCrown seeds the contested cases automatically and gives you a prepared message to send, not a bare link, so the invite reads like an ask a busy reviewer will actually open. If the models agreed on every item, the request is refused (422): there is nothing contested to judge. (A second, endpoint-based review path also exists on the Live traffic page for reviewing a proxied endpoint's captures; that is the path MCP's tools cover, see the MCP tab.)
Not available via the CLI. Use the app or MCP.
MCP's create_review covers a proxied endpoint's captures, not a Prove run report directly (use the UI's Send for review button for that). Call it with the endpoint ID:
{
"endpoint_id": "<endpoint-id>",
"name": "Transcription review"
}Then call invite_reviewer with the resulting session ID:
{
"session_id": "<session-id>",
"reviewer_label": "Dr. Smith"
}invite_reviewer returns the one-time token. The reviewer link is app.redcrown.ai/review/<token>.
2. The expert reviews (blinded)
The reviewer sees Option A and Option B without model names. They vote on each case, optionally adding a confidence level and severity flag.
The reviewer opens app.redcrown.ai/review/<token>. Both reviewer surfaces open by stating what is being asked, that model names are hidden, that there is no account to create, and that verdicts may be published unnamed and in aggregate while written comments are never published. On a run-based review the reviewer then settles the contested cases one at a time, picking Option A or Option B (or one of the escape hatches: both fine, both poor, neither, not sure); a returning reviewer sees how many cases they already settled and picks up where they left off. On the older capture-based review each clip shows the current (incumbent) output, then one card per candidate word-diffed against it, and the reviewer clicks Accept or Flag on each candidate card with an optional note.
Not available via the CLI. Use the app or MCP.
Call get_review_examples with a capture ID (not the session ID) to retrieve the blinded candidate outputs for one captured request:
{
"capture_id": "<capture-id>"
}Each example includes the incumbent output, the challenger output, and the reviewer's vote if already cast.
3. Read the decision
Once votes are in, the decision rollup un-blinds the candidates and names the recommended model based on accept rate and cost.
For a run-based (Send for review) session, open app.redcrown.ai/reviews/<sessionId>: the recommended candidate is shown at the top with a verdict summary, followed by a per-category breakdown of where reviewers agreed or split. For an endpoint-based session, the decision report renders inline on the Live traffic page: the recommended candidate at the top, then a table of accept rate, accepts/flags, mean cost, and mean divergence per candidate.
Not available via the CLI. Use the app or MCP.
Call get_decision_report with the session ID:
{
"session_id": "<session-id>"
}The shape depends on the session type: an endpoint-based session returns the recommended candidate plus accept rate, mean cost, and mean divergence per candidate; a run-based session returns a recommendation, review stats, and a per-category breakdown instead.
4. What the public proof shows
A run-based review is resolved into the public proof page at read time, not baked into the frozen report, so votes cast after a proof was minted still show up when the page is next opened.
A proof with any judged cases shows a Human review section: how many contested cases were judged, reviewer counts split by self versus invited (never summed together, since a self-review is never described as independent), the verdict split, the agreement rate, which model reviewers preferred, and whether that agrees with the ranking. Disagreement is rendered at the same weight as agreement. Reviewer names and written comments are never published, only the counts and the verdict.
Not available via the CLI. Read the section on the proof page itself.
Not exposed as a tool. The same data is the review key on the public proof payload (GET /proof/<token>), null when nobody has judged it yet.
Last verified .