Reviewer portal
Send a run to a non-technical domain expert for a blinded verdict.
1. Create a session and invite
A review session selects the contested cases from a run and sends a magic-link invite to an expert. Model names are hidden from the reviewer.
Open a completed run report and click Send for review. RedCrown seeds the contested cases automatically. Copy the invite link and share it with your expert.
Not available via the CLI. Use the app or MCP.
Call create_review with the run ID and an accept threshold, then call invite_reviewer to mint a magic-link token:
{
"run_id": "<run-id>",
"accept_threshold": 0.8
}Then invite:
{
"session_id": "<session-id>",
"label": "Dr. Smith"
}invite_reviewer returns the one-time token. The reviewer link is app.redcrown.ai/review/<token>.
2. The expert reviews (blinded)
The reviewer sees Option A and Option B without model names. They vote on each case, optionally adding a confidence level and severity flag.
The reviewer opens app.redcrown.ai/review/<token>. No account is needed. Each case shows the incumbent output and a challenger output side by side with word-diff highlighting. The reviewer selects Accept A, Accept B, or Flag for each case.
Not available via the CLI. Use the app or MCP.
Call get_review_examples with the session ID to retrieve the blinded cases and any existing votes:
{
"session_id": "<session-id>"
}Each example includes the incumbent output, the challenger output, and the reviewer's vote if already cast.
3. Read the decision
Once votes are in, the decision rollup un-blinds the candidates and names the recommended model based on accept rate and cost.
Open the un-blinded decision rollup at app.redcrown.ai/reviews/<sessionId>. The recommended candidate is shown at the top, followed by a ranked table of accept rates, mean cost, and divergence per candidate.
Not available via the CLI. Use the app or MCP.
Call get_decision_report with the session ID:
{
"session_id": "<session-id>"
}The response includes the recommended candidate, accept rate per candidate, mean cost, mean divergence, and cross-reviewer agreement.
Last verified .