Compare all three

Run the check in CI

Put redcrown check or redcrown harness compare in a pipeline step, keep its exit status, and let your pipeline decide what a failed check means. This page uses the files from Check your first AI change. Every shell command on it was run on those files before it was written here.

1. The boundary

A CI gate is five parts, and RedCrown owns one of them.

Your test runner, or supplied fixtures, produce the outputs. RedCrown does not run your application.
The evidence artifacts: for redcrown check, one evidence file per version from redcrown score --evidence-json (or redcrown eval --evidence-json) and a policy JSON; for redcrown harness compare, a journal directory with one subdirectory per run.
The command: redcrown check applies the policy to the candidate and the reference; redcrown harness compare diffs two harness runs check by check. Both read local files only, make no network call, and need no account.
The exit status, preserved by your CI system. This is the whole interface.
Your pipeline's decision. Whether a nonzero exit blocks a merge, a deploy, or nothing is a rule you configure in your CI system. A successful command configures no protection anywhere.

What you configure in your CI system, and RedCrown does not: the job and when it runs; where the evidence artifacts come from and where the result file goes; the step that fails the job on a nonzero exit (most runners do this by default for a plain command, and stop doing it the moment the command is followed by || true or is on the left side of an unguarded pipe); and a branch protection rule that requires the job, if you want the failed job to block a merge. RedCrown does not send notifications, does not run on a schedule, and does not know your branches.

2. Exit codes, per command

The commands use different codes. Do not apply the codes of one command to another.

Command	0	1	2
`redcrown check`	pass: every rule passed	error: a command, file or policy problem (unreadable file, a candidate id not in the file, a policy in YAML, a missing baseline when a regression rule needs one)	fail or indeterminate: a rule was breached, or the evidence could not answer a rule
`redcrown harness compare`	no regression	at least one check regressed on a shared case	corpus drift: the two runs are on different corpus versions and `--allow-corpus-drift` was not given
`redcrown harness report` without `--policy`	report printed	the run directory does not exist	not used
`redcrown harness report --policy` (Needs redcrown 0.1.23 or later.)	pass: every rule in the policy passed	error: an input or policy problem (the run or the baseline run does not exist, the policy file cannot be read, a policy in YAML, a policy that is not valid, a `no_regression` rule with no `--baseline`)	fail or indeterminate: a rule was breached, or the journal could not answer a rule

For redcrown check, exit 2 covers both a demonstrated breach and a measurement gap. A gate that must fail closed treats exit 2 as a failure in both cases: an indeterminate required rule is not a pass. The text, Markdown and JSON output all say which it was, with the reason per rule (threshold_breached, incomplete_quality_coverage, incompatible_evidence). Read the reason to decide what to fix; do not read it to decide whether the job passed.

"No new regression" and "requirements met" are different results. redcrown harness compare exits 0 when no check got worse than in the earlier run. A run that failed half its cases in both runs also exits 0. redcrown harness report --policy exits 0 only when the run meets every rule the owner wrote. Use the compare to find what changed. Use the policy to decide whether the run is acceptable.

3. A shell gate that fails closed

The script scores the fixture, then checks the repaired candidate against the reference run. With set -e, the first nonzero exit stops the script, so exit 2 and exit 1 both fail the job. The result file is written by redirect, not by a pipe, so the exit status the shell sees is the check's own.

CLI

#!/usr/bin/env bash
# Fail closed: a nonzero exit from any command fails the job.
set -euo pipefail

redcrown score score-request.json --evidence-json evidence.json

# Exit 0 continues the job. Exit 2 (fail or indeterminate) and exit 1
# (command, file or policy error) both stop it here.
redcrown check evidence.json --candidate-id candidate-v3 \
  --baseline evidence.json --baseline-candidate-id reference-v1 \
  --policy policy.json --format json > check-result.json

echo "check passed"

Run on the fixture: the script prints the score table, then check passed, and exits 0. With candidate-v2-partial in place of candidate-v3, the check exits 2 on maximum_error_rate, the script stops before the echo, and the job fails; that was run too. check-result.json carries the same result as the terminal output, with exit_code, status, every rule, the population counts and the hashes of both sides, so a later step or a reviewer can read it without re-running the check.

Two things never to write in this script: redcrown check ... || true, which turns every failure into a pass; and redcrown check ... | tee log.txt without pipefail, which reports tee's status instead of the check's.

Not available in the app. The check runs where the evidence files are.

MCP

Not available via MCP.

Continue after an expected failure

Sometimes a job must continue past a failing check, for example a job that asserts the fixture still fails the way the quickstart says it does. The only safe form captures the exit code and asserts the exact value. An unasserted continue is the same as || true.

CLI

#!/usr/bin/env bash
set -euo pipefail

redcrown score score-request.json --evidence-json evidence.json

# Continue after the expected failure, then assert the exact code.
set +e
redcrown check evidence.json --candidate-id candidate-v2 \
  --baseline evidence.json --baseline-candidate-id reference-v1 \
  --policy policy.json
code=$?
set -e
test "$code" -eq 2

echo "candidate-v2 failed the check as expected (exit $code)"

Run on the fixture: the check prints redcrown check: fail (exit 2) with the two breached quality rules, test passes, and the script exits 0. If the check had exited 0 or 1, test would fail and the job with it.

Not available in the app.

MCP

Not available via MCP.

The same gate for a harness journal

For a journal your test runner writes (the format is in Start from what you already have), the command is redcrown harness compare, and exit 1 is the regression signal.

CLI

#!/usr/bin/env bash
set -euo pipefail

# Exit 1 (a regression) or exit 2 (corpus drift) stops the job here.
redcrown harness compare 2026-09-16-a 2026-09-17-b --runs runs \
  --format json > compare-result.json

echo "no regression"

Run on the two three-case runs from that guide, where pass-case lost its persisted check between the runs: the compare exits 1, the script stops before the echo, and the job fails. Comparing a run with itself exits 0 and prints no regression. A measured check that became unmeasured is inconclusive, not a regression, and does not fail the job; read compare-result.json if you want to fail on that too.

After two runs are pushed to one experiment, the report view shows the same per-check compare. The exit status exists only on the command line.

MCP

Not available via MCP.

A harness run against a written policy

To gate on the requirements and not only on a regression, write a policy file and pass it to redcrown harness report. This policy asks for a pass rate of at least 0.9 over the graded cases (PASS plus FAIL) and for no regression against the baseline run.

{
  "schema_version": 1,
  "kind": "harness",
  "requirements": {
    "minimum_pass_rate": {"threshold": 0.9, "denominator": "graded"},
    "no_regression": {}
  }
}

CLI

#!/usr/bin/env bash
set -euo pipefail

# Exit 2 (fail or indeterminate) and exit 1 (input or policy error)
# both stop the job here.
redcrown harness report 2026-09-17-b --runs runs \
  --policy policy.json --baseline 2026-09-16-a \
  --format json > policy-result.json

echo "requirements met"

Run on the same two runs: the command exits 2 and the script stops before the echo. policy-result.json names both failed rules:

{"cases":3,"exit_code":2,"kind":"harness","rules":[{"case_ids":["fail-case","pass-case"],"count":0,"denominator":2,"id":"minimum_pass_rate","observed":0.0,"reason":"pass_rate_below_threshold","status":"fail","threshold":0.9},{"case_ids":["pass-case"],"count":1,"id":"no_regression","reason":"regression","status":"fail"}],"schema_version":"1","status":"fail"}

With 2026-09-16-a as the run and as the baseline, harness compare exits 0, but the policy still exits 2: no_regression passes and minimum_pass_rate fails at 0.5. With a threshold of 0.5, that run passes, and the script prints requirements met and exits 0. A rule that has no evidence, such as a required check that no case measured, is indeterminate and also exits 2. A no_regression rule with no --baseline exits 1.

Not available in the app.

MCP

Not available via MCP.

4. A GitHub Actions job

The job installs the released CLI, downloads the quickstart fixture, and runs the first two shell gates above. GitHub Actions fails a step on a nonzero exit and stops the job, which is the behavior the gate relies on; nothing in the YAML re-implements it. This YAML was not executed for this page, because Actions cannot run locally. Every shell command inside it was run on the fixture, on the released CLI, and exited as stated above.

CLI

name: regression-check
on: [pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install redcrown==0.1.23
      - name: Download the quickstart fixture
        run: |
          curl -fsSLO https://redcrown.ai/docs/quickstart/regression-check/score-request.json
          curl -fsSLO https://redcrown.ai/docs/quickstart/regression-check/policy.json
      - name: Score the supplied outputs
        run: redcrown score score-request.json --evidence-json evidence.json
      - name: Check the candidate against the reference run
        # Exit 2 (fail or indeterminate) and exit 1 (error) both fail this step.
        run: |
          redcrown check evidence.json --candidate-id candidate-v3 \
            --baseline evidence.json --baseline-candidate-id reference-v1 \
            --policy policy.json --format json > check-result.json
      - name: Keep the result with the job
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: check-result
          path: check-result.json

Replace the fixture download with the step that produces your own evidence files, and candidate-v3 with the candidate id your scoring request declares. evidence.json can contain your inputs, references and outputs; the artifact upload above keeps check-result.json only, which carries rules, counts, excluded case ids and hashes, not the cases themselves. To block a merge on this job, add it to the branch protection rule for the branch; the job cannot do that by itself.

Not available in the app.

MCP

Not available via MCP.

5. What the gate does not do

It does not run your feature. The outputs come from your runner or your fixtures.
It does not decide whether a failure blocks anything. Your pipeline does.
It does not accept a baseline. A hosted acceptance is a separate decision a workspace member makes in the app; a local reference file is not one.
It does not send anything anywhere. Pushing a report to the app is a separate, explicit command.
It does not run on a schedule and does not notify. Nothing in RedCrown does.

Last verified 2026-09-23.