The 2026 SEC Extraction Challenge

At a quality bar of 0.99 declared before the run, the cheapest model tested lands within 0.0016 F1 of the flagship baseline, at 1/97 of the price: $6.30 against $613.35 per 10,000 filings.

11 models were given the same prompt and the same 80 excerpts from 10-Q filings filed between 2026-07-01 and 2026-09-05, and asked for the same 8 standardized financial facts as strict JSON. Scoring is deterministic field F1 against the filer’s own XBRL facts. There is no judge model. The winner is defined before the run as the cheapest candidate clearing a field F1 of 0.99.

Open the proof: every item, every receipt, every caveat

Cost against accuracy

The cost axis is logarithmic because the candidates span $6.30 to $915.60 per 10,000 filings. The quality axis is zoomed to the range the results actually occupy, so the differences are visible at all. Read that zoom honestly: the best and the worst candidate here are separated by 0.94 percentage points of accuracy, across a 145-fold range of price.

The leaderboard

Ranked by the objective, which is cost. n scored is a column here rather than a footnote: every candidate was scored over all 80 filings, none was truncated, none was unscorable, and the run raises no grading, truncation or baseline caveat.

Cost is RedCrown’s list-price table applied to the tokens each model actually used, projected to 10,000 filings. Latency includes the routing hop. Exact record accuracy is the share of filings where all 8 fields were right at once.

Where the bar would have to be before the answer changed

The answer is the same at all four of those thresholds. It changes only once the bar goes above 0.996875, and the pick is then openai/gpt-5.6-sol at $122.40, 19 times the winner’s cost for 0.0016 more F1.

The residual error is a property of the filings, not of the models

Across 11 candidates and 880 results, only 8 distinct (filing, field) cells were missed by anyone at all. 72 of the 80 filings were extracted perfectly by every candidate. 2 of the 8 fields were correct on every model on every filing: operating_income_current_quarter_usd, diluted_eps_current_quarter. In total 24 field values out of 7,040 were wrong.

One of the 8 cells is task ambiguity, not a model error. On Molson Coors Beverage Co, the answer key takes the gross Sales line, 3,604,400,000, and all 11 candidates return Net sales, 3,096,500,000, printed one row below it. Both are on the face of the statement and the prompt gives no rule for choosing between them. It is published as a limitation rather than corrected, because rescoring a locked key to the answer the models happened to give is not a benchmark.

The error taxonomy

Classified from the filing text the model was actually sent, not from a re-fetch.

Zero wrong-quarter and year-to-date confusions. That was the failure this benchmark was built expecting to headline, and it did not occur once. The zero is measured rather than untested: a synthetic statement carrying a real year-to-date column is asserted into that bucket by the test suite, so the classifier is known to be able to report it.

One caution about a number that looks like corroboration and is not. exact_field_accuracy is numerically identical to field_f1 for every candidate on this run. The strict JSON schema forces every candidate to return all 8 keys, so precision and recall share a denominator and the two metrics cannot disagree. They are one measurement, not two.

gemini-3.7-flash: at the bar, and one repetition cannot say which side

The engine scored google/gemini-3.7-flash at 0.989062, which is 0.000938 below the declared bar, and the proof shows it below the bar because that is what one run computes. This page does not contradict that number. It explains why the benchmark cannot resolve which side of the line the model is on.

The identical frozen spec was executed a second time. Same prompt, same per-item inputs, same answer key, same bar, verified field by field. On that execution the same model scored 0.990625, which is above the bar. 4 of 11 candidates moved between the two runs, each by one field out of 640, and the largest movement was 0.001622. That is larger than the 0.000938 margin in either direction.

repeat_n is 1 and quality_stdev is null, so there is no within-run spread to appeal to. Calling this a failure would assert a precision the measurement does not have, and calling it a pass would do the same in the other direction. It is reported here as at the bar, unresolved at one repetition, with both figures shown.

The shape of its errors is the part worth acting on. 4 of this model’s 7 wrong fields are the correct figure divided by exactly ten, which is the one repeated failure mode in the whole run. No other candidate produced one. The receipts show the magnitude, not the cause, and we are not going to guess at one. What they do show is the operational risk: an answer off by a factor of ten is silent. It parses, it satisfies a strict JSON schema, it survives every check that only asks whether the output was well formed, and on a balance-sheet total it is the kind of number a person downstream will act on.

Reproducing this, and what will differ

Anyone re-running from the tag precedence will get a different revenue for one filing, and that is expected. The field’s ordered tag list puts RevenueFromContractWithCustomerExcludingAssessedTax first. For Oscar Health, Inc., a health insurer, that tag holds only 6,095,000, the ASC 606 slice, while premium revenue sits outside it. The face of the statement prints a total of 4,880,220,000, which is the filing’s own Revenues concept, and that is the value in the key. The departure is recorded in the answer key’s own provenance. Without this rule a reader concludes the key is broken.

11 of 100 resolved filings were moved to the development set because their excerpts did not print all eight labelled values. The split moved so that a flawless extractor can score 1.000 on the locked 80. The 0.99 bar was pre-declared in the spec and was never lowered to fit a result.

The universe was walked with a recorded seed: 5,232 10-Q filings in the window, 5,192 after de-duplicating by filer, 333 examined to resolve 100. The exclusion histogram describes filings examined under that walk, never the full universe.

The frozen spec, the answer key, the prompt hash and the per-file checksums are in the repository beside the run. Every model input is byte-identical across candidates: one prompt, one params dict, max_tokens 8,000, temperature 0.

Five times our configuration looked like a model failure

The honest part of this project is what it took to get a run worth publishing. Five separate times, a configuration artifact of ours looked exactly like a model being bad.

1. A 500-token output cap. It caps reasoning tokens together with answer tokens, so two models were cut off mid-reasoning and returned truncated or empty JSON. Published as-is, that reads as two models failing at extraction.
2. An out-of-credit 402 storm from the gateway. Every call failed. A model that cannot be reached is not a model that got the answer wrong, and a benchmark that scores it zero is lying.
3. A markdown fence. Under a loose JSON response format, one model wrapped correct JSON in a code fence. RedCrown scores an unparseable output as unscorable, so a model that got the answer right dropped out of the ranking entirely. The fix was a strict schema for all eleven, applied before the locked run.
4. One wrong cell in our own answer key. A health insurer's revenue resolved to the ASC 606 slice rather than the total revenue the filing prints, because our tag precedence list is right for an operating company and wrong for an insurer. That single cell was the difference between one model passing the bar and failing it.
5. The token cap again, at 2,000 this time, which emptied one output and left one candidate scored over 79 items while ten were scored over 80. A mean over a subset published beside means over the full set is not a leaderboard.

Each one was caught by asking whether the model was wrong or we were measuring wrong. RedCrown's publish gate now refuses a proof that carries the first, second, third or fifth: a truncated output, a config that could not be reached, an unscorable output that left the ranking, or candidates scored over different numbers of items. The fourth, a wrong cell in our own answer key, is the one no gate can see. A person found it, and it is disclosed above rather than gated.

Limitations

• Every candidate was routed through a single gateway rather than each vendor’s own API. That keeps the comparison fair and it means the quoted latency includes the gateway hop.
• The sample is 80 filings. It is small, and every figure on this page should be read with that n attached.
• The answer key is derived from the filer’s own XBRL tagging and inherits any tagging error the issuer made.
• The filings are recent, which limits memorization but does not eliminate it, because the filing text is supplied in the prompt.
• The set skews to mainstream operating companies. Banks, REITs and trusts fall out during tag resolution, so this does not measure extraction from a bank’s statements.
• repeat_n is 1. Ordering inside a group of equal scores is not stable, and five of the 11 candidates do not expose a temperature setting, so their sampling is not pinned.

Everything we changed, and why

Several of the items below started out looking like a model failing and turned out to be a configuration or answer-key defect of ours. They are disclosed here rather than quietly fixed, because a benchmark that only publishes the version of events that survived is not evidence.

max_tokens was raised from the spec's 500 to 8,000 before the locked call.

max_tokens caps reasoning tokens together with answer tokens. At 500, measured on a real excerpt, google/gemini-3.7-flash (481 reasoning tokens) and deepseek/deepseek-v4-flash (604) were cut off mid-reasoning and returned truncated or empty JSON, so publishing at 500 would have printed a token-budget artifact as a model failure. It was raised to 2,000, and at 2,000 an earlier execution of this same frozen spec still exhausted the budget on one of eighty items for deepseek/deepseek-v4-pro, which left that candidate scored over 79 items while ten were scored over 80. The published run uses 8,000. Output is billed per token actually emitted, so a higher cap costs nothing unless a model uses it, and on this run no candidate was truncated and every candidate was scored over all eighty items. Applies to: all eleven candidates, identically.

response_format is a strict json_schema, not json_object.

Under json_object, anthropic/claude-sonnet-5 returned correct JSON inside a markdown fence. RedCrown scores an unparseable output None, which means unscorable, so a correct model left the ranking instead of losing points. The strict schema was applied to all eleven before the locked run, and no candidate emitted a fenced or unparseable body in it. Applies to: all eleven candidates, identically.

Five candidates do not expose temperature and ran at provider default sampling.

temperature: 0 was sent to all eleven. These five ignore it, so their run is not pinned to greedy decoding and a repeat could differ.

All eleven ran reasoning_effort: low.

Reasoning tokens bill as output tokens, so effort is a cost lever as much as a quality lever. A higher effort would change both the cost ranking and, possibly, the residual errors.

All eleven were called through OpenRouter, not each vendor's own API.

One route for all eleven keeps the comparison fair. It also means quoted latency includes OpenRouter's hop and quoted cost is RedCrown's list-price table for the model, not an OpenRouter invoice.

openai/gpt-6-astra is the declared baseline, chosen as the flagship default, not a claim about anyone's production stack.

Every saving on this leaderboard is measured against it. Declaring a baseline bars that candidate from winning and forces a challenger to beat it strictly on the objective, which costs nothing here because the baseline is the dearest configuration on the board and would not have won under either setting. It takes no scoring advantage either: reference_source is labels, so all eleven were scored against the filer's own XBRL facts and never against each other.

11 of 100 resolved filings were moved to the development set because their excerpts did not print all eight labelled values.

The split moved so that a flawless extractor can score 1.000 on the locked 80. The 0.99 bar was pre-declared in the spec and was never lowered to fit a result.

The prompt carries a parent-attributable clause (rule 8), and the taxonomy measures whether models obeyed it.

Across the 100 resolved filings, stockholders equity has a rival consolidated concept on 20 and net income on 20, both printed on the face of the statement. 27 of those rival opportunities fall in the locked 80. The parent_vs_consolidated count in metrics.json reports what the models did with them.

The set skews to mainstream operating companies.

Banks, REITs and trusts systematically fail the spec's revenue and operating-income tag lists, so they drop out during resolution. 333 filings were examined to resolve 100. The benchmark measures extraction from a conventional condensed statement and does not measure a bank's.

The revenue tag list is right for an operating company and wrong for an insurer, and the resolver now checks the result rather than trusting the list.

The field's ordered tag list puts RevenueFromContractWithCustomerExcludingAssessedTax first. For a health insurer that tag holds only the ASC 606 slice while premium revenue sits outside it, so the tag resolves to a real, auditable number that is not the top line the filing prints. One filing in the locked 80, Oscar Health 1568651_0001568651-26-000069, is in that shape: its revenue is resolved from the filing's own Revenues concept, 4,880,220,000, which is the Total revenue line, and the departure from the tag order is recorded in the answer key's own reference_provenance. The resolver now also rejects any filing whose resolved revenue is below its positive resolved net income (revenue_below_net_income), which is the condition that identifies it. Over the 100 resolved filings that guard flags one, and it is in the development set. See tag_resolution_rules in metrics.json. Applies to: the answer key, not to any candidate.

Cost is RedCrown's list-price table applied to real token usage, not a provider invoice.

Each item's cost was computed as prompt_tokens/1000 * input price + completion_tokens/1000 * output price, using the token counts the provider returned. Reasoning tokens are inside completion_tokens, so they are billed here as the provider bills them. The prices themselves are RedCrown's published table for these models.

Prove your own decision

This page is a worked example of what RedCrown produces: a named decision, the alternatives, the threshold, the economics and the item-level evidence, in one shareable record. RedCrown sells no models and routes no traffic, so the proof has no thumb on the scale.

Prove a decision Open the full proof