The LLM Bake Off, Graded: Summaries 76% to 95% Right, 2 Picks Changed, $0

November 3, 2026 · AI · BloomWatch

Part 2 of The Great LLM Bake Off. Same models, same saved answers, graded against the right answers instead of against Claude Haiku 4.5.

Hypothesis

Can cheaper models replace Claude Haiku 4.5 on BloomWatch's six model calls, measured by accuracy against a hand-checked answer key?

Answer: yes. Accuracy changed two of the six picks. Both now run on Claude Haiku 5.5, the most accurate model on both (tied on extraction) and cheaper than the part 1 picks. The other four stay on GPT-6 Luna.

Why

Part 1 graded each model two ways: against the data BloomWatch had stored, and against Haiku 4.5's answer on the same input. Most of the stored data came from Haiku 4.5. A model could match Haiku 4.5 and still be wrong, and a model could be right and look worse for disagreeing.

Before

Part 1's reference was the weak link. On the careers link, the stored value was often a job board a later step found, not a link on the homepage, so every model “missed” it.

The answer key

• Read by hand from the source: each company's funding article, about page, and homepage link list, the same text every model saw. Never from a model's output.
• 30 companies: 24 with an article, 28 with an about page, 29 with a link list. A small sample; one field moves extraction accuracy by half a point.
• 10 models: part 1's 11 minus Sonnet 5.5, which ran on two other call sites only. Haiku 4.5 is counted once, on its direct route.
• Every field, not only the disputed ones, so a mistake every model made still shows up.
• “Unknown” when the source does not say. Leaving the field empty is then the right answer.
• Two readings allowed where the source allows two: a $25M total made of an $18.5M Series A plus an earlier seed counts either way.

Scoring

Each field gets one of four marks. Right includes a correct blank. Missing: the key has a value, the model left it empty. Wrong: a different value. Invented: the source says nothing, the model gave a value anyway. Accuracy is the share of fields marked right. A call returning broken JSON counts every field as missing.

The other three call sites already had real ground truth in part 1 and keep it: 28 fake websites plus 28 real ones, 4 bad company names among 34, and the outreach length rules.

Results

Before and After are cost per 1,000 calls at list price, from part 1's run. Haiku 4.5 was $6.38 per 1,000 companies. * Length rules only; tone was read by hand in part 1.

Where models differed

The prompt asks for a headcount range “inferred from any signal”, such as the number of open roles, and for null when there is no basis. Most models read a careers page listing 6 open roles as a basis for “10-25” or “25-50”. Haiku 5.5 left the field empty on 20 of the 21 pages with no number.

The gap between the top models is one or two fields out of 192, inside the noise of 30 companies. Price decided this one: Haiku 5.5 costs 46% less than the part 1 pick.

Mistakes the models shared

• All 10, Haiku 4.5 included: a headcount for a Norwegian building engineering startup, from a careers page with no number. Answers ranged from 10-25 to 25-50.
• 8 of 10, Haiku 4.5 included: “Education” as the industry of a household assistant app. The about page held a single blog post on teaching kids about money.
• 2 of 10, Haiku 4.5 included: a travel startup's co-founder named as CEO. The article lists the founders and never says who is CEO.
• 2 of 10: the CEO of a startup the company had just bought, given as the company's own CEO.
• None: no extraction field or careers link was missed by all 10.

One concrete case from part 1's picks, read from production on November 3 (5 companies checked by hand): the extraction model returned “Commander Transformation” as a CEO's name. It is part of an adviser's former NATO title. The article names the real CEO earlier in the text. On the same 5 companies, 3 headcounts were right (the page said 2-10) and 2 were guesses.

The switch

• Extract article and summarize company: Claude Haiku 5.5, through OpenRouter on Anthropic's servers, thinking off. Live in production November 3.
• The other four: GPT-6 Luna, no change.
• Fallback: Haiku 4.5 direct on Anthropic's API, as before.
• Trade-off: two calls and the fallback now depend on Anthropic. The fallback covers an OpenRouter outage, not an Anthropic one.

Cost vs value

Conclusion

The hypothesis held under a stricter test. Graded against the source, summaries went from 76.2% to 95.2% right and cost per 1,000 companies from $1.08 to $0.65.

Summary accuracy

76.2% → 95.2%

Extraction accuracy

99.5% → 100%

Headcounts guessed, 21 pages with no number

17 → 1

Cost per 1,000 companies

$1.08 → $0.65

Cost per 1,000 companies vs Haiku 4.5

$6.38 → $0.65

Rescore cost

$0

What I learned

• Grade against the source, not the last model. The stored data was 75% to 78% right, and 48% on careers links.
• Agreement hid the best summary model. Haiku 5.5 agreed with Haiku 4.5 on 76% of summary fields, the lowest of 10, and was the most accurate at 95.2%.
• The prompt invited the guessing. “Inferred from any signal” got a headcount from 8 of 10 models on more than half the pages with no number. A stricter prompt is next.
• Extraction is close to solved. 8 of 10 models scored 98.4% or better, so price decides.
• 30 companies is a small sample. Gaps under a point are ties; the summary gap of 19 points is not.

See BloomWatch →