The Great LLM Bake Off: 11 Models, $1.07, 83% Cheaper per Company

November 3, 2026 · AI · BloomWatch

Hypothesis

Can I get the same results from other models and vendors, through one router (OpenRouter), for a fraction of the cost?

Answer: yes. The cheapest model matching today's answers on each of the six model calls tested cuts cost per 1,000 companies from $6.38 to $1.08. The new models went live in production on November 3.

Why

BloomWatch tracks startups right after they raise money. The pipeline has more steps than this; six of them call a language model, and all six ran on Anthropic's Claude Haiku 4.5, or Sonnet 5 for one step. Spend was about $0.90 a month, so this was not about the bill. Cheaper calls leave room for more input per company: more pages per company, and a second pass.

Before

* Measured October 31 on 8 companies. Sonnet 5 was not part of this test.

The test

• Same input for every model: saved inputs for 30 real companies.
• Same code: the pipeline's own functions and prompts, with only the model swapped.
• 11 models: Claude Haiku 4.5, Haiku 5.5, and Sonnet 5.5; GPT-6 Luna, GPT-5 nano, GPT-5 mini, and gpt-oss-120b; Gemini 2.5 and 3.1 Flash-Lite; DeepSeek V4.1 Flash; Qwen 3.5 Flash. Sonnet 5.5 ran on two steps only.
• One host per model: through OpenRouter, pinned to the vendor's own servers with fallbacks off. Where the vendor has none on OpenRouter, one named host (DeepInfra for DeepSeek, Groq for gpt-oss-120b).
• Thinking off, or at the lowest setting a model allows. Sonnet 5.5 refuses to turn it off.
• Hard budget: the script tracked billed cost per call and would stop at $2.00 across all runs.

Scoring

Each answer is graded field by field. A missing value costs 1 point, a wrong value costs 3, since a wrong fact shown as true does more harm than a blank. Two steps set their own weights: calling a different company's website "the same" costs 5, and so does flagging a real company name as fake.

Every model is graded twice: against what BloomWatch stored, and against Haiku 4.5's answer on the same input (the model in use before). The stored data is not perfect ground truth, so the second number answers the practical question: does the cheaper model give the same answer?

The website check had no negative cases in the sample, so I added 28: each company paired with a different company's homepage, with the name swapped in. This is the expensive mistake in production, a stranger's website shown as the company's.

Results

Costs per 1,000 calls at list price, read from OpenRouter on test day. Quality is from 24 to 56 calls per model per step; validate names is one batch of 34 names per model.

Where models differed

One of GPT-5 nano's misses: the company is a travel platform for group trips. The test homepage was an identity verification company with the travel company's name swapped in. GPT-5 nano answered same; GPT-6 Luna, at $0.01 more per 1,000 calls, answered not same.

Five cheaper models had all 19 drafts read against the same test resume:

• GPT-6 Luna ($0.19 per 1,000): only facts from the resume, specific numbers, plain tone, and it says so when the background is a stretch.
• Haiku 4.5 ($2.13): good, but vaguer, with one claim not in the resume.
• gpt-oss-120b ($0.29) and GPT-5 mini ($0.59): invented candidate facts, such as languages and skills the resume never mentions.
• Gemini 3.1 Flash-Lite ($0.54): stiff ("I am writing to" in 11 of 19), and claimed a resume was attached in 7 of 19.

Router overhead

Haiku 4.5 ran both through OpenRouter and direct on Anthropic's API, same inputs, to measure the extra hop. Median latency 1.02 s through OpenRouter vs 0.78 s direct, 0.41 s slower at the 90th percentile, over about 110 calls each way. Billed cost was identical to the vendor's list price on every call.

The switch

• One router key: all six calls go through OpenRouter, each pinned to the vendor's own servers.
• Fallback: any error, timeout, rate limit, or outage retries on Haiku 4.5 direct on Anthropic's API.
• Qwen dropped: broken JSON on extraction and the most length-rule misses on outreach.
• Before, on production: 3.0 model calls and $0.0049 per new company (November 3 run, 9 companies).
• First run on the new models: 7 calls for $0.0011, no fallbacks.

Cost vs value

Conclusion

The hypothesis held. Two cheaper vendors match the old answers on all six calls tested, at about a sixth of the cost per company. A hand-checked answer key is next, to measure accuracy instead of agreement.

Part 2: The LLM Bake Off, Graded, the same answers graded against a hand-checked answer key.

Cost per 1,000 companies

$6.38 → $1.08

Outreach draft per 1,000

$2.13 → $0.19

Fake websites accepted

0 of 28

Extraction answers matching Haiku 4.5

97%

Latency added by the router

0.24 s median

Bake-off cost

$1.07

What I learned

• Test the expensive mistake on purpose. On real data every model scored 100% on the website check. The 28 fake cases split them.
• Stored data is not ground truth. On the careers link all 11 models agreed with each other and “missed” the same 9 stored values, which came from a later step.
• Newer is not always closer. Haiku 5.5 is 10× cheaper per token than Haiku 4.5, but sent 37% more input tokens for the same prompt and wrote longer emails than asked.
• Price per token is half the story. Gemini 3.1 Flash-Lite costs 2.5× more per token than 2.5 Flash-Lite and made fewer wrong extractions.

See BloomWatch →