Benchmark
Is routing as good as always calling one strong model, and what does it cost? We measured it on 96 held-out prompts and published every answer, including the results that don’t flatter us.
- Quality vs always-Opus
- 44% (below)
- Quality vs always-Sonnet
- 51% (level)
- Your cost vs Opus
- 111%
- Held-out prompts
- 96
On 3 November 2026, Nivana answered two sets of 48 prompts it had never been tuned on: 8 kinds of request, each at three difficulties. The same prompts went directly to Claude Opus 5.5 and to Claude Sonnet 5.5. Three judges from other labs (GPT-6.1 Sol, Gemini 4 Argon and Kimi K3.1 Pro) compared each pair of answers in both orders. 50% means the judges rated the two as equally good.
Results
- Against always-Opus: rated below Opus, at about the same price. The judges preferred Nivana 44% of the time (95% interval 38–49%). The routed models cost 71% of what Opus did. With Nivana’s 50% service fee, a caller paid 111% of the Opus price. The first set alone looked level (49%), but the second didn’t confirm it (38%).
- Against always-Sonnet: level on quality, 2.6× the price. Nivana’s answers were preferred 51% of the time (45–57%). If Sonnet is good enough for everything you send, calling it directly costs less.
- Easy requests are where routing pays. On 32 easy prompts Nivana was level with Opus (49%) at a quarter of its cost.
- Hard requests and writing are where it falls short. On 32 hard prompts Nivana scored 37% against Opus and cost 1.4× as much. Hard reasoning and code that go to Opus at a high reasoning effort hold their own, and the losses come from hard writing, translation and maths sent to smaller models. Writing was the weakest task (26% against Opus).
- Why. The router’s quality scores rate the small models (Claude Haiku, GPT-6 Luna, DeepSeek Flash) within a few points of the top ones, and this benchmark says the gap is larger. Recalibrating those scores on harder prompts is the next change to the router, and it will be measured here before it ships.
A change tried after the first set (a writing quality floor and stricter difficulty labels) made no measurable difference on the second set and raised costs, so it wasn’t shipped. The benchmark also found a bug, now fixed: non-streamed API requests cut the model off at 180 seconds, which a hard code request at high reasoning effort could exceed. The limit is now 280 seconds.
How it was measured
- Held-out prompts. None of the 96 is in the suite that sets the router’s quality scores, so the router wasn’t tuned on them. The second set was written after the first run, to test changes on prompts they weren’t shaped by.
- Nivana chose freely. Each prompt went through the production router, the same path as
POST /v1/chat, with live prices and health. Baselines got the bare prompt with no system prompt. - Bias controls. Each pair was judged in both orders, so position bias cancels. Because judges favour their own lab’s models, a second score drops every verdict from a judge whose lab made either answer. It moved the results by about a point at most.
- Cost. What a caller pays Nivana (model cost plus the fee, in whole credits) against the gateway’s exact cost of the baseline call.
- Limits. 96 prompts is a modest sample: the two sets differed by 11 points against Opus, and model judges aren’t human raters. This is Nivana measuring itself, so treat it as the operator’s figures until someone else reproduces them.
Run it yourself. The harness, both prompt sets and every answer, verdict and cost are public at github.com/aw3-technology/router-index/benchmark. With a Nivana API key and an AI Gateway key, node run.mjs repeats the run against the public API for a few dollars. Results are dated and re-run as the router and prices change.