LLM routers, explained

An LLM router reads each request and sends it to the model most likely to answer it well, at the lowest cost that still does the job. Here is how they work and when one is worth it.

Why route at all?

No single model is best at everything. On Nivana’s own measurements, the model that wins hard reasoning is not the one that wins hard maths, and neither is the one you want for a one-line answer: the Router Index shows a different winner for most kinds of request. Output prices differ about 100× between the cheapest and the dearest model in the pool, so sending every request to the strongest model wastes money on easy ones, and sending everything to a cheap one fails on hard ones.

How a router decides

• Profile the request. A fast, cheap model classifies it: what kind of task (code, writing, maths, data, a long document), how hard, and how long the answer will be.
• Score every model. Each candidate gets a quality score for that kind of task (Nivana measures these weekly with pairwise evals), an expected cost at live prices, and an expected speed from live traffic.
• Weigh by difficulty. Easy requests weigh cost and speed heavily; hard ones weigh quality almost entirely. The highest score wins, and the next two become fallbacks.
• Decide how hard to think. Reasoning models can think a little or a lot; the router sets the effort per request, which matters as much as the model for cost and latency.

Kinds of router

• Rule-based. If the prompt mentions code, use model X. Cheap and predictable, but brittle.
• Learned classifiers. A model trained on preference data predicts which model would win. Good on average; hard to update when new models arrive.
• Scored routing (Nivana’s approach). A classifier profiles the request, then a transparent formula picks the winner from measured quality, live price and speed. Every decision is explainable, and a new model joins as soon as it has been measured.
• Gateways with an optional auto mode. Model gateways give you hundreds of models behind one API; some offer an automatic option you opt into per request. You still pick by default.

What a router costs you

Routing adds a classification step (on Nivana about a second, with no thinking) and gives up some control: you cannot pin a request to a model. In return you get the best measured model for each request without tracking leaderboards, lower average cost, and failover when a provider is down. If your product depends on one model’s exact behaviour, pin it; if you want the best answer per request, route.

Routing with Nivana

Nivana is auto-only: there is no model parameter. The chat app and the API send every request through the router, and every response says which model answered and why. Start with the quickstart, read how routing works, or see today’s decisions.