{"id":1536,"date":"2026-08-25T08:05:06","date_gmt":"2026-08-25T08:05:06","guid":{"rendered":"https:\/\/dearlottery.net.in\/news\/?p=1536"},"modified":"2026-08-25T08:05:06","modified_gmt":"2026-08-25T08:05:06","slug":"latency-based-routing-picking-the-model-that-answers-fast-enough","status":"publish","type":"post","link":"https:\/\/dearlottery.net.in\/news\/latency-based-routing-picking-the-model-that-answers-fast-enough\/","title":{"rendered":"Latency-Based Routing: Picking the Model That Answers Fast Enough"},"content":{"rendered":"

llm router<\/span><\/a> is the practice of choosing a model by how fast an answer needs to arrive, not by which model is newest or most famous \u2014 and for most applications it is the cheapest performance win you never wired up. The craft lives in three numbers \u2014 time to first token, tokens per second, and the p95 tail \u2014 and it works because it matches the request’s patience to the model’s speed instead of sending everything to one over-qualified flagship. To do that honestly you need comparable measurements across many models \u2014 a budget tier like <\/span>GPT-5.6 Luna<\/span><\/a> makes the latency numbers concrete \u2014 so the latency picture you route on is measured rather than remembered.<\/span><\/p>\n

Teams usually pick models the way they pick hotels: on reputation. The flagship is assigned to everything, the cheapest model to nothing, and the middle of the catalog never gets a look. Latency-based routing inverts that. Each request answers one question first \u2014 how long can this take? \u2014 and the answer chooses the model. It sounds trivial, and mechanically it is; the work is having honest latency data so the choice is made by evidence, not habit.<\/span><\/p>\n

What latency-based routing actually decides<\/b><\/h2>\n

When people say a model is “fast” they usually mean one of three different things, and conflating them is where routing goes wrong.<\/span><\/p>\n

Time to first token (TTFT) is how long between sending a request and receiving the first token of the reply. It sets how a user experiences a chat, a spinner, or an agent’s first step. Tokens per second is the sustained output rate \u2014 how quickly the rest of the answer streams out after that first token. The p95 tail is the version of both numbers that matters when you have real traffic: not the median request, but the one in twenty that arrives late. A model whose median looks fine and whose tail is a disaster will still strand users on the worst day, so honest latency routing plans around the tail, not the average.<\/span><\/p>\n

Latency-based routing simply makes the model a function of the speed requirement. A request that needs a sub-second first token is routed to a fast compact model; a request that can wait a few seconds is routed to a deeper one; a batch job that nobody is watching can be routed to the slowest, strongest model available. The model that answers is the cheapest one fast enough for that request \u2014 not the fastest, and not the biggest.<\/span><\/p>\n

When latency is the binding constraint<\/b><\/h2>\n

Latency stops being a nice-to-have and becomes the binding constraint in three familiar situations.<\/span><\/p>\n

Chat UX is the obvious one. In an interactive conversation, perceived response time is dominated by the first token, not the total answer length. A model that takes a moment to think before speaking feels broken in chat even when its eventual answer is excellent; the user is staring at a cursor.<\/span><\/p>\n

Agents are the hidden one. An agentic loop is a chain of model calls \u2014 plan, act, observe, plan again \u2014 and latency compounds across every step. A single slow call is a pause; a slow model across ten sequential steps is a different product entirely. Agent builders are routinely the ones who abandon the most capable model for a fast one, because throughput, not benchmark score, is what lets the loop close.<\/span><\/p>\n

Streaming is the psychological one. Streaming does not make the first token come faster, but it turns a long total completion into visible progress, so a slow model that streams can feel acceptable while a fast model that returns a blank screen feels broken. The constraint flips from “how long did it take” to “how soon did something move.”<\/span><\/p>\n

How routers turn latency into a budget<\/b><\/h2>\n

Once you accept that speed is a budget, routing becomes a rules problem. The classic shape of adaptive routing is: score the request, set the bar, pick the cheapest model that clears it. Each prompt is graded in under a millisecond \u2014 fast enough that the routing decision costs less than a rounding error on the request itself \u2014 and then sent to the model that meets the requirement at the lowest price. Latency enters as one of the requirements: “first token under a second,” “streams at least this fast,” “completion inside this deadline.”<\/span><\/p>\n

This is where a router that holds every model behind one interface earns its keep. With a single API key reaching 200+ models, latency-based routing stops being a rewrite of your integration layer and becomes a configuration change: set the budget, let the router pick. When the fast route is unavailable, automatic failover reroutes to the next model that fits the same budget, and when the vendor passes list price through at 0% markup, there is no per-call tax quietly changing which model is actually cheapest. OrcaRouter packages exactly that combination \u2014 200+ models behind one key, automatic failover between routes, and vendor list price passed through unchanged \u2014 which is what makes a latency budget a setting rather than a project.<\/span><\/p>\n

The tradeoff you are actually managing<\/b><\/h2>\n

The reason latency routing exists is that model speed and model depth usually move in opposite directions. Fast models are often cheaper, smaller, and shallower \u2014 excellent at the mechanical ninety percent of requests, wrong for the hard ones. Flagship models think longer before answering; that thinking is the whole point, and it is also why their time to first token and their price both sit higher. A router is not choosing the best model. It is choosing which model deserves to spend your latency budget.<\/span><\/p>\n

The live latency spread makes the tradeoff concrete. On any honest board you will see flagship reasoning models posting p50 first-token times measured in seconds while compact fast models answer in well under a second, and the gap is not a defect \u2014 it is the two model families trading thought for speed. Route a question a small model can answer well to the small model and you save both latency and cost; route the genuinely hard question to the flagship and you spend both. The error to avoid is treating the flagship’s latency as a problem to fix instead of a resource to spend on the requests that need it.<\/span><\/p>\n

How to measure latency honestly<\/b><\/h2>\n

Every latency-routing decision rests on measurements, and most teams measure wrong in the same three ways.<\/span><\/p>\n

Median vs p95.<\/b> The median first-token time is the comfortable number; the p95 is the number your users actually feel on a bad day. Measure both, plan around the tail, and watch the tail shift with load and provider capacity \u2014 a leaderboard number is a snapshot, not a contract.<\/span><\/p>\n

Streaming first token vs total completion.<\/b> A model with a fast first token and a slow output rate produces a great “feel” and a long wait for a long answer; a model with a slower first token but high throughput is the reverse. Which one is “faster” depends on whether your request wants a short answer or a long one, which is why the honest question is “fast at what” rather than “fast.”<\/span><\/p>\n

Fixed units.<\/b> Compare like with like: TTFT against TTFT, tokens per second against tokens per second, p50 against p50. Mixing a median from one source with a p95 from another is how teams end up routing on vibes. Pick one board or one router’s telemetry, read the p50 and the p95 together, and re-check when the models change \u2014 in this market, often.<\/span><\/p>\n\n\n\n\n\n\n\n
What to route on<\/b><\/td>\nWhat it captures<\/b><\/td>\nThe decision it drives<\/b><\/td>\n<\/tr>\n
TTFT p50<\/span><\/td>\nMedian wait to first token<\/span><\/td>\nWhich model owns interactive chat traffic<\/span><\/td>\n<\/tr>\n
TTFT p95<\/span><\/td>\nThe one-in-twenty late request<\/span><\/td>\nHow much headroom a model really has under load<\/span><\/td>\n<\/tr>\n
Tokens per second<\/span><\/td>\nSustained output speed<\/span><\/td>\nWhere long-form answers and streaming should go<\/span><\/td>\n<\/tr>\n
Cost per token<\/span><\/td>\nPrice of the choice<\/span><\/td>\nWhether the fast model is also the cheap one<\/span><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

 <\/p>\n

The takeaway<\/b><\/h2>\n

Latency-based routing is not for the team that needs one perfect model \u2014 it is for the team that needs one good experience across many request types. If your traffic is uniformly interactive, set a tight first-token budget and let fast models carry the load; if it is a mix of chat, agents and batch work, set a budget per job shape and let the router pick the cheapest model that clears it. Measure both the median and the tail, compare like units, and re-check when the models change. The outcome is the same either way: faster-feeling answers, a lower bill, and flagships reserved for the questions where their thinking time is the point.<\/span><\/p>\n

Sourcing note: OrcaRouter product facts \u2014 adaptive routing with sub-millisecond prompt grading, one API key reaching 200+ models, automatic failover, 0% markup on vendor list price, and the live leaderboard with per-model latency statistics \u2014 come from the official homepage and product pages, checked 2026-08-22. The general latency spread described here (flagship p50 first-token times in seconds versus well under a second for compact models) reflects the common pattern on third-party latency boards and is directional, not a guarantee; all latency figures move with traffic and provider capacity.<\/i><\/p>\n","protected":false},"excerpt":{"rendered":"

llm router is the practice of choosing a model by how fast an answer needs to arrive, not by which model is newest or most famous \u2014 and for most applications it is the cheapest performance win you never wired up. The craft lives in three numbers \u2014 time to first token, tokens per second, … Read more<\/a><\/p>\n","protected":false},"author":24,"featured_media":1537,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[5],"tags":[],"class_list":["post-1536","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technology"],"_links":{"self":[{"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/posts\/1536","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/users\/24"}],"replies":[{"embeddable":true,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/comments?post=1536"}],"version-history":[{"count":1,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/posts\/1536\/revisions"}],"predecessor-version":[{"id":1538,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/posts\/1536\/revisions\/1538"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/media\/1537"}],"wp:attachment":[{"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/media?parent=1536"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/categories?post=1536"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dearlottery.net.in\/news\/wp-json\/wp\/v2\/tags?post=1536"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}