LiteLLM’s complexity router charges you two model calls on each request: a small one to decide which tier responds, and the real one that responds. Since the September 5, 2026 version, you can compress those two prompts separately, and LiteLLM’s internal testing places the additional savings on the routing call at 32%.
Until this version, both hops shared a single compression configuration. And that’s a real constraint, not cosmetic: the responding model needs enough context to give a good answer, so the compression level you chose was limited by what that model could tolerate. The classifier—which only has to return a tier label—was being handed far more context than it needs, with compression designed for a different job.
By Devy · Category: AI Dev Tools
What is LiteLLM?
LiteLLM is an open source gateway that sits between your application and model providers. You point your OpenAI-compatible client at the LiteLLM proxy instead of pointing it directly at OpenAI or Anthropic, and the gateway handles credentials, budgets, fallbacks, logging, and routing behind a single endpoint. You run it yourself, with litellm --config config.yaml.
If you read our note on CodeGate, the approach will feel familiar: a single controlled checkpoint where your traffic crosses, and where policy lives.
What is AutoRouter and why does it make two calls?
AutoRouter is LiteLLM’s function that classifies each incoming request and assigns it a model tier, instead of forcing you to set one manually. The complexity router defines tiers—SIMPLE, MEDIUM, COMPLEX, REASONING—and associates each one with a model.
The classification can be done three ways, according to LiteLLM’s documentation: a heuristic scorer taking less than a millisecond, keyword-based rules, or a small model. If you set classifier_type: llm, the classifier becomes a real model call. LiteLLM clarifies that this call goes through the same Router instance, so your credentials, budgets, and fallbacks apply to it; and that on timeout, empty content, or a schema mismatch, it falls back to the heuristic scorer.
It’s a sensible design. It’s also the call nobody audits. It doesn’t appear in your application code, it produces no visible output for the user, and it runs on absolutely every request.
If you’re not entirely clear on why this invisible call costs you money, start with What are tokens in AI and why do they charge you for them: that’s where the arithmetic that makes everything else in this note make sense lives.
What changed in this version?
Two new fields inside complexity_router_config, each pointing to a named guardrail:
auto_router_routing_compression— the compression applied to the classifier’s promptauto_router_model_compression— the compression applied to the responding model’s prompt
Separated this way, you can compress the routing hop aggressively and leave the model hop moderate. LiteLLM’s note reports that this “cut classification costs an additional 32% beyond what shared compression was already saving, with no changes to routing accuracy”.
Read that number for exactly what it is. It’s LiteLLM’s internal testing, published without methodology, measuring its own function. It’s a plausible direction and an unverified magnitude. At the time this note was published, no one outside BerriAI has reproduced it.
How do you configure it?
This is the example published by LiteLLM, reproduced exactly as they wrote it:
model_list:
- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: gpt-4o-mini
MEDIUM: gpt-4o
COMPLEX: gpt-4-turbo
classifier_type: llm
auto_router_routing_compression: headroom-aggressive
auto_router_model_compression: headroom-moderate
guardrails:
- guardrail_name: headroom-aggressive
litellm_params:
guardrail: headroom
mode: pre_call
api_base: https://api.berri.ai/headroom
model: o1
tokens_to_retain: 200
- guardrail_name: headroom-moderate
litellm_params:
guardrail: headroom
mode: pre_call
api_base: https://api.berri.ai/headroom
model: gpt-4o-mini
tokens_to_retain: 1000
The mechanism is in tokens_to_retain: 200 tokens for the classifier, 1000 for the responding model. The classifier only has to emit a tier label, so it survives with a fraction of the context.
Keep in mind that the two guardrail names in this example—headroom-aggressive and headroom-moderate—are labels LiteLLM chose for its own example, not reserved words. Yours are called whatever you want; all that matters is that the fields match.
What is Headroom and do you have to use LiteLLM’s hosted service?
Headroom is the prompt compression service that does the real work. We covered it in Headroom: how to cut up to 95% of your tokens without changing responses: it rewrites the prompt before dispatching it, preserving meaning and cutting volume.
Here’s the detail worth not overlooking. LiteLLM’s example config points api_base to https://api.berri.ai/headroom, which is the endpoint hosted by LiteLLM itself. That is, if you copy that block as-is, your prompts travel to a third-party service before reaching the model, even though your gateway is self-hosted.
That’s a property of the example, not of the function. Headroom’s documentation in LiteLLM describes it as a sidecar service you run alongside the proxy, deployed from a Dockerfile they provide, and api_base is simply the address where your Headroom service listens. Point it at your own sidecar and the entire path stays inside your infrastructure. The documented parameters of the guardrail are guardrail, mode (which must be pre_call; applying it on the response side does nothing), api_base, an optional api_key, and an optional default_on that compresses all requests passing through the proxy instead of being activated route by route.
If in your organization prompts leaving your network is a compliance question, this is the line you need to read carefully before pasting the example.
LiteLLM or OpenRouter?
The practical difference is where the routing runs: OpenRouter decides for you in its infrastructure, LiteLLM decides in yours. Everything else follows from there.
OpenRouter is a hosted service. You point your client at https://openrouter.ai/api/v1 with an API key and access hundreds of models through a single endpoint, with automatic fallbacks. LiteLLM is a gateway you run yourself: the proxy lives in your infrastructure, the provider credentials are yours, and you write the routing policies in your own config.yaml.
| LiteLLM | OpenRouter | |
|---|---|---|
| Where it runs | Your infrastructure | Hosted service |
| Provider keys | Yours, in your proxy | OpenRouter’s (also accepts bringing your own) |
| How it chooses provider | The rules you write in config.yaml |
Cost-weighted balancing, excluding providers with recent outages |
| Routing control | Total: tiers, classifier, fallbacks, budgets | Per request, via the provider object |
| Compression per hop | Yes, with the two fields in this note | Not exposed |
| What you operate | A proxy, and optionally a Headroom sidecar | Nothing |
When is OpenRouter worth it?
When you don’t want to operate infrastructure. The control it offers is considerable: in the provider object of each request you can set order to test providers in a given order, sort to prioritize by price, throughput or latency, only and ignore to allow or discard providers, and require_parameters to keep only those that support all the parameters you sent.
For teams with data requirements there are two controls worth knowing about: data_collection: "deny" excludes providers that might store data for training, and zdr: true limits routing to Zero Data Retention endpoints. Their documentation indicates that by default they don’t log prompts or responses, and they offer a discount on usage to those who decide to enable that logging.
As for cost, their FAQ notes that they pass through the underlying provider’s price without markup, and charge a commission on credit purchases—5.5% via Stripe at the time of publishing this note—. Verify that number before projecting anything: it’s one that changes.
When is LiteLLM worth it?
When routing itself is something you need to adjust, audit, or cheapen.
And this note is the exact example. Per-hop compression exists because you control the process that builds the classifier’s prompt: you can decide that prompt travels with 200 tokens while the responding model’s travels with 1000. That lever isn’t exposed when a hosted service decides routing, because the classifier’s prompt isn’t even yours.
The practical rule: if your problem is access to many models, OpenRouter solves it without you deploying anything. If your problem is the cost and behavior of routing itself —and you already have enough traffic that a 32% overhead on the classifier call means something—, that optimization only exists on the side you operate. Choosing LiteLLM means accepting maintenance of an infrastructure piece in exchange for being able to adjust things like this.
Is it worth enabling today?
Three things worth weighing honestly, all true at the time of publishing this note:
AutoRouter appears as Beta in LiteLLM’s documentation, with the config explicitly subject to change. The 32% is self-reported. And LiteLLM is still recruiting design partners for AutoRouter and gathering results in GitHub discussion #32168, which tells you the feature is shipped and usable, but not yet settled.
The counterweight is that the risk is low and reversible. It’s two lines of config on a gateway you already run, the classifier falls back to the heuristic scorer when something fails, and you can measure the effect on your own traffic instead of trusting anyone else’s benchmark. If you’re already running the complexity router with classifier_type: llm, you’re already paying for that second call on all requests. The only question is how much context it needs to keep doing its job, and the answer is almost certainly less than what the responding model needs.
The cheapest token is still the one you never send.