Route 80% of Agent Calls to a Local Model and Cut Your LLM Bill

By the end of this, you will have a routing architecture with one quality gate. It pushes the easy 80% of agent calls to a local model and escalates only the hard 20% to a frontier API. Your output quality stays flat. Your bill does not.

Why does your agent overspend on every single call?

Most agent loops treat every task identically. A frontier model prices every token the same — whether the task is summarising a 200-word support ticket or synthesising a 40-page legal contract.

That pricing symmetry is the problem. The tasks are not symmetric.

Pull last week's call log from your agent harness. Sort tasks by prompt length, output length, and downstream retry rate. You will find a cluster — usually 70–85% of total calls — that share three properties: short input, deterministic output shape, and near-zero retry rate.

Those tasks did not need a frontier model. They needed a fast, cheap model that returns structured JSON reliably.

Here is a concrete before-state. A document-processing agent routes every call to a frontier API. It runs roughly 4,000 calls per day. About 3,200 of those calls are extraction tasks: pull the invoice date, vendor name, and line-item total from a PDF. Each call costs roughly the same token price as a complex reasoning call.

The after-state: those 3,200 extraction calls go to a local model running on the same machine as the orchestrator. Only the 800 calls that require cross-document reasoning or ambiguity resolution escalate.

What counts as an "easy" task for this audit?

Use three filters. First, is the output schema fixed? If the agent always returns a JSON object with the same keys, a local model can handle it. Second, does the task require fewer than five reasoning steps? Count the steps a human would take. Third, does the task tolerate a one-retry policy? If a wrong answer triggers an automatic retry anyway, the cost of an occasional local-model miss is already priced in.

Tasks that pass all three filters are safe to route locally.

Tasks that fail any filter are escalation candidates.

Write that filter logic down before you build anything. The routing layer is only as good as the task taxonomy behind it. A poorly defined taxonomy routes ambiguous tasks to the wrong tier — you pay for escalations that were unnecessary, or you ship bad output from a local model that was under-equipped.

Audit your call log this week. Tag each task type as local-eligible or escalate. That tag list is your routing spec.

Audit.

Which local model is actually good enough for the easy 80%?

"Local model" covers a wide range. A 7-billion-parameter quantised model running on CPU is not the same as a 70-billion-parameter model on a dedicated GPU box. Picking the wrong one ships bad output silently — no error, just wrong JSON that your downstream system accepts.

Start with your real task sample, not synthetic benchmarks. Take 200 examples from last week's call log — the ones your audit tagged as local-eligible. Run each example through three candidate local models. Compare the outputs against the frontier model's outputs on the same prompts.

Score on three axes: schema compliance (did the output match the expected JSON structure?), factual match (did the extracted values match ground truth?), and retry rate (how often did the output require a second call?).

A minimal comparison table for model selection

You are not looking for a model that matches the frontier API. You are looking for a model that clears your acceptable-quality floor on local-eligible tasks.

Define that floor before you run the benchmark. A schema compliance rate below 90% probably breaks your downstream parser. A retry rate above 8% erodes the cost saving because retries burn tokens too.

If no local model clears your floor on the full local-eligible set, split the set further. Some tasks in that set may be borderline. Route only the subset where local models score above your threshold. That is still a meaningful cost reduction — even routing 50% locally is better than routing zero.

Keep the benchmark reproducible. Store the 200-example eval set in your repo. Re-run it every time you update the local model or change the prompt template. Model updates change behaviour in ways that are not always obvious from release notes.

# Run a local model eval against your saved task sample
python eval_router.py \
  --model llama3-8b-q4 \
  --task-sample ./data/local_eligible_200.jsonl \
  --output ./results/llama3_eval.json \
  --score-keys schema_match,factual_match,retry_flag

Benchmark your top three candidates this week. Pick one. Set the quality floor in writing.

Re-baseline.

How do you build a routing layer that escalates only when it must?

The routing layer sits between your orchestrator and your model clients. It reads the task metadata — type tag, input token count, output schema — and makes a binary decision before any LLM call fires.

Keep the router dumb. A complex ML classifier that itself calls an LLM to decide routing adds latency and cost. Use a rule-based router first. Graduate to a lightweight classifier only if your task taxonomy is too ambiguous for rules.

What does a rule-based router look like in practice?

Here is a minimal Python implementation. It reads three metadata fields and returns the model client to use.

from enum import Enum
 
class Tier(Enum):
    LOCAL = "local"
    FRONTIER = "frontier"
 
def route(task_type: str, input_tokens: int, schema_fixed: bool) -> Tier:
    """Return LOCAL if the task is cheap-eligible, FRONTIER otherwise."""
    LOCAL_TYPES = {"extract", "classify", "summarise_short", "format"}
    if task_type not in LOCAL_TYPES:
        return Tier.FRONTIER
    if input_tokens > 2048:
        return Tier.FRONTIER
    if not schema_fixed:
        return Tier.FRONTIER
    return Tier.LOCAL

The router fires before your model client is instantiated. Zero latency overhead on the decision itself.

Now wire it into your orchestrator loop.

tier = route(task.type, task.input_tokens, task.schema_fixed)
if tier == Tier.LOCAL:
    response = local_client.complete(task.prompt)
else:
    response = frontier_client.complete(task.prompt)

That is the entire routing layer. Two function calls. The complexity lives in your task taxonomy, not in the router code.

Where does the quality gate fit?

The quality gate is a post-call check on local model outputs. It runs only on LOCAL-tier responses. It validates schema compliance and — optionally — a confidence heuristic.

If the local response fails the gate, the orchestrator escalates that single call to the frontier tier. Log the escalation with the failure reason. After a week, review the escalation log. If one task type escalates frequently, move it to the FRONTIER set in your taxonomy.

The gate catches silent failures before they hit your downstream system. It also gives you a feedback loop: escalation frequency is a live signal that your taxonomy needs adjustment.

Deploy the router and quality gate to your staging environment this week.

Refactor.

How do you measure whether the routing layer is actually working?

Shipping the router is not the end. The router is only justified if it demonstrably reduces cost without degrading output quality. You need three counters running from day one.

Counter 1: local call rate. Of all agent calls in the period, what percentage routed to the local tier? Your target on day one is the percentage of tasks you tagged as local-eligible in the audit. If your audit said 78% were local-eligible but your counter shows 40% routing locally, your taxonomy is misclassified or your task metadata is wrong.

Counter 2: escalation rate. Of all LOCAL-tier calls, what percentage failed the quality gate and escalated? This is your quality signal. A rate under 5% means the local model is handling its assigned tasks cleanly. A rate above 10% means either the quality floor is too strict or the local model is under-equipped for some task types in the LOCAL set.

Counter 3: gate failure type. Log why each escalation fired — schema mismatch, empty output, confidence below threshold. After 500 escalations, group by failure type. The most common failure type points to the task type you should move out of the LOCAL set.

A worked example of the feedback loop

After week one, the escalation log shows 14% of calls typed summarise_short are failing the schema gate. The failure reason is always the same: the local model returns a plain string instead of {"summary": "...", "word_count": N}.

Two options. Patch the local model prompt to enforce the JSON wrapper. Or move summarise_short to the FRONTIER set temporarily, fix the prompt, and re-run the benchmark before moving it back.

This is the feedback loop the quality gate enables. Without it, those schema mismatches would silently corrupt downstream state.

# metrics/router.yml — example counter config for Prometheus
router_calls_total:
  labels: [tier, task_type]
router_escalations_total:
  labels: [task_type, failure_reason]
router_gate_failures_total:
  labels: [task_type, failure_type]

                                      ┌──▶  Local Model    (easy 80%)
  Orchestrator ──task metadata──▶  Router ──┤
        route()                     └──▶  Frontier API  (hard 20%)

Figure: The routing layer reads task metadata and sends calls to the local model or frontier API. The quality gate sits on the local path and escalates failures.

Set up all three counters before you go live. Review the dashboard on day 7.

Re-baseline.

When should you escalate the whole architecture — not just a single call?

The routing split is not a permanent configuration. Local models are improving fast. A task that required a frontier model six months ago may route cleanly to a local model today.

Run a quarterly promotion review. Take the ten most-escalated task types from the past 90 days. Re-benchmark them against the current local model. If any task type now clears your quality floor, move it to the LOCAL set and re-run your counter baseline.

This is how you keep the cost reduction compounding over time. The initial routing split captures the obvious wins. The quarterly review captures the incremental ones as local model capability catches up.

What signals mean a task should move back to FRONTIER?

Three signals. First, escalation rate for that task type is persistently above 12% after two prompt-tuning attempts. Second, downstream error rate on local-tier outputs for that task type is rising week over week. Third, the task type has changed — a new feature added a reasoning step that was not there in the original audit.

When any of these signals appear, move the task type back to FRONTIER immediately. Do not leave it in the LOCAL set waiting for the next quarterly review. The gate and the counters exist precisely to catch this before it compounds.

How do multiple model tiers affect the architecture longer-term?

The two-tier model (local + frontier) is the simplest version of this architecture. As model providers ship more distinct capability tiers — a lightweight tier, a mid-tier, and a frontier tier — the routing layer can grow to match.

Gemini's model family, for example, now spans meaningfully different capability levels across its tier lineup. That spread exists specifically because different tasks warrant different model weights. The routing architecture described here is the operator-side implementation of the same logic.

Do not add a third tier until you have stable data from the two-tier system. The two-tier router gives you the taxonomy, the counters, and the escalation feedback loop. A third tier adds complexity without proportional benefit until you understand your task distribution well.

Start two-tier. Run it for a quarter. Then decide whether a mid-tier adds value based on your escalation log — not based on a model announcement.

# router_config.yml — promotion review template
quarterly_review:
  review_period_days: 90
  escalation_threshold_pct: 12
  tasks_to_evaluate:
    - task_type: summarise_short
      current_tier: FRONTIER
      retest_local_model: llama3-8b-q4
    - task_type: classify_intent
      current_tier: LOCAL
      confirm_stays_local: true

Schedule the quarterly review now. Put it on the calendar before you ship the router.

Delete.

Wrap-up

Audit your call log, tag local-eligible tasks, deploy the rule-based router with a quality gate, and instrument three counters. That one architecture change routes the easy 80% of calls to a local model and escalates only when output quality demands it.

Made with AI by Qyndex — drafted by an agent, reviewed by the Qyndex team.