My LLM Calls Are Adding 800ms to Every Request — How to Fix It

TL;DR

If a measured live-model call owns the request's latency budget, you have four broad options: test a different live model, fine-tune or self-serve a smaller model, cache eligible repeated inputs, or evaluate a compiled classifier for a bounded decision. Compare quality, p50/p95/p99, current usage cost, reliability, and maintenance on the same workload.

You've Found the Problem

You added profiling. You traced the request. You found the bottleneck:

step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=llm_classify  ms=847.0  ← this one
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=856.8

In the illustrative trace above, the live-model call owns almost all of the request latency. Use your own spans to calculate the share.

In this example trace, the live-model request is the bottleneck. Your numbers will depend on the model, prompt, output, provider, region, and load, so record the full distribution rather than copying an average.

The question isn't what's wrong — you know what's wrong. The question is which fix is right for your situation.

Option 1: Use a Faster LLM Provider

The simplest change: same prompt, faster model, different provider.

How it works

Providers such as Groq and Cerebras optimise model inference, but model, prompt, region, load, and structured-output method still determine the complete request time.

Implementation

from groq import AsyncGroq

client = AsyncGroq(api_key="YOUR_GROQ_KEY")

async def classify_with_groq(text: str) -> str:
    response = await client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject. Respond with only the classification."},
            {"role": "user", "content": text}
        ],
        temperature=0,
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

Latency comparison

| Candidate | P50 | P95 | P99 | |----------|-----|-----|-----| | Current live model | [measure] | [measure] | [measure] | | Optimised live-model candidate | [measure] | [measure] | [measure] | | Compiled candidate, classifier stage | [measure] | [measure] | [measure] | | Combined compiled + escalation path | [measure] | [measure] | [measure] |

Verdict

Best for: Teams that need live-model flexibility and find a candidate that clears the measured budget.

Limitation: It remains an external model-serving dependency with current provider pricing and limits. Verify both rather than relying on old plan details.

Option 2: Fine-Tune a Smaller Model

Train a smaller model specifically for your classification task, then call it via API.

How it works

Take GPT-4o-mini or a similar small model and fine-tune it on your classification data. The fine-tuned model knows your specific domain, so it needs fewer tokens in the prompt and responds more consistently.

Implementation

# Step 1: Prepare training data from your existing LLM outputs
training_examples = []
for log in decision_logs:
    training_examples.append({
        "messages": [
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": log.input_text},
            {"role": "assistant", "content": log.classification}
        ]
    })

# Step 2: Upload and fine-tune
file = client.files.create(file=jsonl_bytes, purpose="fine-tune")
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18"
)

# Step 3: Use the fine-tuned model
async def classify_finetuned(text: str) -> str:
    response = await openai.chat.completions.create(
        model="ft:gpt-4o-mini-2024-07-18:your-org::abc123",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": text}
        ],
        temperature=0,
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

Latency comparison

| Approach | P50 | P95 | Usage price | |----------|-----|-----|-------------| | Current base model | [measure] | [measure] | [current provider rate × observed tokens] | | Smaller base model | [measure] | [measure] | [current provider rate × observed tokens] | | Fine-tuned model | [measure] | [measure] | [training + current inference rate] |

Verdict

Best for: When you want better accuracy on your specific domain while staying within the LLM ecosystem. Fine-tuned models can use shorter prompts, saving tokens.

Limitation: Fine-tuning is more expensive per-token than the base model. Latency improvement is modest (10-15%). You're still making an API call per request, still per-token pricing. And fine-tuning requires maintaining a labeled dataset.

Option 3: Cache Frequent Requests

If the same inputs recur, cache the responses.

How it works

Hash the input, check a cache before calling the LLM. On a hit, return the cached result in sub-millisecond time. On a miss, call the LLM and cache the result.

Implementation

import hashlib
import redis.asyncio as redis

cache = redis.Redis(host="localhost", port=6379)

async def classify_with_cache(text: str) -> str:
    # Normalize and hash
    normalized = text.strip().lower()
    key = f"classify:{hashlib.sha256(normalized.encode()).hexdigest()}"

    # Check cache
    cached = await cache.get(key)
    if cached:
        return cached.decode()  # ~0.2ms

    # Miss: call LLM
    result = await classify_with_llm(text)  # ~800ms

    # Cache for 1 hour
    await cache.setex(key, 3600, result)
    return result

Effectiveness depends on your workload

| Workload Type | What determines the hit rate | Evaluation | |--------------|----------------------|---------------------------| | Spam detection | Template and campaign duplication | Measure exact and semantic repeats | | Content moderation (UGC) | Reposts and copied content | Measure before sizing the cache | | Support ticket triage | Reused templates and issue bursts | Compare freshness requirements | | Custom classification | Input distribution | Instrument the real workload |

Verdict

Best for: Workloads with high duplication rates (spam, templated messages). Implement this regardless of what else you do — it's cheap and easy.

Limitation: It does not help unique inputs. Measure hit rate and staleness; cache misses retain the original model path.

Option 4: Compile Into an ONNX Classifier

Replace the LLM call entirely with a purpose-built classification model.

How it works

An LLM generates synthetic training data for your classification task. That data trains a lightweight classifier (a text encoder for semantic features + a gradient-boosted classifier for the final prediction, with automated hyperparameter tuning). The model exports to ONNX format, and the compiled stage targets under 100ms. That classifier handles the normal runtime path; a cloud deployment can optionally call an LLM when confidence falls below its configured escalation threshold, while an exported edge bundle has no cloud or LLM dependency after download.

Implementation

import httpx

# Replace the LLM call
async def classify_compiled(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}

For even lower latency, run the model locally:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this product!"})
# Inspect result.decision, result.confidence, and result.stage

Latency comparison

| Approach | Evidence to collect | |----------|---------------------| | Current live model | Same-prompt p50/p95/p99, tokens, errors, per-class quality | | Optimised or fine-tuned live model | Same metrics on the same cases | | Sparkient cloud | Classifier-only and combined paths; four controlled synthetic domains report 33–42ms batch-average time per item | | Sparkient edge | Target-hardware latency, concurrency, packaging, and output parity |

Verdict

Best for: Classification tasks where you're choosing from a fixed set of options. Content moderation, spam detection, ticket triage, lead scoring, fraud flagging. Any task where the output is one of N predefined options.

Limitation: Only works for classification. If your LLM call is generating text, summarizing content, or doing anything with unbounded output, you still need an LLM. The compiled model also needs retraining when your classification categories change.

The Decision Matrix

| Question | If Yes → | If No → | |----------|----------|---------| | Is the output one of N fixed options? | Option 4 (compile) | Option 1 (faster LLM) | | Does an optimised live model clear the measured budget? | Option 1 may be sufficient | Evaluate Option 4 or eligible caching | | Do many inputs repeat? | Option 3 (cache) first | Skip caching | | Do you need to change prompts frequently? | Option 1 or 2 | Option 4 (compile) | | Is cost more important than latency? | Option 2 (fine-tune) or Option 4 (compile) | Option 1 (Groq) |

Choose the smallest change that passes the project's quality, latency, cost, reliability, and maintenance criteria.

Combining Approaches

These options aren't mutually exclusive. A production-grade setup often combines them:

async def classify_production(text: str) -> dict:
    # Layer 1: Cache check (~0.2ms)
    cached = await cache.get(cache_key(text))
    if cached:
        return cached

    # Layer 2: Compiled classifier (~41ms)
    result = await sparkient.decide("content-moderation", {"text": text})

    # Layer 3: optional LLM escalation (if enabled and confidence < threshold)
    if result["confidence"] < 0.7:
        result = await llm_classify(text)  # measure separately

    await cache.setex(cache_key(text), 3600, json.dumps(result))
    return result

Instrument each stage rather than assuming a mix:

• Cache: hit rate, freshness, and lookup latency
• Classifier: full API latency and per-class quality
• LLM escalation: rate, quality, tokens, and tail latency
• Combined path: p50, p95, p99, and credits on representative traffic

Illustrative Before and After

The following hypothetical trace shows how replacing one measured 847ms classification call with a measured 38ms candidate would affect the surrounding endpoint. It is arithmetic for an evaluation plan, not a Sparkient customer result:

BEFORE:
step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=llm_classify  ms=847.0
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=856.8

AFTER:
step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=classify      ms=38.0
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=47.8

In this hypothetical trace, total request time falls from 857ms to 48ms, a 94.4% reduction. Do not reuse that percentage as a claim: instrument the existing endpoint and the candidate in the same environment.

FAQ

Q: Will switching to Groq solve the problem permanently? It may. Test the current model and prompt from your deployment region. If the candidate clears the full-path objective and current provider economics, changing providers may be enough; if not, evaluate a compiled candidate for the bounded calls.

Q: What accuracy should I expect from a compiled classifier vs. GPT-4o? Sparkient's four controlled synthetic domains report 0.886–0.951 macro F1 and 33–42ms batch-average time per item. Those runs do not establish a universal gap to GPT-4o; compare the exact prompt and trained model on the same cases.

Q: Can I build my own compiled classifier without Sparkient? Absolutely. The pipeline is: (1) generate or collect labeled training data, (2) encode text into semantic embeddings, (3) train a classifier on the encoded features, (4) export to ONNX, (5) serve with ONNX Runtime. If you have an ML team, this is a well-understood pipeline. Sparkient automates all five steps, including synthetic data generation with an LLM teacher.

Q: How does the LLM escalation work? The classifier returns a confidence score. When optional escalation is enabled and confidence is below the configured threshold, Sparkient can call an LLM. Validate the threshold; confidence does not guarantee ambiguity, and the escalation rate, quality, latency, and credits are workload-specific.

Your latency budget should go to features, not classification. Try Sparkient free — 5,000 credits, no credit card required.