My LLM Calls Are Adding 800ms to Every Request — How to Fix It

TL;DR

If your LLM call is adding 800ms+ to every request, you have four options: switch to a faster LLM provider (Groq: 150-300ms), fine-tune a smaller model (still 400-800ms), cache frequent requests (0ms on hit, same on miss), or compile the LLM's logic into an ONNX classifier (sub-100ms latency). The right choice depends on whether you're doing classification (bounded outputs) or generation (unbounded outputs).

You've Found the Problem

You added profiling. You traced the request. You found the bottleneck:

step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=llm_classify  ms=847.0  ← this one
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=856.8

The LLM call is responsible for 98.8% of your request latency. Everything else — auth, validation, database — takes under 10ms combined.

You're not doing anything wrong. This is just what LLM API calls cost in latency. GPT-4o averages 623ms for a structured classification response. On a bad day, it's 1,500ms. Your P99 might hit 2+ seconds.

The question isn't what's wrong — you know what's wrong. The question is which fix is right for your situation.

Option 1: Use a Faster LLM Provider

The simplest change: same prompt, faster model, different provider.

How it works

Providers like Groq and Cerebras run optimized inference hardware that delivers tokens significantly faster than standard cloud APIs. For structured classification responses (short outputs), this translates to 150-300ms total latency.

Implementation

from groq import AsyncGroq

client = AsyncGroq(api_key="YOUR_GROQ_KEY")

async def classify_with_groq(text: str) -> str:
    response = await client.chat.completions.create(
        model="llama-3.3-70b-versatile",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject. Respond with only the classification."},
            {"role": "user", "content": text}
        ],
        temperature=0,
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

Latency comparison

| Provider | Model | P50 | P95 | P99 | |----------|-------|-----|-----|-----| | OpenAI | GPT-4o | 623ms | 1,100ms | 2,300ms | | OpenAI | GPT-4o-mini | 420ms | 650ms | 1,200ms | | Groq | Llama 3.3 70B | 180ms | 280ms | 450ms | | Cerebras | Llama 3.3 70B | 160ms | 250ms | 400ms |

Verdict

Best for: Teams that need the flexibility of an LLM (free-form reasoning, changing prompts) and can accept 150-300ms latency. Good first step.

Limitation: Still per-token pricing. Still an external dependency. Still 150ms+ even in the best case — too slow for some hot paths. Groq's free tier has aggressive rate limits; paid tiers are necessary for production.

Option 2: Fine-Tune a Smaller Model

Train a smaller model specifically for your classification task, then call it via API.

How it works

Take GPT-4o-mini or a similar small model and fine-tune it on your classification data. The fine-tuned model knows your specific domain, so it needs fewer tokens in the prompt and responds more consistently.

Implementation

# Step 1: Prepare training data from your existing LLM outputs
training_examples = []
for log in decision_logs:
    training_examples.append({
        "messages": [
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": log.input_text},
            {"role": "assistant", "content": log.classification}
        ]
    })

# Step 2: Upload and fine-tune
file = client.files.create(file=jsonl_bytes, purpose="fine-tune")
job = client.fine_tuning.jobs.create(
    training_file=file.id,
    model="gpt-4o-mini-2024-07-18"
)

# Step 3: Use the fine-tuned model
async def classify_finetuned(text: str) -> str:
    response = await openai.chat.completions.create(
        model="ft:gpt-4o-mini-2024-07-18:your-org::abc123",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": text}
        ],
        temperature=0,
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

Latency comparison

| Approach | P50 | P95 | Cost per 1M tokens | |----------|-----|-----|-------------------| | GPT-4o (base) | 623ms | 1,100ms | $2.50 input / $10.00 output | | GPT-4o-mini (base) | 420ms | 650ms | $0.15 input / $0.60 output | | GPT-4o-mini (fine-tuned) | 380ms | 600ms | $0.30 input / $1.20 output |

Verdict

Best for: When you want better accuracy on your specific domain while staying within the LLM ecosystem. Fine-tuned models can use shorter prompts, saving tokens.

Limitation: Fine-tuning is more expensive per-token than the base model. Latency improvement is modest (10-15%). You're still making an API call per request, still per-token pricing. And fine-tuning requires maintaining a labeled dataset.

Option 3: Cache Frequent Requests

If the same inputs recur, cache the responses.

How it works

Hash the input, check a cache before calling the LLM. On a hit, return the cached result in sub-millisecond time. On a miss, call the LLM and cache the result.

Implementation

import hashlib
import redis.asyncio as redis

cache = redis.Redis(host="localhost", port=6379)

async def classify_with_cache(text: str) -> str:
    # Normalize and hash
    normalized = text.strip().lower()
    key = f"classify:{hashlib.sha256(normalized.encode()).hexdigest()}"

    # Check cache
    cached = await cache.get(key)
    if cached:
        return cached.decode()  # ~0.2ms

    # Miss: call LLM
    result = await classify_with_llm(text)  # ~800ms

    # Cache for 1 hour
    await cache.setex(key, 3600, result)
    return result

Effectiveness depends on your workload

| Workload Type | Typical Cache Hit Rate | Effective Latency Reduction | |--------------|----------------------|---------------------------| | Spam detection | 25-40% | Significant | | Content moderation (UGC) | 5-15% | Moderate | | Support ticket triage | 2-5% | Minimal | | Custom classification | 1-10% | Varies |

Verdict

Best for: Workloads with high duplication rates (spam, templated messages). Implement this regardless of what else you do — it's cheap and easy.

Limitation: Doesn't help with unique inputs. Your P95 is still 800ms+ because 60-95% of requests are cache misses. Caching is a complement to other solutions, not a replacement.

Option 4: Compile Into an ONNX Classifier

Replace the LLM call entirely with a purpose-built classification model.

How it works

An LLM generates synthetic training data for your classification task. That data trains a lightweight classifier (a text encoder for semantic features + a gradient-boosted classifier for the final prediction, with automated hyperparameter tuning). The model exports to ONNX format and runs inference in under 100ms. The LLM is used during training, not in production.

Implementation

import httpx

# Replace the LLM call
async def classify_compiled(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}

For even lower latency, run the model locally:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this product!"})
# EdgeDecision(decision="approve", confidence=0.97, latency_ms=3.2)

Latency comparison

| Approach | P50 | P95 | P99 | |----------|-----|-----|-----| | GPT-4o | 623ms | 1,100ms | 2,300ms | | Groq (Llama 3.3 70B) | 180ms | 280ms | 450ms | | GPT-4o-mini (fine-tuned) | 380ms | 600ms | 1,200ms | | Sparkient (cloud API) | 33ms | 41ms | 58ms | | Sparkient Edge (local) | 3ms | 8ms | 12ms |

Verdict

Best for: Classification tasks where you're choosing from a fixed set of options. Content moderation, spam detection, ticket triage, lead scoring, fraud flagging. Any task where the output is one of N predefined options.

Limitation: Only works for classification. If your LLM call is generating text, summarizing content, or doing anything with unbounded output, you still need an LLM. The compiled model also needs retraining when your classification categories change.

The Decision Matrix

| Question | If Yes → | If No → | |----------|----------|---------| | Is the output one of N fixed options? | Option 4 (compile) | Option 1 (faster LLM) | | Can you tolerate 150-300ms? | Option 1 (Groq/Cerebras) | Option 4 (compile) or Option 3 (cache) | | Do many inputs repeat? | Option 3 (cache) first | Skip caching | | Do you need to change prompts frequently? | Option 1 or 2 | Option 4 (compile) | | Is cost more important than latency? | Option 2 (fine-tune) or Option 4 (compile) | Option 1 (Groq) |

Most teams doing classification end up at Option 4 eventually. The question is how many intermediate steps they take first.

Combining Approaches

These options aren't mutually exclusive. A production-grade setup often combines them:

async def classify_production(text: str) -> dict:
    # Layer 1: Cache check (~0.2ms)
    cached = await cache.get(cache_key(text))
    if cached:
        return cached

    # Layer 2: Compiled classifier (~41ms)
    result = await sparkient.decide("content-moderation", {"text": text})

    # Layer 3: LLM escalation (only if confidence < 0.7)
    if result["confidence"] < 0.7:
        result = await llm_classify(text)  # ~600ms, but rare

    await cache.setex(cache_key(text), 3600, json.dumps(result))
    return result

This gives you:

• Cache hit: ~0.2ms (15-30% of requests)
• Classifier: ~41ms (65-80% of requests)
• LLM escalation: ~600ms (3-5% of requests)
• Weighted average: ~35ms

Before and After

Here's a real before/after for a content moderation endpoint:

BEFORE:
step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=llm_classify  ms=847.0
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=856.8

AFTER:
step=auth          ms=1.2
step=validate      ms=0.8
step=db_read       ms=4.3
step=classify      ms=38.0
step=db_write      ms=3.1
step=response      ms=0.4
total              ms=47.8

Total request time dropped from 857ms to 48ms — a 94.4% reduction. The classification step went from dominating the request to being just another step.

FAQ

Q: Will switching to Groq solve the problem permanently? Groq reduces latency to 150-300ms, which is a big improvement over 800ms. But it's still 150ms+ on the hot path, and you're still paying per-token. For many teams, it's a good intermediate step while you evaluate compiled classifiers for your highest-volume endpoints.

Q: What accuracy should I expect from a compiled classifier vs. GPT-4o? Across benchmarked domains, compiled models achieve 89-95% F1 scores. For content moderation specifically, 91.5% F1 at 41ms P95, compared to GPT-4o's ~93% F1 at 623ms P50. The accuracy gap is small; the latency gap is 15x.

Q: Can I build my own compiled classifier without Sparkient? Absolutely. The pipeline is: (1) generate or collect labeled training data, (2) encode text into semantic embeddings, (3) train a classifier on the encoded features, (4) export to ONNX, (5) serve with ONNX Runtime. If you have an ML team, this is a well-understood pipeline. Sparkient automates all five steps, including synthetic data generation with an LLM teacher.

Q: How does the LLM escalation work? The compiled classifier returns a confidence score with every prediction. When confidence is below a threshold (e.g., 0.7), the request automatically escalates to an LLM for a higher-quality decision. This happens for roughly 3-5% of requests — ambiguous cases that the classifier is genuinely uncertain about. The result is near-LLM accuracy at classifier speed for the vast majority of requests.

Your latency budget should go to features, not classification. Try Sparkient free — 5,000 credits, no credit card required.