Why Your API Is Slow: Finding the LLM Bottleneck in Your Stack

TL;DR

If API latency rose after adding an AI feature, trace the request before assigning blame. A live-model call is one common cause, but its impact is model-, prompt-, provider-, and load-dependent. The appropriate response may be caching, asynchronous processing, a different live model, or evaluating a compiled classifier for a repeated bounded decision.

The Symptom: Your P95 Just Tripled

You shipped a new feature — maybe content moderation, maybe a smart routing layer, maybe an auto-categorization step — and your P95 latency went from 80ms to 900ms overnight. Your frontend team is filing bugs. Your SLA is in danger.

You check the usual suspects:

• Database queries — compare current traces with the earlier baseline
• External API calls — same third-party services, same latency
• Memory / CPU — utilization looks normal
• Network — no packet loss, no DNS issues

Everything looks fine. Except it isn't.

How to Find the Bottleneck

Before you start optimizing, you need to know exactly where the time is going. Here are three approaches, from quick-and-dirty to production-grade.

1. Quick Timing Wrapper

Add timing logs around each step in your request handler:

import time
import structlog

logger = structlog.get_logger()

async def handle_request(request):
    t0 = time.perf_counter()

    # Step 1: Validate input
    validated = validate(request)
    t1 = time.perf_counter()
    logger.info("step_timing", step="validate", ms=round((t1 - t0) * 1000, 1))

    # Step 2: Business logic
    result = process(validated)
    t2 = time.perf_counter()
    logger.info("step_timing", step="process", ms=round((t2 - t1) * 1000, 1))

    # Step 3: The "AI enhancement" someone added last sprint
    classification = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {result.text}"}]
    )
    t3 = time.perf_counter()
    logger.info("step_timing", step="classify", ms=round((t3 - t2) * 1000, 1))

    return build_response(result, classification)

You'll see output like:

step=validate ms=0.3
step=process ms=4.2
step=classify ms=847.6

There it is. The classify step — an LLM call — is eating 847ms of your request budget.

2. Distributed Tracing

If you're running OpenTelemetry or Datadog APM, add spans around your LLM calls. Most LLM SDKs now support auto-instrumentation:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def classify_content(text: str):
    with tracer.start_as_current_span("llm.classify") as span:
        span.set_attribute("llm.provider", "openai")
        span.set_attribute("llm.model", "gpt-4o")
        result = await openai.chat.completions.create(...)
        span.set_attribute("llm.latency_ms", result.usage.completion_time_ms)
        return result

This gives you a flame graph where the LLM call sticks out like a skyscraper next to bungalows.

3. Production Metrics Dashboard

Track LLM call latency as a histogram metric. After a week, you'll have P50, P95, and P99 numbers:

| Metric | Typical LLM Call | Everything Else | |--------|-----------------|-----------------| | P50 | 620ms | 12ms | | P95 | 1,100ms | 45ms | | P99 | 2,300ms | 120ms |

The variance is the killer. Your P99 is nearly 4x your P50 because LLM APIs have unpredictable cold starts, queuing delays, and variable generation lengths.

Why LLM Calls Are Uniquely Bad for Latency

Not all slow API calls are equally painful. A database query to a replica might take 50ms — slow, but consistent. LLM calls have three properties that make them particularly damaging:

Workload-dependent baseline. Model, provider, input length, structured-output method, output length, region, and load all affect the request.

Potentially high variance. Inspect p50, p95, and p99; do not infer the tail from the average.

External dependency. HTTP connections can be pooled, but the remote model service, its rate limits, and its queue remain outside your process.

Four Ways to Fix It

Once you've confirmed the LLM call is the bottleneck, here are your options — each with real trade-offs.

Option 1: Move to a Faster LLM Provider

Switch from OpenAI to Groq or Cerebras for inference:

| Candidate | What to measure | Quality check | |-----------|-----------------|---------------| | Current live model | End-to-end p50/p95/p99, tokens, errors | Existing held-out set | | Smaller or optimised live model | Same prompt and traffic profile | Per-class comparison with current path | | Compiled candidate | Classifier-only and combined escalation paths | Per-class errors and calibration |

Best for: When you need generative flexibility and the selected live model clears the measured quality, latency, and cost budget.

Limitation: It remains a provider dependency with current token pricing and limits. Calculate the bill from observed tokens and traffic rather than a request-count threshold.

Option 2: Cache Frequent Inputs

If many of your inputs are identical or near-identical (e.g., users posting the same spam), a cache layer can eliminate redundant calls:

import hashlib
import redis

r = redis.Redis()

async def classify_cached(text: str) -> str:
    cache_key = f"classify:{hashlib.sha256(text.encode()).hexdigest()}"
    cached = r.get(cache_key)
    if cached:
        return cached.decode()

    result = await call_llm(text)
    r.setex(cache_key, 3600, result)
    return result

Best for: High-duplicate workloads like spam detection.

Limitation: Exact-match caching does not help unique inputs. Measure hit rate and freshness instead of assuming either.

Option 3: Move Classification Off the Hot Path

Process classifications asynchronously — return a response immediately and update the classification later:

from celery import Celery

app = Celery("tasks", broker="redis://localhost")

@app.task
def classify_async(item_id: str, text: str):
    result = call_llm(text)
    db.update(item_id, classification=result)

# In your request handler:
async def handle_request(request):
    item = save_to_db(request)
    classify_async.delay(item.id, request.text)  # Fire and forget
    return {"id": item.id, "status": "processing"}

Best for: When the classification result doesn't need to be in the API response (e.g., background moderation, analytics tagging).

Limitation: Many use cases require the classification in the response. If you need to approve/reject content before showing it to users, async doesn't work.

Option 4: Compile the LLM Into a Fast Classifier

If your LLM call is doing classification — picking from a fixed set of options like approve/review/reject or spam/not_spam — you can compile that logic into a purpose-built ONNX classifier. The LLM teaches a lightweight model offline; the lightweight model runs in production.

import httpx

# Before: measure the live model path
async def classify_with_llm(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify as approve/review/reject: {text}"}]
    )
    return response.choices[0].message.content

# After: measure the trained Sparkient path
async def classify_with_sparkient(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}

Best for: Classification and routing tasks where you're choosing from a defined set of options. This covers content moderation, ticket triage, lead scoring, fraud flags.

Limitation: This only works for classification. If your LLM call is generating free-form text (summaries, translations, creative writing), you still need an LLM.

Before and After

Use this before/after template with measurements from the same endpoint and traffic:

| Metric | Before (GPT-4o) | After (Compiled Model) | |--------|-----------------|----------------------| | P50 latency | [measure] | [measure by stage] | | P95 latency | [measure] | [include escalation] | | P99 latency | [measure] | [include escalation] | | Monthly usage cost | [provider tokens] | [plan credits and top-ups] | | Per-class quality | [measure] | [measure] | | Runtime dependency | LLM provider | Sparkient cloud; optional LLM escalation |

Tail latency matters as much as the average. Keep classifier-only and escalated requests separate so the combined p95 and p99 are not understated.

How to Decide Which Fix to Use

Ask yourself two questions:

1. Does the classification need to be in the response? If no → async processing (Option 3).
2. Is the LLM choosing from a fixed set of outcomes? If yes → compiled classifier (Option 4). If no → faster LLM (Option 1) + caching (Option 2).

Start with the smallest change supported by the traces. A compiled classifier is warranted only when a stable bounded call remains a material constraint and the candidate passes a held-out evaluation.

FAQ

Q: How do I know if my LLM call is actually the bottleneck? Add timing spans around each step in the request handler and inspect p50, p95, and p99 on representative traffic. That evidence will show whether the LLM call, the network, the database, or something else owns the latency budget.

Q: Can I just use streaming to make it feel faster? Streaming helps for chat UIs where the user sees tokens arriving. It doesn't help for API-to-API calls where you need the full result before responding. Your server-side latency is the same whether you stream or not.

Q: What accuracy trade-off should I expect with a compiled classifier? Across four controlled synthetic domains—support triage, content moderation, gaming chat, and marketplace review—compiled models report 0.886–0.951 macro F1, 91–96% accuracy, and 33–42ms batch-average time per item. Those runs do not measure per-request p95 or customer traffic. Compare against the exact LLM prompt and traffic in your project before claiming equivalence or savings.

Q: Is Sparkient the only way to compile an LLM into a classifier? No. You can build your own text encoder or train a gradient-boosted classifier yourself. Sparkient provides generation, labelling, training, deployment, and ONNX export as one workflow. If you have an ML team and need complete control, building your own pipeline is a valid choice; compare both options on quality, elapsed setup time, credits, and maintenance.

Latency budgets are finite. Spend yours on things only an LLM can do. For classification, start with the free tier — 5,000 credits, no credit card required.