Why Your API Is Slow: Finding the LLM Bottleneck in Your Stack

TL;DR

If your API response time spiked after adding an AI feature, the culprit is almost certainly an LLM call sitting in your request handler. A single GPT-4o call adds 500-1500ms to every request. You can fix this with caching, async processing, a faster LLM provider (150-300ms), or by compiling the LLM's logic into an ONNX classifier that runs in under 100ms.

The Symptom: Your P95 Just Tripled

You shipped a new feature — maybe content moderation, maybe a smart routing layer, maybe an auto-categorization step — and your P95 latency went from 80ms to 900ms overnight. Your frontend team is filing bugs. Your SLA is in danger.

You check the usual suspects:

• Database queries — no change, still sub-10ms
• External API calls — same third-party services, same latency
• Memory / CPU — utilization looks normal
• Network — no packet loss, no DNS issues

Everything looks fine. Except it isn't.

How to Find the Bottleneck

Before you start optimizing, you need to know exactly where the time is going. Here are three approaches, from quick-and-dirty to production-grade.

1. Quick Timing Wrapper

Add timing logs around each step in your request handler:

import time
import structlog

logger = structlog.get_logger()

async def handle_request(request):
    t0 = time.perf_counter()

    # Step 1: Validate input
    validated = validate(request)
    t1 = time.perf_counter()
    logger.info("step_timing", step="validate", ms=round((t1 - t0) * 1000, 1))

    # Step 2: Business logic
    result = process(validated)
    t2 = time.perf_counter()
    logger.info("step_timing", step="process", ms=round((t2 - t1) * 1000, 1))

    # Step 3: The "AI enhancement" someone added last sprint
    classification = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {result.text}"}]
    )
    t3 = time.perf_counter()
    logger.info("step_timing", step="classify", ms=round((t3 - t2) * 1000, 1))

    return build_response(result, classification)

You'll see output like:

step=validate ms=0.3
step=process ms=4.2
step=classify ms=847.6

There it is. The classify step — an LLM call — is eating 847ms of your request budget.

2. Distributed Tracing

If you're running OpenTelemetry or Datadog APM, add spans around your LLM calls. Most LLM SDKs now support auto-instrumentation:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def classify_content(text: str):
    with tracer.start_as_current_span("llm.classify") as span:
        span.set_attribute("llm.provider", "openai")
        span.set_attribute("llm.model", "gpt-4o")
        result = await openai.chat.completions.create(...)
        span.set_attribute("llm.latency_ms", result.usage.completion_time_ms)
        return result

This gives you a flame graph where the LLM call sticks out like a skyscraper next to bungalows.

3. Production Metrics Dashboard

Track LLM call latency as a histogram metric. After a week, you'll have P50, P95, and P99 numbers:

| Metric | Typical LLM Call | Everything Else | |--------|-----------------|-----------------| | P50 | 620ms | 12ms | | P95 | 1,100ms | 45ms | | P99 | 2,300ms | 120ms |

The variance is the killer. Your P99 is nearly 4x your P50 because LLM APIs have unpredictable cold starts, queuing delays, and variable generation lengths.

Why LLM Calls Are Uniquely Bad for Latency

Not all slow API calls are equally painful. A database query to a replica might take 50ms — slow, but consistent. LLM calls have three properties that make them particularly damaging:

High baseline latency. Even the fastest commercial LLMs (Groq, Cerebras) take 150-300ms for a structured response. Standard APIs like GPT-4o-mini or Gemini Flash sit at 500-1500ms.

High variance. LLM latency depends on input length, output length, model load, and provider queuing. Your P95 might be 2-3x your P50.

No connection pooling. Unlike database connections, you can't pool or reuse LLM connections in a meaningful way. Every call is a cold HTTP request.

Four Ways to Fix It

Once you've confirmed the LLM call is the bottleneck, here are your options — each with real trade-offs.

Option 1: Move to a Faster LLM Provider

Switch from OpenAI to Groq or Cerebras for inference:

| Provider | Model | Typical Latency | Accuracy Trade-off | |----------|-------|-----------------|-------------------| | OpenAI GPT-4o | gpt-4o | 600-1500ms | Baseline | | OpenAI GPT-4o-mini | gpt-4o-mini | 400-800ms | Slight accuracy drop | | Groq | Llama 3 70B | 150-300ms | Comparable for classification | | Cerebras | Llama 3 70B | 150-250ms | Comparable for classification |

Best for: When you need the full flexibility of an LLM and can tolerate 150ms+ latency.

Limitation: You're still paying per-token and still exposed to provider outages. At 50K requests/day, Groq bills add up fast.

Option 2: Cache Frequent Inputs

If many of your inputs are identical or near-identical (e.g., users posting the same spam), a cache layer can eliminate redundant calls:

import hashlib
import redis

r = redis.Redis()

async def classify_cached(text: str) -> str:
    cache_key = f"classify:{hashlib.sha256(text.encode()).hexdigest()}"
    cached = r.get(cache_key)
    if cached:
        return cached.decode()

    result = await call_llm(text)
    r.setex(cache_key, 3600, result)
    return result

Best for: High-duplicate workloads like spam detection.

Limitation: Cache hit rates for unique content (user posts, emails, support tickets) are typically under 10%. This won't help if every input is different.

Option 3: Move Classification Off the Hot Path

Process classifications asynchronously — return a response immediately and update the classification later:

from celery import Celery

app = Celery("tasks", broker="redis://localhost")

@app.task
def classify_async(item_id: str, text: str):
    result = call_llm(text)
    db.update(item_id, classification=result)

# In your request handler:
async def handle_request(request):
    item = save_to_db(request)
    classify_async.delay(item.id, request.text)  # Fire and forget
    return {"id": item.id, "status": "processing"}

Best for: When the classification result doesn't need to be in the API response (e.g., background moderation, analytics tagging).

Limitation: Many use cases require the classification in the response. If you need to approve/reject content before showing it to users, async doesn't work.

Option 4: Compile the LLM Into a Fast Classifier

If your LLM call is doing classification — picking from a fixed set of options like approve/review/reject or spam/not_spam — you can compile that logic into a purpose-built ONNX classifier. The LLM teaches a lightweight model offline; the lightweight model runs in production.

import httpx

# Before: ~800ms per request
async def classify_with_llm(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify as approve/review/reject: {text}"}]
    )
    return response.choices[0].message.content

# After: ~41ms per request
async def classify_with_sparkient(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}

Best for: Classification and routing tasks where you're choosing from a defined set of options. This covers content moderation, ticket triage, lead scoring, fraud flags.

Limitation: This only works for classification. If your LLM call is generating free-form text (summaries, translations, creative writing), you still need an LLM.

Before and After

Here's a realistic before/after for an API endpoint that classifies user-generated content:

| Metric | Before (GPT-4o) | After (Compiled Model) | |--------|-----------------|----------------------| | P50 latency | 623ms | 33ms | | P95 latency | 1,100ms | 41ms | | P99 latency | 2,300ms | 58ms | | Monthly cost (50K req/day) | ~$15,000 | $199 | | Variance (P99/P50) | 3.7x | 1.8x | | Provider dependency | Full | Training only |

The latency improvement matters, but the variance improvement matters more. Your P99 drops from 2.3 seconds to 58ms. Your tail latency becomes predictable.

How to Decide Which Fix to Use

Ask yourself two questions:

1. Does the classification need to be in the response? If no → async processing (Option 3).
2. Is the LLM choosing from a fixed set of outcomes? If yes → compiled classifier (Option 4). If no → faster LLM (Option 1) + caching (Option 2).

Most teams start with caching and a faster provider, then move to a compiled classifier when they realize the cost and variance are still problematic at scale.

FAQ

Q: How do I know if my LLM call is actually the bottleneck? Add timing logs around each step in your request handler. In nearly every case we've seen, the LLM call is 10-100x slower than every other step combined. A quick time.perf_counter() wrapper will confirm it in minutes.

Q: Can I just use streaming to make it feel faster? Streaming helps for chat UIs where the user sees tokens arriving. It doesn't help for API-to-API calls where you need the full result before responding. Your server-side latency is the same whether you stream or not.

Q: What accuracy trade-off should I expect with a compiled classifier? Across benchmarked domains (content moderation, phishing detection, support triage), compiled models achieve 89-95% F1 scores and 91-96% accuracy. That's in the same range as GPT-4o for classification tasks, at 15x less latency.

Q: Is Sparkient the only way to compile an LLM into a classifier? No. You can build your own text encoder or train a gradient-boosted classifier yourself. Sparkient automates the synthetic data generation, training, and ONNX export steps. If you have an ML team and want full control, building your own pipeline is a valid choice. If you want to go from zero to production classifier in an afternoon, the managed service is faster.

Latency budgets are finite. Spend yours on things only an LLM can do. For classification, start with the free tier — 5,000 credits, no credit card required.