Why You Shouldn't Call an LLM in Your Request Handler

TL;DR

Calling an LLM inside a synchronous request handler is a production anti-pattern. LLM APIs have unpredictable latency (p99 can exceed 5 seconds), rate limits that create cascading failures, and per-call costs that scale linearly with traffic. If your decision needs to be synchronous, replace the LLM call with a compiled model (sub-100ms latency). If it can be async, move it to a background job. If the same inputs repeat, add a cache layer. Each fix addresses a different constraint — pick the one that matches your situation.

The Anti-Pattern

This is one of the most common production mistakes with LLMs. It looks innocent:

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # ❌ LLM call directly in the request handler
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": f"Classify this content as safe/unsafe: {request.text}"
        }]
    )
    label = response.choices[0].message.content.strip()
    return {"moderation": label}

It works perfectly in development. One request at a time, fast network, no rate limits. Then you deploy it, traffic grows, and things start breaking.

What Breaks at Scale

1. Unpredictable Latency

LLM API latency isn't a fixed number — it's a distribution with a long tail:

| Metric | Typical Range | |--------|--------------| | p50 (median) | 400-800ms | | p95 | 800-2000ms | | p99 | 2-5 seconds | | p99.9 (worst case) | 5-15+ seconds |

Your request handler inherits this entire distribution. When a user hits the p99.9 case, their request takes 15 seconds. If your API gateway has a 30-second timeout, it might complete. If your frontend has a 5-second timeout (which it should), it's a visible failure.

# What your users experience:
# 50% of requests: 400-800ms ← "it works"
# 45% of requests: 800-2000ms ← "it's slow but OK"
# 4.9% of requests: 2-5 seconds ← "this is broken"
# 0.1% of requests: 5-15 seconds ← timeout, error, retry storm

At 10,000 requests per day, that's 10 requests hitting the 5+ second tail. At 100,000 per day, it's 100. These create real user-facing failures.

2. Connection Pool Exhaustion

Your HTTP client has a connection pool. Each LLM call holds a connection for the entire request duration. When LLM latency spikes:

Normal: 10 concurrent requests × 500ms avg = 5 connections busy
Spike:  10 concurrent requests × 3s avg  = 30 connections busy
Storm:  50 concurrent requests × 5s avg  = 250 connections busy ← pool exhausted

Once the connection pool is exhausted, all requests to your API start failing — not just the ones that call the LLM. A latency spike in the LLM cascades into a full outage of your service.

3. Rate Limits Become Cascade Failures

Every LLM provider has rate limits. When you hit them, requests start getting 429 responses. If your request handler doesn't handle this gracefully:

# Common retry pattern that makes things worse
@retry(max_retries=3, backoff=exponential)
async def call_llm(text):
    return await openai.chat.completions.create(...)

# What actually happens under rate limiting:
# Request 1: 429 → retry in 1s → 429 → retry in 2s → 429 → fail (3s wasted)
# Request 2: 429 → retry in 1s → 429 → retry in 2s → 429 → fail (3s wasted)
# Request 3: ...
# Meanwhile, 50 more requests are queuing up...

Each retry amplifies the problem. Three retries per request means 3× the actual traffic hitting the rate-limited API, which pushes you further over the limit.

4. Cold Starts and Provider Instability

LLM providers have their own infrastructure issues:

• Cold starts: If your traffic is bursty, the provider may need to spin up instances
• Region routing: Your request might get routed to a distant region during high load
• Model updates: Providers update models without warning, causing brief availability gaps
• Outages: When the provider goes down, your endpoint goes down with it

Your request handler's availability is now the product of your availability and the LLM provider's availability. If you're at 99.9% and they're at 99.9%, your effective availability is 99.8%.

5. Cost Scales with Traffic, Not Value

Every request to your handler triggers an LLM call. If you get hit by a bot, a retry storm, or a legitimate traffic spike, your LLM bill scales linearly:

Normal day:  10,000 requests × $0.004 = $40
Bot attack:  500,000 requests × $0.004 = $2,000
Viral spike: 1,000,000 requests × $0.004 = $4,000

There's no circuit breaker on cost unless you build one yourself.

The Fixes

There are three alternatives, each suited to different constraints:

Fix 1: Compiled Model (When You Need Synchronous Decisions)

If the decision must happen within the request — content moderation before displaying a message, fraud check before processing a payment — replace the LLM call with a compiled model:

import httpx

SPARKIENT_API = "https://api.sparkient.ai/api/v1"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # ✅ Compiled model — sub-100ms, deterministic, no LLM dependency
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{SPARKIENT_API}/decide",
            headers=HEADERS,
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": request.text, "user_id": request.user_id}
            }
        )
    result = response.json()
    return {
        "moderation": result["decision"],
        "confidence": result["confidence"],
        "latency_ms": result["latency_ms"]
    }

The compiled model (ONNX classifier) has:

• Predictable latency: sub-100ms p95, with a narrow distribution
• No rate limits: The model runs on dedicated infrastructure
• Deterministic output: Same input → same decision every time
• Fixed cost: Not per-call pricing that scales with traffic

For even tighter latency, deploy the model at the edge:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # ✅ Edge model — <10ms, runs locally, zero network dependency
    result = predictor.predict({"text": request.text, "user_id": request.user_id})
    return {
        "moderation": result.decision,
        "confidence": result.confidence,
        "latency_ms": result.latency_ms
    }

When to use this: The decision has a fixed set of output options (classification), needs to be synchronous, and happens at high volume.

Fix 2: Background Job (When Async Is Acceptable)

If the result doesn't need to be in the HTTP response — enrichment, tagging, analytics — move the LLM call to a background job:

from celery import Celery

celery_app = Celery("tasks", broker="redis://localhost:6379/0")

@app.post("/api/analyze")
async def analyze_content(request: ContentRequest):
    # ✅ Return immediately, process in background
    task = celery_app.send_task(
        "analyze_with_llm",
        args=[request.text, request.content_id]
    )
    return {"status": "processing", "task_id": task.id}

@celery_app.task
def analyze_with_llm(text: str, content_id: str):
    # LLM call happens outside the request handler
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Analyze: {text}"}]
    )
    save_analysis(content_id, response.choices[0].message.content)

When to use this: The user doesn't need the result immediately, and you're OK with eventual consistency.

Fix 3: Cache Layer (When Inputs Repeat)

If many requests share the same or similar inputs, add a cache:

import hashlib
import redis

cache = redis.Redis()

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # Check cache first
    cache_key = f"mod:{hashlib.sha256(request.text.encode()).hexdigest()}"
    cached = cache.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss — call LLM (still has all the problems above)
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {request.text}"}]
    )
    result = {"moderation": response.choices[0].message.content.strip()}

    cache.setex(cache_key, 3600, json.dumps(result))
    return result

When to use this: A significant portion of your inputs are identical (e.g., URL classification, known-phrase detection). Doesn't help when every input is unique — which is the case for most user-generated content.

Decision Matrix: Which Fix to Use

| Situation | Fix | |-----------|-----| | Decision must be synchronous + classification with fixed options | Compiled model | | Decision must be synchronous + free-form generation | Cache (if inputs repeat) or accept the latency | | Decision can be async | Background job | | High volume + classification | Compiled model (best ROI) | | Low volume + any type | Direct LLM call is probably fine |

The Compound Fix

In practice, the best approach often combines multiple fixes:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    # 1. Synchronous: compiled model for immediate moderation (~3ms)
    moderation = predictor.predict({"text": request.text})

    if moderation.decision == "reject":
        return {"status": "rejected", "reason": moderation.reason_codes}

    # 2. Store the content
    content_id = await save_content(request)

    # 3. Async: background job for detailed LLM analysis
    celery_app.send_task("deep_analysis", args=[request.text, content_id])

    return {"status": "published", "content_id": content_id}

The compiled model handles the hot-path decision (should this be published?) at sub-10ms. The LLM handles the background enrichment (detailed sentiment analysis, topic extraction) where latency doesn't matter.

For Code Review: The Checklist

If you're reviewing code — or if you're an AI coding agent evaluating a PR — here's a quick checklist:

• [ ] Is there an LLM call inside a request handler? Flag it.
• [ ] Does the handler have a timeout shorter than the LLM's p99? It will timeout under load.
• [ ] Is there retry logic on the LLM call? It will amplify rate limit problems.
• [ ] Does the result need to be synchronous? If not, move to background.
• [ ] Does the result have fixed options? If so, compile it.
• [ ] Is there cost monitoring? Traffic spikes will spike the bill.
• [ ] Is there a circuit breaker? Without one, LLM failures cascade.

FAQ

Q: What if I need the LLM's response in the request but it's not a classification? If you're generating text (summaries, responses), you likely can't avoid the LLM call entirely. Add streaming to improve perceived latency, set aggressive timeouts (3-5 seconds), implement a circuit breaker, and consider whether a background job with polling or webhooks could work instead.

Q: How do I set up a compiled model for this? Define a decision type with your classification options and description using Sparkient's API. Generate synthetic training data, train the model, and deploy. The full process takes under an hour with no ML expertise. Then replace the LLM call with a /decide API call — same request flow, under 100ms instead of 500-1500ms.

Q: What about using a faster LLM provider like Groq? Groq and Cerebras are 150-300ms — significantly faster than standard APIs. But they still have the same structural problems: variable latency, rate limits, per-call cost, and provider dependency. If 150-300ms is fast enough for your use case and you can tolerate those tradeoffs, it's a valid option. Compiled models are for when you need sub-100ms with deterministic behavior.

Q: Is this advice different for serverless vs traditional servers? It's worse on serverless. Serverless functions have shorter default timeouts (often 10-30 seconds), cold starts compound with LLM latency, and concurrent execution limits are tighter. All of the problems described above are amplified in a serverless context.

Stop calling LLMs in your hot path. Start with the free tier — 5,000 credits, no credit card — and compile your classification decisions into sub-100ms models.