Why You Shouldn't Call an LLM in Your Request Handler

TL;DR

A live-model call inside a synchronous handler is a risk when its p95/p99, provider limits, retries, or usage cost exceed the endpoint's budget. Trace it first. If the work can be asynchronous, move it off the path; if eligible inputs repeat, test a cache; if it is a stable bounded decision, evaluate a compiled classifier. Each remedy solves a different measured constraint.

The Anti-Pattern

This is one of the most common production mistakes with LLMs. It looks innocent:

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # ❌ LLM call directly in the request handler
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{
            "role": "user",
            "content": f"Classify this content as safe/unsafe: {request.text}"
        }]
    )
    label = response.choices[0].message.content.strip()
    return {"moderation": label}

It works perfectly in development. One request at a time, fast network, no rate limits. Then you deploy it, traffic grows, and things start breaking.

What Breaks at Scale

1. Unpredictable Latency

LLM API latency isn't a fixed number — it's a distribution with a long tail:

| Metric | Current endpoint | |--------|------------------| | p50 (median) | [measure] | | p95 | [measure] | | p99 | [measure] | | Timeout/error rate | [measure] |

Your request handler inherits the model call's distribution. Compare it with the client, gateway, and downstream timeouts in the actual system.

# Record p50, p95, p99, timeouts, and retries from production traces.

At 10,000 requests per day, that's 10 requests hitting the 5+ second tail. At 100,000 per day, it's 100. These create real user-facing failures.

2. Connection Pool Exhaustion

Your HTTP client has a connection pool. Each LLM call holds a connection for the entire request duration. When LLM latency spikes:

Normal: 10 concurrent requests × 500ms avg = 5 connections busy
Spike:  10 concurrent requests × 3s avg  = 30 connections busy
Storm:  50 concurrent requests × 5s avg  = 250 connections busy ← pool exhausted

Once the connection pool is exhausted, all requests to your API start failing — not just the ones that call the LLM. A latency spike in the LLM cascades into a full outage of your service.

3. Rate Limits Become Cascade Failures

Every LLM provider has rate limits. When you hit them, requests start getting 429 responses. If your request handler doesn't handle this gracefully:

# Common retry pattern that makes things worse
@retry(max_retries=3, backoff=exponential)
async def call_llm(text):
    return await openai.chat.completions.create(...)

# What actually happens under rate limiting:
# Request 1: 429 → retry in 1s → 429 → retry in 2s → 429 → fail (3s wasted)
# Request 2: 429 → retry in 1s → 429 → retry in 2s → 429 → fail (3s wasted)
# Request 3: ...
# Meanwhile, 50 more requests are queuing up...

Each retry amplifies the problem. Three retries per request means 3× the actual traffic hitting the rate-limited API, which pushes you further over the limit.

4. Cold Starts and Provider Instability

LLM providers have their own infrastructure issues:

• Cold starts: If your traffic is bursty, the provider may need to spin up instances
• Region routing: Your request might get routed to a distant region during high load
• Model updates: Providers update models without warning, causing brief availability gaps
• Outages: When the provider goes down, your endpoint goes down with it

Your request handler's availability is now the product of your availability and the LLM provider's availability. If you're at 99.9% and they're at 99.9%, your effective availability is 99.8%.

5. Cost Scales with Traffic, Not Value

Every request to your handler triggers an LLM call. If you get hit by a bot, a retry storm, or a legitimate traffic spike, your LLM bill scales linearly:

Model the exposure as accepted traffic × observed tokens × current provider prices, plus retry amplification. Put explicit rate, credit, and spend limits in front of the endpoint.

There's no circuit breaker on cost unless you build one yourself.

The Fixes

There are three alternatives, each suited to different constraints:

Fix 1: Compiled Model (When You Need Synchronous Decisions)

If the decision must happen within the request — content moderation before displaying a message, fraud check before processing a payment — replace the LLM call with a compiled model:

import httpx

SPARKIENT_API = "https://api.sparkient.ai/api/v1"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # Compiled normal path; optional escalation is reported in result["stage"]
    async with httpx.AsyncClient() as client:
        response = await client.post(
            f"{SPARKIENT_API}/decide",
            headers=HEADERS,
            json={
                "decision_type": "content-moderation",
                "input": {"text": request.text, "user_id": request.user_id}
            }
        )
    result = response.json()
    return {
        "moderation": result["decision"],
        "confidence": result["confidence"],
        "latency_ms": result["latency_ms"]
    }

The compiled model (ONNX classifier) has:

• Compiled-stage target: under 100ms; four controlled synthetic runs report 33–42ms batch-average time per item, not per-request p95
• Measured capacity: Cloud limits and edge hardware can be tested against the expected load
• Structured output: The response includes the decision, confidence, stage, and reason codes
• Credit-metered operations: No live LLM token charge on the normal compiled-model path

For even tighter latency, deploy the model at the edge:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # Edge model — runs locally; benchmark latency on target hardware
    result = predictor.predict({"text": request.text, "user_id": request.user_id})
    return {
        "moderation": result.decision,
        "confidence": result.confidence,
        "stage": result.stage
    }

When to use this: The decision has fixed, stable outcomes, needs a constrained runtime profile, and a held-out evaluation shows the compiled candidate is suitable.

Fix 2: Background Job (When Async Is Acceptable)

If the result doesn't need to be in the HTTP response — enrichment, tagging, analytics — move the LLM call to a background job:

from celery import Celery

celery_app = Celery("tasks", broker="redis://localhost:6379/0")

@app.post("/api/analyze")
async def analyze_content(request: ContentRequest):
    # ✅ Return immediately, process in background
    task = celery_app.send_task(
        "analyze_with_llm",
        args=[request.text, request.content_id]
    )
    return {"status": "processing", "task_id": task.id}

@celery_app.task
def analyze_with_llm(text: str, content_id: str):
    # LLM call happens outside the request handler
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Analyze: {text}"}]
    )
    save_analysis(content_id, response.choices[0].message.content)

When to use this: The user doesn't need the result immediately, and you're OK with eventual consistency.

Fix 3: Cache Layer (When Inputs Repeat)

If many requests share the same or similar inputs, add a cache:

import hashlib
import redis

cache = redis.Redis()

@app.post("/api/moderate")
async def moderate_content(request: ContentRequest):
    # Check cache first
    cache_key = f"mod:{hashlib.sha256(request.text.encode()).hexdigest()}"
    cached = cache.get(cache_key)
    if cached:
        return json.loads(cached)

    # Cache miss — call LLM (still has all the problems above)
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {request.text}"}]
    )
    result = {"moderation": response.choices[0].message.content.strip()}

    cache.setex(cache_key, 3600, json.dumps(result))
    return result

When to use this: A significant portion of your inputs are identical (e.g., URL classification, known-phrase detection). Doesn't help when every input is unique — which is the case for most user-generated content.

Decision Matrix: Which Fix to Use

| Situation | Fix | |-----------|-----| | Decision must be synchronous + classification with fixed options | Compiled model | | Decision must be synchronous + free-form generation | Cache (if inputs repeat) or accept the latency | | Decision can be async | Background job | | Recurring classification with a measured constraint | Evaluate a compiled model | | Low volume + any type | Direct LLM call is probably fine |

The Compound Fix

In practice, the best approach often combines multiple fixes:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    # 1. Synchronous: compiled model for immediate moderation (~3ms)
    moderation = predictor.predict({"text": request.text})

    if moderation.decision == "reject":
        return {"status": "rejected", "reason": moderation.reason_codes}

    # 2. Store the content
    content_id = await save_content(request)

    # 3. Async: background job for detailed LLM analysis
    celery_app.send_task("deep_analysis", args=[request.text, content_id])

    return {"status": "published", "content_id": content_id}

The compiled model handles the bounded hot-path decision; the LLM handles background enrichment where generation is still useful. Measure the local or cloud classifier latency in the target environment.

For Code Review: The Checklist

If you're reviewing code — or if you're an AI coding agent evaluating a PR — here's a quick checklist:

• [ ] Is there an LLM call inside a request handler? Flag it.
• [ ] Does the handler have a timeout shorter than the LLM's p99? It will timeout under load.
• [ ] Is there retry logic on the LLM call? It will amplify rate limit problems.
• [ ] Does the result need to be synchronous? If not, move to background.
• [ ] Does the result have fixed, stable, measurable options? If so, evaluate a compiled candidate.
• [ ] Is there cost monitoring? Traffic spikes will spike the bill.
• [ ] Is there a circuit breaker? Without one, LLM failures cascade.

FAQ

Q: What if I need the LLM's response in the request but it's not a classification? If you're generating text, you may need the live model. Streaming can improve perceived latency; set timeouts from the product objective and observed distribution, add a circuit breaker, and consider a background job with polling or webhooks.

Q: How do I set up a compiled model for this? Define a decision type, generate or add examples, train, and evaluate the candidate. Duration depends on the data and configuration. If it passes, introduce /decide gradually and measure full-path latency, stage mix, credits, and per-class quality.

Q: What about using a faster LLM provider like Groq? Test the current provider, model, and prompt. Optimised live inference may be sufficient, but it remains a provider path with workload-dependent latency, limits, and pricing. A compiled model is a candidate only for a bounded decision that passes its own evaluation; classifier output is structured, not a guarantee of deterministic business correctness.

Q: Is this advice different for serverless vs traditional servers? Serverless changes the limits rather than making one design universally worse. Check the chosen platform's current timeout, cold-start, concurrency, and networking behaviour alongside the live-model distribution.

Stop calling LLMs in your hot path. Start with the free tier — 5,000 credits, no credit card — and compile your classification decisions into sub-100ms models.