5 Architecture Anti-Patterns That Break at 100K Requests Per Day

TL;DR

Five architecture patterns work fine at prototype scale but break catastrophically at 100K+ requests/day: synchronous LLM calls in request handlers (adding 600ms+ per request), blocking classification in hot paths, zero caching for identical inputs, single-threaded model inference, and unbounded LLM costs without budget caps. Each has a specific fix — and most fixes should be implemented before you hit the wall, not after.

Introduction

Every architecture decision is a bet on scale. Some bets pay off. Others work perfectly at 1,000 requests per day and implode at 100,000.

These five anti-patterns share a common trait: they're all reasonable choices at prototype scale. They're simple, they work, and they ship fast. The problem isn't that you used them — it's that you didn't replace them before traffic caught up.

Here's how to spot each one and what to replace it with.

Anti-Pattern #1: LLM Call in the Request Handler

What it looks like

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    # Validation: 1ms
    validated = validate(request)

    # Database write: 5ms
    item = await db.save(validated)

    # LLM classification: 600-1500ms 😬
    classification = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {validated.text}"}]
    )

    item.category = parse_response(classification)
    await db.update(item)
    return item

Why it breaks

At 100 requests/second, you have 100 concurrent LLM calls in flight. Each takes 600-1500ms. Your connection pool is exhausted. Your response times spike. Clients start timing out, which triggers retries, which makes everything worse.

The math is brutal: 100 concurrent connections × 1 second average = 100 in-flight requests at steady state. Most web servers default to 100-500 max connections. You're at the limit before you've even started scaling.

What to do instead

Option A: Move it off the hot path. Queue the classification for async processing and return immediately:

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    validated = validate(request)
    item = await db.save(validated)
    await queue.send({"item_id": item.id, "text": validated.text})
    return {"id": item.id, "status": "processing"}  # 6ms total

Option B: Replace with a compiled model. If you need the classification in the response, swap the LLM call for a sub-100ms classifier:

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    validated = validate(request)

    # Compiled classifier: 33-41ms
    decision = await sparkient.decide("content-moderation", {"text": validated.text})

    item = await db.save(validated, category=decision["decision"])
    return item  # 45ms total

Option C: Use a faster LLM provider. Groq or Cerebras can deliver structured responses in 150-300ms — still not great for a hot path, but survivable if your concurrency is low.

Anti-Pattern #2: Synchronous Classification in a Hot Path

What it looks like

# Every message goes through this before being displayed
async def process_message(message: Message) -> Message:
    # Must classify before showing to other users
    safety = await classify_safety(message.text)  # Blocks rendering

    if safety == "reject":
        raise ContentPolicyViolation()

    message.safety_score = safety
    return message  # User sees message only after classification completes

Why it breaks

This is a variant of Anti-Pattern #1, but worse because it's in a user-facing hot path. The user is staring at a loading spinner while the classification runs. At 100K messages per day:

• Average user waits 800ms before their message appears
• Chat feels laggy compared to competitors
• Users start leaving before messages send

What to do instead

The fix depends on your tolerance for false negatives:

Optimistic rendering: Show the message immediately, classify in the background, and remove it if it's rejected:

async def process_message(message: Message) -> Message:
    await broadcast(message)  # Show immediately
    asyncio.create_task(classify_and_moderate(message))  # Background
    return message

Fast synchronous check: Use a sub-100ms classifier that's fast enough to be invisible:

async def process_message(message: Message) -> Message:
    # 33-41ms — users won't notice
    decision = await classify_fast(message.text)

    if decision["decision"] == "reject" and decision["confidence"] > 0.9:
        raise ContentPolicyViolation()

    return message

For content moderation specifically, compiled classifiers achieve 91.5% F1 with a P95 latency of 41ms. That's fast enough to sit in the hot path without degrading UX.

Anti-Pattern #3: No Caching for Identical Inputs

What it looks like

async def classify(text: str) -> str:
    # Called for every request, even duplicates
    return await llm.classify(text)

No cache layer. No deduplication. The same spam message gets classified 500 times at the same cost and latency.

Why it breaks

In content moderation and spam detection, duplicate inputs are common. A viral spam campaign might generate 10,000 identical messages. Without caching, you're paying for 10,000 LLM calls to get the same answer 10,000 times.

At $0.005 per call, that's $50 spent to repeatedly answer the same question. At scale, duplicates can account for 10-40% of your total LLM spend.

What to do instead

Add a cache layer with content hashing:

import hashlib
from functools import lru_cache

# In-memory LRU for hot duplicates
@lru_cache(maxsize=10_000)
def classify_cached_sync(text_hash: str, text: str) -> str:
    return classify_sync(text)

# Redis for distributed caching
async def classify_with_cache(text: str) -> str:
    text_hash = hashlib.sha256(text.encode()).hexdigest()

    # Check Redis first
    cached = await redis.get(f"classify:{text_hash}")
    if cached:
        return cached

    # Classify and cache for 1 hour
    result = await classify(text)
    await redis.setex(f"classify:{text_hash}", 3600, result)
    return result

This is the cheapest fix on this list. Implement it regardless of what else you do.

Important caveat: Caching works well for exact duplicates but doesn't help with unique content. If your workload is mostly unique (support tickets, user reviews), your cache hit rate will be under 5%. You need a different approach for those cases.

Anti-Pattern #4: Single-Threaded Model Inference

What it looks like

# Global model, no concurrency control
model = load_onnx_model("classifier.onnx")

async def predict(text: str) -> str:
    # Every request shares one inference session
    # ONNX Runtime is thread-safe, but this creates contention
    features = extract_features(text)
    return model.run(None, {"input": features})[0]

Why it breaks

ONNX Runtime and most ML inference libraries are thread-safe, but performance degrades under contention. With 100 concurrent requests:

• CPU inference queues up, creating latency spikes
• A single model instance becomes the bottleneck
• P99 latency can be 10x the P50

What to do instead

Use multiple inference sessions:

import onnxruntime as ort
from concurrent.futures import ThreadPoolExecutor

# Create a pool of inference sessions
session_pool = [
    ort.InferenceSession("classifier.onnx", providers=["CPUExecutionProvider"])
    for _ in range(4)  # Match your CPU core count
]

executor = ThreadPoolExecutor(max_workers=4)

async def predict(text: str) -> str:
    session = session_pool[hash(text) % len(session_pool)]
    features = extract_features(text)
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(
        executor,
        lambda: session.run(None, {"input": features})[0]
    )

Enable intra-op parallelism:

session_options = ort.SessionOptions()
session_options.intra_op_num_threads = 2     # Threads per inference
session_options.inter_op_num_threads = 4     # Parallel operations
session_options.execution_mode = ort.ExecutionMode.ORT_PARALLEL

session = ort.InferenceSession("classifier.onnx", session_options)

This fix is mechanical — it doesn't require changing your model or your architecture. Just add concurrency control and tune thread counts to your hardware.

Anti-Pattern #5: Unbounded LLM Costs Without Budget Caps

What it looks like

# No rate limiting, no cost tracking, no budget cap
async def handle_request(request):
    result = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": request.text}]
    )
    return result

There's no check on how many calls you're making or how much you're spending. The OpenAI API key has no usage limit. Your monitoring dashboard shows request counts but not dollar amounts.

Why it breaks

A traffic spike — whether from organic growth, a DDoS, or a viral moment — sends your LLM costs through the roof. A bot scraping your API can generate 1M requests in a day. At $0.005 per call, that's $5,000 in a single day from a single bot.

Without budget caps, you won't know until the invoice arrives.

What to do instead

Layer three defenses:

1. Rate limiting per user/API key:

from app.auth.rate_limiter import TokenBucketLimiter

limiter = TokenBucketLimiter(redis, max_tokens=100, refill_rate=10)

@app.post("/api/classify")
async def classify(request: Request):
    if not await limiter.allow(request.api_key):
        raise HTTPException(429, "Rate limit exceeded")
    return await do_classification(request)

2. Budget caps with alerts:

async def check_budget(org_id: str) -> bool:
    today_spend = await redis.get(f"spend:{org_id}:{today()}")
    daily_limit = await get_org_limit(org_id)

    if float(today_spend or 0) >= daily_limit:
        await send_alert(org_id, "Daily LLM budget exhausted")
        return False
    return True

3. Circuit breaker on provider errors:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def call_llm_with_circuit_breaker(text: str):
    try:
        return await openai.chat.completions.create(...)
    except openai.RateLimitError:
        # Fall back to cached result or default classification
        return get_fallback_classification(text)

This isn't optional at scale. Budget controls are infrastructure, not a nice-to-have.

The Compound Effect

These anti-patterns don't exist in isolation. A typical struggling system has three or four of them simultaneously:

1. LLM call in the handler → high latency
2. No caching → redundant calls amplify the cost
3. No budget cap → costs grow unbounded
4. Single-threaded inference → even the fallback path is slow

Fixing any one of them helps. Fixing all of them transforms your system from fragile to production-grade.

Checklist: Are You Ready for 100K Requests/Day?

| Check | Question | If No | |-------|----------|-------| | ☐ | Are all LLM calls off the hot path or under 100ms? | Fix Anti-Pattern #1 or #2 | | ☐ | Do you cache classification results for identical inputs? | Fix Anti-Pattern #3 | | ☐ | Can your inference layer handle 100 concurrent requests? | Fix Anti-Pattern #4 | | ☐ | Do you have per-user rate limits and daily budget caps? | Fix Anti-Pattern #5 | | ☐ | Do you have alerts for spend anomalies? | Add monitoring today |

FAQ

Q: At what scale do these anti-patterns actually cause problems? Anti-patterns #1 and #2 (LLM in the hot path) start hurting at around 10K requests/day — that's when latency variability becomes noticeable. Anti-pattern #5 (no budget cap) can bite you at any scale if you get hit by a bot. Caching (#3) and concurrency (#4) become critical around 50K requests/day.

Q: Should I fix all five at once? Start with caching (#3) and budget caps (#5) — they're the cheapest to implement and protect you immediately. Then tackle the hot-path LLM call (#1 or #2), which requires more architecture work but gives the biggest performance improvement.

Q: How do I know if I've fixed the problems? Set up three dashboards: (1) P50/P95/P99 latency by endpoint, (2) daily LLM spend by model and endpoint, (3) cache hit rate. If your P95 is under 200ms, your daily spend is predictable, and your cache hit rate is above 15%, you're in good shape.

Q: Can I use Sparkient for all five fixes? Sparkient directly addresses anti-patterns #1 and #2 by replacing the LLM call with a sub-100ms compiled classifier. It also includes built-in credit-based billing with caps (addressing #5). For caching (#3) and inference concurrency (#4), those are infrastructure concerns you'll want to handle at your layer regardless of what classification approach you use.

Build for the scale you're growing into, not the scale you're at. Start with the free tier — 5,000 credits, no credit card required.