5 AI Architecture Anti-Patterns to Load-Test Before Scale

TL;DR

Five patterns deserve explicit load testing before traffic grows: live-model calls in request handlers, blocking classification in hot paths, no deduplication for identical inputs, untuned inference concurrency, and unbounded model usage without budget caps. The failure point depends on the workload and infrastructure, so measure it rather than relying on a request-count threshold.

Introduction

Every architecture decision is a bet on scale. Some bets pay off. Others work at prototype traffic and fail once concurrency, tail latency, or spend crosses the system's actual limit.

These five anti-patterns share a common trait: they're all reasonable choices at prototype scale. They're simple, they work, and they ship fast. The problem isn't that you used them — it's that you didn't replace them before traffic caught up.

Here's how to spot each one and what to replace it with.

The timings and request counts in the examples below are illustrative. Replace them with traces, provider usage, and concurrency limits from the system being reviewed.

Anti-Pattern #1: LLM Call in the Request Handler

What it looks like

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    # Validation: 1ms
    validated = validate(request)

    # Database write: 5ms
    item = await db.save(validated)

    # Live-model classification: record actual p50/p95/p99
    classification = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": f"Classify: {validated.text}"}]
    )

    item.category = parse_response(classification)
    await db.update(item)
    return item

Why it breaks

At sustained traffic, long model calls increase the number of requests in flight. Once concurrency, connection, or provider limits are reached, queues grow; timeouts and indiscriminate retries can amplify the problem.

The math is brutal: 100 concurrent connections × 1 second average = 100 in-flight requests at steady state. Most web servers default to 100-500 max connections. You're at the limit before you've even started scaling.

What to do instead

Option A: Move it off the hot path. Queue the classification for async processing and return immediately:

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    validated = validate(request)
    item = await db.save(validated)
    await queue.send({"item_id": item.id, "text": validated.text})
    return {"id": item.id, "status": "processing"}  # 6ms total

Option B: Replace with a compiled model. If you need the classification in the response, swap the LLM call for a sub-100ms classifier:

@app.post("/api/submit")
async def submit_content(request: ContentRequest):
    validated = validate(request)

    # Compiled classifier: inspect the returned latency and stage
    decision = await sparkient.decide("content-moderation", {"text": validated.text})

    item = await db.save(validated, category=decision["decision"])
    return item

Option C: Use a faster live-model provider. Benchmark the current model and prompt end to end; a hosted model can be the simplest answer when it clears the latency, cost, and reliability budget.

Anti-Pattern #2: Synchronous Classification in a Hot Path

What it looks like

# Every message goes through this before being displayed
async def process_message(message: Message) -> Message:
    # Must classify before showing to other users
    safety = await classify_safety(message.text)  # Blocks rendering

    if safety == "reject":
        raise ContentPolicyViolation()

    message.safety_score = safety
    return message  # User sees message only after classification completes

Why it breaks

This is a variant of Anti-Pattern #1, but worse because it's in a user-facing hot path. The user is staring at a loading spinner while the classification runs. At 100K messages per day:

• Every message inherits the moderation path's tail latency
• Variance becomes visible as inconsistent send times
• Timeouts and retries can compound under load

What to do instead

The fix depends on your tolerance for false negatives:

Optimistic rendering: Show the message immediately, classify in the background, and remove it if it's rejected:

async def process_message(message: Message) -> Message:
    await broadcast(message)  # Show immediately
    asyncio.create_task(classify_and_moderate(message))  # Background
    return message

Measured synchronous check: Use a classifier only when its end-to-end tail latency clears the product's interaction budget:

async def process_message(message: Message) -> Message:
    # Compiled path — measure from the real client
    decision = await classify_fast(message.text)

    if decision["decision"] == "reject" and decision["confidence"] > 0.9:
        raise ContentPolicyViolation()

    return message

In Sparkient's controlled synthetic content-moderation run, the compiled model reported 0.900 macro F1, 91.5% accuracy, and 41ms batch-average time per item. That supports evaluation in a runtime path; it does not establish per-request p95, production UX, or policy quality for a new application.

Anti-Pattern #3: No Caching for Identical Inputs

What it looks like

async def classify(text: str) -> str:
    # Called for every request, even duplicates
    return await llm.classify(text)

No cache layer. No deduplication. The same spam message gets classified 500 times at the same cost and latency.

Why it breaks

In content moderation and spam detection, duplicate inputs are common. A viral spam campaign might generate 10,000 identical messages. Without caching, you're paying for 10,000 LLM calls to get the same answer 10,000 times.

The waste equals duplicate calls multiplied by their measured token usage and current provider pricing. Record the real duplicate rate before estimating savings.

What to do instead

Add a cache layer with content hashing:

import hashlib
from functools import lru_cache

# In-memory LRU for hot duplicates
@lru_cache(maxsize=10_000)
def classify_cached_sync(text_hash: str, text: str) -> str:
    return classify_sync(text)

# Redis for distributed caching
async def classify_with_cache(text: str) -> str:
    text_hash = hashlib.sha256(text.encode()).hexdigest()

    # Check Redis first
    cached = await redis.get(f"classify:{text_hash}")
    if cached:
        return cached

    # Classify and cache for 1 hour
    result = await classify(text)
    await redis.setex(f"classify:{text_hash}", 3600, result)
    return result

This is the cheapest fix on this list. Implement it regardless of what else you do.

Important caveat: Exact-match caching does not help unique content. Measure hit rate, freshness requirements, and the cost of a stale classification before relying on it.

Anti-Pattern #4: Single-Threaded Model Inference

What it looks like

# Global model, no concurrency control
model = load_onnx_model("classifier.onnx")

async def predict(text: str) -> str:
    # Every request shares one inference session
    # ONNX Runtime is thread-safe, but this creates contention
    features = extract_features(text)
    return model.run(None, {"input": features})[0]

Why it breaks

Thread safety does not guarantee optimal throughput or tail latency. Under concurrent load:

• CPU inference queues up, creating latency spikes
• A single model instance becomes the bottleneck
• Tail latency can rise as work queues behind constrained compute

What to do instead

Use multiple inference sessions:

import onnxruntime as ort
from concurrent.futures import ThreadPoolExecutor

# Create a pool of inference sessions
session_pool = [
    ort.InferenceSession("classifier.onnx", providers=["CPUExecutionProvider"])
    for _ in range(4)  # Match your CPU core count
]

executor = ThreadPoolExecutor(max_workers=4)

async def predict(text: str) -> str:
    session = session_pool[hash(text) % len(session_pool)]
    features = extract_features(text)
    loop = asyncio.get_event_loop()
    return await loop.run_in_executor(
        executor,
        lambda: session.run(None, {"input": features})[0]
    )

Enable intra-op parallelism:

session_options = ort.SessionOptions()
session_options.intra_op_num_threads = 2     # Threads per inference
session_options.inter_op_num_threads = 4     # Parallel operations
session_options.execution_mode = ort.ExecutionMode.ORT_PARALLEL

session = ort.InferenceSession("classifier.onnx", session_options)

This fix is mechanical — it doesn't require changing your model or your architecture. Just add concurrency control and tune thread counts to your hardware.

Anti-Pattern #5: Unbounded LLM Costs Without Budget Caps

What it looks like

# No rate limiting, no cost tracking, no budget cap
async def handle_request(request):
    result = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": request.text}]
    )
    return result

There's no check on how many calls you're making or how much you're spending. The OpenAI API key has no usage limit. Your monitoring dashboard shows request counts but not dollar amounts.

Why it breaks

A traffic spike—whether organic, abusive, or accidental—can turn token-metered calls into an unplanned bill. Calculate the exposure from the endpoint's actual token distribution, provider prices, rate limits, and maximum accepted traffic.

Without budget caps, you won't know until the invoice arrives.

What to do instead

Layer three defenses:

1. Rate limiting per user/API key:

from app.auth.rate_limiter import TokenBucketLimiter

limiter = TokenBucketLimiter(redis, max_tokens=100, refill_rate=10)

@app.post("/api/classify")
async def classify(request: Request):
    if not await limiter.allow(request.api_key):
        raise HTTPException(429, "Rate limit exceeded")
    return await do_classification(request)

2. Budget caps with alerts:

async def check_budget(org_id: str) -> bool:
    today_spend = await redis.get(f"spend:{org_id}:{today()}")
    daily_limit = await get_org_limit(org_id)

    if float(today_spend or 0) >= daily_limit:
        await send_alert(org_id, "Daily LLM budget exhausted")
        return False
    return True

3. Circuit breaker on provider errors:

from tenacity import retry, stop_after_attempt, wait_exponential

@retry(stop=stop_after_attempt(3), wait=wait_exponential(min=1, max=10))
async def call_llm_with_circuit_breaker(text: str):
    try:
        return await openai.chat.completions.create(...)
    except openai.RateLimitError:
        # Fall back to cached result or default classification
        return get_fallback_classification(text)

This isn't optional at scale. Budget controls are infrastructure, not a nice-to-have.

The Compound Effect

These anti-patterns don't exist in isolation. A typical struggling system has three or four of them simultaneously:

1. LLM call in the handler → high latency
2. No caching → redundant calls amplify the cost
3. No budget cap → costs grow unbounded
4. Single-threaded inference → even the fallback path is slow

Fixing any one of them helps. Fixing all of them transforms your system from fragile to production-grade.

Checklist: Are You Ready for 100K Requests/Day?

| Check | Question | If No | |-------|----------|-------| | ☐ | Do all synchronous model calls clear the endpoint's measured latency and reliability budget? | Fix Anti-Pattern #1 or #2 | | ☐ | Do you cache classification results for identical inputs? | Fix Anti-Pattern #3 | | ☐ | Can your inference layer handle 100 concurrent requests? | Fix Anti-Pattern #4 | | ☐ | Do you have per-user rate limits and daily budget caps? | Fix Anti-Pattern #5 | | ☐ | Do you have alerts for spend anomalies? | Add monitoring today |

FAQ

Q: At what scale do these anti-patterns actually cause problems? There is no universal daily-volume threshold. A low-volume endpoint can still fail its latency objective, while a high-volume endpoint can be safe with sufficient concurrency and budgets. Load-test the full path and monitor queue depth, p95/p99, errors, retries, provider limits, and spend.

Q: Should I fix all five at once? Prioritise from evidence. Budget caps and rate limits bound financial and reliability exposure; tracing identifies the slow path; caching helps only when hit rate and freshness support it; changing the model path requires a quality comparison.

Q: How do I know if I've fixed the problems? Set up three dashboards: (1) p50/p95/p99 latency by endpoint and stage, (2) daily model usage and spend by endpoint, and (3) cache hit rate and staleness. Judge them against the product's own service objective and budget rather than generic thresholds.

Q: Can I use Sparkient for all five fixes? Sparkient directly addresses anti-patterns #1 and #2 by replacing the LLM call with a sub-100ms compiled classifier. It also includes built-in credit-based billing with caps (addressing #5). For caching (#3) and inference concurrency (#4), those are infrastructure concerns you'll want to handle at your layer regardless of what classification approach you use.

Build for the scale you're growing into, not the scale you're at. Start with the free tier — 5,000 credits, no credit card required.