From Prototype to Production: What Breaks When You Scale AI Features

TL;DR

AI features that work perfectly in prototype break at scale in predictable ways: latency creep at 10K requests/day, cost explosion at 100K, rate limits and reliability failures at 500K+. The fix isn't better prompts or bigger infrastructure — it's recognizing that classification workloads don't need an LLM at runtime. Compile the decision once, run it millions of times at under 100ms with 89-95% F1 accuracy.

The Prototype That Works

You built something great. A content moderation system, a ticket triage tool, a lead scoring API. It uses GPT-4o, and it works beautifully:

async def classify_content(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": text}
        ],
        temperature=0
    )
    return response.choices[0].message.content.strip()

At 100 requests per day during testing, this is perfect. Latency? Who cares — it's a demo. Cost? Under $1/day. Reliability? The API has been up every time you've checked.

Then you launch.

Stage 1: 10K Requests/Day — Latency Creep

What breaks

Your users notice lag. Each classification takes 500-1500ms. For a content moderation feature, that's 500-1500ms of dead time between "submit" and "posted." For a ticket triage system, it's a visible delay in the dashboard.

Your frontend team adds loading spinners. Your product manager asks "why is this so slow?"

The numbers

| Metric | At 100 req/day | At 10K req/day | |--------|---------------|----------------| | P50 latency | 620ms | 650ms | | P95 latency | 1,100ms | 1,400ms | | P99 latency | 1,800ms | 2,500ms | | Daily cost | $0.50 | $50 | | User complaints | 0 | "it feels slow" |

The P50 barely changes, but the P95 and P99 get worse. More concurrent requests mean more queuing at the LLM provider. Your tail latency is now 2.5 seconds — long enough for users to abandon actions.

Typical fix at this stage

Most teams add caching and switch to a faster model:

# Switch to GPT-4o-mini and add response caching
import hashlib, redis

cache = redis.Redis()

async def classify_content(text: str) -> str:
    key = f"cls:{hashlib.sha256(text.encode()).hexdigest()}"
    if cached := cache.get(key):
        return cached.decode()

    response = await openai.chat.completions.create(
        model="gpt-4o-mini",  # Faster, cheaper
        messages=[...],
        temperature=0
    )
    result = response.choices[0].message.content.strip()
    cache.setex(key, 3600, result)
    return result

This helps. GPT-4o-mini is faster (400-800ms) and cheaper. Caching eliminates duplicate calls. But you're still making an LLM call for every unique input.

Stage 2: 100K Requests/Day — Cost Explosion

What breaks

The caching helps with duplicates, but most of your inputs are unique. Your cache hit rate is 8%. That means 92% of your 100K daily requests still hit the LLM.

Your finance team flags the invoice:

| Metric | At 10K req/day | At 100K req/day | |--------|---------------|----------------| | Daily LLM cost | $50 | $450 | | Monthly LLM cost | $1,500 | $13,500 | | Cache hit rate | 12% | 8% | | Effective cost per decision | $0.005 | $0.005 |

The per-decision cost hasn't changed. But volume × cost = a bill that's bigger than your compute infrastructure.

The conversation you have

Engineering: "The feature is working as designed." Finance: "It's costing $13K/month. That's more than our entire AWS bill." Product: "We can't remove it — users love it."

Nobody's wrong. The feature works, it costs what it costs, and users value it. But the unit economics don't scale.

Typical fix at this stage

Teams explore three options:

1. Fine-tune a cheaper model — Reduces per-call cost but still linear pricing
2. Batch processing — Classify in bulk during off-hours (only works for non-real-time needs)
3. Start evaluating alternatives — This is usually when teams discover that classification doesn't require an LLM

Stage 3: 500K Requests/Day — Rate Limits and Reliability

What breaks

You hit the LLM provider's rate limits. OpenAI's Tier 3 allows 5,000 requests per minute for GPT-4o-mini. At 500K requests/day, you need ~350 requests/minute sustained — well within limits at average load. But traffic is bursty. Your peak hour does 3x average, hitting 1,050 requests/minute. Close to the limit, with spikes going over.

Then the LLM provider has an outage. It happens — OpenAI has had multiple multi-hour outages. When your classification service is a thin wrapper around OpenAI, an OpenAI outage is your outage.

| Metric | At 100K req/day | At 500K req/day | |--------|----------------|----------------| | Monthly LLM cost | $13,500 | $67,500 | | Rate limit errors/day | 0 | 50-200 | | Provider outage impact | "Slow feature" | "Site is down" | | On-call pages/month | 0 | 3-5 |

What you learn

At this scale, three truths become clear:

1. Per-token pricing doesn't have a volume ceiling. 10x traffic = 10x cost.
2. Third-party dependencies in the hot path are a reliability risk. Every external call is a potential outage.
3. Classification is not generation. You're paying for the full power of a language model to pick from three options.

The Compile, Don't Call Approach

Classification workloads have a specific property that makes them different from generative AI: the output space is bounded. You're choosing from approve/review/reject or spam/not_spam or urgent/normal/low. The LLM isn't creating anything new — it's making a judgment call.

This means you can use the LLM as a teacher instead of a worker:

1. Offline: The LLM generates synthetic training examples covering edge cases
2. Offline: A compiled classifier trains on those examples
3. Offline: The model exports to ONNX format
4. Production: The ONNX model runs classifications at sub-100ms, no LLM calls

import httpx

# The LLM taught the decision. The compiled model makes it.
async def classify_content(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()

For edge-critical deployments where even the API call is too much, you can run the model locally:

from sparkient_edge import EdgePredictor

# No cloud dependency. Sub-10ms inference.
predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this product!"})
# EdgeDecision(decision="approve", confidence=0.94, latency_ms=3.2)

The Scale Comparison

Here's what each approach looks like across the scaling stages:

| Metric | GPT-4o | GPT-4o-mini | Compiled Model | |--------|--------|-------------|----------------| | Latency (P95) | 1,100ms | 600ms | 41ms | | Cost at 10K/day | $1,500/mo | $90/mo | $199/mo | | Cost at 100K/day | $15,000/mo | $900/mo | $499/mo | | Cost at 500K/day | $75,000/mo | $4,500/mo | $999/mo | | Rate limit risk | High | Medium | None | | Provider outage impact | Full | Full | Training only | | Accuracy (F1) | ~93% | ~90% | 89-95% |

At 10K requests/day, GPT-4o-mini is cheapest. At 100K+, the compiled model is both cheaper and faster.

When to Make the Switch

Don't over-engineer prematurely, but don't wait until you're on fire either.

Stay with LLM calls if:

• You're under 5K requests/day
• Your classification needs change weekly (new categories, shifting criteria)
• You're still exploring what the feature should do

Switch to a compiled model if:

• You're above 10K requests/day and costs are a concern
• Your classification categories are stable
• Latency matters (real-time UX, hot path API)
• You need reliability independent of a third-party API

Use edge deployment if:

• You need sub-10ms decisions
• You can't tolerate any cloud dependency
• You're running on-device or in environments with unreliable connectivity

The Migration Path

You don't have to rewrite everything at once. Here's a progressive migration:

Week 1: Add timing and cost logging to all LLM calls. Get the baseline numbers.

Week 2: Identify classification workloads (bounded output space). These are your migration candidates.

Week 3: Set up a compiled classifier for your highest-volume classification task. Run it in shadow mode — call both the LLM and the classifier, log both results, compare.

Week 4: Once accuracy is validated, cut over. Keep the LLM as a fallback for low-confidence results.

This is exactly what the three-stage pipeline does: CEL rules handle hard business logic (<1ms), the compiled classifier handles the vast majority of decisions (<100ms), and the LLM is only called when confidence is below threshold (150ms+, but rarely triggered).

FAQ

Q: How long does it take to go from "I have an LLM call" to "I have a compiled classifier"? With Sparkient, you can define a decision type, generate training data, and have a production classifier in about 2-3 hours. The training pipeline handles synthetic data generation, model training, and ONNX export automatically. Each training run uses 2,000 credits.

Q: What if my classification categories change? Retrain the model. Add the new categories to your decision type definition, trigger a new training run, and deploy the updated model. If your categories change daily, a compiled classifier isn't the right fit — stick with an LLM until your taxonomy stabilizes.

Q: Do I lose accuracy compared to GPT-4o? Across benchmarked domains, compiled models achieve 89-95% F1 scores and 91-96% accuracy. For content moderation specifically, the benchmark shows 91.5% F1. GPT-4o scores slightly higher (~93% F1) on the same tasks, but the gap narrows with domain-specific training data. The three-stage pipeline with LLM escalation for low-confidence results closes the gap further.

Q: What about fine-tuning GPT-4o-mini instead? Fine-tuning is a solid middle ground — you get better accuracy from a cheaper model. But you're still paying per-token and still dependent on the provider's API in production. Fine-tuning makes sense if you need the LLM's flexibility. A compiled classifier makes sense if you need speed, cost predictability, and independence from a third-party API.

Build AI features that scale with your users, not your bill. Start with the free tier — 5,000 credits, no credit card required.