From Prototype to Production: What Breaks When You Scale AI Features

TL;DR

AI features can expose different cost, latency, rate-limit, and reliability constraints as traffic changes. Bounded classification calls are candidates for a compiled model, but there is no universal traffic threshold. Sparkient's four controlled synthetic domains report 0.886–0.951 macro F1 and 33–42ms batch-average time per item.

The traffic, provider, price, cache, and incident figures below are an illustrative scaling scenario, not a customer case study or current provider quote.

The Prototype That Works

You built something great. A content moderation system, a ticket triage tool, a lead scoring API. It uses GPT-4o, and it works beautifully:

async def classify_content(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Classify as: approve, review, or reject."},
            {"role": "user", "content": text}
        ],
        temperature=0
    )
    return response.choices[0].message.content.strip()

At 100 requests per day during testing, this is perfect. Latency? Who cares — it's a demo. Cost? Under $1/day. Reliability? The API has been up every time you've checked.

Then you launch.

Stage 1: 10K Requests/Day — Latency Creep

What breaks

In this illustrative scenario, the live-model classification owns most of the delay between "submit" and "posted." Replace the table below with production traces from the exact endpoint.

Your frontend team adds loading spinners. Your product manager asks "why is this so slow?"

The numbers

| Metric | At 100 req/day | At 10K req/day | |--------|---------------|----------------| | P50 latency | 620ms | 650ms | | P95 latency | 1,100ms | 1,400ms | | P99 latency | 1,800ms | 2,500ms | | Daily cost | $0.50 | $50 | | User complaints | 0 | "it feels slow" |

The P50 barely changes, but the P95 and P99 get worse. More concurrent requests mean more queuing at the LLM provider. Your tail latency is now 2.5 seconds — long enough for users to abandon actions.

Typical fix at this stage

Most teams add caching and switch to a faster model:

# Switch to GPT-4o-mini and add response caching
import hashlib, redis

cache = redis.Redis()

async def classify_content(text: str) -> str:
    key = f"cls:{hashlib.sha256(text.encode()).hexdigest()}"
    if cached := cache.get(key):
        return cached.decode()

    response = await openai.chat.completions.create(
        model="gpt-4o-mini",  # Faster, cheaper
        messages=[...],
        temperature=0
    )
    result = response.choices[0].message.content.strip()
    cache.setex(key, 3600, result)
    return result

In the scenario, the smaller model improves the measured profile and caching removes exact duplicates. Unique inputs still take the live-model path; verify the result with current provider pricing and real traffic.

Stage 2: 100K Requests/Day — Cost Explosion

What breaks

The caching helps with duplicates, but most of your inputs are unique. Your cache hit rate is 8%. That means 92% of your 100K daily requests still hit the LLM.

Your finance team flags the invoice:

| Metric | At 10K req/day | At 100K req/day | |--------|---------------|----------------| | Daily LLM cost | $50 | $450 | | Monthly LLM cost | $1,500 | $13,500 | | Cache hit rate | 12% | 8% | | Effective cost per decision | [illustrative scenario] | [illustrative scenario] |

The per-decision cost hasn't changed. But volume × cost = a bill that's bigger than your compute infrastructure.

The conversation you have

Engineering: "The feature is working as designed." Finance: "It's costing $13K/month. That's more than our entire AWS bill." Product: "We can't remove it — users love it."

Nobody's wrong. The feature works, it costs what it costs, and users value it. But the unit economics don't scale.

Typical fix at this stage

Teams explore three options:

1. Fine-tune a cheaper model — Reduces per-call cost but still linear pricing
2. Batch processing — Classify in bulk during off-hours (only works for non-real-time needs)
3. Start evaluating alternatives — This is usually when teams discover that classification doesn't require an LLM

Stage 3: 500K Requests/Day — Rate Limits and Reliability

What breaks

You hit the LLM provider's rate limits. OpenAI's Tier 3 allows 5,000 requests per minute for GPT-4o-mini. At 500K requests/day, you need ~350 requests/minute sustained — well within limits at average load. But traffic is bursty. Your peak hour does 3x average, hitting 1,050 requests/minute. Close to the limit, with spikes going over.

Then the LLM provider has an outage. It happens — OpenAI has had multiple multi-hour outages. When your classification service is a thin wrapper around OpenAI, an OpenAI outage is your outage.

| Metric | At 100K req/day | At 500K req/day | |--------|----------------|----------------| | Monthly LLM cost | $13,500 | $67,500 | | Rate limit errors/day | 0 | 50-200 | | Provider outage impact | "Slow feature" | "Site is down" | | On-call pages/month | 0 | 3-5 |

What you learn

At this scale, three truths become clear:

1. Per-token pricing doesn't have a volume ceiling. 10x traffic = 10x cost.
2. Third-party dependencies in the hot path are a reliability risk. Every external call is a potential outage.
3. Classification is not generation. You're paying for the full power of a language model to pick from three options.

The Compile, Don't Call Approach

Classification workloads have a specific property that makes them different from generative AI: the output space is bounded. You're choosing from approve/review/reject or spam/not_spam or urgent/normal/low. The LLM isn't creating anything new — it's making a judgment call.

This means you can use the LLM as a teacher instead of a worker:

1. Offline: The LLM generates synthetic training examples covering edge cases
2. Offline: A compiled classifier trains on those examples
3. Offline: The model exports to ONNX format
4. Production: The ONNX model handles the normal path; four controlled synthetic runs reported batch-average time per item below 100ms, while each deployment needs its own measurements; cloud deployments may enable LLM escalation

import httpx

# The LLM taught the decision. The compiled model makes it.
async def classify_content(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text}
            }
        )
        return response.json()

For edge-critical deployments where even the API call is too much, you can run the model locally:

from sparkient_edge import EdgePredictor

# No cloud dependency. Benchmark on the target hardware.
predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this product!"})
# Inspect result.decision, result.confidence, and result.stage

The Scale Comparison

Compare the approaches using the real request distribution rather than a universal volume threshold:

| Metric | Live LLM call | Sparkient cloud | Sparkient edge | |--------|---------------|-----------------|----------------| | Latency | Model, prompt, and provider dependent | 33–42ms batch-average time per item in four controlled synthetic runs; escalation is slower | Local hardware dependent | | Billing | Provider token usage | Monthly plan credits plus top-ups | Subscription plus your local infrastructure | | Runtime dependency | LLM provider | Sparkient API; optional LLM escalation | No cloud or LLM dependency | | Quality evidence | Evaluate the actual prompt | Evaluate a trained model on the same held-out cases | Same exported model; verify local runtime |

The compiled option wins only if the measured quality, latency, credit usage, and maintenance profile beat the current path for that project.

When to Make the Switch

Don't over-engineer prematurely, but don't wait until you're on fire either.

Stay with LLM calls if:

• Current volume and latency make the simpler live call the better trade-off
• Your classification needs change weekly (new categories, shifting criteria)
• You're still exploring what the feature should do

Switch to a compiled model if:

• Repeated calls create a measured latency, cost, reliability, or privacy constraint
• Your classification categories are stable
• Latency matters (real-time UX, hot path API)
• You need reliability independent of a third-party API

Use edge deployment if:

• Local measurement shows the cloud round trip misses the latency budget
• You can't tolerate any cloud dependency
• You're running on-device or in environments with unreliable connectivity

The Migration Path

You don't have to rewrite everything at once. Here's a progressive migration:

Week 1: Add timing and cost logging to all LLM calls. Get the baseline numbers.

Week 2: Identify classification workloads (bounded output space). These are your migration candidates.

Week 3: Set up a compiled classifier for your highest-volume classification task. Run it in shadow mode — call both the LLM and the classifier, log both results, compare.

Week 4: Once accuracy is validated, cut over. Keep the LLM as a fallback for low-confidence results.

This is the structure of Sparkient's three-stage pipeline: CEL rules handle deterministic logic, the compiled classifier targets under 100ms, and optional LLM escalation handles configured low-confidence cloud cases. The stage mix and escalation latency must be measured on representative traffic.

FAQ

Q: How long does it take to go from "I have an LLM call" to "I have a compiled classifier"? Sparkient automates optional example generation, model training, evaluation, and ONNX export. Training time depends on the data and available compute, while the status response reports the current attempt, stage, and heartbeat. Example review, integration, and production evaluation add project-dependent time. Each training run uses 2,000 credits.

Q: What if my classification categories change? Retrain the model. Add the new categories to your decision type definition, trigger a new training run, and deploy the updated model. If your categories change daily, a compiled classifier isn't the right fit — stick with an LLM until your taxonomy stabilizes.

Q: Do I lose accuracy compared to GPT-4o? Across four public domains, compiled models achieve 0.886–0.951 macro F1 and 91–96% accuracy. The benchmarks compare against traditional ML baselines, not GPT-4o. Evaluate the live prompt, classifier, and any optional escalation path separately on the same held-out cases.

Q: What about fine-tuning GPT-4o-mini instead? Fine-tuning is a solid middle ground — you get better accuracy from a cheaper model. But you're still paying per-token and still dependent on the provider's API in production. Fine-tuning makes sense if you need the LLM's flexibility. A compiled classifier makes sense if you need speed, cost predictability, and independence from a third-party API.

Build AI features that scale with your users, not your bill. Start with the free tier — 5,000 credits, no credit card required.