ML Inference Latency Tiers: What to Measure in 2026

TL;DR

Inference options have different runtime shapes: local rules are usually the smallest; exported models remove the network but depend on hardware; compiled cloud models add a network path; live LLMs depend on model, prompt, provider, and output length. Four controlled synthetic Sparkient runs report 33–42ms batch-average time per item. Choose from the actual workload, not a universal provider tier.

The Problem

You need your system to make a decision. Maybe it's moderating a comment, scoring a lead, or triaging a support ticket. The question is: how fast does the answer need to be, and how much are you willing to pay for that speed?

In 2026, the inference landscape is fragmented. GPU inference providers are racing to cut LLM latency. ONNX Runtime is making classical ML faster. Edge inference is becoming viable. But there's no single resource that maps out the actual latency tiers, what runs in each one, and when you should pick one over another.

This article provides a framework for comparing the tiers. Provider latency and pricing change, so verify current documentation and measure the full request in your environment.

The Latency Tiers

Here is a practical map of the tiers; only the Sparkient row below uses the four current public Sparkient benchmarks.

| Latency Tier | What Runs | Latency evidence | Accuracy (Classification) | Cost per 1M Decisions | Best For | |---|---|---|---|---|---| | <1ms target | CEL/Rego rules, lookup tables | Measure locally | Evaluate policy coverage | Application compute | Hard business rules, rate limits, blocklists | | Local compiled | Edge ONNX, local classifiers | Hardware dependent | Model dependent | Subscription plus local compute | On-device decisions, air-gapped systems | | <100ms compiled-stage target | Sparkient compiled models (cloud) | 33–42ms batch-average time per item in four controlled synthetic runs | 0.886–0.951 macro F1, 91–96% accuracy on those synthetic runs | Plan credits | Repeated structured decisions | | Hosted LLM inference | Current model providers | Model, prompt, and provider dependent | Task dependent | Current token pricing | Generation and flexible decisions |

Let's break down each tier.

Tier 1: Sub-1ms — Rules and Lookups

What it is: Deterministic rules evaluated against structured input. CEL (Common Expression Language), Rego, or plain if/else logic.

Example:

# CEL rule: block if user is rate-limited
# Evaluates in <0.1ms
"ctx.requests_last_hour > 100 || ctx.account_age_days < 1"

Coverage: Rules work exactly for the cases they encode. A keyword blocklist can catch obvious spam but miss paraphrases such as "dm me for the real deal 💰" or obfuscation such as "check my b1o for l1nks." Measure coverage on the application's own labelled cases rather than assuming a generic rules-only accuracy.

When to use this tier:

• Hard business logic that should never be overridden by ML (age verification, rate limits, geographic restrictions)
• As the first stage of a multi-stage pipeline — catch the obvious cases before hitting a model

When NOT to use this tier:

• Any decision requiring semantic understanding of text
• Anything where the decision space is too large to enumerate

Tier 2: Local Edge — Hardware Dependent

What it is: Classical ML models (gradient-boosted classifiers, tree ensembles) or small neural networks exported to ONNX, running locally without network round-trips.

Example:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Free money! Click here now!!!"})
# Inspect result.decision, result.confidence, and result.stage

Runtime: Measure the complete bundle on the target CPU, including validation, rule evaluation, text processing, inference, process boundaries, and concurrency.

When to use this tier:

• Air-gapped or offline environments
• On-device decisions (mobile, IoT, embedded)
• When you can't tolerate any network dependency
• High-throughput workloads where a benchmark on the target machine confirms the required capacity

When NOT to use this tier:

• You need the model to update without redeploying the edge bundle
• You need audit logging and centralized monitoring

Tier 3: Sub-100ms — Compiled Models (Cloud)

What it is: The same ONNX models from Tier 2, but served via a cloud API with rules, classification, and escalation in a managed pipeline.

This is where Sparkient operates. Four controlled synthetic runs reported batch-average time per item below 100ms; they did not measure per-request p95. An LLM can generate or label examples during training, and cloud deployments can optionally escalate low-confidence decisions at runtime.

Benchmark data (real numbers):

| Domain | F1 Score | Accuracy | Batch-average time per item | |---|---|---|---| | Support ticket triage | 0.951 | 96.2% | 42ms | | Content moderation | 0.900 | 91.5% | 41ms | | Gaming chat | 0.886 | 91.0% | 34ms | | Marketplace listings | 0.938 | 94.3% | 33ms |

These results compare the compiled model with six traditional machine-learning baselines on the same held-out set. They do not establish a universal speed or quality multiplier against every LLM provider.

When to use this tier:

• Repeated classification with a measured quality, latency, cost, reliability, privacy, or offline constraint
• Latency-sensitive hot paths (inline moderation, real-time scoring)
• When you need custom decision categories, not generic labels
• When you want a measured task-specific model without an LLM call on every request

When NOT to use this tier:

• You need open-ended text generation
• Your decision requires multi-step reasoning
• The volume, quality, latency, or operating-cost improvement would not justify integration and retraining

Tier 4: Optimised Live-LLM Serving — Measure the Prompt

What it is: Hardware-optimized LLM inference providers that run models on custom silicon (LPUs, RDUs) or heavily optimized GPU clusters.

Providers such as Groq, Cerebras, and SambaNova optimise live-model inference. Their end-to-end latency changes with model, prompt, structured-output method, region, load, and output length; verify current documentation and run the exact request.

When to use this tier:

• You need the LLM's full reasoning ability (not just classification)
• The measured quality, latency, and current usage price fit the workload
• You need generative output, not just a label
• You're willing to pay the per-token cost for speed

When NOT to use this tier:

• Repeated classification creates a measured cost, latency, privacy, or reliability constraint
• A compiled candidate beats the live path on the project's held-out evaluation and operating profile
• You need sub-100ms response times

Tier 5: General Live-LLM APIs — Workload Dependent

What it is: The mainstream LLM APIs from the major providers, running on standard GPU infrastructure.

Examples include the current fast models from Google, OpenAI, and Anthropic. Do not reuse a generic latency number: benchmark the selected model and region with the actual prompt and output schema.

When to use this tier:

• General-purpose LLM tasks where the measured response distribution fits the interaction budget
• Prototyping and development — easiest to set up
• Low-volume production workloads
• When you need the flexibility to change prompts without retraining

When NOT to use this tier:

• Real-time user-facing decisions (chat moderation, inline scoring)
• Workloads where observed token usage and current pricing create a material constraint
• Anything in a hot path where 500ms+ is unacceptable

Tier 6: Longer or More Complex Live-Model Work

What it is: The most capable models running complex inference.

Larger models, longer contexts, tool use, and longer outputs can move a request into a different latency regime. Model names alone do not define the tier.

When to use this tier:

• Complex multi-step reasoning
• Open-ended generation (writing, coding, analysis)
• Tasks where accuracy is worth any latency cost
• Escalation fallback for lower-tier systems

The Decision Matrix

Here's a practical framework for choosing your tier:

Do you need generative output (not just a label)?
├── Yes → Compare current live models on quality, p95/p99, tokens, and reliability
└── No → Is the decision deterministic (pure business logic)?
    ├── Yes → Use ordinary code or rules
    └── No → Is it repeated, stable, and measurable?
        ├── No → Keep the simpler current path
        └── Yes → Evaluate a compiled cloud or local candidate on the same cases

Implementation: Comparing Tiers in Code

Here's the same content moderation decision across three tiers, so you can see the trade-off in practice:

Tier 1: Rules Only (<1ms)

BLOCKED_TERMS = {"spam", "buy now", "click here"}

def moderate_rules(text: str) -> str:
    lower = text.lower()
    if any(term in lower for term in BLOCKED_TERMS):
        return "reject"
    return "approve"

# Fast, but misses: "dm me for the real deal 💰"
# Misses paraphrases and obfuscation; measure accuracy on a labelled evaluation set

Tier 3: Sparkient Compiled Model (<100ms)

import httpx

async def moderate_compiled(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text}
            }
        )
    return response.json()

# {"decision": "reject", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}
# Published moderation benchmark: 91.5% accuracy, 0.900 macro F1

Tier 5: Direct LLM Call (~600ms)

from openai import OpenAI

client = OpenAI()

def moderate_llm(text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "system",
            "content": "Classify this content as 'approve', 'review', or 'reject'."
        }, {
            "role": "user",
            "content": text
        }],
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

# "reject" — measure latency, tokens, and current provider price

Same decision. Three very different latency and cost profiles.

FAQ

Is Groq fast enough for real-time classification?

Test the current provider with the actual prompt. A hosted LLM may fit if its end-to-end p95 clears the project's budget; a compiled model is a candidate when it does not.

Why not just use rules and skip ML entirely?

Rules are perfect for deterministic logic. The problem is coverage. "Block messages containing 'buy now'" is a rule. "Block messages that are trying to sell something, even when they don't use obvious sales language" requires understanding intent — that's a classification problem. Most production systems use rules as a first pass and ML for everything the rules don't catch.

How do compiled models compare to fine-tuned LLMs on accuracy?

Sparkient's four controlled synthetic compiled-model runs report 0.886–0.951 macro F1 and 33–42ms batch-average time per item. They do not establish a universal gap to a fine-tuned LLM; evaluate both on the same labelled cases. Open-ended generation remains outside Sparkient's classifier scope.

What about serverless GPU inference (Modal, Replicate, Banana)?

Serverless GPU platforms can add cold-start and queueing latency. Measure cold and warm distributions separately, including model load time, region, concurrency, and scale-to-zero behaviour, before placing them on a latency-sensitive path.

Want to see where your workload falls on the latency spectrum? Start with Sparkient's free tier — 5,000 credits, no credit card — and benchmark compiled models against your current inference setup.