My Rules Engine Can't Handle Edge Cases but LLMs Are Too Slow

TL;DR

Rules are ideal for explicit constraints but can become brittle when meaning and context matter. Sparkient combines CEL rules, a compiled classifier, and optional LLM escalation. Its controlled synthetic content-moderation run reports 0.900 macro F1, 91.5% accuracy, and 41ms batch-average time per item; it does not measure per-request p95, and the full three-stage result depends on the project's stage mix.

The inputs and traffic shares below illustrate the architecture; they are not measured customer results.

The Gap You're Stuck In

You have a rules engine. It's fast. It handles the obvious cases:

# Your current rules
def moderate_content(text: str) -> str:
    text_lower = text.lower()

    # Obvious rejects
    if any(word in text_lower for word in BLOCKED_WORDS):
        return "reject"

    # Known-good patterns
    if len(text) < 20 and not any(c in text for c in "!@#$%"):
        return "approve"

    # Everything else... ???
    return "review"  # 40% of traffic ends up here

The problem: 40% of your traffic falls into the "I don't know" bucket. Your review queue is drowning.

So you tried adding an LLM:

async def moderate_content_v2(text: str) -> str:
    # Try rules first
    rule_result = apply_rules(text)
    if rule_result != "review":
        return rule_result

    # Fall back to LLM for uncertain cases
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Classify as approve, review, or reject."},
            {"role": "user", "content": text}
        ]
    )
    return response.choices[0].message.content.strip()

This may improve quality, but the escalated share now inherits the live model's latency, tokens, and provider dependency. Measure classifier-only, escalated, and combined p95/p99 separately.

The comparison is incomplete until both paths are tested:

• Rules only: Usually fast, but coverage and per-class quality depend on the policy
• Rules + live-model fallback: Potentially broader, but the combined quality, latency, tokens, and failure modes are workload-specific

The Accuracy Problem with Rules

Rules fail on edge cases because language is ambiguous. Here's what a rules engine gets wrong:

| Input | Expected | Rules Output | Why Rules Fail | |-------|----------|-------------|----------------| | "This product is absolutely AMAZING!!!" | approve | review | Caps + exclamation = false positive | | "I want to kill it in this game" | approve | reject | "kill" triggers blocked word list | | "Check my profile for more 😉" | reject (spam) | approve | No blocked words, short enough | | "Not gonna lie, this is trash" | approve (opinion) | reject | "trash" triggers blocked word | | "DM me for exclusive content" | review | approve | Solicitation without blocked words |

Rules are explicit conditions. They do not learn semantic boundaries from examples, and large overlapping rule sets can become hard to reason about. Measure both coverage and correctness on a labelled set.

The Latency Problem with LLMs

On the other end, a live model can add context sensitivity with a different runtime profile:

Rules-only path:
  ├── apply_rules()     → measure
  └── total             → measure

Live-model fallback path:
  ├── apply_rules()     → measure
  ├── model request     → measure p50/p95/p99 and tokens
  └── total             → measure end to end

Weighted averages can hide the slower fallback tail. Record the stage mix and report the combined p95 and p99, not just a mean.

The Three-Stage Pipeline

The solution is to fill the gap between rules and LLMs with a compiled classifier — a lightweight model that's fast enough for the hot path but smart enough to handle edge cases.

Stage 1: CEL Rules (<1ms)
  ├── Hard business logic: rate limits, blocklists, format validation
  ├── Share of requests: measure
  └── Correctness on matched cases: evaluate

Stage 2: ONNX Classifier (<100ms)
  ├── Compiled model: text encoder + classifier
  ├── Share of requests: measure
  └── Quality: evaluate on a representative labelled set

Stage 3: Optional LLM Escalation (model and prompt dependent)
  ├── Gemini fallback: called when enabled and confidence is below the threshold
  ├── Share of traffic: measure on representative production inputs
  └── Quality: evaluate separately from classifier-only results

The classifier does not need to handle every case, but confidence is not a substitute for validation. Tune the threshold against per-class errors. When optional escalation is enabled, low-confidence requests can take the slower LLM stage; measure the actual stage mix.

The Comparison

Here's what each approach actually delivers in production:

| Approach | Runtime profile | Quality evidence | Cost model | |----------|-----------------|------------------|------------| | Rules only | Usually <1ms | Evaluate on labelled cases | Application compute | | Rules + live LLM | Stage-mix dependent | Evaluate the combined path | Provider token usage | | Sparkient classifier | 41ms batch-average time per item in the controlled synthetic content run | 0.900 macro F1 in that run | Plan credits | | Three-stage Sparkient pipeline | Stage-mix dependent | Evaluate separately | Plan credits plus escalation usage |

The public result validates classifier-only quality and latency. A three-stage deployment can route deterministic, learned, and uncertain cases differently, but its combined quality and p95 depend on the project's stage mix.

How the Classifier Handles Edge Cases

Let's revisit those examples that stumped the rules engine:

| Input | Rules | Classifier | Correct? | |-------|-------|-----------|----------| | "This product is absolutely AMAZING!!!" | review ✗ | approve (0.91) ✓ | Yes | | "I want to kill it in this game" | reject ✗ | approve (0.87) ✓ | Yes | | "Check my profile for more 😉" | approve ✗ | reject (0.83) ✓ | Yes | | "Not gonna lie, this is trash" | reject ✗ | approve (0.78) ✓ | Yes | | "DM me for exclusive content" | approve ✗ | review (0.64) → escalate | LLM decides |

Those examples illustrate why a learned model may be worth evaluating; they are not guaranteed predictions. Add difficult cases to a fixed evaluation set and inspect probabilities, per-class errors, and escalation behaviour from the actual trained model.

Implementation

Setting up the three-stage pipeline

With Sparkient, the three stages are built into the /decide endpoint. You define rules, train a classifier, and the pipeline handles routing automatically:

import httpx

async def moderate_content(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type": "content-moderation",
                "input": {"text": text, "user_id": "user_123"}
            }
        )
        return response.json()

# Result includes which stage handled it:
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}
# {"decision": "reject", "confidence": 0.99, "latency_ms": 0.4, "stage": "rules"}
# {"decision": "review", "confidence": 0.88, "latency_ms": 210, "stage": "escalation"}

The stage field tells you exactly how the decision was made:

• "rules" — matched a CEL rule, sub-millisecond
• "classifier" — ONNX model was confident, sub-100ms
• "escalation" — LLM was called because confidence was low

Writing CEL rules for the hard business logic

CEL (Common Expression Language) rules handle absolute constraints that should never be overridden by a model:

# These rules run before the classifier, in under 1ms
rules = [
    # Always reject: known banned patterns
    {
        "expression": 'ctx.text.contains("buy followers") || ctx.text.contains("get rich quick")',
        "outcome": "reject",
        "priority": 1
    },
    # Always approve: verified users with good history
    {
        "expression": "ctx.user_trust_score > 0.95 && ctx.text.size() < 500",
        "outcome": "approve",
        "priority": 2
    },
    # Always review: new accounts with links
    {
        "expression": 'ctx.account_age_days < 7 && ctx.text.contains("http")',
        "outcome": "review",
        "priority": 3
    }
]

Rules should encode business policy, not try to understand language. "New accounts with links always go to review" is a business rule. "Determine if this text is spam" is a classification task — let the model handle it.

Building it yourself

If you want full control, here's the pipeline architecture:

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

class ThreeStageClassifier:
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained("your-text-encoder")
        self.session = ort.InferenceSession("classifier.onnx")
        self.rules = load_rules()
        self.labels = ["approve", "review", "reject"]
        self.confidence_threshold = 0.7

    def decide(self, text: str, context: dict) -> dict:
        # Stage 1: Rules
        rule_result = self.check_rules(text, context)
        if rule_result:
            return {"decision": rule_result, "stage": "rules", "confidence": 1.0}

        # Stage 2: Classifier
        inputs = self.tokenizer(text, return_tensors="np",
                                truncation=True, max_length=512)
        outputs = self.session.run(None, dict(inputs))
        probs = softmax(outputs[0][0])
        confidence = float(np.max(probs))
        prediction = self.labels[np.argmax(probs)]

        if confidence >= self.confidence_threshold:
            return {"decision": prediction, "stage": "classifier",
                    "confidence": confidence}

        # Stage 3: LLM Escalation
        return self.escalate_to_llm(text, context)

This is the core architecture. The production version adds retry logic, caching, feature engineering, and monitoring — but the three-stage routing is the key pattern.

Why the Escalation Rate Must Be Measured

The stage mix controls combined quality, latency, credits, and provider exposure. Record it on representative inputs:

| Metric | Record | |--------|--------| | Rule, classifier, escalation, and fallback share | Percentage by stage and class | | Latency | p50/p95/p99 by stage and combined | | Quality | Per-class precision/recall and costly error counts | | Usage | Sparkient credits plus actual escalation tokens | | Failure behaviour | What happens on low credits, timeout, or provider failure |

FAQ

Q: How do I know whether my rules engine is good enough? Run it against a representative labelled set. Report coverage separately from correctness on matched cases, then inspect false approvals and false rejections by class.

Q: What if I need the LLM escalation to be faster? Choose an escalation model based on measured structured-output quality, end-to-end latency, cost, and availability. Even a small escalation rate can affect tail latency, so include escalated requests in the full-path p95 and p99.

Q: Can the classifier handle new types of edge cases over time? Potentially, through retraining. Add reviewed edge cases or generated candidates, retrain, and compare the new model with the fixed evaluation set before deployment. Each run uses 2,000 credits; deploy only if quality improves without unacceptable regressions.

Q: What confidence threshold should I use for escalation? Use 0.7 only as the product default, not as evidence it is safe for a domain. Sweep candidate thresholds on a labelled validation set, weight costly classes separately, then monitor the real stage mix, latency, credits, and reviewed outcomes.

Get the speed of rules and the accuracy of an LLM. Try Sparkient free — 5,000 credits, no credit card required.