My Rules Engine Can't Handle Edge Cases but LLMs Are Too Slow

TL;DR

Rules engines give you sub-millisecond decisions but only 45-60% accuracy on nuanced content. LLMs give you ~92% accuracy but take 623ms+. The fix is a three-stage pipeline: CEL rules (<1ms) handle clear-cut cases, a compiled ONNX classifier (<100ms) handles the middle ground, and an LLM escalation (150ms+) handles only the genuinely ambiguous cases. Result: 91.5% F1 accuracy at 41ms P95 — rules-like speed with LLM-like accuracy.

The Gap You're Stuck In

You have a rules engine. It's fast. It handles the obvious cases:

# Your current rules
def moderate_content(text: str) -> str:
    text_lower = text.lower()

    # Obvious rejects
    if any(word in text_lower for word in BLOCKED_WORDS):
        return "reject"

    # Known-good patterns
    if len(text) < 20 and not any(c in text for c in "!@#$%"):
        return "approve"

    # Everything else... ???
    return "review"  # 40% of traffic ends up here

The problem: 40% of your traffic falls into the "I don't know" bucket. Your review queue is drowning.

So you tried adding an LLM:

async def moderate_content_v2(text: str) -> str:
    # Try rules first
    rule_result = apply_rules(text)
    if rule_result != "review":
        return rule_result

    # Fall back to LLM for uncertain cases
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Classify as approve, review, or reject."},
            {"role": "user", "content": text}
        ]
    )
    return response.choices[0].message.content.strip()

This helps with accuracy, but now 40% of your requests take 600-1500ms. Your users notice. Your P95 latency is ruined by the LLM calls.

You're stuck between two bad options:

• Rules only: Fast (<1ms) but catches maybe 60% of cases correctly
• Rules + LLM fallback: Accurate (~92%) but the LLM calls destroy your latency

The Accuracy Problem with Rules

Rules fail on edge cases because language is ambiguous. Here's what a rules engine gets wrong:

| Input | Expected | Rules Output | Why Rules Fail | |-------|----------|-------------|----------------| | "This product is absolutely AMAZING!!!" | approve | review | Caps + exclamation = false positive | | "I want to kill it in this game" | approve | reject | "kill" triggers blocked word list | | "Check my profile for more 😉" | reject (spam) | approve | No blocked words, short enough | | "Not gonna lie, this is trash" | approve (opinion) | reject | "trash" triggers blocked word | | "DM me for exclusive content" | review | approve | Solicitation without blocked words |

Rules are binary. They can't weigh context, understand sarcasm, or handle the infinite variety of natural language. You could add more rules, but each new rule creates new edge cases. After 200 rules, your engine becomes unmaintainable — and still only catches 60% of cases correctly.

The Latency Problem with LLMs

On the other end, LLMs understand context beautifully but are slow:

Rules-only path:
  ├── apply_rules()     → 0.3ms
  └── total             → 0.3ms ✓

LLM fallback path:
  ├── apply_rules()     → 0.3ms
  ├── openai.create()   → 623ms (P50)
  │                     → 1,100ms (P95)
  │                     → 2,300ms (P99)
  └── total             → 623-2,300ms ✗

At 40% fallback rate, your weighted average latency is:

• 60% × 0.3ms + 40% × 623ms = 249ms average
• P95: 1,100ms (from the LLM path)

That 249ms average looks tolerable on a dashboard, but your P95 is 1.1 seconds. Users on the LLM path have a noticeably worse experience.

The Three-Stage Pipeline

The solution is to fill the gap between rules and LLMs with a compiled classifier — a lightweight model that's fast enough for the hot path but smart enough to handle edge cases.

Stage 1: CEL Rules (<1ms)
  ├── Hard business logic: rate limits, blocklists, format validation
  ├── Handles: ~20% of requests (clear approvals and rejections)
  └── Accuracy on handled cases: ~99%

Stage 2: ONNX Classifier (<100ms)
  ├── Compiled model: text encoder + classifier
  ├── Handles: ~75% of requests (the middle ground)
  └── Accuracy on handled cases: 89-95% F1

Stage 3: LLM Escalation (150ms+)
  ├── Gemini fallback: only called when classifier confidence is low
  ├── Handles: ~5% of requests (genuinely ambiguous)
  └── Accuracy on handled cases: ~93%

Here's the key insight: the classifier doesn't need to be perfect. It just needs to be confident. When it's confident (confidence > 0.7), it returns the decision in under 100ms. When it's uncertain, it escalates to the LLM — but this only happens for ~5% of requests, not 40%.

The Comparison

Here's what each approach actually delivers in production:

| Approach | Accuracy (F1) | P50 Latency | P95 Latency | Cost (50K/day) | |----------|--------------|-------------|-------------|----------------| | Rules only | ~45-60% | <1ms | <1ms | ~$0 (compute only) | | Rules + LLM fallback (40%) | ~88% | 249ms | 1,100ms | ~$6,000/mo | | Compiled classifier only | ~91.5% | 33ms | 41ms | $199/mo | | Three-stage pipeline | ~92% | 33ms | 42ms | $199/mo | | LLM only (every request) | ~93% | 623ms | 1,100ms | ~$15,000/mo |

The three-stage pipeline achieves 92% accuracy — nearly matching the LLM — at 42ms P95 latency. The rules handle the easy cases. The classifier handles the middle ground. The LLM only sees the genuinely hard cases.

How the Classifier Handles Edge Cases

Let's revisit those examples that stumped the rules engine:

| Input | Rules | Classifier | Correct? | |-------|-------|-----------|----------| | "This product is absolutely AMAZING!!!" | review ✗ | approve (0.91) ✓ | Yes | | "I want to kill it in this game" | reject ✗ | approve (0.87) ✓ | Yes | | "Check my profile for more 😉" | approve ✗ | reject (0.83) ✓ | Yes | | "Not gonna lie, this is trash" | reject ✗ | approve (0.78) ✓ | Yes | | "DM me for exclusive content" | approve ✗ | review (0.64) → escalate | LLM decides |

The classifier learned from thousands of synthetic training examples generated by an LLM. It understands that "kill it" in a gaming context is positive, that excessive caps and exclamation marks are a stylistic choice, not a threat, and that "check my profile" is a solicitation pattern.

The last example — "DM me for exclusive content" — is genuinely ambiguous. It could be a legitimate creator or a spammer. The classifier's confidence is 0.64, below the threshold, so it escalates to the LLM for a nuanced decision.

Implementation

Setting up the three-stage pipeline

With Sparkient, the three stages are built into the /decide endpoint. You define rules, train a classifier, and the pipeline handles routing automatically:

import httpx

async def moderate_content(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": text, "user_id": "user_123"}
            }
        )
        return response.json()

# Result includes which stage handled it:
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}
# {"decision": "reject", "confidence": 0.99, "latency_ms": 0.4, "stage": "rules"}
# {"decision": "review", "confidence": 0.88, "latency_ms": 210, "stage": "escalation"}

The stage field tells you exactly how the decision was made:

• "rules" — matched a CEL rule, sub-millisecond
• "classifier" — ONNX model was confident, sub-100ms
• "escalation" — LLM was called because confidence was low

Writing CEL rules for the hard business logic

CEL (Common Expression Language) rules handle absolute constraints that should never be overridden by a model:

# These rules run before the classifier, in under 1ms
rules = [
    # Always reject: known banned patterns
    {
        "expression": 'ctx.text.contains("buy followers") || ctx.text.contains("get rich quick")',
        "outcome": "reject",
        "priority": 1
    },
    # Always approve: verified users with good history
    {
        "expression": "ctx.user_trust_score > 0.95 && ctx.text.size() < 500",
        "outcome": "approve",
        "priority": 2
    },
    # Always review: new accounts with links
    {
        "expression": 'ctx.account_age_days < 7 && ctx.text.contains("http")',
        "outcome": "review",
        "priority": 3
    }
]

Rules should encode business policy, not try to understand language. "New accounts with links always go to review" is a business rule. "Determine if this text is spam" is a classification task — let the model handle it.

Building it yourself

If you want full control, here's the pipeline architecture:

import onnxruntime as ort
from transformers import AutoTokenizer
import numpy as np

class ThreeStageClassifier:
    def __init__(self):
        self.tokenizer = AutoTokenizer.from_pretrained("your-text-encoder")
        self.session = ort.InferenceSession("classifier.onnx")
        self.rules = load_rules()
        self.labels = ["approve", "review", "reject"]
        self.confidence_threshold = 0.7

    def decide(self, text: str, context: dict) -> dict:
        # Stage 1: Rules
        rule_result = self.check_rules(text, context)
        if rule_result:
            return {"decision": rule_result, "stage": "rules", "confidence": 1.0}

        # Stage 2: Classifier
        inputs = self.tokenizer(text, return_tensors="np",
                                truncation=True, max_length=512)
        outputs = self.session.run(None, dict(inputs))
        probs = softmax(outputs[0][0])
        confidence = float(np.max(probs))
        prediction = self.labels[np.argmax(probs)]

        if confidence >= self.confidence_threshold:
            return {"decision": prediction, "stage": "classifier",
                    "confidence": confidence}

        # Stage 3: LLM Escalation
        return self.escalate_to_llm(text, context)

This is the core architecture. The production version adds retry logic, caching, feature engineering, and monitoring — but the three-stage routing is the key pattern.

Why 5% Escalation Beats 40% Fallback

The difference between your current setup (40% LLM fallback) and the three-stage pipeline (5% LLM escalation) is dramatic:

| Metric | 40% LLM Fallback | 5% LLM Escalation | |--------|-------------------|-------------------| | Weighted avg latency | 249ms | 35ms | | P95 latency | 1,100ms | 42ms | | Monthly LLM cost (50K/day) | $6,000 | ~$375 | | LLM API dependency | 40% of requests | 5% of requests | | Outage impact | Severe | Minimal |

The LLM still exists in the pipeline — it's just not in the hot path for 95% of requests. When the LLM provider has an outage, 95% of your decisions continue at full speed. The remaining 5% can fall back to the classifier's best guess or go to manual review.

FAQ

Q: How do I know if my rules engine accuracy is actually 45-60%? Run your rules against a labeled test set. Take 1,000 manually classified examples and see what percentage your rules get right. Most teams are surprised by how low the number is — rules tend to over-reject (catching legitimate content) and under-catch (missing subtle violations). The 45-60% range comes from content moderation benchmarks where rules use keyword matching and simple heuristics.

Q: What if I need the LLM escalation to be faster? Use Groq or Cerebras for escalation instead of GPT-4o. Groq delivers 150-300ms for structured responses. This doesn't change the overall picture much — since only ~5% of requests escalate, the P95 is still dominated by the classifier at 41ms. But it improves the experience for those 5% of ambiguous cases.

Q: Can the classifier handle new types of edge cases over time? Yes, through retraining. When you identify new edge cases, add them to your training data (or let the LLM generate synthetic examples for the gap), retrain the model, and deploy the update. Each retraining run uses 2,000 credits. The classifier gets better at handling edge cases that were previously escalated to the LLM, which reduces your escalation rate further.

Q: What confidence threshold should I use for escalation? Start at 0.7. This means the classifier escalates when it's less than 70% sure of its top prediction. At 0.7, you'll see roughly 3-8% escalation rate depending on your domain. Lower it (e.g., 0.5) to reduce escalations at the cost of more incorrect classifier decisions. Raise it (e.g., 0.85) for higher accuracy at the cost of more LLM calls. Monitor your escalation rate and accuracy weekly and adjust.

Get the speed of rules and the accuracy of an LLM. Try Sparkient free — 5,000 credits, no credit card required.