The Complete Guide to Compiled Decision Intelligence

TL;DR

Compiled Decision Intelligence (CDI) is a new approach that sits between rules engines and LLM APIs. You use an LLM to teach a decision once, offline, then deploy a compiled model that makes that decision millions of times at sub-100ms latency. Across benchmarked domains, compiled models achieve 89-95% F1 at sub-100ms latency — within a few points of direct LLM accuracy at 15× lower latency and a fraction of the cost.

What Is Compiled Decision Intelligence?

Every software system makes decisions. Should this content be approved? Is this transaction fraudulent? Where should this support ticket go? What priority is this lead?

Today, you have two options:

Rules engines — Fast and predictable, but rigid. A rules engine can check if a transaction exceeds $10,000, but it can't assess whether a user's message is a subtle threat disguised as humour. Rules execute in under 1ms but only achieve 45-60% accuracy on nuanced, text-heavy decisions.

LLM APIs — Smart and flexible, but slow and expensive. GPT-4o can handle nuance, sarcasm, and context brilliantly, but it takes ~623ms per call and costs ~$0.01 each. At 50,000 decisions per day, that's ~$15,000/month and half a second of latency in every user interaction.

Compiled Decision Intelligence is a third option. It bridges the gap:

Use an LLM to teach the decision once. Use a compiled model to make it millions of times.

The metaphor is deliberate. Just as a compiler transforms high-level source code into fast machine code, CDI transforms high-level LLM intelligence into fast inference models. The source code (LLM reasoning) is human-readable and easy to modify. The compiled output (ONNX classifier) is optimised for execution speed.

| Approach | Latency | Accuracy (nuanced text) | Cost at 50K/day | Flexibility | |----------|---------|------------------------|-----------------|-------------| | Rules engine | <1ms | 45-60% | ~$0 | Low — manual updates | | LLM API (GPT-4o) | ~623ms | ~93-95% | ~$15,000/mo | High — prompt changes | | Compiled model | <100ms | 89-95% | $199/mo (Starter) | Medium — retrain to update |

CDI isn't better than LLMs at everything. It's better at one specific thing: making the same type of structured decision, repeatedly, at scale, in real time.

The Three-Stage Pipeline

A compiled decision system doesn't rely on a single model. It uses a three-stage pipeline, where each stage handles what it's best at:

Stage 1: CEL Rules (<1ms)

The first stage evaluates deterministic business logic using CEL (Common Expression Language) expressions. These are hard constraints that always apply, regardless of what any model thinks.

// Block transactions over the daily limit
ctx.amount > ctx.daily_limit ? "block" : null

// Auto-approve known-good senders
ctx.sender_id in ctx.whitelist ? "approve" : null

// Force review for new accounts
ctx.account_age_days < 3 ? "review" : null

Rules execute in under 1ms. If a rule matches, the decision is returned immediately — the classifier never runs. This handles the easy cases cheaply and guarantees that hard business constraints are never violated.

When rules fire: Roughly 15-30% of decisions in a typical deployment are handled entirely by rules. These are the cases that don't need intelligence — they need enforcement.

Stage 2: ONNX Classifier (<100ms)

When no rule matches, the input goes to the compiled classifier. This is where the LLM's intelligence lives, encoded into a fast inference model.

The architecture:

1. A text encoder creates high-dimensional semantic embeddings from text fields
2. A gradient-boosted classifier classifies based on embeddings plus structured features (numbers, categories, booleans)
3. The model outputs a decision and a confidence score

{
    "decision": "approve",
    "confidence": 0.94,
    "latency_ms": 38,
    "stage": "classifier"
}

The classifier handles the bulk of decisions — typically 70-85% — with sub-100ms latency. It captures nuance that rules miss (sarcasm, context, intent) while running fast enough for real-time use.

Benchmark performance:

| Domain | F1 Score | Accuracy | p95 Latency | |--------|----------|----------|-------------| | Content moderation | 91.5% | 91-96% | 41ms | | Phishing detection | 89-93% | 91-95% | ~42ms | | Support ticket triage | 89-93% | 90-95% | 33-38ms | | Lead qualification | 90-95% | 91-96% | 35-40ms |

Stage 3: LLM Escalation (150ms+)

When the classifier's confidence is below a threshold (configurable, typically 0.7), the decision escalates to a full LLM call. This is the safety net for genuinely ambiguous cases.

{
    "decision": "review",
    "confidence": 0.91,
    "latency_ms": 412,
    "stage": "escalation"
}

Escalation happens rarely — typically 5-15% of decisions. But it ensures that the hardest cases still get LLM-quality reasoning, while the easy and medium cases are handled at compiled speed.

The economics of escalation: If 10% of 50,000 daily decisions escalate to an LLM at ~$0.003 per call, that's $15/day on top of the fixed Sparkient subscription. Compared to sending all 50,000 through an LLM at $0.01 each ($500/day), you save 97%.

The Training Process

Compilation happens offline, before production. Here's what the training pipeline does:

Step 1: Define the Decision Type

You specify what you want to decide, what the options are, what input the model sees, and what rules always apply.

import httpx

response = httpx.post(
    "https://api.sparkient.ai/api/v1/decision-types",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "name": "support-triage",
        "description": "Route incoming support tickets by priority",
        "options": ["urgent", "normal", "low"],
        "input_schema": {
            "type": "object",
            "properties": {
                "subject": {"type": "string"},
                "body": {"type": "string"},
                "customer_tier": {"type": "string", "enum": ["free", "pro", "enterprise"]},
                "open_tickets": {"type": "integer"}
            },
            "required": ["subject", "body"]
        },
        "rules": [
            {"expression": "ctx.customer_tier == 'enterprise' ? 'urgent' : null"},
            {"expression": "ctx.body.contains('data loss') || ctx.body.contains('security breach') ? 'urgent' : null"}
        ]
    }
)

Step 2: Synthetic Data Generation

The LLM teacher generates diverse training examples for each class. The pipeline:

1. Generates an initial balanced dataset (200-500 examples per class)
2. Runs gap analysis to identify underrepresented patterns
3. Augments weak classes with targeted additional examples
4. Validates label consistency

You don't write prompts or manage this process. It's automated.

Step 3: Model Training

The training pipeline:

1. Feature extraction — Text is encoded into semantic embeddings, structured features are normalized
2. Hyperparameter optimization — Automated tuning searches for the best classifier configuration
3. Cross-validation — 5-fold CV ensures the model generalises, not memorizes
4. ONNX export — The trained model is compiled to ONNX with INT8 quantization
5. Evaluation — F1, accuracy, precision, recall, and confusion matrix

# Trigger training with one call
response = httpx.post(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/train",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)

Training costs 2,000 credits per run and takes a few minutes. Once complete, the model is deployed and the /decide endpoint starts using it immediately.

Step 4: Production Inference

result = httpx.post(
    "https://api.sparkient.ai/api/v1/decide",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "decision_type_id": "support-triage",
        "input": {
            "subject": "Can't access my account",
            "body": "I've been locked out since yesterday. I have a demo with a client in 2 hours and I can't pull up any of my data. This is urgent.",
            "customer_tier": "pro",
            "open_tickets": 0
        }
    }
).json()

# {"decision": "urgent", "confidence": 0.92, "latency_ms": 35, "stage": "classifier"}

Edge Deployment

Compiled models aren't locked to the cloud. You can export an edge bundle and run it anywhere:

# Export from cloud
response = httpx.get(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/export",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)
with open("triage.zip", "wb") as f:
    f.write(response.content)

# Run locally — no cloud, no API keys, no network
from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("triage.zip")
result = predictor.predict({
    "subject": "Server down",
    "body": "Production is completely offline. All customers affected.",
    "customer_tier": "enterprise"
})
# EdgeDecision(decision="urgent", confidence=0.98, latency_ms=4.1)

Edge deployment drops latency to under 10ms because there's no network round-trip. The entire pipeline — rules, tokenizer, encoder, classifier — runs in-process.

When to Use CDI (and When Not To)

CDI is not a universal hammer. Here's an honest assessment:

CDI wins when:

• You make the same type of decision at high volume. Content moderation, fraud scoring, ticket triage, lead qualification — structured decisions made thousands or millions of times per day.
• Latency matters. Payment flows, content submission UX, real-time routing, game servers.
• Cost matters at scale. Going from $15,000/month (LLM) to $199/month (Sparkient Starter) on the same decision task.
• Determinism matters. The same input always produces the same output. No temperature variation, no prompt sensitivity.

CDI is not the right fit when:

• You need free-form generation. CDI produces classifications, not essays. If you need the model to write a response, summarize a document, or generate code, use an LLM directly.
• The decision space changes weekly. CDI requires retraining when categories change. If your policy shifts constantly, the retraining overhead may outweigh the speed benefit.
• Volume is very low. If you're making 100 decisions a day, the engineering investment of setting up CDI exceeds the cost of just calling an LLM. Below ~1,000 decisions/day, direct LLM calls are simpler and the cost difference is negligible.
• You need open-ended reasoning. "Why was this flagged?" is a generation task, not a classification task. CDI returns a decision and confidence, not an explanation.

The hybrid sweet spot:

Most production systems benefit from combining all three approaches. CDI handles the high-volume, latency-sensitive decisions. LLMs handle the edge cases (via escalation) and one-off analysis tasks. Rules handle the deterministic business logic.

CDI vs. Other Approaches

| Approach | Best for | Weakness | |----------|----------|----------| | Rules/heuristics | Deterministic logic, <1ms | Can't handle nuance | | Traditional ML | Tabular data with labels | Needs labelled data, no text understanding | | Fine-tuned LLM | High accuracy on specific tasks | Expensive to serve, 100ms+ latency | | Prompt engineering | Flexibility, rapid iteration | Per-call cost, 500ms+ latency | | CDI (Sparkient) | High-volume structured decisions | Requires retraining for policy changes |

The Compilation Analogy, Extended

The compilation metaphor runs deeper than it first appears:

• Source code = LLM reasoning. Easy to read, modify, and debug. Slow to execute.
• Compiled binary = ONNX model. Optimized for execution. Hard to modify directly.
• Compiler = Sparkient training pipeline. Transforms one into the other.
• Linker = Rules engine. Connects compiled decisions with deterministic logic.
• Debug mode = Escalation. Falls back to interpreted (LLM) execution for hard cases.

Just as you wouldn't ship interpreted Python for a high-performance game engine, you shouldn't ship raw LLM calls for a high-volume decision pipeline. Compile the intelligence, deploy the binary.

FAQ

How long does training take? Typically 3-8 minutes, depending on the number of examples and the complexity of the input schema. Synthetic data generation is the longest step. Training runs asynchronously — you trigger it via the API and receive a webhook or poll for completion.

What happens if the model gets a completely novel input? The confidence score drops, and the decision escalates to the LLM. This is by design — the classifier knows what it knows, and the confidence score reflects that. Inputs that are radically different from the training data will naturally fall below the escalation threshold.

Can I inspect what the model learned? Yes. The training results include per-class precision, recall, and F1, plus a confusion matrix. You can see exactly which categories the model handles well and where it struggles. If "review" cases are being misclassified as "approve," you know to generate more borderline training examples.

Is CDI just knowledge distillation? CDI uses the teacher-student pattern, which shares principles with knowledge distillation, but the scope is broader. Traditional distillation compresses a large model into a smaller one on the same architecture. CDI compiles LLM reasoning into a completely different model type (text encoder + classifier), adds a rules layer, includes escalation logic, and packages it for deployment. It's a full decision pipeline, not just a model compression technique.

Compiled Decision Intelligence is for teams that need LLM-quality decisions at production speed and cost. Define what you want to decide, let the LLM teach it, and deploy a model that runs in under 100ms.

Start with the free tier — 5,000 credits, no credit card. Train your first compiled decision in minutes.