The Complete Guide to Compiled Decision Intelligence

TL;DR

Compiled Decision Intelligence (CDI) sits between rules engines and live LLM APIs. You use an LLM or your own examples to label a bounded decision offline, then deploy a compiled model for the repeated runtime path. Sparkient's four controlled synthetic domains report 0.886–0.951 macro F1 and 33–42ms batch-average time per item.

What Is Compiled Decision Intelligence?

Every software system makes decisions. Should this content be approved? Is this transaction fraudulent? Where should this support ticket go? What priority is this lead?

Today, you have two options:

Rules engines — Fast and predictable, but rigid. A rule can check whether a transaction exceeds a threshold, but long keyword lists and nested conditionals become difficult to maintain when meaning and context matter.

LLM APIs — Flexible and capable, but each live call adds model latency, token usage, and an external dependency. The impact depends on the chosen model, prompt, output size, provider, and traffic.

Compiled Decision Intelligence is a third option. It bridges the gap:

Use an LLM to teach the decision once. Use a compiled model to make it millions of times.

The metaphor is deliberate. Just as a compiler turns a specification into an executable artifact, CDI turns a decision definition plus labelled examples into a task-specific inference model. It reproduces structured choices, not the teacher's chain-of-thought.

| Approach | Runtime profile | Billing profile | Flexibility | |----------|-----------------|-----------------|-------------| | Rules engine | Usually sub-millisecond | Application compute | Low — manual updates | | Live LLM API | Model- and prompt-dependent | Token-metered per call | High — prompt changes | | Sparkient compiled model | 33–42ms batch-average time per item in four controlled synthetic runs | Plan credits; optional escalation costs more | Medium — retrain to update |

CDI is not better than live LLMs at everything. It is a candidate for making the same bounded decision repeatedly when a measured constraint justifies a trained model.

The Three-Stage Pipeline

A compiled decision system doesn't rely on a single model. It uses a three-stage pipeline, where each stage handles what it's best at:

Stage 1: CEL Rules (<1ms)

The first stage evaluates deterministic business logic using CEL (Common Expression Language) expressions. These are hard constraints that always apply, regardless of what any model thinks.

// Block transactions over the daily limit
ctx.amount > ctx.daily_limit ? "block" : null

// Auto-approve known-good senders
ctx.sender_id in ctx.whitelist ? "approve" : null

// Force review for new accounts
ctx.account_age_days < 3 ? "review" : null

Rules usually execute in under 1ms. If a rule matches, the decision is returned immediately and the classifier does not run. Correctly written rules provide a deterministic enforcement layer; their policy and implementation still require tests.

When rules fire: The share depends entirely on the policy and traffic. Log the response stage so you can measure the real rule, classifier, escalation, and fallback mix.

Stage 2: ONNX Classifier (<100ms)

When no rule matches, the input goes to the compiled classifier. The model has learned a decision boundary from LLM-labelled or user-provided examples.

The architecture:

1. A text encoder creates high-dimensional semantic embeddings from text fields
2. A gradient-boosted classifier classifies based on embeddings plus structured features (numbers, categories, booleans)
3. The model outputs a decision and a confidence score

{
    "decision": "approve",
    "confidence": 0.94,
    "latency_ms": 38,
    "stage": "classifier"
}

The classifier can handle the normal decision path with sub-100ms latency. The actual stage mix depends on the rules, model confidence, escalation setting, and production input distribution.

Benchmark performance:

| Domain | F1 Score | Accuracy | Batch-average time per item | |--------|----------|----------|-------------| | Support ticket triage | 0.951 | 96.2% | 42ms | | Content moderation | 0.900 | 91.5% | 41ms | | Gaming chat | 0.886 | 91.0% | 34ms | | Marketplace listings | 0.938 | 94.3% | 33ms |

Stage 3: LLM Escalation (Model Dependent)

When optional escalation is enabled and the classifier's confidence is below the configured threshold, the decision can escalate to an LLM call. This stage has a different latency and credit profile from compiled inference.

{
    "decision": "review",
    "confidence": 0.91,
    "latency_ms": 412,
    "stage": "escalation"
}

Escalation rate is an outcome to measure, not a universal constant. Teams should tune the threshold against per-class quality, latency, and credit usage on representative traffic.

The economics of escalation: Model it from the observed escalation rate, actual input/output tokens, current provider price, Sparkient plan credits, model-serving hours, training, generation, and likely top-ups. The monthly plan includes credits; it is not a fixed unlimited subscription, and there is no universal savings percentage.

The Training Process

Compilation happens offline, before production. Here's what the training pipeline does:

Step 1: Define the Decision Type

You specify what you want to decide, what the options are, what input the model sees, and what rules always apply.

import httpx

response = httpx.post(
    "https://api.sparkient.ai/api/v1/decision-types",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "name": "support-triage",
        "description": "Route incoming support tickets by priority",
        "options": ["urgent", "normal", "low"],
        "input_schema": {
            "type": "object",
            "properties": {
                "subject": {"type": "string"},
                "body": {"type": "string"},
                "customer_tier": {"type": "string", "enum": ["free", "pro", "enterprise"]},
                "open_tickets": {"type": "integer"}
            },
            "required": ["subject", "body"]
        },
        "rules": [
            {"expression": "ctx.customer_tier == 'enterprise' ? 'urgent' : null"},
            {"expression": "ctx.body.contains('data loss') || ctx.body.contains('security breach') ? 'urgent' : null"}
        ]
    }
)

Step 2: Synthetic Data Generation

The LLM teacher generates diverse training examples for each class. The pipeline:

1. Generates an initial dataset according to the configured target when automatic generation is enabled
2. Runs gap analysis to identify underrepresented patterns
3. Augments weak classes with targeted additional examples
4. Validates label consistency

You don't write prompts or manage this process. It's automated.

Step 3: Model Training

The training pipeline:

1. Feature extraction — Text is encoded into semantic embeddings, structured features are normalized
2. Hyperparameter optimization — Automated tuning searches for the best classifier configuration
3. Cross-validation — Held-out folds estimate generalisation within the available dataset
4. ONNX export — The trained model is compiled to ONNX with INT8 quantization
5. Evaluation — F1, accuracy, precision, recall, and confusion matrix

# Trigger training with one call
response = httpx.post(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/train",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)

Training costs 2,000 credits per run, and its duration depends on the data and available compute. The status response reports the current attempt, stage, and heartbeat. auto_deploy defaults to true; if disabled or a configured quality gate is missed, review and deploy the trained policy manually.

Step 4: Production Inference

result = httpx.post(
    "https://api.sparkient.ai/api/v1/decide",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "decision_type": "support-triage",
        "input": {
            "subject": "Can't access my account",
            "body": "I've been locked out since yesterday. I have a demo with a client in 2 hours and I can't pull up any of my data. This is urgent.",
            "customer_tier": "pro",
            "open_tickets": 0
        }
    }
).json()

# {"decision": "urgent", "confidence": 0.92, "latency_ms": 35, "stage": "classifier"}

Edge Deployment

Compiled models aren't locked to the cloud. You can export an edge bundle and run it anywhere:

# Export from cloud
response = httpx.get(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/export",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)
with open("triage.zip", "wb") as f:
    f.write(response.content)

# Run locally — no cloud, no API keys, no network
from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("triage.zip")
result = predictor.predict({
    "subject": "Server down",
    "body": "Production is completely offline. All customers affected.",
    "customer_tier": "enterprise"
})
# Inspect result.decision, result.confidence, and result.stage

Edge deployment removes the network and cloud dependency. The rules, text processing, and classifier run locally; benchmark latency, memory, packaging, and concurrency on the target hardware.

When to Use CDI (and When Not To)

CDI is not a universal hammer. Here's an honest assessment:

CDI wins when:

• You make the same type of decision repeatedly. Content moderation, routing, scoring, approval, and agent gates are examples; recurrence matters because it makes evaluation and operational trade-offs meaningful, not because there is a fixed traffic threshold.
• Latency matters. Payment flows, content submission UX, real-time routing, game servers.
• Cost matters at scale. Compare current token usage with Sparkient credits for decisions, serving, training, generation, and escalation.
• A stable output contract matters. Sparkient returns a bounded decision plus confidence, reason codes, and stage.

CDI is not the right fit when:

• You need free-form generation. CDI produces classifications, not essays. If you need the model to write a response, summarize a document, or generate code, use an LLM directly.
• The decision space changes weekly. CDI requires retraining when categories change. If your policy shifts constantly, the retraining overhead may outweigh the speed benefit.
• There is no material constraint. A low-volume project can still need offline or fast local execution, while a high-volume call may already be simple and cheap. Keep the current path when quality, latency, cost, reliability, privacy, and maintenance all fit.
• You need open-ended reasoning. "Why was this flagged?" is a generation task, not a classification task. CDI returns a decision and confidence, not an explanation.

The hybrid sweet spot:

Some systems benefit from combining all three approaches. Rules handle deterministic constraints, the compiled model handles the learned normal path, and optional cloud escalation handles selected low-confidence cases. The useful mix is an evaluation result.

CDI vs. Other Approaches

| Approach | Best for | Weakness | |----------|----------|----------| | Rules/heuristics | Deterministic logic, <1ms | Can't handle nuance | | Traditional ML | Teams with suitable data and ML operations | Requires data, evaluation, and serving work | | Fine-tuned LLM | Specialised language-model behaviour | Model- and host-dependent training and runtime cost | | Prompt engineering | Flexibility and rapid iteration | Live-model usage, latency, and provider dependency | | CDI (Sparkient) | Repeated bounded decisions with a measured constraint | Requires training, evaluation, and retraining for learned-policy changes |

The Compilation Analogy, Extended

The compilation metaphor runs deeper than it first appears:

• Source specification = policy plus labelled decisions. Explicit, testable, and changeable.
• Compiled binary = ONNX model. Optimized for execution. Hard to modify directly.
• Compiler = Sparkient training pipeline. Transforms one into the other.
• Linker = Rules engine. Connects compiled decisions with deterministic logic.
• Debug mode = Escalation. Falls back to interpreted (LLM) execution for hard cases.

The analogy is useful only up to a point: a trained classifier approximates a labelled decision boundary; it does not compile a reasoning trace. Use a compiled model when the measured project result justifies the trade-off.

FAQ

How long does training take? Training runs asynchronously, and duration varies with the number of examples, text-model configuration, and available compute. Trigger it via the API and use the progress endpoint or webhook to measure the actual run.

What happens if the model gets a completely novel input? A novel input may produce lower confidence, but confidence is not a guaranteed out-of-distribution detector. If escalation is enabled and the score crosses the configured threshold, Sparkient can call the LLM fallback. Monitor real inputs and add evaluated examples before relying on this behaviour.

Can I inspect what the model learned? Yes. The training results include per-class precision, recall, and F1, plus a confusion matrix. You can see exactly which categories the model handles well and where it struggles. If "review" cases are being misclassified as "approve," you know to generate more borderline training examples.

Is CDI just knowledge distillation? CDI uses a teacher-student pattern, but Sparkient trains a different model type (text encoder plus classifier) to reproduce structured labels. It adds rules, confidence, optional escalation, versioning, and deployment around that model; it does not preserve the teacher's reasoning trace.

Compiled Decision Intelligence is for developers with repeated, measurable choices whose output options are stable enough to train and evaluate. Define the decision, let an LLM or your own examples teach it, then compare the compiled model against the existing approach before integrating it.

Start with the free tier — 5,000 credits, no credit card. Measure one candidate decision against a held-out set before integrating it.