How to Run ML Classifiers Offline with Zero Cloud Dependencies

TL;DR

Not every application can call a cloud API. Air-gapped environments, IoT devices, mobile apps, and regions with unreliable connectivity all need local inference. Sparkient's edge bundle packages a compiled ONNX model with CEL rules into a self-contained ZIP — pip install sparkient-edge, load the bundle, and get decisions in under 10ms with zero network calls.

The Problem: Cloud APIs Aren't Always an Option

Cloud-based ML is the default, but it assumes something that isn't always true: a reliable, low-latency network connection.

Real scenarios where cloud APIs don't work:

• Air-gapped environments. Defence, healthcare, and financial systems that can't make outbound API calls for security reasons.
• IoT and embedded devices. Industrial sensors, point-of-sale terminals, and smart cameras operating on intermittent cellular connections.
• Mobile apps in the field. Field service tools, agricultural monitoring, and emergency response apps used in areas with no signal.
• Privacy requirements. GDPR, HIPAA, or internal policies that prohibit sending data to external APIs.
• Latency-sensitive hot paths. Payment processing, game servers, and real-time bidding where even 40ms of network overhead is too much.

The traditional answer is "deploy the model yourself." But that means managing model serving infrastructure, handling updates, dealing with dependency conflicts, and building the inference pipeline from scratch.

Option 1: Build It Yourself with ONNX Runtime

ONNX Runtime is the standard for portable ML inference. If you have an ONNX model, you can run it anywhere Python (or C++, C#, Java, JavaScript) runs.

import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession("model.onnx")

# Prepare input (you handle all preprocessing)
input_data = np.array([[0.5, 1.2, -0.3, 0.8]], dtype=np.float32)

# Run inference
result = session.run(None, {"input": input_data})
predictions = result[0]

Pros: Full control, no dependencies beyond onnxruntime, works on ARM and x86.

Cons: You handle everything yourself — text tokenization, feature engineering, preprocessing, label mapping, confidence calibration, and the rules layer. For a text classification model using a semantic text encoder, that means bundling the tokenizer, managing the embedding step, and stitching together the full pipeline.

Option 2: TensorFlow Lite

TFLite is optimized for mobile and embedded devices. It supports quantization out of the box and has strong Android/iOS integration.

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]["index"], input_data)
interpreter.invoke()
output = interpreter.get_tensor(output_details[0]["index"])

Pros: Excellent mobile support, GPU delegate on Android, small binary size.

Cons: TensorFlow-only ecosystem. If your model was trained in PyTorch or scikit-learn, you need a conversion step that may lose fidelity. Limited server-side support.

Option 3: Core ML (Apple Only)

If you're building exclusively for Apple platforms, Core ML gives you hardware-accelerated inference on the Neural Engine.

let model = try MyClassifier(configuration: MLModelConfiguration())
let prediction = try model.prediction(text: "Check this content")

Pros: Fastest option on Apple hardware, tight Xcode integration.

Cons: Apple-only. No cross-platform support. You're locked into the Apple ecosystem.

Option 4: Sparkient Edge

Sparkient Edge packages the entire decision pipeline — CEL rules, text tokenizer, text encoder, classifier — into a single ZIP bundle that runs anywhere Python runs.

pip install sparkient-edge

Exporting an Edge Bundle

First, train a model through the Sparkient cloud API. Then export it:

import httpx

response = httpx.get(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/export",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)

with open("moderation.zip", "wb") as f:
    f.write(response.content)

The bundle contains everything needed for offline inference:

• The ONNX model (quantized to INT8 for small size)
• The text tokenizer files
• CEL rule definitions
• Input schema and label mappings
• Bundle metadata (version, training date, metrics)

Running Predictions Locally

from sparkient_edge import EdgePredictor

# Load the bundle — one-time initialization
predictor = EdgePredictor.from_bundle("moderation.zip")

# Make decisions — no network calls
result = predictor.predict({
    "text": "Free money! Click here now!!!",
    "account_age_days": 1,
    "previous_violations": 0
})

print(result)
# EdgeDecision(decision="reject", confidence=0.97, latency_ms=3.2)

That's it. No API keys, no network configuration, no model serving infrastructure. The EdgePredictor loads the ONNX model into memory once, then runs predictions at sub-10ms latency.

Bundle Info and Version Management

from sparkient_edge import load_bundle

bundle = load_bundle("moderation.zip")
print(f"Decision type: {bundle.name}")
print(f"Options: {bundle.options}")
print(f"Trained: {bundle.trained_at}")
print(f"Training F1: {bundle.metrics.get('f1_score')}")
print(f"Rules: {len(bundle.rules)} CEL expressions")

When you retrain in the cloud, export a new bundle and swap the file. The predictor API stays the same.

Using Edge with MCP (Local Mode)

The edge package also includes a local MCP server for AI coding assistants:

python -m edge

This starts a stdio-based MCP server that your IDE can connect to:

{
  "mcpServers": {
    "sparkient-local": {
      "command": "python",
      "args": ["-m", "edge"],
      "env": {
        "BUNDLE_PATH": "/path/to/moderation.zip"
      }
    }
  }
}

Now your coding assistant can make local decisions without any cloud connectivity.

Comparison: Edge Inference Options

| Feature | ONNX Runtime (DIY) | TFLite | Core ML | Sparkient Edge | |---------|-------------------|--------|---------|---------------| | Cross-platform | ✅ | ✅ | ❌ Apple only | ✅ | | Text pipeline included | ❌ | ❌ | ❌ | ✅ | | Rules engine | ❌ | ❌ | ❌ | ✅ CEL | | Tokenizer bundled | ❌ | ❌ | ❌ | ✅ | | Quantization | Manual | Built-in | Built-in | Built-in (INT8) | | Typical latency | 2-5ms* | 1-3ms | <1ms | 3-10ms | | Setup effort | High | Medium | Low (Apple) | Low |

*Raw ONNX Runtime inference is fast, but add tokenization and preprocessing and real latency is higher.

The tradeoff is clear: if you want full control and don't mind building the pipeline, ONNX Runtime standalone is great. If you want a working decision system out of the box, Sparkient Edge bundles everything.

Deployment Patterns

Pattern 1: Air-Gapped Server

# Deploy the bundle to the secure environment via approved transfer
# No outbound network access needed at runtime

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("/secure/models/moderation.zip")

# Run in your application's hot path
for message in message_queue:
    result = predictor.predict({"text": message.content})
    if result.decision == "reject":
        quarantine(message)

Pattern 2: IoT Device with Periodic Sync

import os
from sparkient_edge import EdgePredictor

MODEL_PATH = "/opt/models/current.zip"

# Load whichever bundle is currently deployed
predictor = EdgePredictor.from_bundle(MODEL_PATH)

def on_sensor_event(event_data):
    result = predictor.predict(event_data)
    if result.decision == "alert":
        store_for_sync(result)  # Upload when connectivity returns

# Periodic model update (when connected)
def sync_model():
    if network_available():
        download_latest_bundle(MODEL_PATH)
        global predictor
        predictor = EdgePredictor.from_bundle(MODEL_PATH)

Pattern 3: Embedded in a FastAPI Service

from fastapi import FastAPI
from sparkient_edge import EdgePredictor

app = FastAPI()
predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/moderate")
async def moderate(content: dict):
    result = predictor.predict(content)
    return {
        "decision": result.decision,
        "confidence": result.confidence,
        "latency_ms": result.latency_ms
    }

This gives you a self-contained moderation microservice with zero external dependencies.

FAQ

How large are edge bundles? A typical bundle with a quantized text encoder and classifier is 80-120 MB. Most of that is the tokenizer and ONNX model. INT8 quantization reduces the model size by roughly 2-3× compared to float32.

Can I run edge bundles on ARM devices? Yes. ONNX Runtime supports ARM64 natively. The edge predictor works on Raspberry Pi, Jetson Nano, Apple Silicon, and ARM-based cloud instances. Install with pip install sparkient-edge — no architecture-specific builds needed.

How do I update models in air-gapped environments? Export a new bundle from the Sparkient cloud, transfer it to the air-gapped environment via your approved data transfer process (USB, secure file transfer, etc.), and restart the predictor with the new bundle path. The bundle is a self-contained ZIP file — no dependency resolution needed.

What's the minimum Python version? Python 3.9+. The edge package's dependencies all support Python 3.9 through 3.13.

Cloud APIs are the easy default, but they're not the only option. When you need decisions at the edge — fast, private, and offline — a compiled ONNX bundle gets you there.

Start with the free tier to train your model in the cloud, then export an edge bundle and deploy it wherever you need it. pip install sparkient-edge and you're running.