How to Run ML Classifiers Offline Without a Cloud API

TL;DR

Not every application can call a cloud API. Air-gapped and intermittently connected environments may need local inference. Sparkient packages a compiled ONNX model, text assets, and CEL rules into a ZIP. Install sparkient-edge on a compatible Python 3.10+ host, load the bundle, and benchmark local decisions without network calls. Sparkient does not currently provide or test native Android, iOS, microcontroller, or embedded-device SDKs.

The Problem: Cloud APIs Aren't Always an Option

Cloud-based ML is the default, but it assumes something that isn't always true: a reliable, low-latency network connection.

Real scenarios where cloud APIs don't work:

• Air-gapped environments. Defence, healthcare, and financial systems that can't make outbound API calls for security reasons.
• Remote Python-capable computers. Industrial PCs and edge servers operating on intermittent connections, where compatible dependency wheels and sufficient resources are available.
• Privacy or deployment requirements. A project's legal assessment, contract, threat model, or internal policy may require local processing.
• Latency-sensitive hot paths. Local execution can remove network variance when the cloud path does not meet the measured budget.

The traditional answer is "deploy the model yourself." But that means managing model serving infrastructure, handling updates, dealing with dependency conflicts, and building the inference pipeline from scratch.

Option 1: Build It Yourself with ONNX Runtime

ONNX Runtime is a common option for portable ML inference. It supports several languages and platforms, but you must verify the runtime build and model on each target.

import onnxruntime as ort
import numpy as np

# Load the model
session = ort.InferenceSession("model.onnx")

# Prepare input (you handle all preprocessing)
input_data = np.array([[0.5, 1.2, -0.3, 0.8]], dtype=np.float32)

# Run inference
result = session.run(None, {"input": input_data})
predictions = result[0]

Pros: Full control and a relatively small runtime surface. This example uses ONNX Runtime and NumPy. Platform support depends on compatible wheels or builds for the target operating system, Python version, and architecture.

Cons: You handle everything yourself — text tokenization, feature engineering, preprocessing, label mapping, confidence calibration, and the rules layer. For a text classification model using a semantic text encoder, that means bundling the tokenizer, managing the embedding step, and stitching together the full pipeline.

Option 2: TensorFlow Lite

TFLite is optimized for mobile and embedded devices. It supports quantization out of the box and has strong Android/iOS integration.

import tensorflow as tf

interpreter = tf.lite.Interpreter(model_path="model.tflite")
interpreter.allocate_tensors()

input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

interpreter.set_tensor(input_details[0]["index"], input_data)
interpreter.invoke()
output = interpreter.get_tensor(output_details[0]["index"])

Pros: Excellent mobile support, GPU delegate on Android, small binary size.

Cons: TensorFlow-only ecosystem. If your model was trained in PyTorch or scikit-learn, you need a conversion step that may lose fidelity. Limited server-side support.

Option 3: Core ML (Apple Only)

If you're building exclusively for Apple platforms, Core ML gives you hardware-accelerated inference on the Neural Engine.

let model = try MyClassifier(configuration: MLModelConfiguration())
let prediction = try model.prediction(text: "Check this content")

Pros: Tight Xcode integration and access to Apple hardware acceleration; benchmark the converted model on each target device.

Cons: Apple-only. No cross-platform support. You're locked into the Apple ecosystem.

Option 4: Sparkient Edge

Sparkient Edge packages CEL rules, text assets, and the classifier into a ZIP bundle. The target still needs a compatible Python version and dependency wheels. Verify the complete runtime on every operating system and architecture you plan to support.

pip install "sparkient-edge[all]"

Exporting an Edge Bundle

Create and train a decision type in Sparkient, review its results, and deploy the accepted policy. Edge export requires a Growth or Scale plan and an active deployed policy. The trial can evaluate the cloud path but cannot export a bundle. Then export it:

import httpx

response = httpx.get(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/export",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)

with open("moderation.zip", "wb") as f:
    f.write(response.content)

The bundle contains the model assets needed for offline inference:

• The ONNX model (quantized to INT8 for small size)
• The text tokenizer files
• CEL rule definitions
• Feature configuration and label mappings
• Decision type and policy metadata

Install sparkient-edge and its compatible Python runtime dependencies separately before loading the bundle.

Running Predictions Locally

from sparkient_edge import EdgePredictor

# Load the bundle — one-time initialization
predictor = EdgePredictor.from_bundle("moderation.zip")

# Make decisions — no network calls
result = predictor.predict({
    "text": "Free money! Click here now!!!",
    "account_age_days": 1,
    "previous_violations": 0
})

print(result)
# EdgeDecision includes decision, confidence, stage, reason codes, and class probabilities.

That's it. No API keys or network configuration are needed at runtime. The EdgePredictor loads the ONNX model into memory once; measure latency, memory, and throughput on the target hardware.

Bundle Info and Version Management

from pathlib import Path
from sparkient_edge import load_bundle

bundle = load_bundle(Path("moderation.zip").read_bytes())
meta = bundle["metadata"]
print(f"Decision type: {meta.get('decision_type_name', 'unknown')}")
print(f"Options: {bundle['options']}")
print(f"Trained: {meta.get('trained_at', 'unknown')}")
print(f"Rules: {len(bundle['rules'])} CEL expressions")

When you retrain in the cloud, export a new bundle and swap the file. The predictor API stays the same.

Using Edge with MCP (Local Mode)

The edge package also includes a local MCP server for AI coding assistants:

pip install "sparkient-edge[all]"
python -m sparkient_edge

This starts a stdio-based MCP server that your IDE can connect to:

{
  "mcpServers": {
    "sparkient-local": {
      "command": "python",
      "args": ["-m", "sparkient_edge"]
    }
  }
}

After connecting, call load_edge_bundle with the absolute ZIP path. The assistant can then call make_decision locally without cloud connectivity.

Comparison: Edge Inference Options

| Feature | ONNX Runtime (DIY) | TFLite | Core ML | Sparkient Edge | |---------|-------------------|--------|---------|---------------| | Platform scope | Multiple runtimes; verify the selected build | Mobile/embedded targets | Apple only | Python 3.10+ hosts with compatible dependency wheels | | Text pipeline included | ❌ | ❌ | ❌ | ✅ | | Rules engine | ❌ | ❌ | ❌ | ✅ CEL | | Tokenizer bundled | ❌ | ❌ | ❌ | ✅ | | Quantization | Manual | Built-in | Built-in | Built-in (INT8) | | Latency | Benchmark complete pipeline | Benchmark converted model | Benchmark converted model | Benchmark complete bundle on target hardware | | Setup effort | Own preprocessing and serving | Own conversion and app integration | Apple-specific conversion and app integration | Exported bundle plus local packaging and operations |

Raw ONNX Runtime inference is only one part of the path. Benchmark validation, tokenisation, preprocessing, inference, process boundaries, and concurrency together on the target hardware.

The tradeoff depends on the model and deployment. DIY runtimes maximise control; Sparkient Edge packages the trained decision assets, but you still own target-hardware validation, distribution, updates, and monitoring.

Deployment Patterns

Pattern 1: Air-Gapped Server

# Deploy the bundle to the secure environment via approved transfer
# No outbound network access needed at runtime

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("/secure/models/moderation.zip")

# Run in your application's request path
for message in message_queue:
    result = predictor.predict({"text": message.content})
    if result.decision == "reject":
        quarantine(message)

Pattern 2: Remote Python Host with Periodic Sync

import os
from sparkient_edge import EdgePredictor

MODEL_PATH = "/opt/models/current.zip"

# Load whichever bundle is currently deployed
predictor = EdgePredictor.from_bundle(MODEL_PATH)

def on_sensor_event(event_data):
    result = predictor.predict(event_data)
    if result.decision == "alert":
        store_for_sync(result)  # Upload when connectivity returns

# Periodic model update (when connected)
def sync_model():
    if network_available():
        download_latest_bundle(MODEL_PATH)
        global predictor
        predictor = EdgePredictor.from_bundle(MODEL_PATH)

Pattern 3: Embedded in a FastAPI Service

from fastapi import FastAPI
from sparkient_edge import EdgePredictor

app = FastAPI()
predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/moderate")
async def moderate(content: dict):
    result = predictor.predict(content)
    return {
        "decision": result.decision,
        "confidence": result.confidence,
        "stage": result.stage
    }

This gives you a local moderation service that makes no Sparkient API call after installing the package dependencies and bundle.

FAQ

How large are edge bundles? Bundle size depends on the text model, classifier, tokenizer, rules, and metadata. Inspect the exported ZIP and include transfer, storage, and memory in the target-device test.

Can I run edge bundles on ARM devices? Potentially, if compatible wheels exist and the machine has sufficient memory and compute. Verify the complete sparkient-edge dependency set on the target Python version, operating system, and architecture. Benchmark the actual device before committing to it. This is not a native mobile or embedded SDK.

How do I update models in air-gapped environments? Export a new bundle from Sparkient, transfer it through the approved process, and restart the predictor with the new path. The ZIP contains the model assets. Install or transfer the Python package and compatible runtime dependencies separately.

What's the minimum Python version? The current package declares Python 3.10 or later. Dependency wheel availability still varies by platform.

Cloud APIs are the easy default, but they are not the only option. An exported ONNX bundle can support local or offline decisions when the target environment meets the package requirements. Validate quality, latency, memory, and throughput before deployment.

Use the trial to evaluate one cloud model. If it passes, move to Growth or Scale, export the active deployed policy, and test that bundle on the exact target hardware before planning a local deployment.