How to Train a Classifier Using an LLM as Teacher

TL;DR

You can train a production classifier without any hand-labelled data by using an LLM as a teacher. The process: generate 200-500 diverse examples per class with the LLM, train a compiled model on that data, export to ONNX, and deploy at <100ms. Sparkient's benchmarks show 89-95% F1 across multiple domains using this approach — within a few points of the teacher LLM at 15-75× lower latency.

The Problem: LLMs Are Smart but Slow and Expensive

Large language models are remarkably good at classification. Give GPT-4o a content moderation task, a support ticket triage problem, or a lead qualification prompt, and it'll get 90-95% accuracy out of the box.

The issue is the production profile:

| | LLM (GPT-4o) | LLM (Gemini Flash) | Compiled Classifier | |---|---|---|---| | Latency (p95) | ~623ms | ~500-800ms | <100ms | | Cost per call | ~$0.01 | ~$0.003 | Fixed monthly | | Cost at 50K/day | ~$15,000/mo | ~$4,500/mo | $199/mo | | Scales linearly | Yes (cost grows) | Yes (cost grows) | No (fixed) |

The teacher-student pattern resolves this: use the LLM's intelligence once (during training), then deploy a fast model that carries that intelligence forward.

How Teacher-Student Training Works

The pattern has three phases:

Phase 1: TEACH         Phase 2: TRAIN         Phase 3: DEPLOY
┌─────────────┐       ┌──────────────┐       ┌──────────────┐
│  LLM Teacher │──────▶│  Train Small  │──────▶│  ONNX Model  │
│  generates   │       │  Classifier   │       │  in prod     │
│  examples    │       │  on synthetic │       │  <100ms      │
│              │       │  data         │       │              │
└─────────────┘       └──────────────┘       └──────────────┘
   (offline)             (offline)             (production)

The LLM never runs in production. It's used offline, once, to create the training dataset.

Phase 1: Generating Synthetic Training Data

The quality of your teacher determines the quality of your student. Here's what matters:

What makes a good teacher prompt:

A good prompt gives the LLM enough context to generate realistic, diverse examples. It needs:

1. Clear category definitions — Exactly what each label means, with boundary cases
2. Domain context — What kind of platform, what kind of users, what tone is normal
3. Diversity instructions — Vary length, tone, topic, and style
4. Edge case emphasis — Explicitly ask for borderline examples

teacher_prompt = """You are generating training data for a content moderation
classifier on a gaming community forum.

Categories:
- "approve": Normal gaming discussion, strategy talk, trash talk that stays
  playful, opinions about games, memes, questions. Most content falls here.
- "review": Borderline content that needs human judgement — heated arguments
  that might cross a line, mild personal attacks, possible doxxing attempts,
  content that could be harassment depending on context.
- "reject": Clear violations — slurs, explicit threats, doxxing (sharing
  personal info), sexual content involving minors, coordinated harassment,
  real-world violence threats.

Generate {n} examples. For each, provide:
- "text": The forum post content (realistic gaming community language)
- "label": The correct moderation decision
- "reasoning": Brief explanation of why this label is correct

Requirements:
- Include gaming-specific edge cases: competitive trash talk vs actual toxicity
- Vary post length from one-liners to multi-paragraph
- Include examples that could be misclassified by simple keyword matching
- Mix of English dialects and informal writing styles
- {class_distribution} approximate distribution across labels
"""

How many examples you need:

The sweet spot is 200-500 examples per class. Here's what we've observed:

| Examples per class | Typical F1 | Notes | |---|---|---| | 50 | 70-78% | Underfitting — model hasn't seen enough variety | | 100 | 78-85% | Usable for prototyping, not production | | 200 | 85-90% | Good baseline, sufficient for most use cases | | 500 | 89-95% | Strong production performance | | 1,000+ | 90-96% | Diminishing returns unless domain is very complex |

For a three-class problem, 500 examples per class means 1,500 total. At ~$0.003 per LLM call generating 10 examples each, that's roughly $0.45 in API costs for your entire training set.

Handling class imbalance:

Real-world data is rarely balanced. In content moderation, ~90% of content is "approve," ~7% is "review," and ~3% is "reject." But training on that distribution means the model barely sees the minority classes.

Two strategies:

1. Oversample during generation — Generate more examples for rare classes. Ask the LLM for 500 "reject" examples even though they represent 3% of real traffic.

2. Augment after initial generation — Generate a balanced initial set, evaluate performance, then generate more examples specifically for classes with low recall.

Sparkient's training pipeline does both: it generates a balanced initial set, runs gap analysis to identify weak areas, then uses the LLM to augment underperforming classes.

Phase 2: Training the Student Model

The student model needs to be fast (sub-100ms), small (deploys anywhere), and accurate (matches the teacher as closely as possible).

The DIY approach with scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import Pipeline
import json
import numpy as np

# Load synthetic data
with open("teacher_data.json") as f:
    data = json.load(f)

texts = [d["text"] for d in data]
labels = [d["label"] for d in data]

# Simple baseline: TF-IDF + Logistic Regression
pipeline = Pipeline([
    ("tfidf", TfidfVectorizer(max_features=10000, ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000, class_weight="balanced"))
])

scores = cross_val_score(pipeline, texts, labels, cv=5, scoring="f1_macro")
print(f"Baseline F1: {np.mean(scores):.3f} ± {np.std(scores):.3f}")
# Typical result: 0.78 ± 0.03

This gets you a working baseline, but TF-IDF misses semantic meaning. Two sentences with completely different words but the same intent — "I'll find where you live" and "Your address is about to become public" — won't cluster together.

The better architecture: Text Encoder + Classifier

Sparkient's training pipeline uses a two-stage approach:

1. Text encoder creates high-dimensional semantic embeddings from text
2. Classifier makes predictions based on those embeddings plus any structured features (account age, previous violations, etc.)
3. Automated hyperparameter tuning optimizes the model configuration

# Conceptual outline — this is what Sparkient automates
import numpy as np

# Step 1: Encode text with a semantic text encoder
# The encoder converts text into dense vector representations
# that capture meaning, not just word frequency
def encode_texts(texts, batch_size=32):
    embeddings = []
    for batch in chunks(texts, batch_size):
        # Tokenize and encode each batch
        batch_embeddings = text_encoder.encode(batch)
        embeddings.append(batch_embeddings)
    return np.vstack(embeddings)

X_text = encode_texts(texts)

# Step 2: Combine with structured features
X_structured = np.array([[d.get("account_age_days", 0),
                           d.get("previous_violations", 0)]
                          for d in data])
X = np.hstack([X_text, X_structured])

# Step 3: Train classifier with automated hyperparameter tuning
# The tuning process explores different model configurations
# to find the best-performing combination
best_model = train_with_hyperparameter_optimization(
    X, labels,
    n_trials=50,
    scoring="f1_macro"
)

This two-stage architecture is why Sparkient hits 91.5% F1 on content moderation while a TF-IDF baseline sits around 78%.

Phase 3: Export and Deploy

The trained model needs to run fast in production. ONNX (Open Neural Network Exchange) is the deployment format:

# Export to ONNX for fast inference
# The trained model is converted to ONNX format for
# portable, high-performance inference
onnx_model = convert_to_onnx(best_model, input_shape=X.shape[1])
save_onnx_model(onnx_model, "moderation_model.onnx")

ONNX Runtime inference is fast — sub-100ms including text encoding — because the model is small and there are no network calls to an LLM API.

With Sparkient, all of this is one API call:

import httpx

# Trigger training — synthetic data generation, model training, and
# ONNX export all happen automatically
response = httpx.post(
    f"https://api.sparkient.ai/api/v1/decision-types/{decision_type_id}/train",
    headers={"Authorization": "Bearer YOUR_API_KEY"}
)
# Training runs asynchronously. Poll the status endpoint or wait for
# the webhook notification.

# Once deployed, call /decide
result = httpx.post(
    "https://api.sparkient.ai/api/v1/decide",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "decision_type_id": "content-moderation",
        "input": {"text": "gg ez noob uninstall", "user_id": "player_42"}
    }
).json()
# {"decision": "approve", "confidence": 0.82, "latency_ms": 36, "stage": "classifier"}

Validating the Student Against the Teacher

How do you know the student learned the right lessons? You need a validation strategy.

Hold-out validation: Set aside 20% of synthetic data as a test set. Train on 80%, evaluate on 20%. This measures how well the student learned the teacher's patterns — but not whether those patterns are correct.

Teacher agreement: Run a sample of real-world inputs through both the teacher LLM and the student classifier. Measure agreement rate. For a well-trained student, you should see 85-95% agreement with the teacher on the same inputs.

Confidence calibration: Check that the student's confidence scores are meaningful. When the model says 0.95 confidence, it should be right ~95% of the time. When it says 0.6, expect ~60% accuracy. Poorly calibrated models are dangerous because you can't trust the escalation threshold.

Edge case audit: Manually curate 50-100 tricky examples and check the student's decisions. This is the most important validation step because it catches systematic blind spots — categories of content the teacher didn't generate enough examples for.

When to Use This Pattern (and When Not To)

Use teacher-student training when:

• You need sub-100ms decisions (payment flows, content submission, real-time routing)
• You're making the same type of decision millions of times
• You have a clear set of output categories (2-10 classes)
• Cost matters at scale (>10K decisions/day)

Use direct LLM calls when:

• You need free-form reasoning, not classification
• Decision types change frequently (weekly policy updates)
• Volume is low (<1K decisions/day) and latency doesn't matter
• You need the LLM to explain its reasoning in natural language

Use rules engines when:

• Logic is purely deterministic (rate limits, geo-blocking, age verification)
• You need sub-1ms latency
• The decision tree is small and well-defined

The best systems combine all three — which is exactly what the compiled decision pipeline does.

FAQ

Does the student ever surpass the teacher? Rarely in raw accuracy, but often in consistency. LLMs can give different answers to the same input depending on temperature, prompt phrasing, and API version. A compiled classifier gives the exact same answer every time. For production systems, determinism is often more valuable than a marginal accuracy improvement.

How do I handle concept drift? When your domain changes — new types of spam, new slang, new attack patterns — the student's accuracy will degrade. Monitor confidence distributions over time. If average confidence drops or the "review" rate spikes, it's time to retrain. Generate new synthetic data that covers the emerging patterns, mix with your existing examples, and retrain. Sparkient tracks these metrics in the dashboard.

Can I use a different LLM as the teacher? Yes. Any LLM that can generate structured labelled examples works as a teacher. The quality of the student depends on the quality and diversity of the training data, not the specific teacher model. The architecture is model-agnostic — what matters is the quality of the generated examples.

What if I already have some labelled data? Use it. Even 50-100 real examples significantly improve the student, especially for edge cases specific to your domain. Upload them alongside the synthetic data — real examples anchor the model in your actual distribution, while synthetic data provides breadth and diversity.

The teacher-student pattern isn't new, but LLMs make it accessible to anyone with an API key. You don't need a data science team to build a production classifier. You need a good teacher prompt and a solid training pipeline.

Try it in the playground, or start with the free tier — 5,000 credits, no credit card.