LLM Classification vs Fine-Tuning vs Distillation: Which to Use?

TL;DR

Direct LLM classification is simplest to set up but slowest and most expensive at scale. Fine-tuning trains a smaller LLM on your data — better cost and latency, but still 150ms+ and requires GPU infrastructure. Distillation/compilation trains a non-LLM classifier on LLM outputs — fastest (sub-100ms) and cheapest, but only works for tasks with fixed output categories. Choose based on volume, latency needs, and whether you need generative output.

The Problem

You've validated that an LLM can make a decision accurately — classify content, score leads, triage tickets. Now you need to put it in production. The question isn't if the LLM can do it; it's how to operationalize it at the right cost and speed.

Three approaches dominate in 2026:

1. Direct LLM classification — send a prompt, get a label back
2. Fine-tuning — train a smaller LLM on your specific task
3. Distillation / compilation — train a classical ML model on LLM-generated labels

Each has real trade-offs. This guide breaks them down honestly.

Approach 1: Direct LLM Classification

What it is: You send a prompt to an LLM API (GPT-4o, Gemini, Claude) with instructions and input, and parse the classification from the response.

from openai import OpenAI

client = OpenAI()

def classify_direct(text: str) -> str:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {
                "role": "system",
                "content": (
                    "Classify the text as 'spam', 'legitimate', or 'uncertain'. "
                    "Return only the label."
                ),
            },
            {"role": "user", "content": text},
        ],
        max_tokens=10,
    )
    return response.choices[0].message.content.strip().lower()

Strengths

• Fastest to set up. No training, no infrastructure. Write a prompt, call the API.
• Most flexible. Change the prompt to change the behavior instantly. Add a new category by editing a string.
• Highest accuracy out of the box. State-of-the-art LLMs encode vast world knowledge. For ambiguous or edge-case-heavy tasks, the LLM's reasoning ability is unmatched.
• Zero data requirements. You don't need labelled examples to start.

Weaknesses

• Slowest. 500-1500ms per call for GPT-4o-mini, 1-3s for GPT-4o. Even Groq's optimized inference takes 150-300ms.
• Most expensive at scale. At 50,000 calls/day with GPT-4o, you're looking at ~$15,000/month. Even GPT-4o-mini at that volume costs ~$4,500/month.
• Non-deterministic. The same input can produce different outputs across calls (even with temperature=0, outputs can vary across model versions).
• Vendor dependency. Rate limits, model deprecations, and pricing changes are outside your control.
• Prompt injection risk. Adversarial inputs can manipulate the classification.

Best for

• Prototyping and validation (proving the decision type works)
• Low-volume production (<100 calls/day)
• Tasks where flexibility is more important than cost or speed
• One-off decisions where latency doesn't matter

Approach 2: Fine-Tuning

What it is: You take a smaller, cheaper LLM (GPT-4o-mini, Llama 3, Mistral) and train it on your specific classification task using labelled examples.

# Fine-tuning setup (OpenAI example)
from openai import OpenAI

client = OpenAI()

# Upload training data
training_file = client.files.create(
    file=open("training_data.jsonl", "rb"),
    purpose="fine-tune"
)

# Start fine-tuning
job = client.fine_tuning.jobs.create(
    training_file=training_file.id,
    model="gpt-4o-mini-2024-07-18"
)

Strengths

• Better cost than direct classification. A fine-tuned smaller model is cheaper per token than a large model with a long prompt.
• Better latency. Fine-tuned models with short prompts can run in 150-400ms — faster than a large model with detailed instructions.
• Higher accuracy on your specific task. Fine-tuning specializes the model, often improving accuracy by 5-10% on domain-specific data.
• Shorter prompts. The knowledge is in the weights, not the prompt, so you save on input tokens.

Weaknesses

• Requires training data. You typically need 100-1,000+ labelled examples. Where do they come from? Often from a larger LLM — which means you're already doing a form of distillation.
• Still an LLM. Latency is still 150ms+ even on optimized infrastructure. You're paying per token. The model is still susceptible to prompt injection (though less so).
• Training costs money. Fine-tuning on OpenAI costs roughly $25/million training tokens. Self-hosted fine-tuning requires GPU infrastructure.
• Model management. You own a model now. It needs versioning, evaluation, monitoring, and periodic retraining. Fine-tuned models can also be deprecated by the provider.
• GPU inference. Running a fine-tuned model requires GPU infrastructure — either cloud-hosted (expensive) or self-managed (complex).

Best for

• Tasks where you need the LLM's language understanding but want better cost/latency than direct classification
• Medium volume (1,000-50,000 calls/day) where per-token costs are noticeable
• Tasks where accuracy on domain-specific language is critical and a few percentage points matter
• Teams with ML infrastructure and experience managing model deployments

Approach 3: Distillation / Compilation

What it is: You use an LLM to generate labelled training data, then train a classical ML classifier to reproduce the LLM's classification. The LLM is the teacher; the smaller model is the student.

This is what Sparkient calls "compilation" — using the LLM to teach the decision once, then using the compiled model to make it millions of times.

# The compiled model in production
import httpx

async def classify_compiled(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "spam-detection",
                "input": {"text": text},
            },
        )
    return response.json()

# {"decision": "spam", "confidence": 0.96, "latency_ms": 35, "stage": "classifier"}

Strengths

• Fastest inference. Sub-100ms p95 in production. The model is an ONNX-optimized compiled model, not an LLM. No GPU required.
• Cheapest at scale. No per-token costs. Sparkient's Starter plan is $199/month for 100K credits — at 50,000 decisions/day, that's a fraction of the LLM cost.
• No GPU required. ONNX Runtime runs on CPUs. Edge deployment runs on any machine with Python.
• Deterministic. Same input always produces the same output. No temperature, no sampling variance.
• Resistant to prompt injection. The model processes features, not natural language instructions. It can't be talked into changing its classification.
• Edge deployable. Export the model as an ONNX bundle and run it offline with sparkient-edge — sub-10ms decisions with no cloud dependency.

Weaknesses

• Only works for classification. If you need generative output (not just a label), compilation doesn't apply.
• Requires a training step. You need to define the decision type, generate training data, and train the model. With Sparkient, this takes minutes; with a custom pipeline, it could take days.
• Accuracy ceiling. The compiled model can't be smarter than the teacher LLM. In practice, compiled models achieve 89-95% F1 — within 2-3 points of the source LLM. For tasks where 98%+ accuracy is critical, fine-tuning might edge it out.
• Less flexible. Changing output categories or decision logic requires retraining. You can't just edit a prompt.
• Newer approach. Less community documentation and tooling compared to fine-tuning.

Best for

• High-volume classification (10,000+ decisions/day) where cost and latency dominate
• Latency-sensitive hot paths (<100ms required)
• Tasks with fixed output categories and structured input
• Teams that want LLM-quality decisions without LLM infrastructure
• Edge/offline deployment requirements

Comparison Table

| Dimension | Direct LLM | Fine-Tuning | Compilation | |---|---|---|---| | Setup time | Minutes | Hours to days | ~30 minutes | | Training data needed | None | 100-1,000+ examples | Auto-generated | | Inference latency | 500-1500ms | 150-400ms | <100ms | | Cost at 50K/day | $4,500-15,000/mo | $500-2,000/mo | $199/mo | | Accuracy (F1) | 93-97% | 94-98% | 89-95% | | GPU required | No (API) | Yes (inference) | No | | Edge/offline | No | Difficult | Yes | | Prompt injection risk | High | Medium | Low | | Output type | Any (text, JSON) | Any (text, JSON) | Labels only | | Flexibility | Highest | Medium | Lowest | | Maintenance | Low | High | Medium |

The Decision Tree

What type of output do you need?
│
├── Free-form text (generation, writing, reasoning)
│   └── Use Direct LLM or Fine-Tuning
│       ├── <1,000 calls/day → Direct LLM (simplest)
│       └── >1,000 calls/day → Fine-Tuning (cheaper per call)
│
└── Fixed labels (classification, scoring, triage)
    │
    ├── How many calls per day?
    │   ├── <100 → Direct LLM (cheapest, simplest)
    │   ├── 100-1,000 → Direct LLM or Compilation (depends on latency needs)
    │   └── >1,000 → Compilation (cost advantage kicks in)
    │
    ├── Is latency critical? (<100ms required)
    │   ├── Yes → Compilation (sub-100ms vs 150ms+ minimum)
    │   └── No → Any approach works
    │
    └── Is accuracy above 95% F1 critical?
        ├── Yes → Fine-Tuning (highest accuracy ceiling)
        └── No → Compilation (89-95% F1, much faster and cheaper)

Hybrid Approach: Compilation with LLM Escalation

You don't have to pick just one. The most robust production systems combine approaches:

1. Compiled model handles the 90%+ of decisions where it's confident
2. LLM escalation handles the edge cases where the compiled model's confidence is low

This is exactly how Sparkient's pipeline works:

Input → CEL Rules (<1ms) → Compiled Classifier (<100ms) → LLM Escalation (150ms+)
         |                        |                              |
         Blocklist match?         Confident?                     Fallback
         → Instant decision       → Fast decision                → Accurate decision

The result: most decisions are fast and cheap (compiled model), while ambiguous cases get the LLM's full reasoning ability. You get the cost and speed of compilation with the accuracy ceiling of an LLM.

In benchmarks, this hybrid approach delivers 91.5% F1 at 41ms p95 — because the escalation path only fires on the hardest 5-10% of inputs.

Real-World Migration Path

Here's the practical path most teams follow:

Phase 1: Direct LLM (Week 1)

Build with GPT-4o or Gemini. Validate the decision type, refine the prompt, confirm accuracy. Cost: per-token pricing.

Phase 2: Evaluate Volume (Week 2-4)

Monitor call volume, latency impact, and monthly costs. If you're under 1,000 calls/day and latency isn't an issue, stay on Phase 1.

Phase 3: Compile (When volume justifies it)

When you hit the cost or latency threshold, compile the decision. With Sparkient, this takes ~30 minutes:

1. Define the decision type (options + input schema)
2. Train the model (Sparkient generates data from your policy and trains automatically)
3. Point your API calls to Sparkient's /decide endpoint
4. The compiled model handles most decisions; the LLM handles edge cases via escalation

Phase 4: Edge (If needed)

If you need offline capability or sub-10ms latency, export the compiled model as an edge bundle:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("spam-detection.zip")
result = predictor.predict({"text": "Buy now! Limited time offer!!!"})
# EdgeDecision(decision="spam", confidence=0.97, latency_ms=3.2)

FAQ

Is fine-tuning always more accurate than compilation?

Not always, but it has a higher ceiling. Fine-tuned LLMs retain the base model's language understanding, which helps on highly nuanced or ambiguous inputs. In practice, the difference is often 2-5 percentage points of F1. For many production workloads, 91% F1 at 38ms (compiled) is preferable to 95% F1 at 200ms (fine-tuned) — the latency and cost savings outweigh the accuracy gap.

Can I use my existing labelled data instead of LLM-generated data?

Yes. If you have human-labelled training data, you can use it directly. Sparkient also accepts manually added examples alongside the auto-generated synthetic data. Human labels are often higher quality than LLM-generated labels, so mixing both can improve accuracy.

How does compilation handle new categories?

If you add a new output category (e.g., adding "escalate" to an existing "approve/reject" decision), you need to retrain the model. With Sparkient, this takes a few minutes and costs 2,000 credits per training run. With a custom pipeline, you'd need to regenerate training data and retrain from scratch.

What about distilling into a small LLM (e.g., Llama 3 8B) instead of a classical model?

This is a valid middle ground. Distilling into a small LLM gives you some of the LLM's flexibility (handling unexpected inputs) with better latency than the teacher model. The trade-off: you still need GPU inference (150ms+ minimum), and you still pay per-token costs. Compiling into a classical model eliminates both of those costs, but at the price of flexibility.

Ready to decide which approach fits your workload? If you're leaning toward compilation, start with Sparkient's free tier — 5,000 credits, no credit card — and benchmark a compiled model against your current LLM setup.