I Need to Moderate Content but Can't Add a 1-Second API Call

TL;DR

Content moderation is a latency trap: keyword filters are fast but miss 40%+ of violations, while LLM-based moderation adds 500-1500ms per request. A compiled ONNX classifier achieves 91.5% F1 accuracy at 41ms P95 — fast enough for real-time UX, accurate enough to catch nuanced violations. For edge deployment, the same model runs locally at sub-10ms with no cloud dependency.

The Constraint

You need content moderation. Trust & Safety, regulatory compliance, brand protection — the reasons don't matter. What matters is the constraint:

Your UX cannot tolerate a 1-second delay between "submit" and "posted."

You're building a chat app, a social feed, a marketplace listing flow, or a comment section. Users expect their content to appear instantly. Every millisecond of delay after they hit "submit" feels like friction. At 500ms, it's noticeable. At 1 second, it's annoying. At 2 seconds, users start rage-tapping.

And yet, you need to check every piece of content before it's visible to other users. Pre-publish moderation — not post-publish cleanup.

Let's look at every option, with real numbers.

Option 1: Keyword Filters

How it works

Maintain a list of blocked words and regex patterns. Check each submission against the list.

import re

BLOCKED_PATTERNS = [
    r'\b(spam|scam|fake)\b',
    r'\b(buy\s+followers)\b',
    r'(https?://\S+){3,}',  # 3+ links
    # ... 500 more patterns
]

compiled = [re.compile(p, re.IGNORECASE) for p in BLOCKED_PATTERNS]

def moderate_keywords(text: str) -> str:
    for pattern in compiled:
        if pattern.search(text):
            return "reject"
    return "approve"

Performance

| Metric | Value | |--------|-------| | Latency | <1ms | | Accuracy | ~55% F1 | | False positive rate | 15-25% | | False negative rate | 30-40% | | Setup effort | Low | | Maintenance | High (constant rule updates) |

The problem

Keyword filters can't handle context. "I'm going to kill this presentation" isn't a threat. "Check my profile for more details 😉" doesn't contain any blocked words but is clearly spam. Obfuscation bypasses them trivially: "fr33 m0ney" passes any keyword filter looking for "free money."

The false positive rate is equally painful. Legitimate users get their content rejected because they used a word that appears in a different context. "This product is trash" is a valid negative review, not a policy violation.

Verdict: Useful as a first layer for obvious cases, but completely inadequate as a standalone solution.

Option 2: Perspective API (Google Jigsaw)

How it works

Google's ML-based toxicity scoring API. Send text, get scores for toxicity, severe toxicity, insult, profanity, identity attack, and threat.

from googleapiclient import discovery

client = discovery.build("commentanalyzer", "v1alpha1",
                         developerKey="YOUR_API_KEY")

def moderate_perspective(text: str) -> str:
    response = client.comments().analyze(body={
        "comment": {"text": text},
        "requestedAttributes": {
            "TOXICITY": {},
            "SEVERE_TOXICITY": {},
            "SPAM": {}
        }
    }).execute()

    toxicity = response["attributeScores"]["TOXICITY"]["summaryScore"]["value"]

    if toxicity > 0.8:
        return "reject"
    elif toxicity > 0.5:
        return "review"
    return "approve"

Performance

| Metric | Value | |--------|-------| | Latency | 150-300ms | | Accuracy | ~75% F1 (on general toxicity) | | False positive rate | 10-15% | | False negative rate | 15-20% | | Setup effort | Low | | Cost | Free for moderate volume |

The problem

Perspective API is designed for toxicity detection, not general content moderation. It doesn't understand your specific policies. "Buy my product at discount-pills.com" isn't toxic — it scores low on toxicity — but it's spam that violates your content policy.

The 150-300ms latency is borderline acceptable but adds up in user-perceived delay, especially on mobile connections where network latency is already high.

Verdict: Good for toxicity screening. Not sufficient for custom content policies, spam detection, or brand-specific moderation rules.

Option 3: OpenAI Moderation API

How it works

OpenAI's free moderation endpoint classifies content across categories: hate, harassment, self-harm, sexual, violence, and more.

import openai

async def moderate_openai(text: str) -> str:
    response = await openai.moderations.create(input=text)
    result = response.results[0]

    if result.flagged:
        # Check which categories were flagged
        categories = result.categories
        if categories.violence or categories.hate:
            return "reject"
        return "review"
    return "approve"

Performance

| Metric | Value | |--------|-------| | Latency | 200-500ms | | Accuracy | ~80% F1 (on safety categories) | | False positive rate | 8-12% | | False negative rate | 12-18% | | Setup effort | Very low | | Cost | Free |

The problem

Like Perspective API, OpenAI Moderation covers predefined safety categories. If your moderation policy includes spam detection, self-promotion, misinformation, or platform-specific rules (e.g., "no recruitment posts in this community"), the moderation API doesn't cover those.

The 200-500ms latency is in the "noticeable but not terrible" range. But it pushes your total request time to 250-550ms once you add your own processing — getting close to the 500ms threshold where users start noticing.

Verdict: Excellent free option for safety-category moderation. Not customizable to platform-specific policies. Latency is borderline.

Option 4: GPT-4o Custom Moderation

How it works

Call GPT-4o (or GPT-4o-mini) with a custom prompt that describes your specific moderation policy.

async def moderate_gpt4o(text: str) -> str:
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": """You are a content moderator.
            Classify as: approve, review, or reject.

            Policy:
            - Reject: hate speech, threats, explicit content, scams
            - Review: self-promotion, borderline language, new account spam patterns
            - Approve: legitimate content, opinions, questions

            Respond with only the classification."""},
            {"role": "user", "content": text}
        ],
        temperature=0,
        max_tokens=10
    )
    return response.choices[0].message.content.strip()

Performance

| Metric | Value | |--------|-------| | Latency | 500-1500ms (P50 ~623ms) | | Accuracy | ~93% F1 | | False positive rate | 3-5% | | False negative rate | 5-8% | | Setup effort | Low (prompt engineering) | | Cost | ~$0.005/request ($7,500/mo at 50K/day) |

The problem

This is the highest-accuracy option. GPT-4o understands your custom policy, handles context, detects sarcasm, and catches obfuscation. But at 623ms P50, it blows your latency budget.

For a chat app: user types a message, hits send, waits 623ms before it appears. That's a bad experience.

For a content feed: user posts, sees a loading spinner for over half a second. Every post. Every time.

The cost is also significant. At 50K posts per day, you're spending $7,500/month on moderation alone.

Verdict: Best accuracy available, but too slow and too expensive for real-time pre-publish moderation at scale.

Option 5: Compiled ONNX Classifier

How it works

Use an LLM to teach a lightweight classifier your moderation policy offline. The classifier runs in production at sub-100ms, with LLM escalation only for low-confidence cases (~5% of traffic).

import httpx

async def moderate_compiled(text: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": text, "user_id": "user_123"}
            }
        )
        return response.json()
# {"decision": "approve", "confidence": 0.94, "latency_ms": 38, "stage": "classifier"}

For latency-critical applications, run the model locally:

from sparkient_edge import EdgePredictor

predictor = EdgePredictor.from_bundle("moderation.zip")
result = predictor.predict({"text": "Check out this amazing deal!"})
# EdgeDecision(decision="approve", confidence=0.92, latency_ms=3.2)

Performance

| Metric | Cloud API | Edge (Local) | |--------|-----------|-------------| | Latency (P50) | 33ms | 3ms | | Latency (P95) | 41ms | 8ms | | Accuracy (F1) | 91.5% | 91.5% | | False positive rate | 4-6% | 4-6% | | False negative rate | 5-8% | 5-8% | | Cost (50K/day) | $199/mo | One-time training |

The trade-off

Accuracy is slightly lower than GPT-4o (91.5% vs ~93% F1). The gap comes from genuinely ambiguous cases where an LLM's reasoning ability gives it an edge. The three-stage pipeline closes this gap to ~92% F1 by escalating uncertain cases to the LLM.

Verdict: Best balance of speed, accuracy, and cost for real-time moderation. Fast enough for any hot path. Accurate enough for production content moderation.

The Full Comparison

| Approach | Latency (P95) | Accuracy (F1) | Cost (50K/day) | Custom Policies | Real-Time UX | |----------|--------------|--------------|----------------|----------------|-------------| | Keyword filters | <1ms | ~55% | ~$0 | Yes (manual) | ✓ | | Perspective API | 300ms | ~75% | Free | No | Borderline | | OpenAI Moderation | 500ms | ~80% | Free | No | ✗ | | GPT-4o custom | 1,100ms | ~93% | $7,500/mo | Yes | ✗ | | Compiled classifier | 41ms | 91.5% | $199/mo | Yes | ✓ | | Edge classifier | 8ms | 91.5% | Training only | Yes | ✓ |

Only two approaches satisfy both requirements (real-time UX + custom policy support): keyword filters and compiled classifiers. But keyword filters have unacceptable accuracy. The compiled classifier is the only option that gives you all three: speed, accuracy, and customization.

Replacing an OpenAI Moderation Call

Here's a concrete before/after for a chat application:

Before: OpenAI Moderation + GPT-4o Fallback

async def moderate_message(message: str) -> dict:
    # Step 1: Quick check with free moderation API (~300ms)
    mod_result = await openai.moderations.create(input=message)
    if mod_result.results[0].flagged:
        return {"action": "reject", "reason": "safety_violation", "latency_ms": 300}

    # Step 2: Custom policy check with GPT-4o (~623ms)
    response = await openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Check for: spam, self-promotion, solicitation..."},
            {"role": "user", "content": message}
        ]
    )
    classification = response.choices[0].message.content.strip()
    # Total: 300ms + 623ms = ~923ms for non-flagged content
    return {"action": classification, "latency_ms": 923}

After: Compiled Classifier

import httpx

async def moderate_message(message: str) -> dict:
    async with httpx.AsyncClient() as client:
        response = await client.post(
            "https://api.sparkient.ai/api/v1/decide",
            headers={"Authorization": "Bearer YOUR_API_KEY"},
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": message}
            }
        )
        result = response.json()
        # Total: ~38ms for all content, all policies in one call
        return {
            "action": result["decision"],
            "confidence": result["confidence"],
            "latency_ms": result["latency_ms"]
        }

The before approach chains two API calls: moderation check (~300ms) then custom policy check (~623ms) = ~923ms total. The after approach makes one call (~38ms) that covers both safety categories and your custom policies.

Result: 923ms → 38ms. A 24x improvement in latency. Your chat messages now appear in under 50ms instead of nearly a second.

When to Combine Approaches

For maximum safety coverage, you can layer approaches:

async def moderate_layered(message: str) -> dict:
    # Layer 1: Keyword blocklist (instant, catches obvious violations)
    if matches_blocklist(message):
        return {"action": "reject", "source": "blocklist"}

    # Layer 2: Compiled classifier (fast, handles nuance)
    result = await sparkient_classify(message)

    # Layer 3: Only for low-confidence edge cases
    if result["confidence"] < 0.7:
        result = await llm_escalate(message)

    return result

This gives you:

• Instant rejection of known-bad content (blocklist)
• Fast, accurate moderation for the vast majority of content (classifier)
• LLM-quality decisions for genuinely ambiguous cases (escalation)

FAQ

Q: Is 91.5% F1 accurate enough for content moderation? For most platforms, yes. The 91.5% F1 from benchmarks means roughly 4-6% false positives and 5-8% false negatives. By comparison, human moderators typically achieve 80-85% agreement rates with each other on ambiguous content. The compiled classifier with LLM escalation for uncertain cases reaches ~92% F1, which is within the range of human performance.

Q: What about content in languages other than English? The text encoder used in the compiled model is trained on English. For multilingual moderation, you'd need to train with multilingual examples or use a multilingual base model. Sparkient's training pipeline supports this — provide multilingual training examples and the model will learn to handle them. For full multilingual coverage, an LLM-based approach currently has an advantage.

Q: Can I moderate images and videos with this approach? This article covers text moderation only. For image and video moderation, you'll need dedicated vision models (Google Cloud Vision, AWS Rekognition, or custom models). A common pattern is to moderate text with a compiled classifier and images with a vision API, combining the results. The text moderation is usually the latency bottleneck that this approach solves.

Q: What happens if a harmful message gets through (false negative)? No moderation system is 100% accurate. Build a feedback loop: provide a "report" button, review flagged content, and feed corrections back into training data for the next model version. The three-stage pipeline with LLM escalation reduces false negatives by sending uncertain cases to the LLM, but you should always have a human escalation path for user reports.

Ship moderation that's fast enough to be invisible. Start free — 5,000 credits, no credit card required.