All Articles
guide

How to Add Content Moderation to a Next.js App in 5 Minutes

Step-by-step guide to adding real-time content moderation to your Next.js app with Sparkient — both API route and edge middleware approaches.

Peter Dobson9 July 20268 min read

TL;DR

You can add content moderation to a Next.js app in three steps: create a Sparkient decision type for moderation, add a server-side API route that calls /decide, and wire it into your form submission. The API route approach takes ~5 minutes. For inline moderation on every request, you can also use Next.js middleware with Sparkient's edge SDK for sub-10ms decisions.

The Problem

Your Next.js app accepts user input — comments, posts, messages, reviews, bios — and you need to moderate it before it reaches other users or your database.

The options aren't great:

  • No moderation — one toxic comment in a public feed and you're fighting a PR fire
  • Manual moderation — doesn't scale past a few hundred submissions per day
  • Keyword filters — trivially bypassed ("fr33 m0ney" passes most blocklists)
  • Direct LLM calls — accurate, but 500-1500ms per request adds noticeable lag to form submissions, and costs scale linearly with volume

What you want is a moderation check that's fast enough to run inline (before the form submission completes), accurate enough to catch nuanced content, and cheap enough to run on every single submission.

What You'll Build

A Next.js app with two moderation approaches:

  1. API Route approach — a server-side route that checks content before saving it, using Sparkient's REST API (~38ms latency)
  2. Edge Middleware approach — a middleware function that checks content on every POST request, using Sparkient's edge SDK (~3ms latency, fully offline)

Both return a structured decision: approve, review, or reject.

Prerequisites

  • A Next.js 14+ app (App Router)
  • A Sparkient account (free tier works — 5,000 credits, no credit card)
  • A trained content moderation decision type (you'll set this up in Step 1)

Step 1: Create Your Moderation Decision Type

Log into the Sparkient dashboard and create a new decision type:

  • Name: content-moderation
  • Options: approve, review, reject
  • Input schema:
json
{
  "text": "string",
  "user_id": "string",
  "context": "string"
}

Add any CEL rules for hard business logic. For example:

ctx.text.size() > 10000

This rule instantly rejects any text over 10,000 characters — no ML needed.

Click Train to compile your model. This takes a few minutes. Sparkient's teacher LLM generates synthetic training examples from your policy definition, then trains and exports a compiled ONNX classifier.

Once training completes, grab your API key from the dashboard.

Step 2: API Route Approach (Server-Side)

This is the most common pattern: check content in your API route before saving it to the database.

Create the moderation utility

typescript
// src/lib/moderation.ts

interface ModerationResult {
  decision: "approve" | "review" | "reject";
  confidence: number;
  latency_ms: number;
  stage: string;
}

export async function moderateContent(
  text: string,
  userId: string,
  context: string = "general"
): Promise<ModerationResult> {
  const response = await fetch("https://api.sparkient.ai/api/v1/decide", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SPARKIENT_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      decision_type_id: "content-moderation",
      input: { text, user_id: userId, context },
    }),
  });

  if (!response.ok) {
    throw new Error(`Moderation API error: ${response.status}`);
  }

  return response.json();
}

Use it in your API route

typescript
// src/app/api/comments/route.ts

import { NextRequest, NextResponse } from "next/server";
import { moderateContent } from "@/lib/moderation";

export async function POST(request: NextRequest) {
  const { text, userId, postId } = await request.json();

  // Validate input
  if (!text || !userId || !postId) {
    return NextResponse.json(
      { error: "Missing required fields" },
      { status: 400 }
    );
  }

  // Moderate content before saving
  const moderation = await moderateContent(text, userId);

  if (moderation.decision === "reject") {
    return NextResponse.json(
      {
        error: "Your comment was flagged by our content policy.",
        decision: moderation.decision,
      },
      { status: 422 }
    );
  }

  if (moderation.decision === "review") {
    // Save to database with a "pending_review" status
    await saveComment({
      text,
      userId,
      postId,
      status: "pending_review",
      moderationConfidence: moderation.confidence,
    });

    return NextResponse.json({
      message: "Your comment has been submitted for review.",
      decision: moderation.decision,
    });
  }

  // Approved — save and publish immediately
  await saveComment({
    text,
    userId,
    postId,
    status: "published",
    moderationConfidence: moderation.confidence,
  });

  return NextResponse.json({
    message: "Comment posted.",
    decision: moderation.decision,
  });
}

async function saveComment(data: Record<string, unknown>) {
  // Your database logic here (Prisma, Drizzle, etc.)
  console.log("Saving comment:", data);
}

Wire it into your frontend

typescript
// src/components/CommentForm.tsx
"use client";

import { useState } from "react";

export function CommentForm({ postId }: { postId: string }) {
  const [text, setText] = useState("");
  const [status, setStatus] = useState<string | null>(null);
  const [submitting, setSubmitting] = useState(false);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    setSubmitting(true);
    setStatus(null);

    const response = await fetch("/api/comments", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text, userId: "current-user", postId }),
    });

    const result = await response.json();
    setStatus(result.message || result.error);
    if (response.ok) setText("");
    setSubmitting(false);
  }

  return (
    <form onSubmit={handleSubmit}>
      <textarea
        value={text}
        onChange={(e) => setText(e.target.value)}
        placeholder="Write a comment..."
        rows={3}
      />
      <button type="submit" disabled={submitting}>
        {submitting ? "Checking..." : "Post Comment"}
      </button>
      {status && <p>{status}</p>}
    </form>
  );
}

Add your environment variable

bash
# .env.local
SPARKIENT_API_KEY=sk_your_api_key_here

That's it. Every comment submission now runs through moderation in ~38ms before hitting your database.

Step 3: Edge Middleware Approach (Optional)

For teams that want to moderate content at the edge — closer to the user, with no cloud API dependency — Sparkient's edge SDK runs the same compiled model locally.

This approach uses Next.js middleware to intercept POST requests before they reach your API routes.

Install the edge SDK

bash
pip install sparkient-edge

For Next.js middleware, you'll need a small edge-compatible service that runs the predictor. One approach: run a lightweight sidecar that your middleware calls locally.

python
# sidecar/moderation_server.py
from fastapi import FastAPI
from sparkient_edge import EdgePredictor

app = FastAPI()
predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/moderate")
async def moderate(payload: dict):
    result = predictor.predict({
        "text": payload["text"],
        "user_id": payload.get("user_id", "unknown"),
        "context": payload.get("context", "general")
    })
    return {
        "decision": result.decision,
        "confidence": result.confidence,
        "latency_ms": result.latency_ms
    }

Then in your Next.js middleware:

typescript
// src/middleware.ts

import { NextRequest, NextResponse } from "next/server";

export async function middleware(request: NextRequest) {
  // Only moderate POST requests to content endpoints
  if (request.method !== "POST") return NextResponse.next();

  const body = await request.json();
  if (!body.text) return NextResponse.next();

  try {
    // Call the local moderation sidecar
    const moderation = await fetch("http://localhost:8000/moderate", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text: body.text, user_id: body.userId }),
    });

    const result = await moderation.json();

    if (result.decision === "reject") {
      return NextResponse.json(
        { error: "Content rejected by moderation policy." },
        { status: 422 }
      );
    }
  } catch {
    // If the sidecar is down, fail open (let the request through)
    // In production, you might want to fail closed instead
    console.error("Moderation sidecar unavailable, failing open");
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/api/comments/:path*", "/api/posts/:path*"],
};

The edge approach gives you sub-10ms moderation with zero cloud dependencies. The trade-off is managing the sidecar process.

Which Approach Should You Use?

| | API Route | Edge Middleware | |---|---|---| | Latency | ~38ms | ~3ms | | Setup effort | 5 minutes | 30 minutes | | Cloud dependency | Yes (Sparkient API) | No | | Model updates | Automatic | Manual (redeploy bundle) | | Best for | Most teams | Offline/air-gapped, ultra-low latency |

Start with the API route approach. It's simpler, keeps your model updates automatic, and 38ms is fast enough for virtually all user-facing form submissions. Move to edge if you need offline capability or sub-10ms latency.

FAQ

Does the moderation check add noticeable latency to form submissions?

At 38ms (API route) or 3ms (edge), the moderation check is imperceptible to users. A typical form submission involves database writes, image processing, and notification dispatch — each of which takes longer than the moderation step.

What if the Sparkient API is down?

Build a fallback. The simplest approach: if the API returns an error or times out, save the content with a pending_review status and let a human check it later. Never silently publish unmoderated content in a fail-open scenario without a review queue behind it.

Can I moderate content in languages other than English?

Yes. Sparkient's compiled models use semantic text embeddings, which support multilingual text. Define your moderation policy to cover the languages your platform supports, and the teacher LLM will generate training examples in those languages. Accuracy may vary by language — test on your actual content.

How do I handle appeals?

Moderation decisions include a confidence score. Store this alongside the decision in your database. When a user appeals, you can prioritize low-confidence rejections for human review. You can also re-run the content through the LLM escalation path for a second opinion.

Ready to add moderation to your Next.js app? Create a free Sparkient account — 5,000 credits, no credit card — and have moderation running in production in under 10 minutes.

Ready to get started?

Start with 5,000 free credits and 250 decisions. No credit card required.

Start Free