How to Evaluate Content Moderation in a Next.js App

TL;DR

You can evaluate content moderation in a Next.js app in three steps: create a decision type, add a server-side route that calls /decide, and wire it into a test flow. For local or offline inference, use the edge SDK and benchmark it on the deployment hardware.

The Problem

Your Next.js app accepts user input — comments, posts, messages, reviews, bios — and you need to moderate it before it reaches other users or your database.

The options aren't great:

• No moderation — one toxic comment in a public feed and you're fighting a PR fire
• Manual moderation — doesn't scale past a few hundred submissions per day
• Keyword filters — trivially bypassed ("fr33 m0ney" passes most blocklists)
• Direct LLM calls — flexible, but their measured quality, latency, token use, and provider limits may not fit every form submission

What you want is a moderation check that's fast enough to run inline (before the form submission completes), accurate enough to catch nuanced content, and cheap enough to run on every single submission.

What You'll Build

A Next.js app with two moderation approaches:

1. API Route approach — a server-side route that checks content before saving it through Sparkient's REST API
2. Local sidecar approach — a local service that loads an exported edge bundle and removes the cloud dependency

Both return a structured decision: approve, review, or reject.

Prerequisites

• A Next.js 14+ app (App Router)
• A Sparkient account (free tier works — 5,000 credits, no credit card)
• A trained content moderation decision type (you'll set this up in Step 1)

Step 1: Create Your Moderation Decision Type

Log into the Sparkient dashboard and create a new decision type:

• Name: content-moderation
• Options: approve, review, reject
• Input schema:

{
  "type": "object",
  "properties": {
    "text": {"type": "string"},
    "user_id": {"type": "string"},
    "context": {"type": "string"}
  },
  "required": ["text"]
}

Add any CEL rules for hard business logic. For example:

ctx.text.size() > 10000

This rule rejects any text over 10,000 characters on the deterministic rules path — no classifier needed.

Click Train to compile your model. Training time depends on the data and available compute, and the status response reports the current attempt, stage, and heartbeat. When example generation is enabled, Sparkient's teacher LLM creates synthetic examples from your policy definition before the pipeline trains and exports a compiled ONNX classifier.

Once training completes, grab your API key from the dashboard.

Step 2: API Route Approach (Server-Side)

This is the most common pattern: check content in your API route before saving it to the database.

Create the moderation utility

// src/lib/moderation.ts

interface ModerationResult {
  decision: "approve" | "review" | "reject";
  confidence: number;
  latency_ms: number;
  stage: string;
}

export async function moderateContent(
  text: string,
  userId: string,
  context: string = "general"
): Promise<ModerationResult> {
  const response = await fetch("https://api.sparkient.ai/api/v1/decide", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SPARKIENT_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      decision_type: "content-moderation",
      input: { text, user_id: userId, context },
    }),
  });

  if (!response.ok) {
    throw new Error(`Moderation API error: ${response.status}`);
  }

  return response.json();
}

Use it in your API route

// src/app/api/comments/route.ts

import { NextRequest, NextResponse } from "next/server";
import { moderateContent } from "@/lib/moderation";

export async function POST(request: NextRequest) {
  const { text, userId, postId } = await request.json();

  // Validate input
  if (!text || !userId || !postId) {
    return NextResponse.json(
      { error: "Missing required fields" },
      { status: 400 }
    );
  }

  // Moderate content before saving
  const moderation = await moderateContent(text, userId);

  if (moderation.decision === "reject") {
    return NextResponse.json(
      {
        error: "Your comment was flagged by our content policy.",
        decision: moderation.decision,
      },
      { status: 422 }
    );
  }

  if (moderation.decision === "review") {
    // Save to database with a "pending_review" status
    await saveComment({
      text,
      userId,
      postId,
      status: "pending_review",
      moderationConfidence: moderation.confidence,
    });

    return NextResponse.json({
      message: "Your comment has been submitted for review.",
      decision: moderation.decision,
    });
  }

  // Approved — save and publish immediately
  await saveComment({
    text,
    userId,
    postId,
    status: "published",
    moderationConfidence: moderation.confidence,
  });

  return NextResponse.json({
    message: "Comment posted.",
    decision: moderation.decision,
  });
}

async function saveComment(data: Record<string, unknown>) {
  // Your database logic here (Prisma, Drizzle, etc.)
  console.log("Saving comment:", data);
}

Wire it into your frontend

// src/components/CommentForm.tsx
"use client";

import { useState } from "react";

export function CommentForm({ postId }: { postId: string }) {
  const [text, setText] = useState("");
  const [status, setStatus] = useState<string | null>(null);
  const [submitting, setSubmitting] = useState(false);

  async function handleSubmit(e: React.FormEvent) {
    e.preventDefault();
    setSubmitting(true);
    setStatus(null);

    const response = await fetch("/api/comments", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text, userId: "current-user", postId }),
    });

    const result = await response.json();
    setStatus(result.message || result.error);
    if (response.ok) setText("");
    setSubmitting(false);
  }

  return (
    <form onSubmit={handleSubmit}>
      <textarea
        value={text}
        onChange={(e) => setText(e.target.value)}
        placeholder="Write a comment..."
        rows={3}
      />
      <button type="submit" disabled={submitting}>
        {submitting ? "Checking..." : "Post Comment"}
      </button>
      {status && <p>{status}</p>}
    </form>
  );
}

Add your environment variable

# .env.local
SPARKIENT_API_KEY=sk_your_api_key_here

Every comment submission now runs through the moderation route before hitting the database. Inspect the returned stage and latency_ms, then load-test the complete form path.

Step 3: Local Sidecar Approach (Optional)

For teams that want to moderate content at the edge — closer to the user, with no cloud API dependency — Sparkient's edge SDK runs the same compiled model locally.

This approach uses Next.js middleware to intercept POST requests before they reach your API routes.

Install the edge SDK

pip install sparkient-edge

For Next.js middleware, you'll need a small edge-compatible service that runs the predictor. One approach: run a lightweight sidecar that your middleware calls locally.

# sidecar/moderation_server.py
from fastapi import FastAPI
from sparkient_edge import EdgePredictor

app = FastAPI()
predictor = EdgePredictor.from_bundle("moderation.zip")

@app.post("/moderate")
async def moderate(payload: dict):
    result = predictor.predict({
        "text": payload["text"],
        "user_id": payload.get("user_id", "unknown"),
        "context": payload.get("context", "general")
    })
    return {
        "decision": result.decision,
        "confidence": result.confidence,
        "stage": result.stage
    }

Then in your Next.js middleware:

// src/middleware.ts

import { NextRequest, NextResponse } from "next/server";

export async function middleware(request: NextRequest) {
  // Only moderate POST requests to content endpoints
  if (request.method !== "POST") return NextResponse.next();

  const body = await request.json();
  if (!body.text) return NextResponse.next();

  try {
    // Call the local moderation sidecar
    const moderation = await fetch("http://localhost:8000/moderate", {
      method: "POST",
      headers: { "Content-Type": "application/json" },
      body: JSON.stringify({ text: body.text, user_id: body.userId }),
    });

    const result = await moderation.json();

    if (result.decision === "reject") {
      return NextResponse.json(
        { error: "Content rejected by moderation policy." },
        { status: 422 }
      );
    }
  } catch {
    // If the sidecar is down, fail open (let the request through)
    // In production, you might want to fail closed instead
    console.error("Moderation sidecar unavailable, failing open");
  }

  return NextResponse.next();
}

export const config = {
  matcher: ["/api/comments/:path*", "/api/posts/:path*"],
};

The edge approach removes the cloud dependency. Its latency depends on the model, hardware, process boundary, and load; the trade-off is managing the local sidecar.

Which Approach Should You Use?

| | Cloud API Route | Local Sidecar | |---|---|---| | Latency | Network and stage dependent; 41ms batch-average time per item in the controlled synthetic moderation run | Model, hardware, process boundary, and load dependent | | Setup effort | Central cloud integration | Local packaging and operations | | Cloud dependency | Yes (Sparkient API) | No | | Model updates | Automatic | Manual (redeploy bundle) | | Best for | Central management | Offline/air-gapped deployments or a measured local-runtime requirement |

Start with the API route approach. It is simpler and keeps model deployment centralised. Move to edge when offline operation or measured network latency justifies local model operations.

FAQ

Does the moderation check add noticeable latency to form submissions?

Measure the full submission path. The controlled synthetic moderation run reports 41ms batch-average time per item for the compiled stage, not per-request p95; cloud networking, optional escalation, local hardware, database work, and the rest of the route determine what the user experiences.

What if the Sparkient API is down?

Build a fallback. The simplest approach: if the API returns an error or times out, save the content with a pending_review status and let a human check it later. Never silently publish unmoderated content in a fail-open scenario without a review queue behind it.

Can I moderate content in languages other than English?

Yes. Sparkient's compiled models use semantic text embeddings, which support multilingual text. Define your moderation policy to cover the languages your platform supports, and the teacher LLM will generate training examples in those languages. Accuracy may vary by language — test on your actual content.

How do I handle appeals?

Moderation decisions include a confidence score. Store this alongside the decision in your database. When a user appeals, you can prioritize low-confidence rejections for human review. You can also re-run the content through the LLM escalation path for a second opinion.

Ready to evaluate moderation in your Next.js app? Create a free Sparkient account — 5,000 credits, no credit card — and test one decision type against representative held-out examples before routing production traffic.