My AI Agent Takes 3 Seconds Per Decision — How to Get Under 100ms

TL;DR

AI agents built on LangChain, CrewAI, or custom frameworks typically make 3-5 LLM calls per loop iteration, taking 3-9 seconds per decision cycle. Many of those calls are repetitive classification decisions (route this ticket, moderate this message, approve this request) that return the same structured options every time. Replace those with compiled classifiers running at sub-100ms latency, and your agent loop drops from seconds to milliseconds for the decisions that matter most.

The Problem: Every Decision Is an LLM Call

Here's a typical agent loop. The agent receives a user message, decides what to do, calls a tool, evaluates the result, and decides the next step. Each "decide" step is an LLM call:

# Typical agent loop — 3 LLM calls minimum
async def agent_step(message: str):
    # Step 1: Classify intent (LLM call — 500-1500ms)
    intent = await llm.classify(message, options=["question", "complaint", "request"])

    # Step 2: Route to handler (LLM call — 500-1500ms)
    handler = await llm.route(intent, context=get_context())

    # Step 3: Generate response (LLM call — 500-1500ms)
    response = await llm.generate(handler.prompt, context=handler.context)

    return response

Three LLM calls. At best, 1.5 seconds total (Groq-speed). At worst, 4.5 seconds (standard APIs). Add retries for rate limits, and you're looking at 3-9 seconds per agent cycle.

Your users are waiting.

Which Decisions Can Be Compiled?

Not every LLM call in your agent is the same. Some genuinely need the full reasoning power of a large language model. Others are doing the same classification over and over with the same set of options.

Here's how to tell the difference:

| Signal | Compilable | Needs Full LLM | |--------|-----------|----------------| | Fixed set of output options | ✅ | | | Structured input (JSON, form data) | ✅ | | | Same decision made thousands of times | ✅ | | | Decision logic is well-understood | ✅ | | | Requires multi-step reasoning | | ✅ | | Output is free-form text generation | | ✅ | | Novel situations every time | | ✅ | | Needs access to live external context | | ✅ |

In most agent architectures, Steps 1 and 2 are compilable. Intent classification and routing are structured decisions with fixed options. Step 3 — generating a response — genuinely needs the LLM.

The Fix: Compile the Repetitive Decisions

The pattern is straightforward: use an LLM to teach the decision once, then use a compiled model to make it millions of times.

For the agent loop above, you'd compile the intent classification and routing decisions into sub-100ms classifiers, and keep the LLM only for response generation:

import httpx

SPARKIENT_API = "https://api.sparkient.ai/api/v1"
HEADERS = {"Authorization": "Bearer YOUR_API_KEY"}

async def agent_step(message: str):
    # Step 1: Classify intent — compiled model, ~38ms
    intent_resp = await httpx.AsyncClient().post(
        f"{SPARKIENT_API}/decide",
        headers=HEADERS,
        json={
            "decision_type_id": "agent-intent-classification",
            "input": {"text": message, "channel": "chat"}
        }
    )
    intent = intent_resp.json()["decision"]  # "question", "complaint", or "request"

    # Step 2: Route to handler — compiled model, ~35ms
    route_resp = await httpx.AsyncClient().post(
        f"{SPARKIENT_API}/decide",
        headers=HEADERS,
        json={
            "decision_type_id": "agent-routing",
            "input": {"intent": intent, "user_tier": "premium", "text": message}
        }
    )
    handler = route_resp.json()["decision"]  # "faq_bot", "human_agent", "escalation"

    # Step 3: Generate response — still needs LLM, 500-1500ms
    response = await llm.generate(handler_prompt(handler), context=message)

    return response

Before: 1.5-4.5 seconds (3 LLM calls) After: 570-1570ms (2 compiled decisions + 1 LLM call)

You've cut the total latency by 40-65%, and the two classification steps now have predictable, consistent latency instead of the variance that comes with LLM APIs.

Going Faster: Parallel Compiled Decisions

Since the compiled decisions are independent and fast, you can run them in parallel:

async def agent_step(message: str):
    async with httpx.AsyncClient() as client:
        # Run both classifications in parallel — total ~40ms
        intent_task = client.post(
            f"{SPARKIENT_API}/decide",
            headers=HEADERS,
            json={
                "decision_type_id": "agent-intent-classification",
                "input": {"text": message}
            }
        )
        moderation_task = client.post(
            f"{SPARKIENT_API}/decide",
            headers=HEADERS,
            json={
                "decision_type_id": "content-moderation",
                "input": {"text": message}
            }
        )

        intent_resp, mod_resp = await asyncio.gather(
            intent_task, moderation_task
        )

    intent = intent_resp.json()
    moderation = mod_resp.json()

    if moderation["decision"] == "reject":
        return {"error": "Message blocked", "reason": moderation["reason_codes"]}

    # Only call the LLM for response generation
    response = await llm.generate(route(intent["decision"]), context=message)
    return response

Now your agent does intent classification and content moderation in parallel in ~40ms, then only calls the LLM for the part that actually needs it.

MCP Integration: Agents That Create Their Own Decision Types

If your agent framework supports MCP (Model Context Protocol), your agent can create and train decision types directly — no dashboard needed. Sparkient's MCP server exposes tools for the full lifecycle:

// Claude Desktop / Cursor MCP config
{
  "mcpServers": {
    "sparkient": {
      "url": "https://mcp.sparkient.ai/mcp",
      "headers": {
        "Authorization": "Bearer YOUR_API_KEY"
      }
    }
  }
}

With this configured, an AI coding agent can:

1. Create a decision type using create_decision_type — define the name, options, and description
2. Generate training examples using generate_examples — Sparkient's LLM teacher creates synthetic data
3. Train the model using train_model — kicks off async training with a chosen preset
4. Make decisions using make_decision — call the compiled model at sub-100ms

Here's what that looks like in practice. An agent working on a support ticket system might reason:

"This codebase routes tickets by calling GPT-4o on every request. The routing options are always the same: billing, technical, account, general. I'll create a compiled decision type for this."

The agent then calls create_decision_type with name: "ticket_routing", options: ["billing", "technical", "account", "general"], followed by generate_examples and train_model. Once trained, every ticket routing decision takes ~38ms instead of 500-1500ms.

The Audit: Find Your Compilable Decisions

Here's a quick way to audit your agent's decision points:

1. Log every LLM call your agent makes for 24 hours
2. Categorize each call: Is it classification, routing, extraction, or generation?
3. Check the output space: Does it always return one of a fixed set of options?
4. Count frequency: How many times per day is this exact decision type made?

Any call that is classification/routing, has fixed options, and runs hundreds+ times per day is a compilation candidate.

Across benchmarked domains, compiled models achieve 89-95% F1 scores at sub-100ms latency. GPT-4o achieves similar accuracy at ~623ms. For classification decisions, the accuracy is comparable — but the latency and cost are not.

When to Keep the Full LLM

Be honest about what compilation doesn't do well:

• Open-ended generation — writing emails, crafting responses, summarizing documents. These need the LLM.
• Novel reasoning — decisions your system has never seen before, with no pattern to learn from.
• Context-heavy decisions — where the decision depends on a large, changing context window (full conversation history, live database state).

The sweet spot for compilation is the boring, repetitive, structured decisions that your agent makes thousands of times with the same options. Those are the ones burning your latency budget.

FAQ

Q: How accurate are compiled models compared to the full LLM? Across content moderation, phishing detection, and other benchmarked domains, compiled models achieve 89-95% F1 scores. GPT-4o achieves similar accuracy on the same tasks but at ~623ms per decision vs under 100ms. For edge cases where the compiled model is uncertain, Sparkient automatically escalates to an LLM fallback.

Q: Can I use this with LangChain or CrewAI? Yes. Both frameworks support custom tool definitions. Define a tool that calls Sparkient's /decide endpoint, and the agent will use it like any other tool. For MCP-compatible frameworks, add the Sparkient MCP server directly to your config — no custom tool code needed.

Q: How much training data do I need? None. Sparkient generates synthetic training data using an LLM teacher based on your decision type definition. You define the options and describe what the decision means in plain English, and the system generates examples, trains the model, and deploys it. A training run costs 2,000 credits.

Q: What if my agent's decision types change over time? Retrain. If you add a new routing option or the decision logic shifts, trigger a new training run. The pipeline generates fresh synthetic data, trains a new model, and deploys it — typically in a few minutes. Your agent keeps running on the old model until the new one is deployed.

Your agent doesn't need 3 seconds to classify an intent. Start with the free tier — 5,000 credits, no credit card — and compile your first decision type in minutes.