Skip to content
← all posts
6 min readproduction

Mentiko Best Practices: Building Reliable Agent Chains

best-practicesreliabilityproductionpatterns

Mentiko Team

You can build an agent chain that works once. Getting it to work reliably -- across different inputs, under load, over weeks of scheduled runs -- requires deliberate design choices. These are the patterns that separate chains that demo well from chains that run in production without waking anyone up at 3am.

Keep agents focused

Each agent should do one thing. A "research, summarize, and format" agent is three agents pretending to be one. When it fails, you don't know which step failed. When you want to change the summary format, you risk breaking the research logic. When you want to reuse the researcher in a different chain, you can't -- it's coupled to the summarizer and formatter.

Split it:

{
  "agents": [
    {
      "name": "researcher",
      "prompt": "Research {TOPIC}. Output structured findings with sources.",
      "triggers": ["chain:start"],
      "emits": ["research:complete"]
    },
    {
      "name": "summarizer",
      "prompt": "Condense the research into 3 key findings. Preserve source citations.",
      "triggers": ["research:complete"],
      "emits": ["summary:complete"]
    },
    {
      "name": "formatter",
      "prompt": "Format the summary as {OUTPUT_FORMAT}. Apply {BRAND_VOICE} guidelines.",
      "triggers": ["summary:complete"],
      "emits": ["chain:complete"]
    }
  ]
}

Three agents, each with a clear responsibility. You can swap the formatter without touching the researcher. You can reuse the summarizer in a different chain. When the chain fails at the summarizer step, you know exactly where to look.

The rule of thumb: if your agent prompt contains the word "and" describing two distinct tasks, split it into two agents.

Name events descriptively

Events are the connective tissue of your chain. Generic names like step-1:done or agent:finished are meaningless when you're debugging at 2am.

Use the pattern noun:verb or domain:action:

research:complete        (not step1:done)
draft:ready              (not output:generated)
review:approved          (not check:passed)
payment:failed           (not error:occurred)

When you read a chain definition, the event names should tell the story: research completes, draft is ready, review approves, content publishes. No ambiguity, no need to cross-reference which agent emits what.

Write prompts that fail visibly

A bad prompt doesn't throw an error. It produces bad output silently. Your chain completes successfully, but the summarizer hallucinated statistics and the formatter applied the wrong brand voice. You don't find out until a customer reads the output.

Write prompts that make failures obvious:

Bad:  "Summarize the research."
Good: "Summarize the research into exactly 3 bullet points.
       Each bullet must cite a specific source URL from the input.
       If fewer than 2 source URLs exist in the input, output
       ERROR: INSUFFICIENT_SOURCES and stop."

The good prompt has validation built into the instructions. If the input is insufficient, the agent produces an error string instead of a plausible-looking hallucination. Downstream agents can check for ERROR: prefixes and route to error handlers.

Other patterns for visible failure:

  • Require structured output (JSON with required fields) so missing data is syntactically obvious
  • Include a confidence score in the prompt instructions: "Rate your confidence 1-10. If below 6, prefix output with LOW_CONFIDENCE."
  • Ask the agent to list its sources or reasoning steps so you can verify them

Use variables for everything that changes

Hardcoded values in prompts are a maintenance problem. When you want to switch models, change the output length, or adjust a threshold, you're editing prompt text -- and any edit to a prompt risks changing the agent's behavior in unexpected ways.

Extract everything configurable into variables:

{
  "variables": {
    "TOPIC": "",
    "MODEL": "gpt-5.4",
    "MAX_WORDS": "1200",
    "BRAND_VOICE": "professional, concise",
    "OUTPUT_FORMAT": "markdown",
    "CONFIDENCE_THRESHOLD": "6"
  },
  "agents": [
    {
      "name": "writer",
      "prompt": "Write a {MAX_WORDS}-word article about {TOPIC} in {OUTPUT_FORMAT} format. Use a {BRAND_VOICE} tone. If your confidence is below {CONFIDENCE_THRESHOLD}/10, prefix with LOW_CONFIDENCE.",
      "triggers": ["chain:start"],
      "emits": ["chain:complete"]
    }
  ]
}

Now you can adjust the word count, tone, format, and confidence threshold without editing the prompt structure. Different environments can use different variable values (staging with MAX_WORDS=200 for faster testing, production with MAX_WORDS=1200). The chain definition stays identical.

Add error handling before you need it

The most common mistake: building the happy path, deploying, and adding error handling after the first production failure. By then, the failure has already caused damage -- missed output, broken downstream systems, silent data loss.

Add retries and error events from the start:

{
  "name": "researcher",
  "prompt": "Research {TOPIC}...",
  "triggers": ["chain:start"],
  "emits": ["research:complete"],
  "on_error": "research:failed",
  "retry": {
    "max_attempts": 3,
    "backoff": "exponential",
    "initial_delay_ms": 2000
  }
}

Every agent that calls an external service should have retries. Every agent should have an on_error event. Whether you route that error event to a recovery chain, a notification, or just a log entry is secondary -- the important thing is that failures are captured as events instead of disappearing silently.

Test with real inputs, not toy data

A chain that works with "test topic" as input might fail spectacularly with real input. Real topics are longer, messier, and more ambiguous. Real data has edge cases your test data doesn't.

Build a set of test inputs that represent your actual usage:

# Run with a variety of real inputs
mentiko run content-pipeline --var TOPIC="kubernetes networking troubleshooting"
mentiko run content-pipeline --var TOPIC="Q1 2026 revenue analysis for Series B deck"
mentiko run content-pipeline --var TOPIC=""  # empty input -- should fail gracefully
mentiko run content-pipeline --var TOPIC="$(cat very-long-input.txt)"  # large input

Test the edges. Each reveals a different failure mode that won't show up with "test topic."

Set timeouts

An agent without a timeout can hang forever. Without one, a stuck API call or infinite loop blocks downstream agents and consumes resources indefinitely.

{
  "name": "researcher",
  "prompt": "Research {TOPIC}...",
  "triggers": ["chain:start"],
  "emits": ["research:complete"],
  "timeout_seconds": 120,
  "on_timeout": "research:timeout"
}

Set timeouts based on observed execution times. If your researcher typically completes in 30 seconds, a 120-second timeout gives 4x headroom for slow runs while preventing indefinite hangs.

The on_timeout event lets you handle timeouts differently from errors. A timeout might route to a retry with a simpler prompt, while an error might route to a completely different agent.

Version your chain definitions

Chain definitions are code. Store them in git, review changes in PRs, tag releases. Write commit messages that explain why you changed the chain, not just what changed.

When a chain breaks in production, git log chains/content-pipeline.json shows exactly what changed and when. git diff between the working version and the broken version reveals the precise edit. Roll back with git revert and you're running the previous version immediately.

Monitor in production

A chain that works on day one might degrade over time. Model behavior changes. API rate limits tighten. Input patterns shift. Track success rate, duration percentiles, token usage, and retry/fallback rates for every chain.

{
  "monitoring": {
    "error_rate_threshold": 0.05,
    "duration_p95_threshold_ms": 60000,
    "token_budget": 50000,
    "alert_channels": ["slack:chain-alerts"]
  }
}

These thresholds catch degradation before it becomes a user-visible problem. A chain that's technically succeeding but hitting its fallback path 40% of the time is a chain with a problem.

The checklist

Before deploying any chain to production:

  • Each agent does exactly one thing
  • Events are named descriptively
  • Prompts include validation criteria for visible failures
  • All configurable values use variables
  • Every agent has retries and an on_error event
  • Timeouts are set on every agent
  • Tested with real inputs including edge cases
  • Chain definition is committed to git
  • Monitoring thresholds are configured

None of these are complicated. They're just the difference between a chain that works in a demo and one that runs for months without intervention.

For error handling patterns in detail, see Error Handling Patterns for AI Agent Chains. For deployment considerations, see 10 Things to Know Before Deploying AI Agents.

Get new posts in your inbox

No spam. Unsubscribe anytime.

More from the blog

7 min readproduction

Error Handling Patterns for AI Agent Chains

4 min readproduction

10 Things to Know Before Deploying AI Agents in Production