10 Things to Know Before Deploying AI Agents in Production
Mentiko Team
Running AI agents in a demo is easy. Running them in production -- reliably, securely, affordably -- is where teams get surprised. Here are ten things we've learned from building Mentiko and watching early access teams deploy their first chains.
1. Agents fail silently
An agent that returns a confident-sounding but wrong answer looks like success in your logs. Unlike traditional software where errors throw exceptions, LLM-based agents fail by producing plausible garbage.
Fix: Add validation agents to your chains. A fact-checker after a researcher. A code linter after a code generator. Trust but verify.
2. Costs multiply faster than you expect
A 4-agent chain where each agent makes 3 LLM calls = 12 API calls per run. At $0.01 per call, that's $0.12 per run. Run it 100 times a day and you're at $360/month in API costs alone -- before platform fees.
Fix: Monitor per-run costs from day one. Set budget alerts. Use cheaper models for simple tasks (classification, formatting) and expensive models only for reasoning.
3. Retry logic needs a ceiling
When an agent fails, the natural instinct is to retry. But retrying an LLM call that failed because the prompt is wrong will fail the same way every time. And retrying an agent that's hitting rate limits will make rate limiting worse.
Fix: Retry with exponential backoff, maximum 3 attempts. After 3 failures, route to a fallback agent or alert a human. Never retry indefinitely.
4. Prompts are your most fragile dependency
A model update from your LLM provider can change how your prompts behave. A prompt that worked perfectly with Claude might produce different output with Claude. This isn't a bug -- it's the nature of language models.
Fix: Version control your prompts (Mentiko stores them in JSON chain files). Test chains after model updates. Keep a "known good" model version pinned for critical chains.
5. You need observability from day one
When a 4-agent chain produces wrong output, which agent made the mistake? Without per-agent logging, you're debugging a black box.
Fix: Log every agent's input, output, and execution time. Mentiko does this automatically with per-agent activity capture. For custom setups, instrument every handoff.
6. Secrets management is non-negotiable
Your agents need API keys, database credentials, and access tokens. Hardcoding them in chain definitions is the number one security mistake teams make.
Fix: Use a secrets vault (Mentiko's is AES-256-GCM encrypted). Inject secrets at runtime via environment variables. Never commit secrets to chain JSON files.
7. Agent outputs need schemas
If agent A's output is "a summary" and agent B expects "a JSON object with a summary field," the chain breaks. Unstructured outputs are the leading cause of chain failures.
Fix: Define output schemas for each agent. Use structured output modes when your LLM supports them. Validate outputs before passing to the next agent.
8. Scheduling is harder than it looks
"Run this chain every morning at 9am" sounds simple. Until: timezone handling, daylight saving time, overlapping runs when a chain takes longer than the interval, and handling missed runs after downtime.
Fix: Use a proper scheduler, not a bare cron job. Mentiko's scheduler handles timezone awareness, overlap prevention, and missed-run detection.
9. Multi-tenancy can't be retrofitted
If you're building agent orchestration for a team (or multiple teams), data isolation needs to be designed in from the start. Retrofitting multi-tenancy onto a single-tenant system is one of the most expensive engineering projects you can undertake.
Fix: Choose a platform that was built multi-tenant from day one. Or accept that your homegrown solution will be single-tenant forever.
10. Start with the chain, not the agent
Most teams start by building the perfect agent. They spend weeks tuning prompts, adding tools, optimizing context windows. Then they realize they need orchestration and have to refactor everything.
Fix: Start with the chain definition. What's the workflow? What are the handoffs? What events connect the steps? Then build agents to fill each slot. The chain architecture matters more than any individual agent's perfection.
These aren't theoretical concerns. Every team running agents in production has hit at least half of these. The good news: platforms like Mentiko handle most of them out of the box -- observability, scheduling, secrets, multi-tenancy, error routing.
The ones you still own: prompt quality, output validation, and cost monitoring. Those will always be your responsibility, because they're specific to your use case.
Building your first production chain? Start here or see the patterns.
Get new posts in your inbox
No spam. Unsubscribe anytime.