AI Agents in Production: The Operator's Handbook

There are a thousand tutorials on building an agent. Give it a system prompt, wire up a couple of tools, watch it call a function, ship a demo video. That part is genuinely easy now — any competent engineer can go from zero to a working agent in an afternoon, and the barrier keeps dropping every quarter.

Nobody teaches the other part. What happens when that agent is still running in six months, when it's one of 325 agents you're responsible for, when it burns through an API budget overnight because a retry loop went sideways, when it "successfully" completes a task by fabricating the result instead of doing the work. Building an agent and running ai agents in production are two different disciplines, and the industry has almost entirely optimized content for the first one. Every framework's quickstart ends at "it works." None of them cover what happens the two-hundredth time it runs, unattended, against real data, with real cost attached to every token.

I run 325 agents in production. Not as a thought experiment — as the actual operating model behind an engineering org, a trading fleet managing live capital, a content pipeline, and a public-facing Academy platform. Some of those agents write and ship code. Some watch infrastructure and page me when something drifts. Some manage positions where a bad tool call has a dollar figure attached to it, not just an embarrassing log line. The gap between "I built an agent" and "I operate a fleet of agents that doesn't fall over" is where almost everyone gets stuck, and it's the gap this handbook is written to close.

Here's roughly what one week of that operating model produces, pulled from an audited window rather than a marketing estimate:

None of that is possible without the discipline this handbook covers. A fleet that size run without guardrails doesn't produce more output — it produces more incidents, faster, at greater cost.

This post is the map. Ten spokes, each one a discipline you need if you're going to run agents instead of just demo them: how to start, what stack to use, why running is harder than building, which framework fits, which design patterns survive contact with reality, the seven ways fleets actually fail, how to watch 325 agents without babysitting them, how memory has to work so agents don't wake up with amnesia, where no-code tools help and where they break, and how a single tool becomes a fleet through skills. Read the section that matches your current problem, then go deep with the linked post.

1. How to Start Building AI Agents

Before you can run a fleet, you need one agent that actually works. The mistake most people make at this stage is reaching for a heavyweight framework before they understand the loop underneath it: a model, a set of tools, a way to call those tools, and a way to feed the result back in. Strip away the abstraction and every agent is that loop, repeated until the task is done or the agent gives up. Tool calling isn't magic — it's a structured contract between the model and your code, and understanding that contract at the raw level is what lets you debug it later instead of guessing.

Most people skip straight to a multi-agent framework because it looks more impressive, then spend a week debugging why their single agent won't call a tool correctly — a problem that's trivial to diagnose once you understand the loop directly. Agentic AI isn't a different category of software; it's software with a probabilistic core, and the engineering discipline around it doesn't change just because the middle step involves a model instead of deterministic logic.

How to Start Building AI Agents: From Zero to First Agent walks through building that first agent without the framework overhead — so when you do adopt a framework, you know exactly what problem it's solving for you instead of trusting it blindly. Get the loop right first. Everything else in this handbook assumes you have.

2. The AI Agent Tech Stack

A fleet of 325 agents isn't running on one clever prompt. It's running on a stack: a model layer, a tool-calling layer, a memory layer, an orchestration layer, and a deployment layer, each with real tradeoffs. Pick the wrong piece — a memory store that can't survive a restart, a tool-calling format that doesn't support parallel calls — and you'll rebuild that layer under production pressure instead of by choice, usually during the exact week you can least afford the downtime.

The stack question is really a boundary question: which layer is the model responsible for, and which layer is deterministic code responsible for. MCP (Model Context Protocol) earns its place here because it standardizes that boundary — a consistent way for an agent to discover and call tools without every integration being a bespoke wire-up. That standardization is what makes it realistic to run hundreds of agents against a shared tool surface instead of maintaining hundreds of one-off integrations.

The AI Agent Tech Stack Behind 325 Agents in Production breaks down the actual stack behind that fleet, layer by layer, including where MCP fits as the connective tissue between agents and tools. This is the reference architecture — not a hypothetical one, the one actually running.

3. Running AI Agents in Production

This is the thesis of the whole series, stated directly: building was the easy part. An agent that works in a notebook and an agent that survives a production environment — real traffic, real cost pressure, real failure modes, real humans depending on the output — are not the same artifact. The gap is operational discipline, not model capability, and it's the reason two teams can use the identical model and framework and end up with wildly different reliability.

What actually changes when an agent goes from demo to infrastructure: it needs a way to fail safely instead of just failing loudly, it needs cost limits that trip before a runaway loop becomes a bill, and it needs someone — a person or a system — accountable for what it did overnight while nobody was watching. None of that shows up in a framework's README. It shows up in the postmortem after the first incident, which is a more expensive way to learn it.

Running AI Agents in Production: Why Building Was the Easy Part is the deep dive on that gap: what changes when an agent stops being a demo and starts being infrastructure. If you read one spoke besides this pillar, read that one.

4. Picking an AI Agent Framework

Framework choice is a real decision with real lock-in, and most comparisons are shallow — feature checklists instead of "what happens when this breaks at 2am." LangGraph gives you explicit graph control at the cost of more code up front. CrewAI gives you fast multi-agent scaffolding at the cost of less control once you need something the abstraction didn't anticipate. Claude Code gives you a coding-native agent loop with a skill system, which is a different shape of tool entirely — closer to an operator's console than a library you import.

The right question isn't "which one is more powerful," it's "which one fails in a way I can debug at 2am with no one else awake." An agent framework you can't introspect under pressure will cost you more in incident time than it ever saved you in scaffolding time. This is also where agent framework choice intersects with your team's actual skill set — the best framework on paper is the wrong one if nobody on the team can read its stack trace.

LangGraph vs CrewAI vs Claude Code: Picking an AI Agent Framework compares them on the axes that matter once you're operating, not prototyping: debuggability, cost predictability, and how each one fails. Pick based on your failure tolerance, not the demo.

5. AI Agent Design Patterns That Survive Production

Most agent design patterns you'll find online look great in a single-turn demo and collapse the first time an agent has to recover from a bad tool call, coordinate with a second agent, or resume after a crash mid-task. The patterns that actually matter are the boring ones: explicit state, idempotent tool calls, human-in-the-loop checkpoints at the right altitude, and a clear boundary between what the agent decides and what it's allowed to execute unsupervised.

The pattern I see skipped most often is the boundary one. Teams give an agent broad tool access because it's convenient during development, then discover in production that "convenient" and "safe to run unattended" are different properties. The fix isn't less capability — it's a deliberate seam between decisions the agent can make freely and actions that require a checkpoint, whether that checkpoint is a human approval or an automated policy check. Multi-agent systems make this harder, not easier: coordination failures compound, and a pattern that's safe for one agent can become unsafe the moment a second agent is reading and writing the same state.

5 AI Agent Design Patterns That Survive Production documents the five that have held up across hundreds of live agent runs, with the failure each pattern specifically prevents. Guardrails aren't a separate feature you bolt on later — they're a design pattern you build in from the first line.

6. Why AI Agents Fail in Production

Every operator eventually collects the same list of failure modes, usually the hard way. An agent reports success without doing the work. A retry loop quietly triples your API bill. A tool call succeeds against a stale schema and corrupts data downstream. Context gets silently truncated and the agent starts hallucinating state that never existed. None of these show up as a crash — that's what makes them dangerous. They show up as quietly wrong output that looks fine until someone downstream trusts it.

The common thread across almost every failure mode I've hit across 325 agents isn't model quality — it's a missing verification step somewhere between "the agent said it was done" and "the work was actually checked." Evals catch this at the model-output level; a verification gate catches it at the task-completion level. You need both, because an agent can pass an eval on the exact task it was tested on and still fabricate completion on a task it wasn't.

Why AI Agents Fail in Production: 7 Failure Modes From a Live Fleet catalogs seven of these, drawn from a live fleet rather than a hypothetical. Knowing the failure mode in advance is the difference between a five-minute fix and a multi-hour incident — this is the pattern-matching library you want in your head before you need it.

7. AI Agent Observability

You cannot watch 325 agents. You can't even watch 10 agents in real time and stay sane. The only way this scales is if the agents surface their own state — structured logs, cost telemetry, task-completion signals — and you build the dashboards and alerts that pull your attention only when something actually needs it. Observability at this scale isn't a nice-to-have dashboard; it's the load-bearing wall the entire operating model stands on.

LLM observability means something different than traditional application observability. You're not just tracking whether a request succeeded — you're tracking whether the reasoning that produced the result was sound, whether the cost per task is drifting, and whether an agent's tool-call pattern this week looks different from its pattern last week in a way that predicts a problem before it becomes one. Alert fatigue is the real enemy here: an alerting setup that pages you for every anomaly trains you to ignore it, which defeats the entire purpose.

AI Agent Observability: Monitoring 325 Agents Without Watching Them covers what to instrument, what LLM observability actually means beyond "log the prompts," and how to build alerting that pages you for the right reasons instead of every reason. This is the discipline that makes a fleet of this size operable by one person instead of a team of ten.

8. AI Agent Memory and Agentic RAG

An agent that forgets everything on restart is expensive in a specific way: it re-derives context it already had, re-asks questions it already answered, and repeats mistakes it already made. Memory is what turns a stateless function call into something that compounds — but only if it survives restarts, deploys, and the occasional corrupted index. I learned that last part the hard way, watching a memory index quietly corrupt and take a week of accumulated context down with it before anyone noticed.

Agentic RAG is a step past plain retrieval-augmented generation: instead of a single retrieve-then-generate pass, the agent decides when to query memory, reformulates the query if the first pass didn't return anything useful, and writes new memories back as it works — closer to how a person actually uses notes than to a static lookup. That loop is what lets an agent get measurably better at a recurring task over weeks instead of repeating the same discovery process every single run.

AI Agent Memory: Agentic RAG That Survives Restarts covers the memory architecture question directly: what agentic RAG buys you over plain retrieval, how to structure memory so it's actually queryable months later, and what "survives restarts" means in practice when your index lives on disk and your agents don't. Memory is infrastructure, not a nice-to-have.

9. No-Code AI Agents

No-code agent tools solve a real problem — they get a non-engineer from idea to working automation in an afternoon, no framework, no deploy pipeline. They also hit a ceiling fast: once you need custom tool calling, real error handling, or an agent that coordinates with other agents, the no-code layer usually becomes the thing you're fighting instead of the thing helping you.

The honest way to think about it: no-code agent builders are excellent at automating a single well-defined workflow with predictable inputs, and they get noticeably worse as the task becomes more open-ended or the failure modes become more consequential. A no-code agent that drafts a weekly summary email is a fine use of the tool. A no-code agent making decisions with financial or safety consequences unsupervised is usually a sign the task graduated past the tool before anyone noticed. Knowing which side of that line your automation sits on is the actual skill — not the tool choice itself.

No-Code AI Agents: What They're Good For and Where They Break is an honest accounting of both sides — where the no-code path is genuinely the right call, and the specific signals that tell you it's time to graduate to code. Not every automation needs to be a production agent. Know which one you're building before you pick the tool.

10. Claude Code Skills: One Tool, Six Stacks

The fleet behind this handbook doesn't run on 325 separately-engineered agents. It runs on a skill system — reusable, composable units of agent behavior — layered on top of one tool. A skill for code review, a skill for deployment verification, a skill for content publishing. Combine them and one tool becomes a fleet, without 325 bespoke codebases to maintain, each with its own dependency tree and its own way of breaking.

This is the answer to a question every operator eventually asks: how do you scale agent count without scaling maintenance burden linearly. The answer isn't more infrastructure — it's composability. A well-designed skill is narrow enough to verify, general enough to reuse across a dozen different workflows, and cheap enough to swap out when it stops earning its keep. That's a fundamentally different engineering posture than standing up a new agent framework per use case, and it's the one that actually holds up once the agent count gets into the hundreds.

Claude Code Skills: 6 Stacks That Turn One Tool Into a Fleet walks through six concrete skill stacks and how they compose. This is the leverage mechanism that makes 325 agents a maintainable number instead of a liability — and it's the closest thing to a cheat code in this whole handbook.

The Handbook, Restated

Building an agent is a weekend project. Running ai agents in production — a fleet that survives contact with real cost, real failure, real memory pressure, and real scale — is an operating discipline, and it's the discipline nobody packages as a tutorial because it isn't a single skill. It's guardrails plus observability plus memory plus failure recovery plus the judgment to know which one is breaking when something goes wrong at 3am.

I wrote the case for treating orchestration itself as the unit of leverage after auditing what running agents in parallel actually produces over a real work window — the receipts behind why "operator" is the right word for this job, not "prompt engineer." I've also written about what changes for engineering leadership specifically when the agents on your team write and execute code themselves, not just suggest it. Both are worth reading alongside this handbook if you're trying to figure out where your own role sits in this shift.

None of this replaces good engineering judgment — agent orchestration is a force multiplier on the discipline you already have, not a substitute for it, which is the same argument tesseractintelligence.io makes about applying that same operator discipline to systematic decision-making under uncertainty.

If you want to try the operational side of this handbook instead of just reading about it, start with verify — a free, no-signup skill that runs your full build → lint → test → secrets-scan loop before you push, and gives you a PASS/FAIL report instead of a hunch. It's a small piece of the discipline, but it's the piece most people skip, and skipping it is where a lot of "it worked in the demo" incidents start. Start with the 7 free skills at jeremyknox.ai/skills-library — no signup.