Production Deployment of Agent Systems

Building an agent that works in a notebook is very different from deploying one that serves real users reliably. Production agent systems must handle asynchronous execution, comprehensive observability, graceful error recovery, horizontal scaling, and detailed trace logging. This lesson covers the patterns and practices for deploying agents to production.

---

Production Architecture

┌──────────┐     ┌──────────────┐     ┌────────────────────────────┐
│  Client   │────▶│  API Gateway │────▶│  Agent Orchestrator        │
│           │     │  (Auth, Rate │     │  (Task queue, routing)     │
│           │     │   Limiting)  │     └────────────┬───────────────┘
└──────────┘     └──────────────┘                   │
                                        ┌───────────┼───────────────┐
                                        │           │               │
                                        ▼           ▼               ▼
                                  ┌──────────┐ ┌──────────┐  ┌──────────┐
                                  │ Agent    │ │ Agent    │  │ Agent    │
                                  │ Worker 1 │ │ Worker 2 │  │ Worker N │
                                  └────┬─────┘ └────┬─────┘  └────┬─────┘
                                       │            │              │
                              ┌────────┴────────────┴──────────────┘
                              │
                   ┌──────────┼──────────────┐
                   │          │              │
                   ▼          ▼              ▼
            ┌──────────┐ ┌──────────┐ ┌──────────┐
            │ LLM APIs │ │ Tool     │ │ State    │
            │          │ │ Services │ │ Store    │
            └──────────┘ └──────────┘ │ (Redis/  │
                                      │  Postgres)│
                                      └──────────┘

---

Async Execution

Agents can take seconds to minutes to complete. Never block the request thread.

Task Queue Pattern

import asyncio
import uuid
from enum import Enum