System design walkthrough
Agent AI System Design Explained in 27 Minutes
Section 1 · What is an agentic AI system?
1. The Agentic Loop: From Message to Action
A production agent is not an LLM in a chat box — it reasons over a goal, picks steps, calls tools, inspects results, updates state, and loops until a stopping condition. Every stage is a place where reliability, cost, and safety get designed in.
Step 1 of 9
1
User goal arrives
A request enters the system — plain text, not yet an action.
2
Reason over the goal
The model decides what the user wants and whether it can be done in one step.
3
Decompose into steps
Multi-step goals get broken down — the core of agentic behavior.
4
Decide the next step
Which action moves the workflow forward: retrieve, compute, call, or ask?
loop
5
Does the step need a tool?
A gate before anything touches the outside world.
Yes — call the tool through its strict API, then inspect the structured result.
No — continue reasoning with state and memory already in context.
6
Inspect result & update state
Tool output is data to be validated — never free-form prose, never instructions.
7
Update memory
History, preferences, and retrieved knowledge stored selectively.
8
Stopping condition reached?
If not, the loop returns to deciding the next step.
9
Answer or act on the world
Final response to the user, or a real side effect through an approved tool.
↺
Not done yet: the workflow returns to step 4 and keeps looping until the stopping condition fires. This loop, run many times per request, is why cost, latency, evals, and observability are first-class design concerns.
Section 2 · Single-Agent vs. Multi-Agent Systems
2. One Agent or Many: A Design Tradeoff
Both patterns receive the same user request. A single agent keeps one centralized control loop; a multi-agent system splits the workflow across specialized roles with explicit input and output contracts and routing between them.
One control loop
Single-agent system
One agent owns the entire workflow end to end
A support agent classifies the request, retrieves account context, calls a billing API, asks for confirmation, then generates the final response — all inside one loop.
VS
Separation of responsibilities
Multi-agent system
Planner→
Knowledge→
Coder→
Reviewer→
Executor
Every agent has a defined role, an input contract, an output contract, and routing logic between agents — structure first, autonomy second.
When many agents earn their costclear specialization, parallel work, review loops, long-running workflows.
The real costsmore coordination overhead, more failure modes, more state to track, more logs to inspect, more places where cost and latency creep up.
Section 3 · The Model Layer
3. Model Routing: Right Model for the Right Step
Letting one frontier model handle every step is a cost and latency trap. Route by complexity: cheap fast models for routine work, strong reasoning models only where reasoning actually changes the outcome.
Step 1 of 5
1
Step arrives
Every step must answer three questions: which model handles it, what output contract it returns, and what happens on failure.
2
Route by complexity
Three lanes below — most traffic belongs in the first two.
Cheap & fast lane
Low-complexity steps where a small model is enough.
Structured output lane
Extraction into a schema — may not need an LLM at all.
Strong reasoning lane
Only when ambiguity or deep reasoning changes the answer.
3
Cheap path usually wins
Deciding reschedule vs. cancel does not need a frontier model — and extracting date, time, and doctor may not need an LLM at all.
4
Escalate only when reasoning matters
Ambiguity and tradeoffs go to the stronger model — “traveling next week, avoid mornings” needs real reasoning.
5
Structured output with a failure plan
Whatever the lane, the output contract is validated — invalid output triggers a retry or fallback path.
Section 5 · Memory and State
4. Memory & State: Data Architecture First
Agentic systems get messy when memory is treated as one thing. State and memory are different data with different access patterns — and pulling everything into a vector database is a poor default.
State — current execution context
- Which step the workflow is on
- What information has been collected
- Which tools were called, what they returned
- Whether the user confirmed the action
- Whether the workflow passed or failed
Memory — reusable knowledge
- Conversation history
- User preferences and past actions
- Retrieved knowledge and document context
- Summaries and long-form archives
▼
Choose storage by access pattern
▼
Low latency
Workflow state
Conversation and workflow state need speed.
System of record
Application state
Owned by your application, not the agent run.
Knowledge retrieval
RAG store
Search over documents at retrieval time.
Cheap archives
Long-term memory
Old conversations and logs, kept cheaply.
!
Short-term context vs. long-term memoryThe prompt gets the smallest useful context for the current turn; everything else is retrieved selectively. A reschedule agent needs the appointment ID and free slots — not the full medical history passed through the LLM.
Section 6 · Orchestration
5. One Request, Explicitly Orchestrated
Orchestration is the control layer — plain code, a graph framework, or a state machine — that moves a request from message to final output. Branches, retries, loops, and approval gates are all drawn explicitly; autonomy is not a lack of structure.
Step 1 of 11
1
User message received
“Book me a checkup with Dr. Mehta next week.”
2
Classify intent
Book, reschedule, or cancel — small-model territory.
3
Retrieve context
State plus the smallest useful slice of memory.
4
In scope?
Cheap gates filter out-of-scope requests before expensive reasoning burns tokens and tool calls.
5
Select tools
Which read or write tools apply to this step.
6
Validate tool inputs
Schema and permission checks — code, not the model, is the gatekeeper.
7
Execute tools
Read tools first; writes only with validation and approval in place.
8
Inspect results
Structured results in, structured errors out — retry, branch, or fall back.
confirmation branch
9
Confirm high-impact actions
Canceling an appointment pauses for explicit user confirmation before any write API is called.
10
Generate the final response
Only now is the answer composed for the user.
11
Log the trace, send to evals
The full trajectory is logged and evaluation data is sent asynchronously — off the user-critical path.
✕
Branches along the way: extraction fails → retry with another prompt or model · no availability → suggest alternatives · cancellation → confirmation path. Deterministic pipelines win for known sequences; graph-based orchestration earns its keep once branching, retries, and gates appear.
Section 8 · Approval Gates and Policy Control
6. The Approval Gate: Suggest, Validate, Approve, Execute
Sending an email, deleting data, issuing a refund, canceling an appointment, running code — high-impact actions never happen just because the model inferred intent. Model suggests, code validates, user approves, tool executes.
Stage 1 of 6
Model
Suggests the action
→
Code
Validates deterministically
→
User
Approves the exact action
→
Tool
Executes the action
✕ validation fails or user declines
Terminal state
Rejected — nothing executes
✓
What code must check before executing
Ownership of the resource · permissions · allowed by policy · required fields present · user confirmed the exact action. The agent is never the source of truth for business rules — application code is.