Agent AI System Design

Not an LLM Wrapped in a
Chat Interface

A builder's walkthrough of the system design behind production agentic AI: model routing, tool contracts, memory and state, orchestration, evaluation, approval gates — and the production principles that make an agent reliable, fast, and safe enough to connect to real tools.

Source: "Agent AI System Design Explained in 27 Minutes" — Aishwarya Srinivasan · youtube.com/watch?v=mwN75EiGfCE

Agenda

What We'll Cover

Foundations & Architecture

What is an agentic AI system?
Single-agent vs multi-agent systems
The model layer: routing & structured output

The Building Blocks

Tools: the interface to the real world
Memory & state
Orchestration
Evaluation
Approval gates & policy control

Production Principles

Reliability
Cost & latency
Context & RAG design
Observability, security & privacy

Wrap-Up

Putting it all together
Key takeaways checklist

Architecture Patterns

Single-Agent vs Multi-Agent

Single-Agent

One primary agent owns the entire workflow.

It may still call multiple tools, retrieve documents, update its memory, and run many reasoning steps — but the control loop stays centralized.

Example: a customer-support agent that classifies an incoming request, retrieves the account context, calls a billing API, asks for confirmation, then generates the final response.

Multi-Agent

Workflow split across specialized agents — planning, knowledge retrieval, code writing, review, execution.

Key idea: separation of responsibilities — defined roles, explicit input contracts, explicit output contracts, and routing logic between agents.

Multi-agent earns its complexity when…

Task has clear specialization
Work benefits from parallel work
Review loops are needed
Long-running workflows

But multi-agent adds…

More coordination overhead
More failure modes & state to track
More logs to inspect
More places where cost and latency creep up

The choice is a design tradeoff — not a matter of which pattern sounds more advanced.

Building Block 1 · The Model Layer

Route Models by Step Complexity

The default — one powerful model handling every step — gets expensive and slow very quickly.

Step type	Model tier
Intent classification, routing, extraction, schema filling, simple summarization	Small, cheap, fast model
Book vs reschedule vs cancel vs just asking a question	A small model classifies this fine
Date, time, doctor name, appointment type → JSON schema	May not need an LLM at all
Ambiguous constraints, deeper reasoning "I'm going to be traveling next week, I want to avoid mornings" · "make sure this happens after my lab results come in"	Stronger reasoning model

Structured outputs

Any step whose output feeds another system returns a predictable structure, never free-form prose — via JSON schema, Pydantic models, or function/tool calling.

3-question contract per step

1. Which model runs this step?
2. What output contract does it return?
3. What happens if it fails or returns invalid output?

Building Block 2 · Tools

Design Tools Like Strict APIs

Tools are the interface between the model and the external world: database lookups, CRM, calendar, payment APIs, code interpreters, search, ticketing, Slack/Google actions, any backend function the agent may call.

Name Description Input schema Output schema Permission boundaries Timeout behavior Retry behavior Error format

Anti-pattern

update_user accepts one vague string:
"update this user based on the request" — far too open-ended.

Safer contract

Explicit structured fields: user_id · field_to_update · new_value · reason · source_request_id · confirmation_required

Tool outputs must be machine-readable: a structured error on failure, a structured result on success. The agent should never parse messy prose from your backend.

Building Block 2 · Tools (cont.)

Separate Read Tools, Then Phase In Risk

Read-only capabilities first Low-risk write tools next High-risk write tools — only with validation + human approval

Fetching available appointment slots is very different from canceling an appointment — understand the risk factor of each tool.

MCP — Model Context Protocol

An emerging pattern for exposing tools, resources, and context to agents in a standardized way. Optional. Even without MCP, the system-design principle is identical: tools need contracts, permissions, boundaries, and logs.

Building Block 3 · Memory & State

Memory Is Not One Thing

State

The current execution context of the workflow.

Which step are we on?
What has already been collected?
Which tools were called — what did they return?
Has the user confirmed the action?
Did the workflow pass or fail?

Memory

Broader, cross-run information.

Conversation history
User preferences
Past actions
Retrieved knowledge & document context
Summaries & long-form info useful later

Rescheduling a doctor's appointment: store the appointment ID, proposed new time, confirmation status, and workflow step as structured state — don't pass the full medical history through the LLM when it only needs the appointment ID and the available slots.

Building Block 3 · Memory & State (cont.)

Storage by Access Pattern

A common mistake is pulling everything into a vector database — that is not a good default. Choose each store by access pattern.

Data	Store
Conversation & workflow state	Low-latency store: Redis, DynamoDB, Postgres, MongoDB
Application state	Your application database
Knowledge retrieval (RAG)	Pinecone, pgvector, Weaviate, Elasticsearch, OpenSearch, or a managed knowledge base
Long-term archives	Cheaper object storage — e.g., an S3 bucket

Short-term vs long-term

Short-term: what you pass into the prompt for the current turn.
Long-term: retrieved selectively.

Never stuff everything into the context window — that creates context bloat. Retrieve the smallest useful context for the current step.

Memory design = data architecture

What to store
Where to store it
How long to keep it
How to retrieve it
What is safe to send to the model — or not

Building Block 4 · Orchestration

Define the Control Flow Explicitly

Orchestration is the control layer — moving from a user request, through intermediate steps and tool calls, to a final output. Implementable in plain application code, LangGraph, Temporal, LlamaIndex workflows, LangChain, custom state machines — often a combination.

Receive message Classify intent Retrieve context Check scope Select tools Validate inputs Execute tools Inspect results Confirm if needed Final response Log trace + async evals

Do not confuse autonomy with lack of structure. Production agents need a very clear control flow.

Building Block 4 · Orchestration (cont.)

Pipelines, Graphs & Agent Routing

Simple flows → deterministic pipeline

Not every workflow needs planning and reflection. If the sequence of steps is mostly known in advance, design it as a pipeline or a state machine — and use agentic reasoning only when the workflow genuinely needs dynamic decision-making.

Complex flows → graph orchestration

Graphs let you represent branching, retries, loops, approval gates, and fallback paths:

Extraction fails → retry with a different prompt or model
"No available appointment" → branch to suggest alternatives
Cancellation request → branch into a confirmation path

Agent-to-agent routing (multi-agent)

Define for every handoff: which agent owns the next step · what information gets passed along · what output format is expected · how conflicts are resolved. Without that, multi-agent systems become extremely difficult to debug.

Building Block 5 · Evaluation

Eval the Trace, Not Just the Final Answer

In agentic systems, "no exception thrown" no longer means it works. Evaluate every important step of the trajectory: intent classification, retrieval quality, tool selection, tool arguments, policy compliance, confirmation behavior, final answer quality, task success.

Valid JSON, semantically wrong Correct tool, wrong arguments Irrelevant context retrieved Answering from stale information Fails to ask for confirmation Refuses a valid request Complies with an unsafe request

A support agent can produce a very polished final answer yet have applied the wrong refund policy. Eval the trace and you see the retrieval pulled the wrong policy document, or the model misclassified the user's plan type.

Building Block 5 · Evaluation (cont.)

Test Sets, Production Evals & Metrics

Happy paths Ambiguous requests Out-of-scope requests Tool failures Malicious inputs Partial information Policy edge cases Escalation cases

This test set becomes your regression suite whenever you change the model, the prompt, the retrieval logic, or your tool schemas.

Production evals

Sampled async evals — a judge LLM scores a percentage of conversations offline
High-signal user feedback goes directly into evals
LLM-as-judge is useful, not perfect — combine model grading with deterministic checks and human review

Produce metrics, not just examples

Intent accuracy Tool-call success rate Invalid-schema rate Retrieval hit rate Refusal accuracy Escalation rate Task completion rate User flag rate Cost per successful task

Building Block 6 · Approval & Policy Control

Human in the Loop for High Impact

These should never happen just because the model inferred the intent:

Send email Delete data Issue refund Cancel appointment Change billing Update CRM record Run code Place order Financial transaction

Model suggests Code validates User approves Tool executes

Deterministic validation

Check the user owns the target
Check permissions
Check the action is allowed under policy
Check required fields are present
Check the user confirmed the exact action

Source of truth

Your application code — not the agent — is the source of truth for business rules. If one agent plans and another executes, the execution layer must not blindly trust the planning layer.

Example: "cancel my appointment" → verify ownership, cancellability, policy, and explicit confirmation before the cancellation API is called.

Production Principle 1 · Reliability

Reliable Even When the Model Isn't

The system behaves predictably even when the model does not — through decomposition, contracts, retries, validation, fallbacks, and monitoring.

Decompose prompts

A single joint prompt that classifies, retrieves, decides policy, calls tools, and writes the final response is extremely hard to test. Smaller steps are far easier to evaluate and debug.

Validate → retry → fallback

If the next step depends on the model's output, never rely on free-form text. Validate the schema; retry on invalid output; use a fallback path if it still fails.

Model call = unreliable dependency

Timeouts, malformed outputs, rate limits, provider errors, degraded quality — every one needs a handling path.

Deterministic validation ≠ model reasoning

Model extracts a date → code validates it. Model picks a user ID → code verifies permissions. Model proposes a tool call → code validates the arguments.

Reliability is not about making the model perfect — it is about designing the system so model imperfections do not immediately become product failures.

Production Principle 2 · Cost & Latency

Design Cost and Latency Together

One interaction may run many model calls — classification, query rewriting, retrieval, planning, tool selection, tool arguments, result interpretation, final response, evaluation. All-large-models everywhere = extremely slow and extremely expensive.

Lever	Rule
Model routing	Small models for simple classification and extraction; larger models only when ambiguity, reasoning, or synthesis is required
Token limits	Output tokens are both a cost and a latency lever — two-sentence answer, no essay; downstream needs JSON, no prose
Caching	Cache retrieval results, repeated policy lookups, tool metadata, and stable context
Batching / async	For non-blocking work like evaluation and summarization
Streaming	Always stream user-facing responses that may take time
Progress states	Long-running tool calls → show progress, never a blank screen
Early scope gates	Cheap filters, rules, and intent gates before the expensive reasoning step — out-of-scope requests burn tokens and tool calls

Cost observability: tokens in · tokens out · cost per step · cost per conversation · cost per successful task.

Production Principle 3 · Context & RAG Design

Pass the Right Context, Not Everything

Each context source has different freshness, trust, privacy, and latency characteristics — choose per step, don't dump everything into the prompt.

User message Conversation state Application database Retrieved documents Tool results User profile Long-term memory

Retrieval quality > vector DB

Document chunking
Metadata filters
Hybrid search where useful
Reranking
Freshness controls
Source attribution when trust matters

Trusted vs untrusted content

Retrieved docs must not override system instructions
Tool outputs are data, not instructions
User content isolated from developer/system instructions
Long conversations → summarization + checkpoints, stored separately from raw logs

Production Layer · Observability

Log the Anatomy of Every Run

Category	Fields
Identity	Model name · model version · prompt version · step name · workflow ID · conversation ID
Tool use	Tool name · tool arguments — sensitive values masked
Performance	Latency · time to first token · tokens in · tokens out · cost
Resilience	Retries · fallbacks · errors
Feedback & evals	Was feedback used · eval scores

From the logs you must be able to answer where the failure happened: intent classification · retrieval · planning · tool selection · tool execution · policy validation · final response.

Production Layer · Security & Privacy

Attacker-Controlled Input, Minimum Data

Security — treat everything touching the model as attacker-controlled unless proven otherwise

User messages: direct prompt injection Retrieved documents: indirect prompt injection Tool outputs: poisoned data Model outputs: unsafe commands — SQL, HTML, code

Separate instructions from data Never execute raw model output No SQL/shell from generated text Least-privilege tool permissions Approval gates for risky actions

Privacy — send the minimum data needed

Model needs appointment ID + availability → no full patient report
Summarizing a ticket → no full customer history
Mask PII at the right time; set retention policies for logs, traces, and archives
Keep sensitive fields out of the prompt unless required for the task
Data boundary = model provider + vector DB + observability + logging systems

Key Takeaways

The Complete Checklist

Treat the agent as a full production software system, not an LLM in a chat interface

Choose single vs multi-agent deliberately — specialization & review loops vs coordination overhead

Route models by step complexity — cheap fast models for simple steps, strong models where reasoning changes the outcome

Design tools like strict APIs: names, schemas, permissions, timeouts, retries, structured errors

Separate memory from state; store by access pattern; pass the smallest useful context

Define explicit control flow — deterministic pipelines for known steps, graphs for branching and gates

Gate high-impact actions: model suggests · code validates · user approves · tool executes

Evaluate the trace, not just the final answer, against a realistic regression test set

Decompose prompts; validate, retry, fall back; treat each model call as an unreliable dependency

Design cost and latency in: route, limit tokens, cache, batch, stream, gate scope early

Log the anatomy of every run — and know exactly where each failure happened

Treat everything touching the model as attacker-controlled; send the minimum data

The End

Go Build It Right

A production-grade agentic AI system needs all of the pieces together. And agentic AI system design is not just prompt engineering — it is backend design, data design, security design, and product design with an LLM in the loop.

Source: "Agent AI System Design Explained in 27 Minutes" — Aishwarya Srinivasan (youtube.com/watch?v=mwN75EiGfCE)
Going deeper: a free ~2-hour masterclass recording on Agentic AI System Design · Gen Academy's live, project-based certification program — weekly classes, low-code/no-code or code-heavy tracks, open to anyone becoming an AI professional.

Not an LLM Wrapped in aChat Interface