Aishwarya Srinivasan · 27-Minute System Design Deep Dive

Agent AI System Design
Explained in 27 Minutes

An agentic AI system is not an LLM wrapped in a chat interface — a builder's walkthrough of the full system design: model routing, tools, memory, orchestration, trace-level evals, approval gates, and the production controls that take an agent from demo to a system real users can rely on.

Source: youtube.com/watch?v=mwN75EiGfCE · Channel: Aishwarya Srinivasan

Agenda

What We'll Cover

Core Building Blocks

What is an agentic AI system?
Single-agent vs multi-agent systems
The model layer: routing & structured output
Tools & tool contracts
Memory & state
Orchestration
Evaluation

Approvals & Production Layers

Approval gates & policy control
Reliability
Cost & latency
Context & RAG design
Observability, security & privacy
Putting it all together

The Big Picture

A System That Acts on the World

An agentic AI system is a production software system: the model reasons over a goal, decomposes it into steps, decides what to do next, calls tools, inspects the results, updates its state — and repeats until a stopping condition.

The agentic loop

Decompose the goal into steps
Pick the next step; does it need a tool?
Call the tool through a strict API
Inspect the structured result
Update state and memory
Stopping condition reached? Answer or act

Why system design matters

The hard part is not getting the model to answer.

Agents trigger real actions through APIs — a full software system with reliability, security, and operations concerns. Design for:

Reliability Cost awareness Speed & UX Context awareness Observability Safety on real tools

System Architecture

Single-Agent vs Multi-Agent

Single-Agent

One primary agent owns the entire workflow.

It may call multiple tools, retrieve documents, update memory, and run many reasoning steps — but the control loop stays centralized.

Example: a customer support agent that classifies the request, retrieves account context, calls a billing API, asks for confirmation, then generates the final response.

Multi-Agent

Workflow split across specialized agents: planning, knowledge retrieval, code writing, review, execution.

Key idea: separation of responsibilities — defined roles, explicit input contracts, output contracts, and routing logic between agents.

Multi-agent earns its complexity when…

The task has clear specialization
Parallel work helps
Review loops are needed
Workflows run long

But it adds real costs…

More coordination overhead
More failure modes & state to track
More logs to inspect
More places cost & latency creep up

The choice is a design tradeoff — not a matter of which pattern sounds more advanced.

Building Block 1 · The Model Layer

Route Models by Step Complexity

The default — one powerful model for every step — is a trap: it gets expensive and slow very quickly.

Step type	Model tier
Intent classification, routing, extraction, schema filling, simple summarization	Small, cheap, fast model
"Book, reschedule, cancel, or just a question?"	Small model classifies it fine
Date, time, doctor name, appointment type into a JSON schema	May not need an LLM at all
Ambiguous constraints, real reasoning "Traveling next week, avoid mornings" · "after my lab results come in"	Stronger reasoning model

Structured outputs everywhere

Any step whose output feeds another system returns a predictable structure — JSON schema, Pydantic models, function/tool calling. Never free-form prose.

3-question contract per step

1. Which model runs this step?
2. What output contract does it return?
3. What happens if it fails or returns invalid output?

Building Block 2 · Tools

Tools Are the Interface to the Real World

Database lookups, CRM/calendar/payment APIs, a code interpreter, search, ticketing, Slack/Google actions — any backend function the agent is allowed to call. In production, design tools like APIs with strict contracts.

Name Description Input schema Output schema Permission boundaries Timeout behavior Retry behavior Error format

Anti-pattern: too open-ended

update_user accepting one vague string — "update this user based on the request" — lets the model send arbitrary instructions into your backend.

Safer contract: explicit fields

user_id · field_to_update · new_value · reason · source_request_id · confirmation_required

Outputs must be machine-readable too: a structured error on failure, a structured result on success. The agent should never parse messy prose from your backend.

Building Block 2 · Tools (cont.)

Layer Risk In Gradually

A good pattern: separate read tools from write tools, then phase in risk in stages.

Read-only capabilities first Add low-risk write tools High-risk write tools — only with validation + human approval

Fetching available appointment slots is very different from canceling an appointment — know the risk factor of each tool.

MCP — Model Context Protocol

An emerging pattern for exposing tools, resources, and context to agents in a standardized way. Optional. Even without MCP, the principle is identical: tools need contracts, permissions, boundaries, and logs.

Building Block 3 · Memory & State

Memory Is Not One Thing

Many agentic systems get messy here because they treat memory as one thing — the root cause is usually not separating memory from state.

State

Current execution context of the workflow.

Which step are we on?
What has been collected so far?
Which tools were called, what did they return?
Has the user confirmed the action?
Did the workflow pass or fail?

Memory

Broader, cross-run information.

Conversation history
User preferences
Past actions
Retrieved knowledge & document context
Summaries & long-form info useful later

Keep workflow state structured — appointment ID, proposed new time, confirmation status, workflow step. Don't pass the full medical history through the LLM when it only needs the appointment ID and the available slots.

Building Block 3 · Memory & State (cont.)

Choose Storage by Access Pattern

A common mistake is pulling everything into one vector database — not a good default.

Data	Store
Conversation & workflow state	Low-latency store: Redis, DynamoDB, Postgres, MongoDB
Application state	Your application database
Knowledge retrieval (RAG)	Pinecone, pgvector, Weaviate, Elasticsearch, OpenSearch, or a managed knowledge base
Long-term archives	Cheaper object storage (e.g., an S3 bucket)

Short-term vs long-term

Short-term context: what you pass into the prompt for the current turn.
Long-term memory: retrieved selectively.

Never stuff everything into the context window — context bloat. Retrieve the smallest useful context for the current step.

Memory design = data architecture

What to store
Where to store it
How long to keep it
How to retrieve it
What is safe to send to the model

Building Block 4 · Orchestration

Explicit Control Flow

Orchestration is the control layer: user request → intermediate steps → tool calls → final output. Built with plain code, LangGraph, Temporal, LlamaIndex workflows, LangChain, or custom state machines — often a combination.

Receive message Classify intent Retrieve context Check scope Select tools Validate inputs Execute tools Inspect results Confirm if needed Generate response Log trace + async evals

Do not confuse autonomy with lack of structure. Production agents need a very clear control flow.

Building Block 4 · Orchestration (cont.)

Pipelines, Graphs & Routing

Simple flows → deterministic pipeline

Not every workflow needs planning and reflection. If the sequence of steps is mostly known in advance, design it as a pipeline or state machine. Use agentic reasoning only when the workflow genuinely needs dynamic decisions.

Complex flows → graph orchestration

Graphs represent branching, retries, loops, approval gates, and fallback paths:

Extraction fails → retry with a different prompt or model
"No available appointment" → branch to suggest alternatives
Cancellation request → branch into a confirmation path

Agent-to-agent routing (multi-agent)

Define for every handoff: which agent owns the next step · what information gets passed along · what output format is expected · how conflicts are resolved. Without this, multi-agent systems become extremely difficult to debug.

Building Block 5 · Evaluation

Evaluate the Trace, Not Just the Answer

In traditional software, no exception thrown means it works — in agentic systems that assumption breaks quickly. Evaluate every important step of the trajectory: intent classification, retrieval quality, tool selection, tool arguments, policy compliance, confirmation behavior, final answer quality, task success.

Valid JSON, semantically wrong Correct tool, wrong arguments Irrelevant context retrieved Stale information used No confirmation asked Refuses a valid request Complies with an unsafe request

A support agent can produce a polished final answer yet use the wrong refund policy. Final-text-only eval misses the failure — trace eval shows the retrieval pulled the wrong policy document or the model misclassified the user's plan type.

Building Block 5 · Evaluation (cont.)

Test Sets, Production Evals & Metrics

Happy paths Ambiguous requests Out-of-scope requests Tool failures Malicious inputs Partial information Policy edge cases Escalation cases

This set becomes your regression suite whenever you change the model, the prompt, the retrieval logic, or a tool schema.

Production evals

Sampled async evals — an LLM-as-judge scores a percentage of conversations offline
High-signal user feedback goes directly into the eval queue
LLM-as-judge is useful, not perfect: combine model grading + deterministic checks + human review

Produce metrics, not just examples

Intent accuracy Tool-call success rate Invalid-schema rate Retrieval hit rate Refusal accuracy Escalation rate Task completion rate User flag rate Cost per successful task

Building Block 6 · Approval & Policy Control

Human in the Loop for High-Impact Actions

These should never happen just because the model inferred the intent:

Send email Delete data Issue refund Cancel appointment Change billing Update CRM record Run code Place an order Financial transaction

Model suggests Code validates User approves Tool executes

Deterministic validation

Check the user owns the target (e.g., the appointment)
Check permissions
Check the action is allowed under policy
Check required fields are present
Check the user confirmed the exact action

Source of truth

Your application code — not the agent — is the source of truth for business rules. If a planner agent generates a plan and an executor runs it, the execution layer must not blindly trust the planning layer: tool execution still needs validation and authorization.

Production Principle 1 · Reliability

System Reliable Even When the Model Isn't

Decompose prompts

A single joint prompt that classifies, retrieves, decides policy, calls tools, and writes the response is extremely hard to test. Smaller steps are far easier to evaluate and debug.

Validate → retry → fallback

If the next step depends on the model's output, never rely on free-form text. Validate the schema; retry on invalid; use a fallback path if it still fails.

Model calls = unreliable dependencies

Timeouts, malformed outputs, rate limits, provider errors, degraded quality — every one needs a handling path.

Validation ≠ model reasoning

Model extracts a date → code validates it. Model picks a user ID → code verifies permissions. Model proposes a tool call → code validates the arguments.

Reliability is not about making the model perfect — it is about designing the system so model imperfections do not immediately become product failures.

Production Principle 2 · Cost & Latency

Design Cost and Latency Together

One interaction can include intent classification, query rewriting, retrieval, planning, tool selection, tool arguments, result interpretation, response generation, and evaluation — every step a model call.

Lever	Rule
Model routing	Small models for simple classification/extraction; larger models only when ambiguity, reasoning, or synthesis is required
Token limits	Output tokens are a cost and a latency lever — two-sentence answer → no essay; downstream needs JSON → no prose
Caching	Cache retrieval results, repeated policy lookups, tool metadata, and stable context
Batching / async	For non-blocking work like evaluation and summarization
Streaming	Always stream user-facing responses that may take time
Progress states	Long-running tool calls → show progress, never a blank screen
Scope gates early	Cheap filters, rules, and intent gates before the expensive reasoning step — out-of-scope requests burn tokens and tool calls

Track: tokens in · tokens out · cost per step · cost per conversation · cost per successful task.

Production Principle 3 · Context & RAG Design

Pass the Right Context, Not Everything

Context design is one of the biggest differences between a toy agent and a useful production agent. Context can come from many places:

User message Conversation state Application database Retrieved documents Tool results User profile Long-term memory

Each source has different freshness, trust, privacy, and latency characteristics — choose per step.

Retrieval quality > having a vector DB

Document chunking
Metadata filters
Hybrid search where useful
Reranking
Freshness controls
Source attribution when trust matters

Trusted vs untrusted content

Retrieved documents must not override system instructions
Tool outputs are data, not instructions
User content isolated from developer/system instructions
Long conversations → summarization + checkpoints, stored separately from raw logs

Production Layers · Observability, Security & Privacy

Observe Everything · Trust Nothing · Send the Minimum

Observability

Log the anatomy of every run.

Model name & version, prompt version, step, workflow & conversation ID
Tool name + arguments — sensitive values masked
Latency, time to first token, tokens in/out, cost
Retries, fallbacks, errors
Feedback used, eval scores

Security

Everything touching the model is attacker-controlled until proven otherwise.

User messages → direct prompt injection
Retrieved documents → indirect injection
Tool outputs → poisoned data
Model outputs → unsafe SQL/HTML/code

Separate instructions from data · never execute raw model output · no SQL/shell from generated text · least-privilege tools · approval gates.

Privacy

Send the minimum data needed.

Needs an appointment ID → don't send the patient report
Mask PII at the right time
Retention policies for logs, traces, archives
Sensitive fields stay out of prompts unless required
Provider, vector DB, observability & logs = your data boundary

From your logs you must be able to answer where the failure happened: intent classification, retrieval, planning, tool selection, tool execution, policy validation, or the final response.

Key Takeaways

The Checklist

An agentic AI system is a production software system — not an LLM wrapped in a chat interface

Choose single vs multi-agent deliberately: specialization and review loops vs coordination, state, and cost

Route models by step complexity — strong reasoning only where it changes the outcome

Design tools like strict APIs: schemas, permissions, timeouts, retries, structured errors

Separate memory from state; store each by access pattern; pass the smallest useful context

Explicit control flow — pipelines for known steps, graphs for branching, retries, and approval gates

Evaluate the trace, not just the final answer, against a realistic regression test set

High-impact actions need gates: model suggests, code validates, user approves, tool executes

Design reliability, cost, latency, and context together — decompose, validate, cache, stream, gate scope early

Log every run's anatomy; treat everything touching the model as attacker-controlled; send the minimum data

Putting It All Together

All of It, Together

A production-grade agentic AI system needs every piece at once: model routing, strict tool contracts, explicit memory and state management, orchestration, trace-level evals, approval gates, reliability, cost and latency controls, context design, observability, security, and privacy.

Agentic AI system design is not just prompt engineering — it is backend design, data design, security design, and product design with an LLM in the loop.

Source: "Agent AI System Design Explained in 27 Minutes" · Aishwarya Srinivasan · youtube.com/watch?v=mwN75EiGfCE

Agent AI System DesignExplained in 27 Minutes