📝 Agentic AI System

Description

< What is it? >

• [Agentic AI System]
  • An agentic AI system is a type of artificial intelligence that can autonomously perform tasks, make decisions, and interact with its environment. Unlike traditional AI systems that require human intervention for every action, agentic AI systems can operate independently, adapting to new situations and learning from their experiences.
  • Agentic AI systems are designed to exhibit characteristics of agency, such as goal-directed behavior, decision-making capabilities, and the ability to perceive and respond to changes in their environment. These systems can be applied in various domains, including robotics, autonomous vehicles, virtual assistants, and more.
• [ReAct]
  • ReAct ("Reasoning + Acting") is a prompting framework for LLMs where the model interleaves explicit reasoning steps with actions (tool calls), rather than just producing an answer directly or just calling tools blindly. It's the core loop underlying most agentic systems.
  
• [LangGraph]
  • LangGraph is a Python library designed to build stateful, multi-step applications that integrate LLMs with external tools. It uses a graph-based approach to define workflows, where each step (or node) represents a specific operation. LangGraph also supports:
    1. Conditional edges: Directing workflows dynamically based on conditions.
    2. Persistent states: Retaining context across workflow executions.
    3. Tool integration: Seamless interaction between LLMs and external tools or APIs.
  • LangGraph State
    • A LangGraph state is a structured representation of the current knowledge, context, and goals of an agentic AI system => the information/data being passed around the graph.
  
• [Tool]
  • A tool is an external resource or API that an agentic AI system can call to perform specific actions or retrieve information. Tools can include search engines, databases, calculators, code execution environments, and more. By leveraging tools, agentic systems can extend their capabilities beyond text generation and interact with the real world.
  • Each tool is described with:
    • Name - identifier for the tool
    • Description - what the tool does and when to use it (LLM use the description to decide if the tool is relevant to the task)
    • Parameters - what inputs are needed, their types, and descriptions
    • Expected Output - what the tool returns

    {
      "name": "get_weather",
      "description": "Get current weather for a specific city. Use this when users ask about weather conditions.",
      "parameters": {
        "city": {
          "type": "string",
          "description": "Name of the city to get weather for"
        }
      }
    }

  • Agent Decision-Making process
    1. Parse the user request - Understand what's being asked
    2. Review available tools - Check descriptions to see which tools might help
    3. Select appropriate tools - Based on description matches with the task
    4. Generate tool calls - Format the call with proper parameters (Extracts parameters from user input based on tool parameter descriptions)
    5. Execute and process results - Use the output to continue the task

Key points

< Main advantages >

RAG vs LLM call

1. [Autonomy] it decides what to do next rather than following a fixed script. Given a goal like "book me a flight to Tokyo under $800," it figures out the steps itself.
2. [Tool use] it can call external tools/APIs (search, code execution, databases, calculators) to gather information or take actions in the world, not just generate text.
3. [Multi-step reasoning / planning] it breaks a complex goal into subtasks, executes them in sequence (or parallel), and adapts the plan if something fails or new information appears.
4. [Memory / state] it tracks what it's already done and what it's learned across steps, so it doesn't repeat work or lose context mid-task.
5. [Self-correction / reflection] many agentic systems evaluate their own intermediate outputs and retry or revise if something looks wrong.

< How does an agent implement memory? >

LangChain is a framework for building agents that combine language models, tools, and application logic. Its agents run on LangGraph, which provides workflow execution and state persistence. This section uses LangChain and LangGraph to explain a concrete memory implementation. See the LangChain overview.

In an agent built with LangChain and LangGraph, memory is application-managed data that the agent saves and retrieves. The application supplies relevant information with each model call; saving a conversation does not update the model's weights. The same storage-and-retrieval principle can also be implemented in other agent frameworks.

Memory	What it stores	Scope and storage	How the agent uses it
Short-term memory / state	Recent messages, tool results, task progress, and a summary of earlier steps	One conversation or task, identified by a thread_id; persisted through checkpoints	Restores the current task and supplies context for the next step
Long-term memory	User preferences, durable facts, and useful records of previous tasks	Across conversations, scoped to a user or application; stored in a database	Retrieves relevant records and includes them in the current context

“Short-term” describes the scope, not whether the data lives in RAM. A database checkpoint can preserve a conversation across process restarts. LangGraph separates thread state from cross-thread memory through checkpointers and stores. See its memory overview and persistence documentation.

• Example: while planning a trip, the current destination and completed searches belong to thread state. An explicitly stated preference such as “I prefer trains” can be stored under the user's profile and reused when they start another trip-planning conversation.

Long-term memory can use ordinary key-based database lookups: a profile record does not need embeddings. For a large collection of past notes, semantic retrieval can search embeddings for relevant records. Retrieve the underlying text or structured data to include in the prompt. Google's Machine Learning Crash Course: Embedding space explains the similarity relationships that support this retrieval.

< Improving Agentic RAG with post-training >

• Start with a measured baseline: use an existing instruction-tuned model and evaluate its behavior with the application's prompts, retrieval tools, and document index. Improve those components first, then consider model training for persistent behavior failures. Agentic RAG does not inherently require additional training. See the Agentic RAG architecture guide.

• Separate the objective from the update method: Supervised Fine-Tuning (SFT) learns from examples of desired assistant actions and responses. Low-Rank Adaptation (LoRA) determines how parameters are updated; QLoRA combines LoRA updates with a quantized frozen base model. SFT with LoRA or QLoRA is therefore one training configuration, rather than a choice between competing techniques. See Fine-Tuning for the general comparison and TRL's PEFT integration for combining them.

• Target the component that fails: use execution traces and the existing debugging checklist to identify the earliest failure before selecting a training method.

  Observed limitation	Improvement to investigate
  The required evidence is not retrieved	Repair parsing, chunking, filters, query formulation, embeddings, or reranking; evaluate retrieval separately
  The model chooses the wrong tool or arguments	Clarify tool descriptions and schemas; consider SFT on correct tool-use examples if the behavior persists
  Relevant evidence is present, but the answer is unsupported or poorly formatted	Improve grounding instructions; consider SFT on retrieved-context and answer examples
  Answers are correct, but the agent takes unnecessary steps	Evaluate complete trajectories and stopping rules; consider preference optimization or reinforcement learning when reliable feedback is available

• Keep knowledge and behavior distinct: changing document facts belong in the retrieval index. Training can improve how the model searches, calls tools, uses evidence, and formats citations. It does not replace retrieval or ensure that every generated claim is supported. Compare the adapted system with the baseline on held-out tasks, including groundedness, tool correctness, task completion, latency, and cost. See Agentic RAG Evaluation.

Architecture

The LangChain ecosystem

• LangSmith: debug, evaluation, monitoring - A platform for debugging, evaluating and monitoring LLMs and agentic systems.
• LangGraph: agent customization - A Python library for building agentic systems with LLMs and external tools.
• LangGraph Platform: agent deployment - A cloud-based service for deploying and managing LangGraph applications.

Comparison

< Basic RAG vs. Agentic RAG >

• Core difference: basic RAG follows a predefined retrieval-and-generation pipeline. Agentic RAG adds an agent that chooses its next action based on the question and intermediate results, such as selecting a search tool, rewriting a query, or retrieving more evidence. It is useful when the required steps cannot all be determined in advance. See LlamaIndex's agentic strategies.

• Where agentic RAG can help:

  Aspect	Basic RAG	Agentic RAG
  Retrieval	Runs a predefined search pipeline	Chooses sources and queries based on the task
  Complex questions	Uses the evidence returned by that pipeline	Breaks a question into smaller searches and combines the results
  Missing evidence	May return an incomplete answer or report insufficient information	Can detect a gap, rewrite the query, and search again
  Data sources	Uses configured retrieval sources	Can choose between document search, SQL, and APIs exposed as tools
  Answer checking	Uses predefined checks, if configured	Can inspect evidence and decide whether another step is needed

  Fixed RAG pipelines can also include query rewriting, reranking, multiple sources, and validation. The distinction is whether the model dynamically controls the next steps; these individual features do not require an agent.

• Example: a user asks, “Can I return order 123, delivered 20 days ago?” Suppose the policy allows returns within 30 days, but excludes final-sale products.

  A basic document-search pipeline can retrieve the return policy but may lack the order's final-sale status. An agent with access to both tools can retrieve the policy, look up order 123, and check the exception. If the order is marked final sale, it can explain that the exception applies and cite the policy. If the lookup fails, it should report that eligibility cannot yet be confirmed. A fixed workflow with the same lookups could also handle this predictable task.

• Tradeoffs: agentic RAG is not automatically more accurate. Extra model calls and tool calls increase latency and cost, and incorrect tool choices can compound errors. Checking its own answer does not guarantee correctness. Set limits on steps, retries, and execution time. Anthropic's Building effective agents recommends starting with the simplest solution that meets the task's needs.

• When to choose each: use a fixed RAG pipeline for straightforward document questions or predictable workflows. Consider agentic RAG for open-ended research, questions that require several linked searches, or tasks whose next step depends on newly retrieved information. Compare both on the same evaluation dataset, measuring answer quality, groundedness, latency, and cost before adopting the extra complexity.

< Adaptation approaches for Agentic RAG >

• Roles and selection criteria: the approaches below operate at different levels and can be combined. Prompt engineering configures the application without training; SFT, preference optimization, and reinforcement learning change learned behavior; LoRA and related methods control the trainable parameters.

  Approach	Role in Agentic RAG	When to consider it
  Prompt engineering	Defines retrieval instructions, tool descriptions, citation rules, and answer format	Establish the initial baseline; no model training is required
  SFT	Learns from examples of correct tool selection, arguments, and grounded answers	The model repeatedly mishandles the workflow despite clear prompts and valid tool interfaces
  LoRA / QLoRA	Makes SFT or other model training more memory-efficient	Model weights are accessible for training and compact task-specific updates are useful
  Prompt tuning / prefix tuning	Learns continuous prompt or prefix vectors while freezing base weights	The training and serving stack supports these parameter-efficient methods
  Direct Preference Optimization (DPO)	Learns from preferred versus rejected responses or actions given the same context	Reliable preference pairs are available
  Reinforcement learning	Optimizes multi-step behavior using rewards	A trustworthy task environment and measurable outcomes support training and evaluation

• Prompt engineering and prompt tuning: editing a system prompt does not update model parameters and is not post-training. Prompt tuning learns numerical vectors through gradient updates. See the soft-prompt guide. The choice between SFT, DPO, and reinforcement learning depends on the available supervision; TRL provides implementations of these training approaches.

Implementation

• Agent Memory covers the memory read–write loop and implementations with LangGraph and LlamaIndex.

< SFT training examples for tool use >

• Illustrative training conversation: the following example demonstrates the desired interaction, not an executed training run. Assume search_documents is an available tool that accepts a string query and returns matching passages with document IDs.

  User:
  "What is the return period for headphones?"
  
  Assistant tool call:
  search_documents(query="headphones return period")
  
  Tool result:
  Document D17: "Headphones may be returned within 30 days."
  
  Assistant answer:
  "Headphones can be returned within 30 days. [D17]"

• Training record: store the available tool definitions, conversation messages, assistant tool calls, tool results, and desired final response. The example above is a readable representation; a trainer expects structured fields such as tools, messages, tool_calls, and tool-role messages. See TRL's tool-calling SFT requirements.

• Learning targets: with assistant-only loss, the model learns to generate the tool call from the user's request and tool definitions, then generate the final answer after receiving the retrieved passage. User messages and tool results provide context rather than assistant prediction targets. Include examples with missing evidence, failed searches, and valid alternative tool choices so the model learns when to retry, clarify, or report insufficient information.

• Application responsibilities: the application executes the generated tool call and supplies its result. It also maps D17 to a source link and checks whether the cited passage supports the answer. SFT can teach citation formatting, but the citation identifier must refer to evidence actually available in the current interaction.

• Parameter updates: when this SFT objective is combined with LoRA, gradients update the configured adapter parameters while the base weights remain frozen. QLoRA further reduces base-weight memory through quantization. Evaluate the resulting model in the complete agent workflow; a lower training loss alone does not establish better retrieval or grounded answers.

Troubleshoot

< How do we debug an agentic AI system? >

Debug an agent by tracing its execution and finding the first step where the result differs from the intended behavior. This workflow applies to both LangChain/LangGraph and LlamaIndex. A trace records the sequence of model calls, memory reads, retrievals, tool calls, and state updates for one request.

For each step, capture its input and output, errors, duration, and available token usage. Include a request ID, conversation ID, and model/prompt versions so related events can be connected. Inspect the actual messages sent to the model, including retrieved context and tool results. Record observable actions and state changes; a generated explanation alone does not establish why the system failed.

• Example: a trip-planning agent recommends flights despite a stored train preference. Check whether the preference was saved, retrieved for the correct user, included in the model's input, and reflected in the search tool's arguments. Fix the earliest failing step.

< What should we inspect first? >

Symptom	Inspect	Possible fix
Chooses the wrong tool	Tool descriptions, available tools, and the model's tool call	Clarify when each tool applies; remove overlapping descriptions
Tool fails	Arguments, schema validation, credentials, timeout, and returned error	Validate arguments and handle the specific failure
Forgets earlier information	Session ID, saved memory, retrieved records, and final prompt	Reuse the correct session and include relevant memory within the context budget
Gives an unsupported answer	Retrieved documents and tool results, then the answer	Repair retrieval if evidence is missing; improve answer generation if evidence was present
Repeats the same action	Repeated tool arguments, unchanged state, and stop conditions	Save progress, bound retries, and stop when no progress is made
Is slow or expensive	Per-step latency, token counts, tool-call count, and retries	Reduce unnecessary calls or context at the measured bottleneck

< Tracing with LangChain and LlamaIndex >

LangSmith is a tracing and evaluation platform that integrates with LangChain and LangGraph. For an existing LangChain application, a minimal tracing configuration is:

export LANGSMITH_TRACING=true
export LANGSMITH_API_KEY="<your-api-key>"
export LANGSMITH_PROJECT="agent-debug"

Run the application and inspect its trace in LangSmith. Instrument custom steps when they are not captured automatically. Tracing sends recorded data to the configured service, so redact credentials and sensitive fields. Account-specific endpoint and workspace settings are covered in the LangSmith tracing guide.

LlamaIndex provides instrumentation for recording events and spans, where a span represents an operation and its duration. Connect it to a supported tracing integration to inspect model calls, retrieval, and other instrumented steps. See the LlamaIndex observability guide for setup options. The LangSmith environment variables above configure LangChain tracing; LlamaIndex needs its own integration setup.

< Reproduce the failure and verify the fix >

1. Save a failing case: capture the request, relevant starting state, configuration, and tool responses.
2. Isolate the failing step: rerun it with recorded inputs and mocked external tools. A normal code debugger can inspect tool implementations and state updates.
3. Change one cause: adjust the tool, retrieval, memory handling, or prompt indicated by the trace.
4. Add a regression case: check the expected answer or action and any required constraints. Repeat model-dependent cases to measure reliability, since outputs can vary.

LangGraph can resume or fork execution from a saved checkpoint to investigate a specific step. Downstream model and tool calls execute again, so use mocked tools when replaying actions that would change external state. See LangGraph time travel.

Video Tutorial

Related ideas

• AI Guardrails validates inputs and outputs and constrains data access, tool actions, and agent budgets.
• Fine-Tuning compares training objectives and parameter-update methods.
• Low-Rank Adaptation (LoRA) explains the adapters used for parameter-efficient training.
• Agentic RAG Observability explains traces, logs, metrics, and how they support evaluation.
• Agentic RAG Evaluation explains test datasets, metrics, and evaluation runners.

Reference

• Agentic RAG architecture (Microsoft)
• SFT tool calling (Hugging Face)
• PEFT integration (Hugging Face)
• Soft prompts (Hugging Face)
• Trace LangChain applications (LangSmith docs)
• Use time travel (LangGraph docs)
• Observability (LlamaIndex docs)
• Memory overview (LangChain docs)
• Persistence (LangChain docs)
• Embedding space and static embeddings (Google Machine Learning Crash Course)
• Designing Agentic Systems with LangChain (by DataCamp)
• LangGraph for Beginners: A Simple and Intuitive Guide to Getting Started (by Jyoti Dabass, Ph.D.)
• Building an Agentic RAG with LangGraph: A Step-by-step Guide (by Wendell Rodrigues, Ph.D.)
• Graph API overview (LangChain docs)
• Agentic RAG with Milvus and LangGraph (Milvus docs)