📝 Retrieval Augmented Generation (RAG)

Description

< What is it? >

Retrieval Augmented Generation (RAG) is a technique that combines the power of large language models (LLMs) with external knowledge sources to improve the quality and relevance of generated content. By retrieving relevant information from a knowledge base or database, RAG enhances the model's ability to generate accurate and contextually appropriate responses.

Langchain helps to implement RAG by providing tools and frameworks for integrating LLMs with external data sources, enabling developers to build applications that leverage both the generative capabilities of LLMs and the structured information from databases.

< Main advantages >

1. Up-to-date knowledge — you don't need to retrain the LLM when documents change.
2. Private knowledge — company documents can be supplied at query time.
3. Reduced hallucination — the model can ground its answer in retrieved documents.
4. Citations/sources — you can tell the user which documents support the answer.

Key points

• Data Preparation
  • Load docs (.txt, .pdf, csv, html, etc.)
    • Split into passages (e.g., 100–500 words)
      • Semantic splitting (e.g., using sentence embeddings) is preferred over naive splitting (e.g., by 100 tokens or 1 paragraph).
        • Semantic Chunking for RAG (by Plaban Nayak)

        • # Instantiate an OpenAI embeddings model
          embedding_model = OpenAIEmbeddings(api_key="<OPENAI_API_TOKEN>", model='text-embedding-3-small')
          
          # Create the semantic text splitter with desired parameters
          semantic_splitter = SemanticChunker(
            embeddings=embedding_model, breakpoint_threshold_type="gradient", breakpoint_threshold_amount=0.8
          )
          
          # Split the document
          chunks = semantic_splitter.split_documents(document)

  • Embed each passage into a vector space (e.g., OpenAI embeddings)
  • Store the embeddings in a vector database (e.g., Pinecone, Weaviate, Milvus, Chroma, FAISS)
• Graph Database
  • Neo4j is a graph database that can be used to store and query relationships between entities in the knowledge base. It allows for efficient retrieval of relevant information based on the relationships between entities.
  • LLMGraphTransformer
  • LangChain Cypher search
• RAG evaluation
  • Building an Agentic RAG evaluation system covers answers, retrieval, tool use, memory, and efficiency.
  • Evaluate a RAG application (by langsmith)
  • Application-specific evaluation approaches (by langsmith)
  • Evaluating RAG pipelines with Ragas + LangSmith (by The LangChain Team)
  • Intro to AI Evaluation with Langsmith (by Adrián Jiménez)
• APIs
  • langchain doc loaders
  • langchain text splitters
  • langchain SemanticChunker
  • Evaluating RAG pipelines
    • Context precision
    • Context recall
    • Faithfulness

Architecture

RAG architecture

Neo4j graph database

Comparison

< Python frameworks for a RAG backend and frontend >

• Suggested starting stack: use LlamaIndex + Chainlit for a RAG chatbot. Write the retrieval logic and chat handlers in Python; Chainlit provides the browser interface.

  Layer	Framework	Responsibility
  RAG logic	LlamaIndex	Load documents, create indexes, retrieve evidence, and generate answers. Docs
  Backend API, when needed	FastAPI	Expose the Python application to clients through APIs and WebSockets. Docs
  Chat frontend	Chainlit	Provide a chat UI, stream responses, and display images or other elements through Python code. LlamaIndex integration

  A small application does not need a separate FastAPI service: Chainlit's Python handlers can call LlamaIndex directly. Add FastAPI when other applications also need to call the RAG backend.

  Browser: Chainlit chat UI
               ↕
  Python: Chainlit message handlers
               ↓
           LlamaIndex
               ↓
       Vector database + LLM

• Frontend choices: both frameworks let you build a browser UI using Python.

  Framework	When to choose it	Main tradeoff
  Chainlit	A RAG chatbot with streaming responses and supporting documents/images	Focused on conversational interfaces
  Streamlit	A RAG demo or internal dashboard with chat, tables, filters, and charts	Less control over a highly customized chat experience

  Streamlit provides st.chat_input, st.chat_message, and st.write_stream. See the Streamlit API.

• Agentic RAG: add LangGraph when you need explicit control over branching workflows, state, and repeated tool calls. It handles orchestration; retrieval components and a UI are still needed. See the LangGraph overview.

• Example — images in streamed answers: Chainlit's Image element accepts a URL, local path, or bytes. The backend can send a completed image element separately from streamed text, avoiding image replacement based on incomplete Markdown such as [Figure-1](partial-url. Exact placement between paragraphs still needs application logic. See the Chainlit image documentation.

Video Tutorial

Related ideas

• AI Guardrails covers retrieval permissions, citation checks, and response validation.
• Search & Retrieval explains lexical, semantic, and hybrid retrieval followed by optional reranking.
• Term Frequency–Inverse Document Frequency (TF-IDF) explains a lexical method for weighting terms and retrieving passages.

Reference

• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (by Patrick Lewis, etc 2020)
• Retrieval Augmented Generation (RAG) with Deep Agents | LangChain
• Retrieval Augmented Generation (RAG) with LangChain | DataCamp