Technical branch for developers Assemble manually

RAG: How to Give a Model Company Knowledge

Why doesn’t the LLM know about your CRM? How to connect internal documents without fine-tuning? We’ll walk through building a production-ready RAG pipeline—with code, pitfalls, and evals.

Discuss implementation All Academy Topics

What is RAG and why is it needed?

RAG (Retrieval-Augmented Generation) — architecture where the LLM receives context not from training, but from an external source at the moment of generating a response. This enables:

Use current, private data (knowledge bases, documents, CRM exports).
Avoid overfitting and reduced model speed.
Control the source of the response (important for compliance and auditing).

Without RAG, the LLM “hallucinates”—it doesn’t know what’s in your database until you explicitly provide that context.

RAG Pipeline: 4 Stages

flowchart TD  
    A[Data Source: PDF, DB, Notion] --> B[Preprocessing]  
    B --> C[Vectorization (Embedding)]  
    C --> D[Vector DB (Chroma, Qdrant)]  
    D --> E[Relevance Search]  
    E --> F[Context + Query → LLM]  
    F --> G[Answer with Sources]

Key idea: Divide and conquer — data processing separate from generation.

Minimal RAG Pipeline (Python + LangChain)

from langchain_community.document_loaders import TextLoader  
from langchain_text_splitters import RecursiveCharacterTextSplitter  
from langchain_chroma import Chroma  
from langchain_openai import OpenAIEmbeddings, ChatOpenAI  
from langchain_core.prompts import ChatPromptTemplate  

# 1. Load and split  
loader = TextLoader("docs/internal_knowledge.txt")  
docs = loader.load()  
splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)  
chunks = splitter.split_documents(docs)  

# 2. Vectorize and store  
vectorstore = Chroma.from_documents(  
    chunks,  
    embedding=OpenAIEmbeddings(),  
    persist_directory="./chroma_db"  
)  

# 3. Retrieval + generation  
retriever = vectorstore.as_retriever(search_kwargs={"k": 3})  
prompt = ChatPromptTemplate.from_template("""  
Answer the question using only the provided context.  
Question: {question}  
Context: {context}  
""")  
llm = ChatOpenAI(model="gpt-4o-mini")  

# Pipeline  
def rag_chain(question):  
    docs = retriever.invoke(question)  
    context = "\n\n".join([d.page_content for d in docs])  
    return llm.invoke(prompt.format(question=question, context=context))  

print(rag_chain("How to apply for childcare leave?").content)

In production, replace `TextLoader` with real sources (PDF, API, databases), add post-processing and evals.

Common mistakes (and how to avoid them)

Chunks are too large → the model loses context. Solution: Chunks of 300–600 tokens, with 10–20% overlap.
Bad embeddings → Search returns “similar, but not quite.” Solution: Use specialized models (e.g., `bge-small-en-v1.5` for Russian—`cointegrated/rubert-tiny2`).
No noise filtering → the model generates outputs based on outdated/false data. Solution: Add post-filtering by relevance (BM25, cosine similarity + metrics).
Ignoring metadata → impossible to trace the source. Solution: Store `source`, `page`, and `updated_at`, and pass them to the prompt.

How to Check RAG Quality: Simple Evaluation

Create a test set of 10–20 questions with model answers and sources. For each question:

Get a response from RAG.
Check if the source is in the top-3 most relevant chunks (precision@3).
Rate the answer on a 0–1 scale (yes/no based on accuracy and completeness).

# Metric example
def eval_rag(question, expected_answer, expected_sources):
    response = rag_chain(question)
    retrieved = retriever.invoke(question)
    sources_found = any(src in [d.metadata.get('source') for d in retrieved] 
                        for src in expected_sources)
    # Simple check: whether there are keywords from the standard
    has_keywords = all(kw.lower() in response.content.lower() 
                       for kw in ["vacation", "child", "law"])
    return int(sources_found and has_keywords)

# Result: 0.8 - 80% correct answers

Goal: >0.7 precision@3 and >0.6 factual accuracy.

Pre-Launch Production Checklist

[ ] Data normalized (encoding, HTML cleanup, structuring).
[ ] Chunks with overlap and metadata (source, date, type).
[ ] Embeddings tested for relevance (visualization via UMAP).
[ ] Vector DB with index (HNSW/IVF) and search timeout.
[ ] Request logging + retries on timeouts.
[ ] Prompt with instruction: “If there is no answer in the context, say so.”
[ ] Metrics: latency (p95 < 1.5s), precision@3, user satisfaction (A/B).

Next step

After basic RAG—move on to:

Agents with tool-calling — RAG as an agent tool.
AI Agent Testing — How to evaluate RAG within an agent-based system.
From prompt to production — What else needs to be added (logging, safety, guardrails)?

Don’t forget: RAG isn’t “set and forget.” It’s a cycle: monitor → audit → improve chunks and embeddings.