Why is the claim 'you need LangChain to do RAG' wrong, and when does LangChain earn its keep in a RAG pipeline?

Short answer·Medium·4.0 · 0·~3 min·Asked atMicrosoft Cursor Perplexity

Attempt it

An engineer insists you need LangChain to do RAG. Refute that claim concretely. Describe the minimum viable RAG pipeline and then say when LangChain (or LlamaIndex) does actually earn its keep on top of it.

Free · 2 AI evals / day

RAG, retrieval-augmented generation, is a pattern, not a framework. The pattern is four steps: 1. **Embed** the user's question with an embedding model (`text-embedding-3-small`, `bge-large`, etc.). 2. **Retrieve** the top-k most similar chunks from a vector store (Pinecone, pgvector, Qdrant, Weaviate, Chroma). 3. **Format** the retrieved chunks into a prompt with the question. 4. **Complete** by calling a chat completion API and returning the response. Written directly, this is about 80 lines of Python. No framework required. You can build it with the OpenAI SDK plus your vector store's client library. It works. LangChain (or LlamaIndex) becomes valuable when the pipeline grows past that simple shape. Common triggers: - **Hybrid retrieval.** Combine vector similarity with BM25 keyword search, then rerank with a cross-encoder. Each piece is a few lines; orchestrating the combination by hand becomes verbose. - **Multi-index routing.** Some queries should hit a vector index, others a SQL database, others a graph index. A router that picks the right one per query is messy to hand-roll. - **Response synthesizer strategies.** When the retrieved context is too long for one model call, you need refine, tree-summarize, or accumulate patterns. LlamaIndex packages these. - **RAG embedded in an agent.** When RAG is one tool a larger agent uses, the agent framework handles the orchestration. For the simple case the framework is overhead. For the complex case hand-rolling becomes more expensive than the framework tax. The misconception comes from confusing 'a popular tool exists for X' with 'X requires that tool'.

RAG is a four-step pattern: embed the query, retrieve nearest neighbors from a vector store, format the chunks into a prompt, call the model. Done directly with the OpenAI SDK and a vector store client (Pinecone, pgvector, Qdrant, Weaviate), this is roughly 80 lines of Python. No framework required to make RAG work. LangChain and LlamaIndex earn their keep when composition complexity grows past the simple shape. The common triggers are: **Hybrid retrieval.** Production RAG typically combines vector similarity with BM25 keyword search and a cross-encoder reranker. Each piece is a few lines, but orchestrating the combination by hand becomes verbose. LangChain's `EnsembleRetriever` and LlamaIndex's `QueryFusionRetriever` package this cleanly. **Multi-index routing.** A real product often has questions that should hit different backends. Vector for QA, SQL for counts, summary for overview. A router that picks per-query is reasonable to hand-roll once and painful to extend. LlamaIndex's `RouterQueryEngine` and LangChain's `RunnableBranch` make this maintainable. **Response synthesizer strategies.** When retrieved context exceeds the model's effective window, you need refine (iterate over chunks, building up the answer), tree-summarize (recursively summarize chunk groups), or accumulate (collect per-chunk answers and merge). LlamaIndex packages these as `ResponseSynthesizer` strategies. **RAG as one step in an agent.** When RAG is a tool the agent uses among many, the agent framework (LangGraph, LangChain agents) handles the orchestration, observability, and persistence. The right framing: frameworks are insurance against future complexity. Pay the premium when you expect to need it. For an MVP serving a single use case, the 80-line direct version is faster to build and easier to debug. The 'you need LangChain' framing confuses framework popularity with framework requirement.

Quick reference

TL;DR

Minimal RAG is four steps (embed, retrieve, format, complete) in 80 lines with no framework.

Memory aid

Easy to grasp

Picture a kid claiming you need a kitchen to make a sandwich. You point out that bread, peanut butter, and a knife do the job. They are right that the kitchen helps once you want to make ten different sandwiches with toasted bread and a side soup. But the kitchen is not the sandwich. RAG is the same. The 'kitchen' is the framework. The 'sandwich' is the four steps. Confusing the two leads teams to import five megabytes of code to do something the standard library could do in a screen of code.

Key concepts

LLM frameworks Retrieval-augmented generation

Detailed answer & concept explanation

~6 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

4 minutes: state the four-step minimal pipeline, name a vector store concretely, list the complexity classes that flip the framework calculus, and frame the production answer as 'piecemeal adoption when buying something with the tax'.

Code example

python

# Minimum viable RAG: no framework, ~30 lines
from openai import OpenAI
import pinecone

client = OpenAI()
index = pinecone.Index('my-index')

def rag(question: str, k: int = 5) -> str:
    # 1. Embed
    q_vec = client.embeddings.create(
        model='text-embedding-3-small', input=question
    ).data[0].embedding

    # 2. Retrieve
    matches = index.query(vector=q_vec, top_k=k, include_metadata=True).matches
    chunks = [m.metadata['text'] for m in matches]

    # 3. Format
    context = '\n\n'.join(f'[{i+1}] {c}' for i, c in enumerate(chunks))
    prompt = f'Answer using ONLY the context. If unsure, say so.\n\nContext:\n{context}\n\nQuestion: {question}'

    # 4. Complete
    resp = client.chat.completions.create(
        model='gpt-4o-mini',
        messages=[{'role': 'user', 'content': prompt}],
    )
    return resp.choices[0].message.content

print(rag('What is the company refund policy?'))

Real-world examples

Real products, models, and research that use this idea.

Perplexity AI's RAG-style search backend is built directly on model APIs and a custom retrieval stack, not on LangChain
Many enterprise search rebuilds in 2025-2026 use OpenAI or Anthropic SDKs plus pgvector and stay framework-free
LlamaIndex's QueryFusionRetriever is a common pickup once teams need hybrid retrieval with reranking
LangGraph is frequently adopted specifically for RAG-as agent tool patterns where the orchestration outgrows direct code

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QAt what point in a project's lifecycle does adopting a framework become net-positive?

QWhy might LlamaIndex be preferable to LangChain specifically for RAG workloads?

QWhat does 'eject from LangChain' look like operationally, and why does it matter for tool choice?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Mistaking framework popularity for framework requirement. The framework being widely used does not mean the underlying technique depends on it.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.