How does LlamaIndex ChatEngine differ from QueryEngine?

MCQ·Medium·4.0 · 0·~1 min·Asked atBcg Databricks Midjourney

Attempt it

LlamaIndex has two related abstractions on top of an Index: QueryEngine and ChatEngine. QueryEngine is the stateless RAG primitive. You hand it a query, it retrieves the most relevant Nodes from the Index, a response synthesizer composes them into an answer, and the call returns. No memory of previous queries, no notion of a conversation. Each call is independent. ChatEngine wraps a QueryEngine (or directly wraps an Index) and adds two things: conversation memory and a strategy for handling follow-up turns. The strategy depends on the mode you pick: - `condense_question`. Rewrites a follow-up into a standalone question using the chat history, then sends that standalone question to the underlying retrieve and synthesize step. - `context`. Keeps the conversation history in the prompt and lets the model reason over both the history and the retrieved context. - `react`. Runs an agent loop that may or may not retrieve, depending on the model's tool-calling decision. The key difference is multi-turn awareness, not the index type, not streaming, and not the provider.

**QueryEngine is stateless RAG; ChatEngine adds conversation handling.** Both sit on top of an Index, both can use the same retriever and response synthesizer, but they answer different questions. `QueryEngine.query(q)` is one-shot: retrieve relevant Nodes for `q`, synthesize an answer, return. No memory of prior calls. This is the right primitive for a search box or a single-turn Q&A endpoint. `ChatEngine.chat(message)` is multi-turn: maintain the conversation history, decide how to handle a follow-up, and only then drop into the retrieve and synthesize layer. The 'how to handle the follow-up' part is what the modes encode: - **condense_question**. Uses the chat history to rewrite an underspecified follow-up ('and what about its side effects?') into a standalone query ('what are the side effects of ibuprofen?') before retrieval. Solves the pronoun with no antecedent retrieval bug. - **context**. Injects the chat history into the synthesis prompt alongside the retrieved context, letting the model reason over both. Cheaper than condense_question (no extra rewrite call) but worse retrieval when the follow-up is underspecified. - **react**. Runs a tool-calling loop where retrieval is one tool the model can choose to call. Right when some turns need retrieval and others (small talk, clarifications) do not. The other three options are wrong on the facts. ChatEngine uses whatever Index you give it; both stream; both work with any provider LlamaIndex supports.

TL;DR

QueryEngine is stateless retrieve and synthesize; ChatEngine adds conversation memory and can rewrite underspecified follow-ups into standalone queries before retrieval.

Memory aid

Easy to grasp

Picture a research assistant at a library. The QueryEngine version is the desk attendant. You walk up, ask one question, they pull a book, read you the relevant page, you leave. Every visit starts fresh; they do not remember you. The ChatEngine version is a personal researcher you book by the hour. They remember what you asked last time. When you say 'and what about its side effects?', they know 'its' means the drug you discussed three minutes ago, and they rewrite the question into something they can actually search before pulling the next book. The difference is memory and follow-up rewriting, not the books or the search.

Key concepts

RAG LLM Frameworks

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RAG over a Q&A endpoint and RAG inside a chatbot look like the same problem from a distance. From inside the engine, they are not. The chatbot has to handle follow-up turns where the user references earlier context, and the literal turn text often does not carry enough signal for retrieval. LlamaIndex's split between QueryEngine and ChatEngine encodes exactly that distinction.

This dive walks through the stateless primitive, the three ChatEngine modes, the antecedent-resolution bug they each handle differently, and the operational signals that tell you which mode to pick.

QueryEngine. The stateless RAG primitive

QueryEngine is the canonical retrieve and synthesize flow. You construct it from an Index (index.as_query_engine(...)), call .query(q), and the engine handles three steps internally: a Retriever pulls the top-k most similar Nodes for q, NodePostprocessors optionally rerank or filter, and a Response Synthesizer composes the final answer from the Nodes.

No state survives the call. The next .query(q2) starts fresh. This is the right primitive when each query is independent: a search box, a one-shot Q&A endpoint, a 'summarise this document' button, an internal tool that takes a question and returns an answer.

The failure mode is using it for a chatbot. A second turn that says 'and what about its side effects?' has no antecedent. The pronoun 'its' refers to something said in turn one, which QueryEngine never saw. The retriever embeds the literal text, gets a noisy embedding, and pulls Nodes that are at best loosely related to what the user actually meant.

python

query_engine = index.as_query_engine(similarity_top_k=5, response_mode='tree_summarize')
result = query_engine.query('What are the side effects of ibuprofen?')
# Stateless: nothing carries to the next call.

ChatEngine. State plus a turn-handling strategy

Picking a mode and the failure modes of each

How this composes with the rest of the stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Axis	QueryEngine	ChatEngine
State	Stateless per call	Conversation memory across turns
Turn handling	Literal query → retrieve → synthesize	Mode-dependent (condense / context / react)
Follow-up rewriting	No	Yes in condense_question mode
Tool-calling option	No (just retrieve)	Yes in react mode
Cost per turn	1 retrieval + 1 synthesis	Up to 1 rewrite + 1 retrieval + 1 synthesis
Best fit	Search box, single-turn Q&A	Multi-turn chatbot with referential follow-ups

Real products, models, and research that use this idea.

LlamaIndex's documentation chatbot uses ChatEngine in `condense_question` mode because doc questions chain into 'how does this interact with X' follow-ups.
Notion AI and Linear's assistant patterns rely on the agentic mode equivalent. Retrieval as a tool, not as the default action per turn.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide between condense_question and react?

If every turn needs retrieval, condense_question is cheaper and just as accurate. If turns mix retrieval-requiring and chit-chat, react avoids the unnecessary rewrite and retrieve on non-retrieval turns at the cost of a tool-calling decision.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using QueryEngine for a chatbot and discovering that follow-up questions like 'and what about its side effects?' retrieve garbage because the pronoun gets embedded with no antecedent.

60 second bullets to scan on the way to the call.

What stateless primitive does QueryEngine represent?
How does ChatEngine add state on top of that primitive?

Primary sources. Browse if you want the original framing.