How does LlamaIndex ChatEngine differ from QueryEngine?

MCQ·Medium·4.0 · 0·~1 min·Asked atDatabricks Midjourney

Attempt it

LlamaIndex has two related abstractions on top of an Index: QueryEngine and ChatEngine. QueryEngine is the stateless RAG primitive. You hand it a query, it retrieves the most relevant Nodes from the Index, a response synthesizer composes them into an answer, and the call returns. No memory of previous queries. Each call is independent. ChatEngine wraps a QueryEngine (or directly wraps an Index) and adds two things: conversation memory and a strategy for handling follow-up turns. The strategy depends on the mode you pick: - `condense_question`. Rewrites a follow-up into a standalone question using the chat history, then sends that standalone question to the underlying retrieve and synthesize step. - `context`. Keeps the conversation history in the prompt and lets the model reason over both the history and the retrieved context. - `condense_plus_context` (the default). Rewrites the follow-up, retrieves for it, then answers with the retrieved context plus the chat history. For chats where only some turns need retrieval, an agent (`FunctionAgent` or `ReActAgent`) with the query engine as a tool decides per turn whether to retrieve. The key difference is multi-turn awareness, not the index type, not streaming, and not the provider.

**QueryEngine is stateless RAG; ChatEngine adds conversation handling.** Both sit on top of an Index and can share the same retriever and response synthesizer. `QueryEngine.query(q)` is one-shot: retrieve relevant Nodes for `q`, synthesize an answer, return. No memory of prior calls. This is the right primitive for a search box or a single-turn Q&A endpoint. `ChatEngine.chat(message)` is multi-turn: maintain the conversation history, decide how to handle a follow-up, and only then retrieve and synthesize. The 'how to handle the follow-up' part is what the modes encode: - **condense_question**. Uses the chat history to rewrite an underspecified follow-up ('and what about its side effects?') into a standalone query ('what are the side effects of ibuprofen?') before retrieval. Solves the pronoun with no antecedent retrieval bug. - **context**. Injects the chat history into the synthesis prompt alongside the retrieved context, letting the model reason over both. Cheaper than condense_question (no extra rewrite call) but worse retrieval when the follow-up is underspecified. - **condense_plus_context** (the default). Rewrites the follow-up, retrieves for the standalone question, then answers with the retrieved context plus the chat history. When some turns need retrieval and others (small talk, clarifications) do not, use an agent (`FunctionAgent` or `ReActAgent`) with the query engine as a tool. The model then decides per turn whether to retrieve. Options A, C and D are wrong on the facts. ChatEngine uses whatever Index you give it; both stream; both work with any provider LlamaIndex supports.

TL;DR

QueryEngine is stateless retrieve and synthesize; ChatEngine adds conversation memory and can rewrite underspecified follow-ups into standalone queries before retrieval.

Memory aid

Easy to grasp

Picture a research assistant at a library. The QueryEngine version is the desk attendant. You walk up, ask one question, they pull a book, read you the relevant page, you leave. Every visit starts fresh; they do not remember you. The ChatEngine version is a personal researcher you book by the hour. They remember what you asked last time. When you say 'and what about its side effects?', they know 'its' means the drug you discussed three minutes ago, and they rewrite the question into something they can actually search before pulling the next book. The difference is memory and follow-up rewriting, not the books or the search.

Key concepts

RAG LLM Frameworks

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

RAG over a Q&A endpoint and RAG inside a chatbot can look like the same problem. They are not. The chatbot has to handle follow-up turns where the user references earlier context, and the literal turn text often does not carry enough signal for retrieval. LlamaIndex's split between QueryEngine and ChatEngine encodes exactly that distinction.

QueryEngine. The stateless RAG primitive

QueryEngine is the canonical retrieve and synthesize flow. You construct it from an Index (index.as_query_engine(...)), call .query(q), and the engine handles three steps internally: a Retriever pulls the top-k most similar Nodes for q, NodePostprocessors optionally rerank or filter, and a Response Synthesizer composes the final answer from the Nodes.

No state survives the call. The next .query(q2) starts fresh. This is the right primitive when each query is independent: a search box, a one-shot Q&A endpoint, a 'summarise this document' button, an internal tool that takes a question and returns an answer.

The failure mode is using it for a chatbot. A second turn that says 'and what about its side effects?' has no antecedent. The pronoun 'its' refers to something said in turn one, which QueryEngine never saw. The retriever embeds the literal text, gets a noisy embedding, and pulls Nodes that are at best loosely related to what the user actually meant.

python

query_engine = index.as_query_engine(similarity_top_k=5, response_mode='tree_summarize')
result = query_engine.query('What are the side effects of ibuprofen?')
# Stateless: nothing carries to the next call.

ChatEngine. State plus a turn-handling strategy

Picking a mode and the failure modes of each

How this composes with the rest of the stack

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Axis	QueryEngine	ChatEngine
State	Stateless per call	Conversation memory across turns
Turn handling	Literal query → retrieve → synthesize	Mode-dependent (condense / context / condense plus context)
Follow-up rewriting	No	Yes in condense_question and condense_plus_context modes
Tool-calling option	No (just retrieve)	No (use an agent with the query engine as a tool)
Cost per turn	1 retrieval + 1 synthesis	Up to 1 rewrite + 1 retrieval + 1 synthesis
Best fit	Search box, single-turn Q&A	Multi-turn chatbot with referential follow-ups

Real products, models, and research that use this idea.

Documentation chatbots, where questions chain into 'how does this interact with X' follow-ups, fit ChatEngine in condense_question mode.
Assistants that mix small talk with document questions use an agent with retrieval as a tool, not as the default action per turn.

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you decide between condense_question and an agent with a retrieval tool?

If every turn needs retrieval, condense_question is cheaper and just as accurate. If turns mix retrieval-requiring and chit-chat, the agent avoids the unnecessary rewrite and retrieve on non-retrieval turns at the cost of a tool-calling decision.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Using QueryEngine for a chatbot and discovering that follow-up questions like 'and what about its side effects?' retrieve irrelevant nodes because the pronoun gets embedded with no antecedent.

60 second bullets to scan on the way to the call.

What stateless primitive does QueryEngine represent?
How does ChatEngine add state on top of that primitive?

Primary sources. Browse if you want the original framing.