Why does composing two LCEL Runnables with `|` give you batch and streaming for free?

MCQ·Medium·4.0 · 0·~1 min·Asked atMicrosoft Cursor Perplexity

Attempt it

**LCEL** (LangChain Expression Language) is built around one interface: the **Runnable**. Every Runnable exposes four execution methods: invoke for a single sync call, batch for a list of calls (often parallelized), stream for token by token sync streaming, and astream for async streaming. The pipe operator `|` is overloaded so that `chain = prompt | model | parser` returns a new Runnable, a RunnableSequence. The new Runnable's four methods are mechanically derived from the steps. Invoke calls invoke on each step in order. Batch calls batch on each step on the whole list. Stream chains the streams so tokens flow through each step. Astream does the same async. This is why you get batch and streaming "for free". There is no compilation, no caching, no CUDA kernel. The composite Runnable just delegates to the underlying steps. Anyone could implement the same pattern in a few hundred lines of Python. The practical implication is the symmetry: if every step in your chain implements all four methods correctly, the chain does too. If one step only implements invoke, the chain's stream won't actually stream tokens through that step.

**The Runnable interface is the LCEL contract.** Every Runnable (prompts, chat models, output parsers, retrievers, tools, custom functions) implements four execution methods: invoke, batch, stream, astream. Composing Runnables with `|` produces a new Runnable (a RunnableSequence) whose four methods are mechanically derived from the underlying steps. **Concretely.** `chain.invoke(x)` is `step3.invoke(step2.invoke(step1.invoke(x)))`. `chain.batch(xs)` calls batch on each step in turn, so if step 2 is a chat model that supports concurrent calls, the batch flows through naturally. `chain.stream(x)` chains generators: step 3 consumes the tokens step 2 is yielding, which consumes what step 1 produced. `chain.astream(x)` is the async-iterator version of the same composition. **No magic.** There is no compilation pass, no JIT, no CUDA kernel, no implicit cross-user batching. The pipe operator is `__or__` returning a RunnableSequence that delegates to the steps. You could implement this pattern in a few hundred lines of Python. **The practical implication is symmetry.** The chain inherits each step's strengths AND each step's weaknesses. If one step only properly implements invoke and falls back to a non-streaming default for stream, the chain's streaming breaks at that step. The contract works because every step is expected to implement all four methods faithfully.

Quick reference

TL;DR

Every LCEL Runnable implements four execution methods; piping them produces a Runnable whose four methods derive from the underlying steps. No magic, just interface composition.

Memory aid

Easy to grasp

Picture a relay race where every runner trains to do four things: walk, sprint, batch-sprint with a teammate, and stream messages while running. When you string four such runners together as a team, the team also walks, sprints, batch-sprints, and streams messages, because each member knows how. Nobody had to invent a new kind of running for the team. That is how LCEL chains work: each step promises four execution shapes, and the chain inherits them for free by handing the work step by step.

Key concepts

LLM Frameworks

Detailed answer & concept explanation

~5 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example.

Quick scan · 2–4 min

Everything important, quickly.

Name the four Runnable methods, explain that `|` returns a RunnableSequence whose methods delegate per-step, debunk the magic (no compile, no cache, no JIT), call out the weakest-link consequence, mention per-step batch concurrency, and finish with where to apply retry/fallback.

Real-world examples

Real products, models, and research that use this idea.

The default LangChain chat-completion chain `prompt | ChatAnthropic() | StrOutputParser()` streams Claude Sonnet 4.6 tokens through the parser because all three steps implement `.stream` faithfully.
Vercel AI SDK's React `useChat` hook consumes the chain's `.astream` to drive a token by token UI, the same chain serves `.invoke` from a server-side handler.
LangChain's `RunnableParallel` shows the same contract for fan-out: `{"a": chain_a, "b": chain_b}` is a Runnable whose `.invoke` invokes both in parallel and returns a dict.
Open-source production deployments commonly wrap third-party tools as Runnables (`RunnableLambda(fn)`), the wrapping adds the four-method interface so the tool plugs cleanly into the chain.

Interviewer follow-ups

What an interviewer would ask next. Try answering before peeking at the approach.

QWalk through what happens when one step in a chain implements only `.invoke` and the user calls `chain.stream(x)`.

QHow does `RunnableParallel` differ from `RunnableSequence`, and what's the interface contract for each?

QWhere do you apply `.with_retry` and `.with_fallbacks` in a multi-step chain to maximize robustness?

Don't say this

Red flags and common mistakes that signal junior thinking. Click to expand.

Most common mistake

Believing LCEL does compilation, caching, or implicit parallelism. The pipe operator is plain interface composition: methods on the chain call methods on each step.

60-second cheat sheet

The night-before-the-interview bullets. Scan these on the way to the call.

References

Primary sources. Skim if you want the original framing.