Zenaique

Pick the right CrewAI Process for a fan out research pipeline and defend the choice

Short answer·Hard·4.0 · 0·~3 min·Asked atHebbiaSpotifyTuring
Attempt it

You have a pipeline: one query goes to four parallel researchers (web, arxiv, internal docs, news), then one synthesizer combines their outputs into a single brief. Which CrewAI Process do you choose, and what concrete configuration makes the fan out actually parallel rather than serial?

Free · 2 AI evals / day
TL;DR

Sequential Process with async_execution=True on the four researcher Tasks and an explicit context=[...] fan-in on the synthesizer, hierarchical would add a manager LLM for a topology you already know.

Memory aid
Sign in to see the mnemonic that makes this stick.
Easy to grasp

Think of running a small newsroom. You know exactly what you want: four reporters file stories, one editor stitches them into a brief. You do not need to hire an editor in chief to decide who writes what, the assignment is obvious. You just need the four reporters to write at the same time instead of one after another, then have the editor wait for all four before starting. That is what async_execution does in CrewAI: it tells the four researcher Tasks to go in parallel, while the synthesizer Task stays patient and reads all four when they finish.

Key concepts

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

CrewAI ships two Processes, Sequential and Hierarchical, and an async-execution flag on each Task. Those three primitives compose into most real-world Crew topologies. The interview question is whether you can pick the right combination for a fan-out / fan-in shape without reflexively reaching for Hierarchical.

The pipeline here is deliberately the canonical case: four parallel researchers feeding one synthesizer. The graph is fixed in your code, so there is no routing decision an LLM needs to make at runtime. That eliminates Hierarchical immediately. What is left is wiring Sequential to actually run in parallel, which is where the practical knowledge lives.

Mental model: the Process is the orchestration shape; async_execution and context are the wiring. Pick the cheaper shape, then wire it correctly.

Why Sequential, not Hierarchical

What each Process actually does

Sequential Process runs Tasks in the order you list them. Each Task's output is appended to a shared context that subsequent Tasks can read. There is no manager LLM, no routing decision, no extra call per turn.

Hierarchical Process spins up a manager Agent (you provide a manager_llm) that reads the Crew's overall goal and decides, per turn, which Agent should work on which Task. It is a real LLM call per routing decision, with a real cost and a real failure mode (manager picks the wrong agent, skips a step, or loops).

When the manager earns its keep

Hierarchical pays off when the graph is dynamic: you do not know which agent should run until you see the query. A general research assistant where some questions need code execution and others need only web search benefits from the manager picking.

Why the fixed 4-to-1 case does not

In the pipeline at hand, the assignment is hard-coded: web Task always runs the web researcher, arxiv Task always runs the arxiv researcher, the synthesizer always runs last. There is zero routing for the manager to perform, so Hierarchical only adds latency, tokens, and an extra failure point. Sequential is the strictly better fit.

Making Sequential parallel with async_execution
Fan-in: why context=[...] is non-optional
Production caveats that actually bite
Sign in to unlock the full deep dive.

Situations where this technique stops working.

Sign in to see when this approach fails.

2–4 min · Everything important, quickly.

Sign in to see the quick scan of the deep dive.

Real products, models, and research that use this idea.

  • Perplexity-style 'deep research' agents use exactly this pattern, fan out to several search backends, fan in to a synthesizer.
  • Bloomberg-internal research bots fan out across earnings, news, and filings retrievers, then a writer Task synthesizes.
Sign in to see more production examples.

What an interviewer would ask next. Try answering before peeking at the approach.

QIf one researcher times out, how do you keep the synthesizer from blocking forever and still produce a useful brief?
A

Wrap each researcher Task's tools with per-call timeout + retry, catch the failure in the synthesizer prompt by passing a default 'no data from source X' string, and consider Task-level fallbacks so one dead retriever degrades gracefully instead of failing the Crew.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Reaching for Process.hierarchical because the pipeline 'feels' multi-step. Hierarchical pays a manager-LLM call per routing decision, on a fixed 4-to-1 shape, that cost buys nothing.

Sign in to see all red flags and common mistakes.

60 second bullets to scan on the way to the call.

  • When does Sequential beat Hierarchical?

  • What does async_execution do on each Task during a fan-out?

Sign in to unlock the revision sheet.

Primary sources. Browse if you want the original framing.

Similar questions

Same topic, related formats. Practice these next.

4 curated
Next question
Defend the call to…
Short answer·Hard