AI System Design
Also known as: LLM system design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Designing end to end AI-powered systems that handle ingestion, processing, model serving, retrieval, caching, monitoring, and scaling. A key interview topic for senior roles.
In practice
The make-or-break round for senior and staff GenAI roles. Practice drawing the full system for RAG, agents, and multi-tenant inference.
Related topics
Questions that mention this term
- Order the cache layers an LLM request should check before paying for tokens
- How do you keep a long conversation inside the context window without losing state?
- Why is failing over…
- Order the layers you'd investigate when debugging a RAG faithfulness regression that just shipped to production, from most likely to least likely cause.
- What is the practical guidance on golden eval set size for LLM systems?
- Predict the hardware cost vs gross margin for serving $15/M output tokens on H100
Related terms
BM25
Classical TF-IDF-style lexical scoring: a surprisingly strong baseline, still the first stage of many hybrid RAG pipelines.
Chunking
Cutting documents into retrievable pieces before embedding them.
Context Engineering
Deciding what makes it into the model's context window (prompt + RAG + history + memory), and why.
Context Window
The max number of tokens a model can attend to at once.
Fixed-size Chunking
Slice documents every N tokens with a small overlap. Fast and format-agnostic, but cuts through headings and mid-topic.
Hallucination
When a model confidently makes up something that isn't true.