AI System Design
Also known as: LLM system design
End-to-end design of production LLM systems: ingestion, retrieval, serving, eval, monitoring.
Designing end to end AI-powered systems that handle ingestion, processing, model serving, retrieval, caching, monitoring, and scaling. A key interview topic for senior roles.
In practice
The make-or-break round for senior and staff GenAI roles. Practice drawing the full system for RAG, agents, and multi-tenant inference.
Related topics
Related terms
Retrieval-Augmented Generation (RAG)
Fetch relevant docs at inference time and stuff them into the prompt so the model can answer from real data.
Chunking
Cutting documents into retrievable pieces before embedding them.
Context Window
The max number of tokens a model can attend to at once.
Hallucination
When a model confidently makes up something that isn't true.
HNSW (Hierarchical Navigable Small World)
A graph-based ANN index with log-time approximate search, excellent recall/latency trade-off.
vLLM
An open-source serving engine with PagedAttention. Much higher throughput than naive HF Transformers serving.