LLMOps Engineer

Design an eval framework that tells you whether a production prompt survives a model migration (Claude 4.7 to 5.0, or Claude to GPT-5.5). What do you measure and what's the gate?

You're migrating a production prompt from Claude Opus 4.7 to GPT-5.5 to cut cost. What's the most reliable indicator that you can or cannot move the prompt as-is?

Design a production RAG system serving 10,000 concurrent users over a 100M-document corpus, with a P95 latency target of 2 seconds. Walk through the architecture and the bottleneck at each layer.

Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1-point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?

Multi-GPU FT: pick the right sharding strategy for a 70B full FT on 8 H100s

Match multi-GPU training strategy to what it shards

Explain bf16's advantage over fp16 and what gradient checkpointing trades

Why does fp16 fine-tuning need dynamic loss scaling but bf16 doesn't?

Estimate peak VRAM for a 7B full-FT run in bf16 + Adam

Fill in the full-FT memory contributions for a 7B model

LoRA vs full FT on 7B + 10k examples: order of magnitude cost comparison

When is gradient accumulation NOT equivalent to a real larger batch?

Which statements about FP8, INT8 and INT4 weight quantization are correct?

Match each weight-quantization regime to what it buys you and where it breaks

What does W8A8 vs W4A16 mean and which regime is appropriate for memory-bound decode?

Match each weight/activation quant regime to its mechanism and best-fit workload

How does PagedAttention enable prefix sharing across requests and what data structure makes it cheap?

Define TTFT, TPOT and throughput. Which should a chatbot vs a batch summarizer optimize?

A chatbot team reports TTFT P99 jumped from 300ms to 1.4s overnight. Which root cause is most likely?

Why does tree-based speculative decoding outperform sequential draft chains at the same target-pass cost?

Spot the errors in this tensor-core utilization claim

How does vanilla speculative decoding work and why is verification a single forward pass?

Put the steps of one round of vanilla speculative decoding in the correct order

Spot the errors in this claim about speculative decoding

Which regimes can make speculative decoding fail to speed up, or actively slow down, a serving stack?