LLMOps Engineer

Name TGI's maintainer and its niche among serving stacks

Identify PagedAttention and the project it ships with

Pair HBM and SRAM with their roles on an inference GPU

Order the pieces of state that must be restored to resume a training run bit-exactly

QLoRA OOMs at step 7000 after one clean epoch on a 24 GB GPU: which non-obvious causes deserve a look?

Estimate the trainable param count for a Llama-3 8B LoRA at r=16 targeting all-linear modules.

Same LoRA recipe on H100 vs A100: what precision should each team default to?

DDP vs FSDP for a 13B full fine-tune on 4xA100-40GB: which is feasible?

Paged optimizer in QLoRA, what gets paged, and to where?

Gradient checkpointing, what does it trade, and roughly by how much?

Compute the effective batch from per-device batch, accum steps, and GPU count

Fill in the DeepSpeed config keys that select a ZeRO stage

Accelerate launch: what does that one CLI actually wire up?

A DPO run spikes to NaN around step 400 after a clean start: explain the most likely cause

Fill in the memory delta when a 7B weight tensor moves from fp32 to bf16

Walk through KV-cache updates when speculative decoding rejects some draft tokens

Why not just retrain models to eliminate attention sinks entirely?

Prefill and decode: name the two phases of LLM inference and say which one is compute-bound

vLLM defaults to a page size of 16. Pick what makes 1 and 1024 worse choices.

Spot the masking bug in this packed-sequence training setup

DeepSeek trains models to predict K tokens per step. Pick how that stays causal.

INT8 KV cache vs INT8 weight quantization, which one is easier in production?

How does torch.nn.functional.scaled_dot_product_attention relate to FlashAttention?

Does FlashAttention change the attention output relative to standard attention?

Flash-Decoding parallelizes an axis FlashAttention v2 left alone. Pick which one.