Short Answer

Name two production KV-cache eviction policies and what each prioritizes

Translating one source sentence into many target languages, what attention-side trick scales?

Where inside the attention sub-layer is dropout typically applied during training, and what does dropping there teach the model?

Decoder-only LLMs ship without a cross-attention sub-layer, so where does the context go?

Decode-phase attention is memory bandwidth bound. Explain what flips the regime.

Describe how continuous batching interacts with the KV cache

How does prefix KV-cache sharing reduce serving cost across requests?

In a 7B transformer, do attention or MLP layers eat more FLOPs, and does context length change the answer?

Name two recovery moves a serving stack can pull when KV memory runs short mid-request.

Design a diagnostic eval for an agentic workflow with 62% task success across 5 tools and 10+ steps that reveals WHY tasks fail, not just whether they pass.

LLM-as-judge scores drift up across 50k weekly evals while user complaints hold steady. Diagnose likely causes and design a calibration system preventing silent score inflation.

Design a multi-dimensional eval dashboard for a production LLM product. What dimensions, views, and alerts does it need, and what is the one chart that the VP of Engineering should check every Monday?

Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.

Writing the LLM-as-judge prompt for an eval pipeline: name three biases to design against and one mitigation each.

A RAG pipeline just shipped for internal docs. How would the team know if retrieval is actually helping?

How do you evaluate a customer service chatbot that handles multi-turn conversations? What dimensions matter?

You suspect your custom eval set leaked into a model's training data. Describe two practical tests you would run to detect contamination.

An 18-month-old golden eval set, departed annotators, three model updates, shifted production traffic. Design the lifecycle management process.

A consumer-facing LLM product is about to launch. Describe the safety evaluation that should run first.

A production model has no ground-truth labels on live traffic. Design a monitoring plan that catches degradation before users complain.

The PM asks how to tell if the chatbot got better after the last deploy. Walk through setting up an A/B test.

Building a golden dataset for RAG eval from scratch: walk through the steps and name the mistake that wastes the most labeling budget.

The model provider just updated the model behind the API endpoint. Describe the regression testing process before allowing traffic.

Design a multi-tenant remote MCP server that exposes each customer's Postgres database. Walk through transport, auth, isolation, and observability.

A 40-team org already has OpenAPI specs and three LLM providers. Defend layering MCP on top instead of picking one integration approach.