AI Product Engineer

How does the MCP Prompts primitive differ from the Tools primitive in terms of who controls invocation?

How should a production MCP host handle a tool call that exceeds the latency budget?

What is the correct approach when an MCP tool call exceeds its per-tool timeout?

What is the MCP host approval model and at what granularity should consent be obtained?

Which practices represent good host-side approval UX for MCP tool calls?

When should automated eval escalate to human review in a production LLM product?

Which user-visible failure modes should trigger escalation from automated eval to human review?

What makes multi-turn dialog evaluation harder than single-turn, and how does MT-Bench address it?

Which quality dimension is uniquely challenging about multi turn dialog evaluation vs single turn?

How do you detect quality drift in a production LLM system without ground-truth labels?

Which proxy signals are valid indicators of potential quality drift in a production LLM system?

How do you set up an A/B test for an LLM system change, and what's different from standard A/B testing?

Which considerations are uniquely important for A/B testing LLM systems vs standard UI testing?

How can an agent confidently produce a wrong answer without raising any runtime exception?

How does prompt injection in an agent differ from injection in a chat interface, and why is it OWASP LLM Top-10 #1?

Why is prompt injection more dangerous in an agent than in a chat only LLM?

Describe three patterns for inserting human approval into an agent workflow and their tradeoffs

Match each HITL pattern to its defining tradeoff

Where in an agent workflow should a human approval gate be placed for an irreversible action?

What does cascading error compounding mean for production agent design and reliability budgeting?

Predict: what is the end to end success rate of a 10-step agent task if each step succeeds with 90% probability?

Predict: what per-step accuracy is required to achieve 90% end to end success on a 10-step agent task?

Browser agents drive a real UI instead of calling text APIs. What extra failure modes show up?