AI Product Manager

What are the strengths and weaknesses of MMLU as an LLM benchmark?

How do you set up an A/B test for an LLM system change, and what's different from standard A/B testing?

Which considerations are uniquely important for A/B testing LLM systems vs standard UI testing?

What does SWE-bench measure that AgentBench does not, and why are both needed?

Match each agent benchmark to what it primarily measures

When is running multiple agents worth the coordination overhead?

What coordination overhead does a multi-agent system introduce that a single-agent system avoids?

When is using an agent architecture an engineering mistake, even when LLMs are involved?

Compare CrewAI, AutoGen, and LangGraph coordination models and their reliability tradeoffs

Match each multi-agent framework to its coordination model

How do you define a cost per task budget for an agent and enforce it at runtime?

Why is scoring only the final answer a weak way to evaluate agents?

What does trajectory evaluation measure that final answer accuracy alone cannot capture?

Estimate the per call cost of a typical RAG chatbot using GPT-4o-mini.

When should you fine-tune instead of using RAG?

How does MCP differ from OpenAI-style function calling?

What is the Model Context Protocol (MCP) and what problem does it solve?

What does RAG primarily help with in LLM-based applications?