AI Product Manager

Walk through SWE-bench, AgentBench, GAIA, and TAU-bench: what they measure, their shared blind spots, and how to interpret agent benchmark numbers.

You're shipping a prompt change to production. What is the strongest reason to treat prompt-template version IDs (e.g. v1.7.2) as first-class artifacts, separate from your code release version?

Explain how a DSPy-style 'prompt compiler' optimizer (like BootstrapFewShot) actually optimizes a prompt program and what makes it different from hand-tuned prompt engineering.

Your team is comparing a vanilla LLM chatbot against a RAG chatbot. What is the single most important difference between them at query time?

Design a golden eval set for a production RAG system serving a legal research product. Explain how you build it, how big it should be, what each example contains, and what you measure with it.

Your production RAG costs $1M/month. The CFO wants this cut in half with a max 1-point faithfulness regression. What highest leverage cost optimizations do you deploy, in priority order?

When IS fine-tuning the right call?

Three concrete situations where fine-tuning is the wrong tool

Which of these are valid reasons NOT to fine-tune?

Why 'fine-tune on our company docs' usually fails: and what to do instead

Best architecture for a chatbot over a frequently-updated company knowledge base?

Match open base model to its license consideration

Closed-API fine-tuning in 2025-2026: options and trade-offs vs open-weight FT

How do Anthropic and OpenAI prompt-cache pricing models work and what counts as a cache hit?

How do online (interactive) and offline (batch) inference workloads differ in optimization target?

Why are output tokens typically 3-5× more expensive than input tokens?

What is the 'token tax' and how does it create downstream accuracy disparities across languages?

Define 'fertility' as a tokenizer metric and explain why high fertility for a language is both a fairness and a model performance problem.

Estimate the token cost difference for processing 1M Spanish/French product descriptions vs. an English-only baseline, and describe correct budget planning.

You need an LLM app to answer questions about your company's internal docs, which are updated weekly. Should you RAG or fine-tune, and what factor most cleanly tips the decision?

Match each RAGAS metric to what it specifically measures about a RAG pipeline.

Complete the claim: the two-axis decomposition of RAG eval that diagnoses which layer failed.

Your prompt iteration has plateaued. Walk through the decision of whether to fine-tune or invest in further prompt engineering, with explicit signals for each path.

When should you fine-tune the model instead of iterating on the prompt?

How does DSPy reframe prompt engineering and what does it gain over hand-written prompts?