AI Product Manager

Explain why string-match accuracy is systematically misleading for LLM evaluation

Why string-match accuracy fails as an LLM evaluation metric

Which properties correctly describe the complementary roles of the three LLM evaluation modes?

Which statement best describes what each of the three LLM evaluation modes covers?

Design a slice-level eval strategy for an LLM system where mean score is insufficient

Why is reporting only mean eval score dangerous for LLM quality tracking?

Design an automated red-teaming pipeline for LLM safety evaluation

What is red-teaming in LLM safety evaluation and how does automated red-teaming scale it?

Spot the design flaws in this LLM evaluation rubric

Which properties characterize a well-designed LLM eval rubric?

Order the stages of a production LLM quality assurance pipeline from development to release

Compare offline and online LLM evaluation: when is each the right tool?

How do you navigate the latency, cost, quality triangle when choosing an eval approach?

Order the three eval approaches from cheapest/fastest to most expensive/slowest

How large should a golden eval set be, and what signals tell you when you have enough examples?

What is the practical guidance on golden eval set size for LLM systems?

Spot the flaw in this eval set construction approach

What is the primary pitfall of building your eval set exclusively from production traffic logs?

Compare Promptfoo, DeepEval, and LangSmith as LLM eval frameworks: when to use each?

Match each LLM eval framework to its primary differentiating capability

Explain benchmark contamination and its effect on reported LLM capability scores

How does Chatbot Arena compute ELO rankings, and what bias affects its validity?

What is the primary validity threat to Chatbot Arena ELO rankings as a measure of general model quality?

How do you measure demographic or stereotyping bias in an LLM using a standardized eval?

Analyze MMLU's strengths and failure modes as an LLM benchmark