AI Product Manager

Understand what LLMs can and can't do: capabilities, limitations, cost structures, and evaluation.

• Start hereExplain scaled dot product attention.
• Which metric best measures whether a RAG answer is grounded in the retrieved context?
• Design a RAG evaluation harness, what do you measure and how?
• Explain why transformers replaced RNNs for language modeling.
• Which components make up a standard transformer block?
• Which of these are valid concerns when using LLM-as-judge for evaluation?

When to use RAG vs fine-tuning vs prompting. Map business problems to AI solution patterns.

• Start hereWhat is RLHF, and why is it used after pretraining?
• What does LoRA do, and why is it popular for fine-tuning?
• Which metric best measures whether a RAG answer is grounded in the retrieved context?
• When should you fine-tune instead of using RAG?
• What does RAG primarily help with in LLM based applications?
• Why combine BM25 with dense embeddings for retrieval?

AI product strategy, vendor evaluation, ROI analysis, and responsible AI practices.

• Start hereWhat is RLHF, and why is it used after pretraining?
• Which metric best measures whether a RAG answer is grounded in the retrieved context?
• Design a RAG evaluation harness, what do you measure and how?
• PPO vs DPO, what's the practical difference?
• Which of these are valid concerns when using LLM-as-judge for evaluation?
• Running LLM-as-judge on your…