AI Product Manager

Pick the right vector database for a startup with 2M documents, no DevOps headcount, and a one week launch deadline.

Which of these 2026 vector databases are credible at billion-vector scale? Select all that apply.

A vendor's benchmark chart plots QPS against recall@10. What should you look for, and what claim should make you skeptical?

At what scale does 'just put it in Postgres with pgvector' stop being the right answer?

Select all dimensions that meaningfully drive cost on a managed vector database bill in 2026.

A team's Pinecone bill 5x'd month over month with no traffic change. Order the most likely causes to investigate, from most likely to least likely.

Sketch a Llama 3.1 70B self-host versus hosted-API break-even on the back of an envelope

Qwen 3.5: when does running your own GPU beat paying the hosted API per token?

Pick the cheaper to serve option at equal quality: Llama 4 Maverick MoE or Llama 3.1 70B dense

Why does a reasoning model like DeepSeek V4 quietly inflate the output-token bill?

Pick Claude Sonnet 4.6 over Opus 4.7, when is that the right call?

Pick the workload where a provider's Batch API beats the realtime endpoint

Walk through the triage order when a customer-support LoRA tanks MMLU by 4 points post-deploy

A small medical fine-tune leaks verbatim credit-card snippets under red-teaming: identify the failure mode and the standard mitigation

Llama-4 fine-tune shipping to 800M MAU: which license clause must the team re-read?

Defend the claim that full fine-tuning still beats LoRA for some 2026 production workloads

Design a multi-dimensional eval dashboard for a production LLM product. What dimensions, views, and alerts does it need, and what is the one chart that the VP of Engineering should check every Monday?

Build an eval framework from scratch for three LLM products (support bot, code assistant, summarizer). Cover architecture, shared infra, per-product customization, and six-month failure modes.

A colleague presents this claim: 'Our chatbot improved because BLEU score went from 0.32 to 0.41 on our eval set of open-ended customer questions.' Spot the flaw in this reasoning.

Pick the best definition: in LLM evaluation, a metric is _____.

A teammate asks what 'running evals' means in the LLM context. How would an engineer explain it without jargon?

A model aces a benchmark with suspiciously high scores. The first hypothesis is contamination. Explain what that means to a non-technical PM.

Every model release cites benchmark numbers. Define what a benchmark actually is and name two things it cannot tell you.

You have no reference answers for your eval set. Does that mean you cannot evaluate the model?

Offline eval now or wait for the A/B test results? Pick the key difference between online and offline evaluation.