AI Researcher

HELM from Stanford claims to be a 'holistic' evaluation. What makes it different from running MMLU alone?

Your eval shows Model B scores 82.1% vs Model A at 80.5% on 200 test cases. Your manager wants to ship Model B immediately. What question should you ask first?

Spot the flaw: 'We fine-tuned on 10K examples and eval on a random 500 from the same dataset. Accuracy is 94%, so the model is ready for production.'

Your team debates 32K vs 128K vs 256K vocab. What is the core tradeoff they should frame?

You are building a 7B model for 25 languages including Hindi, Arabic, and Swahili. Design a fair tokenizer and name what it costs.

$50K compute budget, 50B tokens of US legal text, 1.5B parameter model. Defend your tokenizer design.

You are fine-tuning on Python code only. Should you reuse the base tokenizer, train a new one, or extend the vocab?

BPE and WordPiece both merge subwords. What is the one thing they disagree on?

BPE grows the vocabulary by adding merges. Unigram does the opposite. Explain the difference.

What is the core mechanism that distinguishes Self-RAG (Asai et al. 2023) and Corrective-RAG (CRAG, 2024) from a standard single-shot retrieve then generate pipeline?

Non-obvious modes of train/test leakage in instruction-tuning evaluation

Which of these are real train/test leakage modes for an instruction-tuning project?

SimPO: what does it drop vs DPO, and what's the cost win?

Spot the error in this description of SFT loss masking

Explain SFT's loss and the role of prompt-token masking

Spot the bug in this sequence-packing setup

Sequence packing: what it does, and the attention-mask gotcha

Over-refusal: what it is and how naive safety FT causes it

Naive safety FT failed: model now refuses 'how does anesthesia work?'. Best fix?

Where do SFT and DPO sit in the classical RLHF pipeline?

CoT distillation and STaR: what made rejection-sampled reasoning FT effective?

Why has prefix/prompt tuning largely lost to LoRA in production?

Prompt-tuning vs LoRA: pick the architectural reason LoRA tends to win

ORPO: what does it combine into one stage, and why is that attractive?

ORPO vs DPO: which architectural difference enables 'single-stage' training?