PremiumA tenant demands full…
Multiple ChoiceMediumKpmgPatronus
Unlock with Pro to study this question. See Pro pricing.
The comparison table for this question.
Your team serves 50M requests per day and must choose between (a) a single hybrid model with a per request thinking toggle and budget knob, or (b) a two model fleet, cheap fast model plus dedicated reasoning model, behind a router. Walk through the tradeoffs and make a recommendation.
| Dimension | One hybrid model | Routed two-model fleet |
|---|---|---|
| Example | Claude Opus 4.7 (toggle thinking) | Haiku + Opus 4.7 with router |
| Per-request cost control | Toggle on/off + budget | Routing decision |
| Latency on easy requests | Low (thinking off) | Lowest (small model) |
| Latency on hard requests | Reasoning latency | Reasoning latency |
| Operational complexity | One model identifier | Two models + router + eval per leg |
| Cost ceiling | Bounded by budget per call | Bounded by router quality |
OpenAI's o-series reasoning models expose a reasoning ______ parameter (low / medium / high) that controls how much internal deliberation a request gets, while Claude's extended ______ mode takes an explicit token budget that caps the internal reasoning the model may spend before answering.
| Aspect | OpenAI reasoning_effort | Anthropic max_thinking_tokens |
|---|---|---|
| Type | Discrete enum | Continuous integer |
| Values | minimal / low / medium / high | 0 to ~64k tokens |
| User mental model | Tier name | Explicit token budget |
| Reported in usage | reasoning_tokens count | output_tokens (aggregated) |
| Best for | Product teams not thinking in tokens | Cost-engineering teams wanting hard caps |
You are choosing a pretrained vision encoder for a new VLM and the shortlist is CLIP versus SigLIP. How does SigLIP's training objective differ from CLIP's, and why do VLM builders care?
| Aspect | CLIP | SigLIP |
|---|---|---|
| Loss | Softmax (InfoNCE) over batch | Per-pair sigmoid |
| Normalisation | Global across batch | None |
| Batch size | 32k+ for clean signal | Stable from 4k upward |
| Scaling law | Plateaus past a regime | Monotonic up to 32k |
| Default in 2026 VLMs | Retrieval baselines | LLaVA-NeXT, Qwen2-VL, PaliGemma |
Per token latency becomes dominated by PCIe transfers of expert weights, with high variance depending on which experts each token's layers route to
Latency is unchanged because active FLOPs per token are identical; only GPU memory pressure improves
Only the first token is slower; afterwards every needed expert is cached on the GPU for the rest of the generation
Throughput rises because freeing GPU memory lets the scheduler admit larger batches at the same step time
| Path | Approximate bandwidth | Implication for offloading |
|---|---|---|
| GPU HBM3 | About 3 TB/s | Resident experts read at full speed |
| NVLink, GPU to GPU | Hundreds of GB/s | Sharding experts across GPUs beats host offload |
| PCIe Gen4 x16, CPU to GPU | About 32 GB/s | Each cold expert miss costs milliseconds |
| CPU DRAM, compute in place | About 100 GB/s | Often faster to run the expert on CPU than to move it |
Design a support copilot for a SaaS company handling 1,000,000 tickets per month. It drafts replies for human agents, must ground answers in help center docs and resolved past tickets, keep p95 draft latency under 8 seconds, and stay under $0.02 per ticket. Walk through the end to end architecture and the tradeoffs you would call out.
| Cost lever | What it cuts | Typical effect |
|---|---|---|
| Prompt caching | Input cost of the static system prompt and policies | Up to 90 percent off the cached prefix |
| Tight retrieval | Input tokens from context chunks | 3 to 5 chunks instead of 20 cuts most input spend |
| Response caching | Entire calls for repeated intents | Near zero marginal cost on cache hits |
| Output caps | Output tokens, the priciest line | Bounded worst-case cost per draft |
| Tiered routing | Premium model spend | Strong model only on the hard minority |
When someone says 'we use the NIST framework,' what is the difference between the AI RMF 1.0 and the Generative AI profile, and which one applies to an LLM product?
| Aspect | AI RMF 1.0 | GenAI Profile (NIST AI 600-1) |
|---|---|---|
| Published | January 2023 | July 2024 |
| Scope | Any AI system, including classical ML | Generative AI specifically |
| Structure | Four functions: Govern, Map, Measure, Manage | Risk categories + control mapping into the four functions |
| Names risks like prompt injection | No | Yes |
| Applies to an LLM product | Yes, as the umbrella structure | Yes, as the LLM-specific control layer |
You run a customer support agent that can issue refunds and update accounts. Outline a tiered escalation path for guardrail trips, what happens at each tier, who is paged, and what state is preserved.
| Tier | Trigger | User-facing action | Human notification |
|---|---|---|---|
| Tier 1 (soft) | Low-confidence classifier hit, output rail rewrite | Polite fallback or modified response | None; sampled review queue |
| Tier 2 (hard) | High-confidence injection, PII leak attempt, policy violation | Templated apology, conversation short-circuited | Async customer-success queue, hours SLA |
| Tier 3 (irreversible action) | Refund / account modification / external API blocked | Templated message with callback offer | Synchronous SRE page, SEV-3 incident |
Your bank deploys an LLM support chatbot. Product says it should only answer banking questions, not politics, not coding help, not therapy. Sketch the off topic gate: what signal do you use, where does it sit, and how do you handle false positives?
| Approach | Latency | Cost per call | Accuracy |
|---|---|---|---|
| Embedding similarity to centroids | ~5 ms | fractions of a cent | Good on clear cases, weak on mixed-intent |
| Fine-tuned BERT-class classifier | ~20 ms on GPU | low if self-hosted | Strong if training data is fresh |
| LLM-as-judge (small model) | ~100-200 ms | noticeable at scale | Strongest on ambiguous and rare intents |
What is attack success rate (ASR) for a safety eval, and what metric must you track alongside to avoid a false win?
| Metric | What it measures | Failure mode if reported alone |
|---|---|---|
| ASR | Fraction of adversarial prompts that produced unsafe output | Refuse-everything model scores zero and looks 'safe' |
| Benign pass rate | Fraction of on-policy prompts answered correctly | Says nothing about how the model behaves under attack |
| Refusal rate (benign) | Fraction of benign prompts the model refuses | Tells you over-refusal cost but not adversarial robustness |
An executive asks why your team budgets for both expensive RLHF tuned models AND a separate guardrail stack,isn't that paying twice for safety? Defend the asymmetry: alignment reduces frequency, guardrails reduce blast radius. Cite one concrete attack class for each side.
| Property | Alignment (RLHF / CAI) | Guardrails (runtime) |
|---|---|---|
| What it changes | Model's next-token distribution | What the system does with the output |
| Where it acts | Training-time, baked into weights | Request-path, runs on every call |
| Distribution | In-distribution / benign traffic | Every request, including adversarial |
| Statistical effect | Lowers P(harm | benign) | Lowers E[damage | harm] |
| Catches indirect injection? | Rarely,never trained on it | Yes,content tagging, tool allowlist |
| Catches benign over-refusal? | Yes,preference data tunes tone | No,operates on bounds, not tone |
Semantic similarity is time blind: a year old document can be as similar to a query as today's. For a corpus where recency matters, how do you make retrieval prefer fresh information, and when should you NOT?
| Lever | How it works | Best when |
|---|---|---|
| Time-decay weight | Scale similarity by a function of age (e.g. exp(-age/τ)) | Old content is less preferred but still valid |
| Date pre-filter | Restrict candidates to a recency window before ranking | Anything past a cutoff is simply invalid |
| Rerank feature | Feed age as one input to a learned reranker | At scale, when freshness-relevance trade varies by query |
Sentence window retrieval
Auto merging retrieval
Naive fixed chunk retrieval
Shared problem all three address
Hierarchical node tree
Matches on a single precise sentence, then expands to a window of neighboring sentences for context
The precision versus context tension: a chunk small enough to match well is often too small to answer from
Matches and returns the same flat chunk, forcing one size to serve both precision and context
The parent/child structure auto merging walks to roll leaf hits up into a larger passage
Retrieves small leaf chunks, then merges them into their shared parent when enough siblings are hit
| Technique | Match unit | How context expands | Trigger |
|---|---|---|---|
| Naive fixed-chunk | Flat chunk | It does not — match = return | Always the same chunk |
| Sentence-window | Single sentence | Fixed radius of neighbor sentences | Every hit, fixed window |
| Auto-merging | Small leaf chunk | Roll up to shared parent in a node tree | Sibling-count threshold under a parent |
Route numeric and aggregate questions to structured retrieval, text to SQL or table extraction, so the values are computed exactly, instead of relying on the LLM to add up scattered chunks
Increase top-k so the model receives every chunk that mentions a number and can add them up itself
Lower the chunk size so each figure lands in its own chunk and embeds more precisely
Swap the embedding model for a larger one trained on financial text to improve numeric similarity
| Question type | Right retrieval approach |
|---|---|
| Exact figures or aggregates (sum, count, max, compare) | Structured query: text to SQL or table extraction, executed for an exact result |
| Fuzzy or semantic lookup over prose | Vector retrieval over chunks |
| Mixed (narrative plus a number) | Route per sub-question, then combine the results |
Your RAG dashboard shows faithfulness consistently above 0.9, but users complain the answers don't actually help them. Explain what high faithfulness does and does not guarantee, and which evaluation axes you are failing to measure. Name the additional metrics you would add.
| Axis | Question it answers | Stage it isolates |
|---|---|---|
| Context recall | Did retrieval surface the chunks with the answer? | Retriever |
| Faithfulness | Is the answer grounded in the retrieved chunks? | Generation grounding |
| Answer relevance | Does the reply address the user's intent? | Generation usefulness |
A RAG app must answer over several distinct sources: product documentation, a relational database, and live web results. Design a query routing layer: what it does, why a single index is not enough, and how routing decisions are made.
| Routing mechanism | Latency / cost | Best fit |
|---|---|---|
| LLM classifier / function calling | One model call; non-deterministic | Long-tail queries, fast to ship, multi-source selection |
| Small trained classifier | Very low once trained | High-volume head of the query distribution |
| Embedding similarity to source descriptions | One embed + lookup | Cheap default, easy to add sources by writing a description |