Design a medical imaging triage assistant under real compliance constraints

Short answer·Hard·4.0 · 0·~3 min·Asked atBain Contextual Ai Zilliz

Attempt it

A hospital group wants a VLM based assistant that flags likely critical radiology studies so they jump the reading queue. Design it, and be explicit about what regulation, clinical risk, and data protection force you to do differently than in an ordinary AI product.

Free · 2 AI evals / day

Medical AI is not just AI with stricter logging. The product category, the failure mode, and the data type each change what you are allowed to do, and the answer has to address all three. Start with the claim. Software that reorders a reading queue so the radiologist looks at urgent studies first is triage. Software that says this scan shows a hemorrhage is diagnostic. Triage is a meaningfully smaller regulatory burden than diagnostic, but both are usually regulated as medical devices in the US (FDA) and the EU (MDR). Picking the smaller scope on purpose is half the design. The failure mode that matters is missing a critical finding. That is the catastrophic error, so the threshold is tuned for high sensitivity even at the cost of more false positives. The exact tradeoff is set with clinicians, not by the engineering team alone, and validation runs across finding types, scanner vendors, and patient demographics so you catch bias before it reaches a patient. The workflow is strictly additive. Flagged studies move up the queue. Nothing moves down based on the model's say-so. The radiologist still reads every study and the model's flag is provenance, not a decision.

**Scope the claim first.** Frame the product as a worklist prioritization aid, not a diagnostic device. The radiologist makes every final read. This single decision moves the product from FDA Class II diagnostic territory toward the lighter Computer-Aided Triage and Notification class (still regulated, still needs 510(k) clearance for most use cases, but materially more achievable). In the EU, MDR rules apply with similar gradation. Picking the claim deliberately is half the regulatory work. **Clinical risk drives the operating point.** A missed critical finding (intracranial hemorrhage, pulmonary embolism, large pneumothorax) is the catastrophic error. The threshold is tuned for high sensitivity, with the tolerable false-positive rate negotiated with clinicians per finding type. False positives cost reading time; false negatives cost lives. Validation runs stratified per finding type, per scanner vendor, per scanner model, per acquisition protocol, and per patient demographic subgroup to surface bias before deployment. **The workflow is additive only.** Flagged studies move up the queue. Nothing moves down. The radiologist sees provenance: which model, which version, what the model claims to have seen, with attention heatmaps where supported. **Data protection is non-negotiable.** Imaging is PHI under HIPAA in the US and special category data under GDPR in the EU. Business associate agreements with any cloud vendor. De-identification for any secondary use. Deployment on-premises or in a HIPAA-eligible cloud. Audit trails recording model version, study id, input, output, and clinician action.

TL;DR

Scope the claim as triage so the radiologist remains the diagnostician, clear it as a medical device, tune for sensitivity on critical findings, validate across subgroups, and treat PHI and post-market monitoring as

Memory aid

Easy to grasp

Imagine an air traffic controller's screen with hundreds of planes. A helper draws a red ring around the ones that look like they might be in trouble so the controller looks at those first. The helper never decides what is wrong with a plane and never tells one to land somewhere else. It only changes the order in which the controller looks. That is the difference between a triage assistant and a diagnostic system: the assistant changes attention, not action. A hospital VLM that flags scans for the radiologist works the same way. It speeds up urgent work without ever replacing the human reader, and the rules around it exist because the cost of a missed urgent case is somebody's life.

Key concepts

Multimodal AI Safety and Guardrails

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Medical imaging is the highest-stakes multimodal AI domain that ships in 2026, and the design playbook differs from ordinary AI products on almost every axis. The product claim determines the regulatory pathway. The failure mode is asymmetric in a way ordinary ML metrics do not capture. The data is PHI from the first byte. And the product does not stop when it ships, because post-market surveillance is a legal obligation.

This deep dive treats the question as a real device design exercise, not a generic AI safety conversation. The structure follows the order a regulatory-aware engineering team would actually use: claim and class first, then clinical operating point, then workflow, then data handling, then post-market.

Claim first, because the claim is the class

FDA and equivalent regulators (EMA in Europe, PMDA in Japan, NMPA in China) classify medical software by the claim it makes, not by the underlying algorithm. Software that diagnoses a condition sits in Class II or Class III with a corresponding clinical evidence burden. Software that prioritizes a worklist for clinician review sits in the lighter Computer-Aided Triage and Notification (CADt) class, with a more focused 510(k) pathway.

The two product categories can do almost identical model inference under the hood. What differs is the indication for use, the labeling, and the human-in-the-loop structure. CADt requires that the clinician make the final read and that the model's output be additive (it can move studies up the queue but not down). Diagnostic software does not require these structural constraints but pays for that freedom with a much larger evidence package.

The practical implication: scope the claim deliberately. If the business need is satisfied by getting urgent studies in front of a radiologist faster, claim triage. If the business need really is autonomous diagnosis, commit to the longer pathway and the larger trial. Trying to write a claim that lives in between, where the model influences action without a radiologist final read, is the regulatory danger zone.

In the EU, MDR follows similar logic with the risk-class system. The specific terminology differs but the structural insight, that the claim determines the burden, is universal. Building a medical AI product without regulatory counsel involved from the start is the most common founder mistake in this space.

Operating point: sensitivity is safety

Additive workflow, PHI handling, change control

Post-market surveillance is the product

Production realities and 2026 model lineup

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

Aidoc and Viz.ai both ship FDA-cleared triage and notification products that fit the CADt class, with claims explicitly scoped to flagging studies for prioritization rather than diagnosing
RapidAI's stroke triage software notifies stroke teams about large vessel occlusion candidates, sitting in the same CADt class with documented post-market surveillance

What an interviewer would ask next. Try answering before peeking at the approach.

QHow would you design the model-update cadence under change control without becoming uncompetitive?

Use the FDA's Predetermined Change Control Plan (PCCP) pathway to pre-authorize the kinds of updates you expect, like retraining on new scanner data or threshold recalibration within a documented envelope. Updates inside the envelope go through internal validation and notification; updates outside require a new submission. The PCCP is how modern ML medical devices stay current without monthly resubmissions.

1 more follow-up an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Designing a beautiful diagnostic UI that competes with the radiologist's read, then discovering you have built a Class II or Class III diagnostic device with a multi-year regulatory pathway you cannot afford.

60 second bullets to scan on the way to the call.

Why the regulatory claim is the most consequential design decision
How the FDA CADt class differs from Class II diagnostic device pathways

Primary sources. Browse if you want the original framing.