Design an off-topic filter for a banking support chatbot

The product rule is simple: the bot answers banking questions, nothing else. The design question is how to enforce that cheaply and without alienating users when the filter is wrong. The basic structure is a topic classifier sitting on the input rail, before the main model runs. The cheapest workable choice is embedding similarity, embed the user's query, compare against a small set of topic centroids built from canonical banking questions, and accept if the similarity to any centroid is above a threshold. This runs in single-digit milliseconds and is essentially free. For ambiguous cases (similarity in a middle band), you cascade to a heavier check: a small fine-tuned classifier, or an LLM-as-judge with a short policy prompt that returns yes/no with a one-line justification. The cascade keeps the average latency low while preserving accuracy on the hard cases. On a block, the worst thing you can do is silently refuse. Show a templated message that names the policy ('I can only help with account, card, and transaction questions') and offers two or three clarifying buttons. That way a false-positive turns into a one-click correction instead of a frustrated user. Log every block and review a sample weekly.

**Cascade two checks on the input rail; route blocks to a templated message, not a silent refusal.** Start with embedding similarity. Build a centroid per allowed topic (accounts, cards, transactions, loans, branches, security) from a few hundred canonical questions per topic. At runtime, embed the user query and check the maximum cosine similarity against any centroid. Above a high threshold, accept; below a low threshold, block; in the middle band, cascade to a heavier check. Total latency on the fast path is around 5 ms. For the middle band, route to an LLM-as-judge with a short policy prompt: 'Is the following query about a banking topic? Reply yes or no.' This costs around 100-200 ms and uses a small model. Accuracy on the ambiguous slice is much higher than the embedding alone would deliver. On a block, return a templated response that names the policy and offers options: 'I can only help with account, card, transaction, loan, branch, or security questions. Did you mean one of these?' with two or three clarification buttons. False positives become one-click corrections. Close the loop by logging every classification, sampling low-confidence blocks to a review queue, and retraining the classifier or updating centroids weekly against a versioned golden set. The off-topic rate and the false-positive rate are both worth a dashboard.