Pick the most reliable trigger for refreshing a rolling conversation summary

MCQ·Medium·4.0 · 0·~1 min·Asked atAutodesk Sourcegraph Wandb

Attempt it

A rolling summary needs a trigger that decides when to refresh. Four candidate triggers sound reasonable in isolation, but only one is reliable in production. **Every turn.** This wastes most of its work. Many turns add only a few tokens, and re-summarizing the same content every turn burns LLM calls for no gain. **User reminder.** Relying on the user to say "please remember this" is fragile. Most users never do, and by the time someone thinks to mention it, the relevant context may already be out of the window. **Downstream symptom (refusal or low confidence).** By the time the model refuses or hedges, the summary refresh is already late. The bot has already failed to answer the user. **Soft token threshold.** When the live conversation tail crosses, say, 70 percent of the reserved window, the summarizer fires. This is the reliable choice. It is proactive (fires before the budget is exceeded), cheap (does not run on every turn), and observable (the trigger is a number, not a behavior signal). The soft threshold is the same shape as budget watermarks elsewhere in the system: cache eviction high-water marks, autoscaling triggers, queue backpressure. Borrow the pattern.

A reliable summary-refresh trigger has to be **proactive** (fires before the budget overflows), **cheap** (does not run when no work is needed), and **observable** (mechanical to monitor and tune). A soft token threshold on the live conversation tail meets all three; the other options each fail at least one. **Every-turn refresh** fails on cheapness. Many turns add only a handful of tokens, and re-summarizing the same content on every turn means most summarizer calls do no useful work. Total compute and latency overhead scale linearly with conversation length for no quality benefit. **User-reminder refresh** fails on proactivity and reliability. Most users never explicitly ask the model to remember anything. The few who do tend to do so after they have already noticed something was forgotten, which means the context they want preserved is already gone. The trigger fires after the failure, not before. **Downstream-symptom refresh** (refusal, low confidence) also fails on proactivity. Refusals and low confidence are end of pipeline observations. By the time they appear in production, the user has already gotten a bad response. The summary refresh is downstream of the failure, not upstream. **Soft-threshold refresh** fires when the live conversation tail crosses a fraction of the reserved window (commonly 65-80 percent). The trigger is proactive (fires before overflow), cheap (only when the budget pressure is real), and observable (the threshold is a number, easy to monitor and tune). It is the same pattern as cache eviction high-water marks, autoscaler triggers, or queue backpressure: a soft early signal that gives the system time to act before a hard limit is hit. Production systems pair the soft threshold with a hard ceiling: if the soft refresh fails or is async-pending, the hard ceiling forces a synchronous refresh to prevent overflow. The soft threshold does almost all the work; the hard ceiling is overflow insurance.

TL;DR

A soft token threshold is the reliable trigger because it fires proactively before the budget is exceeded while staying cheap by not running on every turn or chasing downstream symptoms.

Memory aid

Easy to grasp

Imagine your inbox. You could clean it after every single email (exhausting), only when your friend asks if you got their message (too late, you already lost it), or when the model says it gave up. The smart move is to clean it when it starts looking full but before it overflows. A summary refresh follows the same shape. Don't summarize on every turn, don't wait for the user to complain, don't wait for the bot to fail. Fire when the live window crosses a comfortable fraction of its budget, while there is still room to compress without panic.

Key concepts

Context Engineering Agents

Concept explanation~2 min read

Everything you need to truly understand this topic: intuition, mechanics, step by step explanation, code, formulas, and worked example. Click to expand.

Choosing the trigger for refreshing a rolling summary is a control-theory problem in disguise. The cost of refreshing too often is wasted compute; the cost of refreshing too rarely is overflow and lost context. The four candidate triggers in this question each implement a different control shape, and only one of them is both proactive enough to catch overflow before it happens and cheap enough to run in production at scale.

This deep dive walks through each option, explains why three of them fail, and details the watermark shape that makes the soft threshold the production standard.

What a reliable trigger has to do

Three properties separate good triggers from bad ones.

Proactive. The trigger has to fire before the failure (budget overflow) happens. Reactive triggers, ones that fire after the user has already gotten a bad response, are too late to repair the turn that exhibited the symptom.

Cheap. The trigger should not run the summarizer on turns where no real work is needed. Every summarizer call is an LLM call with its own latency and cost. Wasted calls compound across long sessions.

Observable. The trigger has to be something the system can monitor and tune. A trigger that depends on user behavior or on noisy downstream signals is hard to debug when it misfires.

Good triggers in adjacent systems all have this shape: cache eviction high-water marks, autoscaler thresholds, queue backpressure, garbage collection triggers. The summary-refresh problem is structurally the same problem, and the same kind of trigger works.

Why the three plausible alternatives fail

The watermark shape: how soft thresholds work

Production patterns and observability

Situations where this technique stops working.

2–4 min · Everything important, quickly.

Real products, models, and research that use this idea.

ChatGPT's session memory uses watermark-style triggers internally to manage rolling context in 2026.
Claude Sonnet 4.6 long-context handling exposes soft-threshold semantics through the API for client-managed memory.

What an interviewer would ask next. Try answering before peeking at the approach.

QWhat is the right threshold percentage, and how do you tune it?

Start at 70-75 percent of the reserved window. Monitor two things: how often the threshold fires per session, and how often the hard ceiling has to backstop the soft trigger. If the hard ceiling fires often, lower the threshold. If the soft trigger fires too often (driving up cost), raise it. Production stacks typically converge in the 65-80 percent band.

2 more follow-ups an interviewer would ask next. Sign in to reveal them.

Red flags & common mistakes

The phrases that signal junior thinking. Click to expand.

Most common mistake

Waiting for a downstream symptom (refusal, low confidence, user reminder) before refreshing. By the time the symptom appears, the relevant context is already out of the window.

60 second bullets to scan on the way to the call.

List the three properties of a reliable summary-refresh trigger
Identify why every-turn refresh wastes compute

Primary sources. Browse if you want the original framing.