Sliding Window Attention

Explain effective context for Mistral's 4096 sliding window over a 16k input

Sliding-window attention can't see beyond the window. Describe the trick that recovers older context.

Identify the attention-pattern difference between Mistral 7B and Llama-2 7B

Name two production KV-cache eviction policies and what each prioritizes

Spot the errors in this KV-eviction strategy for long-running generation

Why does naive sliding-window KV eviction break and how does StreamingLLM's attention-sink fix it?

What's the 'attention sink' phenomenon, and how does StreamingLLM exploit it?