Kara: Innovative Sliding-Window KV Cache Compression for Efficient Reasoning LLMs

Kara, a new sliding-window KV cache compression method, improves the efficiency of reasoning language models. Developed by researchers Shen Han and Yuyang Wu, the method addresses challenges in decoding latency and throughput, which are critical for applications involving long chain-of-thought (CoT) reasoning. This breakthrough was submitted on May 1, 2026, and presents a solution to existing limitations in KV cache compression techniques.

Understanding KV Cache Compression Challenges

Reasoning language models often accumulate a massive KV cache during decoding, leading to high latency and limited throughput. Existing methods face two main issues:

• The threshold-triggered compression policy may not significantly improve throughput and can even worsen it.
• Current techniques typically retain isolated KV pairs or fixed-size chunks, failing to preserve important flexible-sized chunks at arbitrary token positions.

These limitations hinder the performance of language models, necessitating a more effective approach.

Kara's Innovative Approach to KV Cache Management

The Kara method proposes a novel approach to KV cache compression by utilizing a sliding window during decoding. This technique focuses on recently generated context, allowing for improved selection of informative KV pairs.

Key features of Kara include:

1. Bidirectional attention to score and select relevant KV pairs within the sliding window.
2. A Token2Chunk module that expands selected KV pairs into chunks, enhancing the preservation of important semantic information.

Improvements in Throughput and Memory Usage

Kara is adapted to work with PagedAttention and is integrated into an inference framework called KvLLM, built upon vLLM. This integration significantly reduces KV cache memory usage and enhances output throughput.

Extensive experiments demonstrate that both Kara and KvLLM consistently outperform existing methods, offering a promising solution for applications requiring efficient reasoning in language models.