1 Introduction↩︎

Sparse attention is emerging as a fundamental technique for reducing inference costs in large language models (LLMs), driven by the rapid growth of generation lengths in applications such as reasoning [1], agentic systems [2], and reinforcement learning [1], [3]. In these workloads, key-value (KV) cache movement during decoding has become the primary system bottleneck. As a result, sparse attention is being adopted both as a core architectural component in state-of-the-art models such as DeepSeek [4]–[6] and GLM-5.1 [7], and as a drop-in optimization for pretrained models [8]–[19]. To accelerate progress in sparse attention algorithms, reducing implementation complexity while maintaining high serving efficiency is becoming critical.

Deploying and validating new sparse attention algorithms at scale while achieving end-to-end speedups remains highly challenging and engineering-intensive, substantially slowing both human researchers and emerging AI agents in exploring the sparse attention design space. The core difficulty arises from modern LLM serving systems that adopt paged attention to reduce memory fragmentation, in which the KV cache is stored in a physically non-contiguous, block-sparse layout accessed through indirect addressing [20]. This breaks the memory assumptions underlying existing tensor frameworks such as PyTorch [21] for batch (contiguous) tensor layouts (illustrated in 2). Consequently, sparse attention algorithms are difficult to express within existing tensor abstractions. Despite substantial progress toward addressing these challenges, three key gaps remain.

1. Difficult to Generalize to Dynamic Sparsity. Prior work (e.g., FlashInfer [22], FlexAttention [23]) generalized the attention operator to tensors with a paged layout. Thus, they can optimize sparse attention kernels once the sparsity patterns (i.e., the set of KV pairs each query attends to) are known. While these approaches achieve substantial speedups for static sparse attention, they do not naturally extend to dynamic sparse attention, which proved to be more accurate. In dynamic settings, determining the sparsity pattern on the fly is a central component of the algorithm and incurs computational overhead that exceeds that of the attention operator.

2. Lack of Programmability. Existing LLM serving systems [20], [24], [25] support several sparse attention methods but provide limited programmability for integrating new algorithms. As a result, adding a new variant can require substantial engineering effort. For example, implementing Double Sparse [26] in SGLang [24] requires roughly 2000 lines of code, much of which reimplements operators such as GeMM, Reduce, and Top-K over paged tensors. This significantly hinders both researchers and AI agents [27], [28] from rapidly prototyping and iterating on new sparse attention ideas.

3. Incompatibility with Evolving Serving Systems. Many sparse attention methods rely on custom kernels [8], [9], [11], [29], but remain incompatible with key components of modern LLM serving stacks, including paged attention [20], prefix caching [24], and rapidly evolving attention backends [22], [30]–[32]. As a result, their theoretical speedups often fail to translate into practical end-to-end gains. For example, Quest’s original implementation is \(44.4\times\) slower than SGLang full attention, taking nearly two hours to evaluate Llama 3.1-8B on AMC23, despite optimized sparse kernels. These results suggest that evaluating sparse attention on long-generation workloads is impractical without tight integration with modern serving systems, even during prototyping and iteration.

An ideal framework should allow AI agents to express sparse attention mechanisms by specifying what sparsity pattern to apply and how attention is computed, while abstracting away from low-level tensor layouts and memory-management details. At the same time, the framework should transparently optimize execution and remain fully compatible with existing LLM serving infrastructures, enabling researchers to obtain rapid and realistic performance feedback.

A natural approach toward such a framework is to implement each sparse attention variant as a dedicated custom operator. However, this strategy fundamentally lacks compositionality: every new sparsity pattern or algorithm requires a separate implementation, making the system difficult to extend and unable to scale with the rapidly growing diversity of sparse attention designs.

Inspired by abstractions for sparse tensor linear algebra [22], [33]–[35], which provide systematic representations and optimizations for sparse computation, we propose vTensor, a page-centric tensor system. vTensor generalizes sparse tensor computation to a paged layout, enabling sparse attention algorithms to be expressed in a flexible and composable manner without exposing low-level memory layouts or addressing details. However, leveraging vTensor to program sparse attention and achieve speedups for LLM serving requires overcoming several technical challenges.

• Simple and Modular Programming Interface. The framework should provide an intuitive tensor view that hides low-level layout details and reduces user complexity. Sparse attention algorithms should also be expressed as modular components, even when they involve computation across stages of the serving pipeline.

• Efficient Execution on Paged Layouts. Supporting heterogeneous tensor layouts makes kernel implementation challenging. Operators must execute efficiently on non-contiguous memory layouts while minimizing gather operations and redundant data movement.

To address these challenges, we propose Vortex, a system for efficient and programmable sparse attention serving. As shown in 1 (a), Vortex consists of three components: (1) a frontend language (vFlow) embedded in Python for expressing sparse attention algorithms; (2) an interpreter that translates vFlow programs to executable vTensor operators; and (3) an execution backend that integrates seamlessly with existing serving systems.

• In 3.1, we introduce the vFlow programming model through an example based on block top-\(k\) attention, and demonstrate its expressiveness across a wide range of sparse attention algorithms in 3.2.

• In 4.1, we present the vTensor abstraction for paged tensor computation. We then describe in 4.2 how vFlow programs are compiled and executed efficiently as vTensor programs within modern serving systems.

• In 5, we present three key optimizations in Vortex: a workload planner for balanced GPU thread-block scheduling, kernel fusion to reduce intermediate memory traffic (see 5.1), and stochastic optimizations for radix top-\(k\), a critical operator in sparse attention (see 5.2). In 5.3, we present Vortex’s compatibility with existing optimized attention backends, including GQA and MLA.

In 6.1, we first demonstrate that Vortex enables rapid innovation and iteration of sparse attention algorithms with AI agents, as shown in 1 (b). In 6.1.1, Claude Code and Codex are prompted to generate a diverse set of sparse attention algorithms that achieve strong accuracy-throughput trade-offs. In 6.1.2, an 18-hour autonomous optimization loop improves throughput by up to \(3.46\times\) over full attention while preserving accuracy. In 6.4, we then evaluate the serving efficiency of Vortex. Compared to SGLang, Vortex achieves up to \(3.60\times\) and \(2.98\times\) throughput improvements over full attention for block top-\(k\) and Quest, respectively, while maintaining accuracy. In terms of user latency, Vortex reduces P95 latency by up to \(11.7\times\) for block top-\(k\) attention and \(12.8\times\) for Quest under long-input workloads, all evaluated on NVIDIA H200 SXM.

Beyond these, Vortex readily extends to new attention architectures and to very large models that are otherwise hard to even experiment with. On the MLA-based GLM-4.7-Flash, running on an NVIDIA B200 (6.2), a rope-aware sparse attention we design in vFlow achieves up to a \(4.7\times\) speedup over full attention, and on the 229B-parameter MiniMax-M2.7 served across four NVIDIA B200 GPUs (6.4), block top-\(k\) achieves up to a \(1.37\times\) speedup and even exceeds full-attention accuracy. These results show that Vortex not only delivers high-performance sparse attention serving but also provides a scalable step towards automated research and optimization of sparse attention by AI agents.

2 Related Work↩︎

LLM Serving Systems. A large body of work has improved the efficiency of LLM serving [20], [24], [25], [36]–[47]. Orca [36] introduced continuous batching. vLLM addresses the memory fragmentation bottleneck with paged-attention. SGLang [24] focuses on prefix caching, which reuses KV cache across repeated prefixes to reduce redundant computation.

Sparse Attention. Sparse attention for LLMs can be broadly divided into training-time and inference-time approaches. Training-time methods incorporate sparsity directly into model architectures during pre-training  [4]–[6], [48]–[51]. Inference-time methods instead accelerate pretrained dense models with little or no retraining. These approaches are commonly categorized as static or dynamic. Static sparse attention [52]–[56] applies predefined sparsity patterns independent of input content, typically based on token positions. Dynamic sparse attention [8], [9], [11]–[13], [16], [17], [29], [57]–[62] instead constructs input-dependent sparsity patterns by selecting KV entries according to the current query, hidden states, or content relevance.

System Optimizations for Attention. Prior systems such as FlashAttention [30], [63]–[65], Flash-Decoding [66], [67], and SlimAttention [68] accelerate attention through improved hardware utilization and parallelism. Frameworks including FlashInfer [22], FlexAttention [23] further expose sparse attention interfaces based on KV block indices or masks for paged attention. However, these systems generally assume the sparse KV indices are already available, leaving the challenge of efficiently constructing them unresolved. More recent systems, such as LServe [14] and SparseServe [69], explore end-to-end sparse attention optimization but focus on specific algorithms or deployment settings, including Quest [9] and block top-\(k\) attention under memory-constrained workloads.

Currently, Vortex focuses on the decoding phase, where KV memory is the primary bottleneck, and leaves the prefilling phase for future work.

3 Programming Model: vFlow↩︎

In this section, we present the programming model of Vortex, namely vFlow, a unified abstraction for expressing sparse attention algorithms. We begin in 3.1 with a concrete case study on block top-\(k\) attention to illustrate the core design principles of vFlow. In particular, vFlow adopts a single-request mental model, treating tensors as batch size 1, while exposing flexible and composable primitives for constructing sparse attention mechanisms. We then show in 3.2 how this abstraction naturally generalizes to a broad range of sparse attention algorithms.

3.1 A Running Example: Block Top-\(k\) Attention↩︎

Let \(\boldsymbol{q} \in \mathbb{R}^{g \times d}\) denote the query, and let \(\boldsymbol{K} \in \mathbb{R}^{n \times d}\) and \(\boldsymbol{V} \in \mathbb{R}^{n \times d}\) denote the key and value caches, respectively. We partition the sequence into \(B = \lfloor n / b \rfloor\) non-overlapping blocks of size \(b\). The mathematical formulation for block top-\(k\) attention (for a single key value head) is defined in [fig:mathofblocktopk] and the corresponding vFlow implementation is presented in 3, where primitives by vFlow are shown in purple.

This formulation naturally decomposes into two stages with distinct computational characteristics:

(1) Query-independent preprocessing. 1 computes block-level key summaries \(\widetilde{\boldsymbol{k}}_j\) by averaging keys within each block. These summaries depend only on the key cache \(\boldsymbol{K}\) and are independent of the query, and thus can be precomputed once during cache construction and reused across decoding steps. This stage corresponds to forward_cache() in 3, where vFlow presents a logical view of c["centroids"] as a contiguous tensor of shape [1, B, d].

(2) Query-dependent execution. [eq:block_topk_2,eq:block_topk_4,eq:block_topk_5] define the dynamic computation performed for each incoming query \(\boldsymbol{q}\). Specifically, the model (i) computes block relevance scores \(\widetilde{s}_j\), (ii) selects the top-\(k\) blocks, and (iii) performs attention over the corresponding tokens. This stage must be executed per token, as both the selected index set \(\mathcal{I}\) and the output \(\boldsymbol{o}\) depend on the current query. This stage is implemented as forward_indexer(), where vFlow exposes logical contiguous views for all tensors: c["centroids"] as [1, B, d], c["k"] and c["v"] as [1, B, b, d], and q as [1, g, d]. Under this logical view, users can easily reason about intermediate tensor shapes; for example, s \(\in \mathbb{R}^{1 \times B \times g}\) and i \(\in \mathbb{R}^{1 \times 1 \times k}\).

3.2 Generalizability↩︎

The vFlow model is highly general and can express a wide range of sparse attention algorithms. At a high level, its two-stage decomposition mirrors the standard paradigm in retrieval and similarity search [70]–[73]: query-independent auxiliary structures (e.g., summaries, centroids) are constructed offline to enable efficient query-time selection. We further demonstrate the expressiveness of vFlow by answering the following questions.

Q1. Can vFlow easily express more complicated dynamic sparse attention algorithms? Yes. vFlow can naturally express a wide range of dynamic sparse attention algorithms by composing a small set of reusable operators. As shown in the examples (DoubleSparse [26], BlockTopK_NSA [5], Quest [9], and H2O [13], in [fig:ds,fig:nsa,fig:quest,fig:h2o].), different methods can be implemented by combining common building blocks such as GeMM, Softmax, Top-K, Gather, and reduction operators (the full operator set is listed in 1). This compositionality is enabled by vTensor(see 4.1), which provides a unified interface for logical tensors while handling physical layout transparently.

Q2. Can intermediate computation results of forward_indexer() be cached? Yes. vFlow enables caching of intermediate results by allowing operator outputs to be stored in a named cache, provided the user explicitly declares the cache in forward_cache(). A concrete example is H2O, where c["acc"] accumulates attention scores across decoding steps. This design facilitates the implementation of algorithms that require runtime statistics [13], [19], [74].

Q3. Can vFlow extend to static sparse attention?

Yes. Static sparse attention can be naturally expressed in vFlow by constructing index sets that depend only on sequence positions rather than input content. For example, one can obtain the current sequence length via B = c["k"].shape[1] and derive the token position accordingly. The sparsity pattern can then be explicitly specified by constructing the index set \(i\) (e.g., i = List[...]), independent of the query values.

4 Interpretation: vTensor↩︎

The vFlow programs provide a simple yet effective logical tensor abstraction that allows programmers to assume tensors are contiguous and processed with an effective batch size of 1. However, tensors in real LLM serving systems are often stored using more complex layouts to support continuous batching and paged KV caches (8). Thus, we need an underlying tensor system that explicitly accounts for paged layouts to interpret the semantics of vFlow into executable programs. To ensure the flexibility and simplicity of vFlow, the tensor system must satisfy two key properties:

Compositional. Complex algorithms should be constructed by composing a small set of primitive tensor operators, rather than relying on highly specialized monolithic kernels. This design improves modularity, extensibility, and portability across different serving workloads.

Self-Contained. The outputs of tensor operators should remain in formats directly consumable by subsequent operators. In particular, intermediate results should avoid unnecessary materialization or layout conversion, enabling efficient operator chaining on paged tensors.

Therefore, we introduce a new abstraction, vTensor, which extends PyTorch tensors with explicit layout information. In 4.1, we present the definition of vTensor and its operator. In 4.2, we present how to interpret vFlow programs into lower-level vTensor functions, which are executable in serving systems.

4.1 Definitions↩︎

Tensor Definitions. Inspired by [22], we first introduce a unified abstraction for representing tensors with different memory layouts. A vTensor is defined as \[\boldsymbol{x}^{v} = (\boldsymbol{x}, \mathcal{C}) \label{eq:tensordef}\tag{5}\] where \(\boldsymbol{x}\) denotes the underlying Pytorch tensor and \(\mathcal{C}\) captures layout metadata defined as \(\mathcal{C} = (b, \mathbf{p}, \mathbf{I})\), where \(b\) is the batch size, \(\mathbf{p}\) is an index pointer array, and \(\mathbf{I}\) is an index structure. In the paged layout, both \(\mathbf{p}\) and \(\mathbf{I}\) are present, where \(\mathbf{p}\) defines sequence-level offsets and \(\mathbf{I}\) specifies the mapping from sequences to page indices in a ragged form.

Operator Definitions. The operators of vTensor are defined by applying standard PyTorch operators independently to each sequence within the batch. Consider input tensors \(\boldsymbol{x}^{v}_0, \boldsymbol{x}^{v}_1, \dots, \boldsymbol{x}^{v}_{n-1}\) that share the same batch size \(b\), and a PyTorch operator \(\mathcal{F}\) producing \(k\) outputs. For each sequence \(0 \le i < b\), the operator is evaluated as \(\boldsymbol{y}^{v}_{0,i}, \boldsymbol{y}^{v}_{1,i}, \dots, \boldsymbol{y}^{v}_{k-1,i} = \mathcal{F}( \boldsymbol{x}^{v}_{0,i}, \boldsymbol{x}^{v}_{1,i}, \dots, \boldsymbol{x}^{v}_{n-1,i} ).\)

The resulting tensors are then aggregated into output vTensor objects \(\boldsymbol{y}^{v}_0, \dots, \boldsymbol{y}^{v}_{k-1}\) through shape and layout propagation. We present the details in 9.

Based on this abstraction, we implement a broad set of tensor operators and build a compiler that automatically performs kernel synthesis and operator fusion (see 5.1).

4.2 Lowering vFlow to vTensor↩︎

In this section, we present how vFlow programs are translated into sequences of vTensor operators by interpreting operator semantics and specifying tensor layout information (i.e., \(\mathcal{C}\) in 5 ).

Operator Interpretation. Each variable in vFlow is interpreted as a vTensor. For example, the statement \(\texttt{s = GeMM(c["centroids"], q.T)}\) is interpreted as \(\boldsymbol{s}^{v}_i = \texttt{GeMM}\big( \texttt{c["centroids"]}^{v}_i, (\boldsymbol{q}^{v}_i)^T \big), \;0 \le i < b,\) where \(\boldsymbol{q}^{v}_i \in \mathbb{R}^{g \times d}\) and \(\texttt{c["centroids"]}^{v}_i \in \mathbb{R}^{B_i \times d}\). The outputs \(\boldsymbol{s}^{v}_i \in \mathbb{R}^{B_i \times g}\) are aggregated into a ragged vTensor with pointer array \(\mathbf{p}\) satisfying \(\mathbf{p}[i+1] = \mathbf{p}[i] + B_i\), yielding a tensor of shape \(\left(\sum_i B_i\right)\times g\).

Tensor Layouts. Tensor layouts in Vortex are either inherited from the serving system or assigned by the compiler. q uses standard batch layouts, while cache tensors such as c["k"] and c["v"] use paged layouts managed by the serving runtime. Named caches in Vortex, e.g., c["acc"] and c["centroids"], are also stored as paged vTensor objects, whereas temporary intermediates use ragged layouts by default. By sharing the same paging and allocation mechanism as the KV cache, Vortex naturally preserves compatibility with optimizations such as prefix caching.

5 Execution Optimizations↩︎

While vTensor provides a unified abstraction over heterogeneous tensor layouts, efficient execution still requires additional system-level optimizations. To address this, Vortex incorporates several compiler and runtime optimizations. First, Vortex plans workloads according to tensor layouts and data dependencies to improve execution efficiency. Second, it applies kernel fusion to reduce intermediate memory traffic and improve locality (see 5.1). Third, it introduces stochastic optimizations for radix top-\(k\) to accelerate sparse attention indexing (see 5.2). Finally, to execute the selected sparse blocks efficiently, Vortex builds on existing, highly optimized attention backends across attention variants and precisions, complementing them with a custom MLA decode kernel for geometries they do not cover, so that it automatically supports diverse models (see 5.3).

5.1 Workload Planning and Kernel Fusion↩︎

At the beginning of each decoding iteration, the batch size and sequence lengths are known, allowing Vortex to plan workloads and generate optimized GPU kernels [22], [75]. Different operators follow different execution templates depending on their computation patterns. For example, operators such as Softmax and Top-K require specialized kernels, while operators including GeMM, elementwise operations, and reductions share a unified chunk-based execution template as shown in [lst:seq-local-template].

Based on the templates, Vortex further performs kernel fusion to reduce intermediate memory traffic. Computation is represented as a directed acyclic graph, and operators sharing compatible execution templates are greedily fused to eliminate unnecessary intermediate reads and writes.

5.2 Optimization for Radix Top-\(k\)↩︎

As a non-matmul operator, top-\(k\) selection becomes a bottleneck in forward_indexer. A radix-based top-\(k\) algorithm that iteratively partitions values into bins based on their radix digits is widely adopted. Its performance, however, is highly sensitive to the score distribution. In particular, when many large values fall into the same radix bin, the algorithm must repeatedly refine a large candidate set, leading to highly variable runtime.

To mitigate this issue, we introduce stochastic early termination, which is a lossy variant that trades exactness for speed. Instead of fully refining the threshold bin, the algorithm terminates early once enough high-confidence candidates have been collected and then randomly samples the remaining elements from the threshold bin. This stochastic relaxation significantly reduces refinement overhead while providing a tunable trade-off between accuracy and performance.

5.3 Attention Backend Compatibility: GQA and MLA↩︎

To turn sparse selection into end-to-end speedups, Vortex reuses existing, highly optimized attention backends rather than re-implementing attention, feeding the sparse block table from forward_indexer into a paged decode kernel. For grouped-query attention (GQA), Vortex supports the flashinfer [22] and trtllm_mha (TensorRT-LLM) backends; for multi-head latent attention (MLA), used by DeepSeek [4] and GLM [7], it supports trtllm_mla. Reusing these backends also inherits their precision support, such as fp8 KV cache and weights.

Vendor MLA kernels, however, are tied to a fixed geometry: trtllm_mla assumes a specific latent shape and supports only block sizes of 32 or 64. Vortex therefore adds cuda_mla, its own MLA decode kernel supporting general latent geometries and finer blocks (down to 16), so the block granularity can match the sparsity budget. Together, these backends let Vortex automatically support diverse models.

6 Evaluation↩︎

We evaluate Vortex to answer the following two questions:

• Q1. Is Vortex really helpful in innovating and iterating sparse attention algorithms? Yes, and we show it along three axes.

  • Innovation and iteration by AI agents ([sec:innovation,sec:iteration]). Claude Code and Codex generate a broad set of structurally distinct sparse attention algorithms with Pareto-efficient accuracy-throughput trade-offs, and an 18-hour autonomous loop improves AIME24 throughput by up to \(3.46\times\) over dense attention for Qwen3-1.7B while preserving accuracy.

  • Design for new architectures (6.2). Multi-head latent attention (MLA) is a new architecture adopted by frontier models such as DeepSeek and GLM; Vortex lets us rapidly design a rope-aware sparse attention for it, a \(4\times\) end-to-end speedup over full attention at matched accuracy on GLM-4.7-Flash.

  • New understanding (6.3). Vortex acts as a research instrument that reveals where the routing signal lives, identifying which query-key channels and which MLA components are critical.

• Q2. Is Vortex making sparse attention algorithms really faster in deployment? In 6.4.1, for server throughput, Vortex achieves up to \(3.60\times\) and \(2.98\times\) higher throughput than full attention for block top-\(k\) and Quest, respectively, while maintaining accuracy. This advantage holds at the largest scale we test; on the 229B-parameter MiniMax-M2.7 served across four NVIDIA B200 GPUs with tensor parallelism (TP\(=4\)), block top-\(k\) reaches up to \(1.37\times\) higher throughput than full attention with the same accuracy. In 6.4.2, for user latency, Vortex reduces P95 latency by \(11.7\times\) for block top-\(k\) and \(12.8\times\) for Quest at a 16K prompt length under a request rate of 8.0.

Besides, we present the ablation study of the kernel efficiency in 11.

6.1 Researching Sparse Attention with AI Agents↩︎

In this section, we present two use cases of Vortex for AI-assisted research on sparse attention.

Setup. Our target model is Qwen3-1.7B deployed on NVIDIA H200 SXM GPU. We evaluate both reasoning accuracy and serving efficiency using pass@16 on AMC23 and AIME24 (# Max Gen Tokens = 16K), as well as decoding throughput. To support AI-agent-driven research, we provide the agents with 8 representative sparse attention papers as references, along with the Vortex documentation. The template for AI agents to submit algorithms is present in 12.

6.1.1 Experiment 1: Algorithm Innovation↩︎

We task Claude Code [76] (Opus-4.7 and Sonnet-4.6) and Codex [77] (GPT-5.5 [78]) with proposing and implementing 20 sparse attention algorithms each in a one-shot setting, explicitly prompting them to generate novel designs; we describe all of the proposed algorithms in 14. This experiment evaluates whether AI agents can directly produce diverse and effective sparse attention algorithms when equipped with Vortex.

Algorithm Diversity. As shown in 8, we measure diversity using a structural distance metric computed from the generated implementations. For each program, we extract (1) cache definitions, (2) instantiated operators, and (3) operator invocation patterns using abstract syntax tree parsing, and compute pairwise Jaccard-style distances over these components. Claude Sonnet 4.6 achieves the highest average pairwise distance (\(0.789\)), followed by Claude Opus 4.7 (\(0.770\)) and GPT-5 (\(0.709\)). All models produce highly diverse implementations, indicating that AI agents can generate substantially different sparse attention designs in a single attempt when paired with Vortex.

Algorithm Effectiveness. All generated algorithms compile and execute successfully in Vortex without manual intervention, demonstrating the robustness and expressiveness of the programming model. We evaluate the generated algorithms using a staged pipeline: algorithms first run on RULER [79] 4K retrieval, those exceeding \(85\%\) accuracy are evaluated on AMC23, and frontier algorithms are further tested on AIME24. As shown in [fig: amcinno,fig: rulerinno,fig: aimeinno], all three AI models produce Pareto-efficient sparse attention algorithms that maintain dense-attention-level accuracy while achieving \(2\times\)–\(3.1\times\) higher throughput across benchmarks. Sonnet 4.6 produces the strongest algorithms on RULER and AMC23, while GPT-5 achieves the best AIME24 result. These results suggest that AI agents, when paired with Vortex, can directly generate competitive sparse attention designs and accuracy–throughput trade-offs without iterative search or manual systems engineering.

6.1.2 Experiment 2: Algorithm Iteration↩︎

The second study focuses on algorithm iteration. The target benchmark is AIME24. We allow Claude Code to continuously modify, evaluate, and refine sparse attention algorithms over an 18-hour optimization loop. Unlike the previous study, this experiment does not require algorithmic novelty; the AI agent is free to optimize any aspect of the serving pipeline, including sparse attention structures, block sizes, KV-cache data types, and other system-level parameters. This experiment studies whether Vortex can support rapid end-to-end experimentation and iterative improvement driven by AI agents.

We illustrate the results in [fig: mean16iter,fig: throughput16iter,fig: iterationscatter]. The autonomous optimization loop produces sparse-attention flows that achieve up to a \(3.46\times\) throughput speedup over dense attention on AIME24 (\(11{,}894\) vs.\(3{,}437\) tok/s) while preserving accuracy (\(38.96\) vs.\(38.54\)).

Although the final algorithms converge to block top-\(k\) attention rather than fundamentally new sparse attention mechanisms, the AI agent explores and evaluates many diverse algorithmic hypotheses throughout the optimization process, many of which outperform full attention. The improvements arise from a combination of algorithmic and systems-level refinements, including tuning block sizes, top-\(k\) selection strategies, and skipped-layer configurations. More broadly, discovering fundamentally better sparse attention algorithms remains an open challenge, and we believe stronger experimental capabilities and longer optimization horizons could further improve AI-driven algorithm search

6.2 Designing Sparse Attention on MLA Architectures↩︎

Beyond grouped-query attention, we study whether Vortex generalizes to multi-head latent attention (MLA), the compressed-KV design used by DeepSeek [4] and GLM [7]. This offers a second perspective on Q1. Instead of asking an agent to invent flows, we ask whether a researcher can design and validate architecture-specific sparse attention for a new attention family. We evaluate GLM-4.7-Flash on AIME26 with a 32K-token generation budget on a single NVIDIA B200 GPU,¹ designing a rope-aware block-sparse flow in vFlow and comparing it against Quest. The rope-aware flow scores each KV block using the full fused-latent dot product, combining the compressed content and the decoupled rope components, a design specific to the MLA layout. (11 additionally includes a rope-unaware ablation, which we analyze in 6.3.)

As shown in 11, full attention reaches a mean@16 of \(0.765\) but sustains only \(1{,}031\) tokens/s. The rope-aware design matches this accuracy within about one point (\(0.752\)) while delivering roughly \(4\times\) higher throughput, and up to \(4.7\times\) with a tighter block budget, and it even matches or exceeds full attention on pass@4 and pass@8; Quest stays consistently behind the rope-aware frontier. This answers Q1 from a second angle. With Vortex, a researcher can rapidly design and validate sparse attention tailored to a new architecture, here turning the MLA layout into a \(4\times\) end-to-end speedup at full-attention accuracy.

6.3 Gaining New Understanding on Sparse Attention↩︎

Vortex is also a research instrument. Because a scoring rule is only a few lines of vFlow, we can probe which parts of the key representation actually drive block selection, turning sparse attention into a controlled experiment. We highlight two findings.

Query-key channel importance [10]. We split the \(128\) query-key channels into eight groups of sixteen, labeled g0 through g7, and use vFlow to mask one group at a time during routing on Qwen3-4B and Qwen3-8B with RULER, then measure accuracy (12).² Routing information turns out to be highly concentrated. Of the eight groups, only two (g3 and g7) are critical. Masking either one alone is nearly harmless, yet masking both collapses every method to near-random even when six of the eight groups are kept, so which channels are kept matters far more than how many. Conversely, keeping just the half that contains g3 and g7 is lossless, and Quest even improves slightly. The same two groups are critical at both model sizes, indicating a stable Qwen3-family property of the channel geometry rather than a size-specific artifact.

Rope component in MLA routing. The MLA study in 6.2 reveals an analogous effect along a different axis. The rope-unaware variant in 11, which scores blocks from the compressed content alone and ignores the decoupled rope component, drops sharply to \(0.60\) to \(0.63\) mean@16, far below the rope-aware design at \(0.75\). The positional rope component, not just the content, carries the signal needed for accurate block selection under MLA.

Together these studies expose a third facet of Q1. Vortex is useful not only for building and deploying sparse attention but for understanding it, revealing where the routing signal lives across both GQA channels and MLA components.

6.4 Efficiency Evaluations↩︎

Experimental Setup. We evaluate dense Qwen3 models [80] ranging from 0.6B to 8B parameters on an NVIDIA H200 SXM GPU with 141GB memory. We study two sparse attention algorithms, block top-\(k\) attention and Quest [9]. Following prior work [8], [13], [52], each query always attends to the first block and the last two blocks, using a block size of 16 and varying the number of attended blocks from 32 to 256. We compare against SGLang v0.5.9 [24], into which Vortex is integrated. To test scaling to larger models, longer generation, and fp8 precision, we additionally evaluate the 229B-parameter MiniMax-M2.7 [81] across four NVIDIA B200 GPUs with tensor parallelism (TP\(=4\)); these scaling runs use up to 32K-token generation and sweep block sizes of 16, 32, and 64.

6.4.1 Server-side Throughput↩︎

Dense models. We evaluate server-side throughput on the long-generation benchmarks AMC23 and AIME24, with input lengths up to 4K tokens and outputs up to 16K tokens. We report end-to-end offline generation throughput and construct throughput–accuracy Pareto frontiers. As shown in [fig: 0.6baime,fig: 0.6bamc,fig: 1.7baime,fig: 1.7bamc,fig: 4baime,fig: 4bamc,fig: 8baime,fig: 8bamc], sparse attention achieves substantial throughput gains. Under a 5 pp accuracy budget, block top-\(k\) attains up to \(3.46\times\) and \(3.60\times\) speedup on AMC23 and AIME24, respectively, while Quest achieves up to \(2.73\times\) and \(2.98\times\). Even under a stricter 1 pp accuracy budget, block top-\(k\) still delivers \(2.14\)–\(2.31\times\) speedup on AMC23 and \(2.42\)–\(3.60\times\) on AIME24, while Quest achieves \(1.99\)–\(2.11\times\) and \(1.90\)–\(2.59\times\), respectively.

Scaling to a 229B MoE with tensor parallelism. We push further to MiniMax-M2.7, a 229B-parameter model served across four NVIDIA B200 GPUs with tensor parallelism (TP\(=4\)), again on AIME26 with 32K-token generation (14). Full attention reaches a mean@16 of \(0.83\) at \(3{,}341\) tokens/s. Block top-\(k\) even slightly exceeds this accuracy (\(0.84\)) while running \(1.23\times\) faster (\(4{,}110\) tokens/s), and reaches up to \(1.37\times\) with a tighter block budget, while Quest attains \(1.19\times\) at comparable accuracy. The relative gains are smaller than for the dense and 30B-MoE models because at this scale, with TP\(=4\), a larger share of decode time goes to parameter and communication traffic rather than KV-cache access, yet sparse attention still delivers a consistent end-to-end speedup at the largest scale we test.

Choosing block size and budget. Sweeping the block size (16, 32, 64) and the top-\(k\) budget across these experiments, we observe a consistent pattern. A small block size of 16 with a moderate budget of around 125 attended blocks sits near the accuracy ceiling while retaining most of the throughput gain, larger blocks favor throughput, and raising the budget further yields diminishing accuracy returns; block-16 with top-\(k\approx125\) is thus a robust default across model scales. Vortex made these sweeps, spanning 0.6B to 229B models, practical to run fully end-to-end.

6.4.2 User-side Latency↩︎

We evaluate user-side latency using synthetic 16K-token prompts at request rates from 1.0 to 8.0 req/s, with each request generating 512 tokens. We report P95 time per output token (TPOT). All experiments use a P1D1-disaggregated setup to isolate decoding performance, the primary target of Vortex’s sparse-attention optimizations. As shown in [fig: 0.6b16k,fig: 1.7b16k,fig: 4b16k,fig: 8b16k], sparse attention substantially reduces latency under high request rates, achieving up to \(11.7\times\) lower P95 TPOT with block top-\(k\) and \(12.8\times\) lower P95 TPOT with Quest at 8 req/s. At moderate load (4 req/s), block top-\(k\) reduces latency by up to \(11.8\times\), while Quest achieves up to \(11.1\times\) reduction.

7 Conclusion↩︎

We presented Vortex, a system for rapidly developing and deploying sparse-attention algorithms in modern LLM-serving environments. Vortex combines a Python-embedded frontend language (vFlow), an interpreter that lowers programs into executable vTensor operators, and an efficient backend integrated with existing serving systems to realize end-to-end sparse attention speedups. We further demonstrated that Vortex enables AI-agent-driven exploration of the sparse attention design space, producing algorithms with strong accuracy–throughput trade-offs.

Acknowledgements↩︎

We gratefully acknowledge access to NVIDIA computing resources. This work was partially supported by Google Research Award, Google ML & System Junior Faculty Award, Amazon Research Award, Fireworks AI, Intel, Li Auto, Moffett AI, and CMU CyLab Seed funding. This material is also based upon work supported by the National Science Foundation under Grant Nos. CCF-2504353 and CCF-2247014, and by IARPA. Any opinions, findings, conclusions or recommendations expressed are those of the authors and do not necessarily reflect the views of the National Science Foundation.

Limitations and Future Work↩︎

Limitations. Prefill support. Currently, Vortex focuses on the decoding stage of LLM serving and does not yet support sparse attention optimization during the prefill phase. While decoding dominates the cost of long-generation workloads, prefill efficiency remains important for many real-world applications, and extending Vortex to execute efficient sparse attention during prefill is an important direction. Training support. Vortex is designed for inference serving and targets the forward decoding path. It does not yet support sparse attention during training, where the backward pass introduces additional gradient and memory layout requirements that the current vTensor abstraction and backend do not address. Supporting sparse attention training would let researchers study algorithms that are co-designed with the training objective rather than applied only at inference time.

Future work. RNN and Mamba architectures. Vortex currently centers on attention-based models with an explicit key-value cache. Extending the vTensor abstraction to recurrent and state-space architectures such as RNNs and Mamba [82], whose state is a fixed-size recurrent representation rather than a growing cache, would broaden Vortex to the emerging class of hybrid and linear-attention models [83]. End-to-end sparse attention algorithm discovery. Although Vortex enables rapid programming, deployment, and large-scale validation of sparse attention algorithms, it is not itself a fully end-to-end research system. Building end-to-end autonomous systems that jointly propose, analyze, optimize, implementing kernels and validate sparse attention algorithms remains an open research direction.

More broadly, we view Vortex as a foundation for future systems that tightly integrate programmable serving infrastructures with AI-driven systems research.

Broader Impact↩︎

We focus on systems for sparse attention serving in LLMs. While there are numerous application scenarios for large language models that warrant further study of potential societal impact, we would like to highlight that our work does not advance the capabilities of these models. Our work is primarily an algorithmic study with no specific usage limitations, and while LLMs themselves can be used with malicious purposes, we believe that none of such use cases are specific to this paper.

Declaration of LLM Usage↩︎

We used large language models (LLMs) to assist with polishing the writing of this paper and to generate illustrative figures unrelated to the experimental results. In addition, this work includes experiments with LLM agents on sparse attention research and optimization.

8 Tensor Layout in LLM Serving↩︎

Consider a batch of size \(b\) with sequence lengths \({s_0,\dots,s_{b-1}}\) and per-token feature shape \(\mathbf{h}\).

Batch Layout. A set of independent tensors \({\boldsymbol{x}_i \in \mathbb{R}^{s \times \mathbf{h}} \mid 0 \le i < b}\).

Ragged Layout. A contiguous buffer \(\boldsymbol{x}_{\mathrm{flat}} \in \mathbb{R}^{(\sum_i s_i)\times \mathbf{h}}\) with pointer array \(\mathbf{p}\in\mathbb{N}^{b+1}\), where \(\mathbf{p}[0]=0\) and \(\mathbf{p}[i+1]=\mathbf{p}[i]+s_i\). Each sequence is recovered as \(\boldsymbol{x}_i=\boldsymbol{x}_{\mathrm{flat}}[\mathbf{p}[i]:\mathbf{p}[i+1]]\).

Paged Layout. With a shared tensor storage \(\boldsymbol{x}_{\mathrm{storage}} \in \mathbb{R}^{N \times \mathbf{h}}\), each sequence is represented by a list of page indices \(I_i\) and reconstructed as \(\boldsymbol{x}_i=\mathrm{concat}(\boldsymbol{x}_{\mathrm{storage}}[k] \mid k \in I_i)\). Different sequences may share pages, i.e., \(I_i\) and \(I_j\) may contain the same indices. The index structure \(\mathbf{I}=\{I_0,\dots,I_{b-1}\}\) follows a ragged layout. Both batched and ragged layouts can be constructed as instances of the paged layout.

9 Details of vTensor↩︎

Shape Propagation. For each output tensor \(\boldsymbol{y}^{v}_m\), we require all sequences to share the same non-leading dimensions, while allowing the leading dimension to vary across sequences. Formally, \(\mathrm{shape}(\boldsymbol{y}^{v}_{m,i}) = (s_{m,i}, \mathbf{h}_m),\) where \(\mathbf{h}_m\) is identical for all \(0 \le i < b\). Under this constraint, the output shape is represented using a ragged layout with pointer array \(\mathbf{p}\) satisfying \(\mathbf{p}[0]=0\) and \(\mathbf{p}[i+1]=\mathbf{p}[i]+s_{m,i}\). The aggregated tensor therefore has shape \(\left(\sum_i s_{m,i}\right)\times \mathbf{h}_m\).

Layout Propagation. In addition to shape propagation, each output tensor must also determine its page layout. Specifically, for every sequence \(i\), the output tensor \(\boldsymbol{y}^{v}_{m,i}\) is associated with a page index list \(I_{m,i}\) describing where its data resides in shared storage. These page indices may either be newly allocated or reused from existing tensors, depending on the operator semantics. Therefore, layout propagation is controlled by compiler or user-provided policies that specify how pages are allocated, reused, and propagated across operators.

Discussion. Our abstraction imposes a simple constraint: operators are not allowed to transform the batch dimension. Fortunately, this restriction aligns naturally with practical LLM serving workloads, where sequences within a batch are typically processed independently and cross-sequence operations are rare. By extending PyTorch tensors with layout semantics, vTensor systematically inherits the semantics of existing PyTorch operators while augmenting them with shape and layout propagation rules. This design enables a simple and compositional programming model for paged tensor computation (see 3).

10 Kernel Templates↩︎

11 Ablation Study of Vortex Efficiency↩︎

In this section, we study two aspects of Vortex. First, we analyze the execution breakdown of Vortex during a decoding step and quantify the impact of sparse attention kernels. Second, we evaluate the effectiveness of our stochastic radix top-\(k\) optimizations.

11.1 Kernel Profiling.↩︎

Replacing dense attention with sparse attention significantly compresses the decode block latency (17). At BS=16 and 32k context length, block top-\(k\) achieves \(3.78\text{--}4.81\times\) end-to-end speedup across Qwen3 model sizes, while Quest achieves \(3.46\text{--}4.33\times\). At shorter contexts, sparse attention still provides substantial gains, including \(2.08\text{--}2.96\times\) at 16k and \(1.27\text{--}1.74\times\) at 8k. Even at smaller batch sizes (BS=4, 32k), both methods maintain \(1.28\text{--}1.81\times\) speedup.

At the kernel level, sparse attention is substantially faster than dense attention. For example, on Qwen3-8B with BS=16 and 32k context length, the sparse attention kernel requires only \(0.025\) ms compared to \(0.760\) ms for dense attention, yielding over \(30\times\) kernel-level acceleration. Meanwhile, indexer and cache-management kernels contribute only a few microseconds, demonstrating that the overhead introduced by Vortex is negligible compared to attention computation.

11.2 Radix Top-\(k\) Kernels.↩︎

Additional Optimization: Remapping. Before radix partitioning, we apply a monotonic transformation \(x'=f(x)\) that preserves ordering while reshaping the value distribution. The transformation improves bin separation and reduces refinement cost. The function \(f\) is selected offline through profiling and fused into the histogram pass without additional memory traffic.

At recall@\(k > 0.97\), our approx_radix_topk + remap kernel consistently outperforms the radix_topk baseline across all tested settings (18). The speedup ranges from \(1.30\text{--}1.62\times\), with an average improvement of \(1.49\times\). The gains remain stable across different \(k\) values, prefill lengths, and score distributions, including bimodal, normal, lognormal, and uniform distributions. Across all evaluated configurations, approx_radix_topk + remap is the fastest implementation among the four compared kernels.

11.3 Topk Kernels Pareto Analysis↩︎

11.3.0.1 Pareto coverage.

The bar comparison in Figure 18 fixes a single recall floor; Figure 19 reports the full latency–recall trade-off so that the choice of operating point is auditable. Each point represents the fastest variant of a kernel family at one (model, distribution, prefill, \(k\)) cell. The structure of the cloud confirms the reading from Figure 18: approx_radix_topk + remap sits to the left of every other family at every recall level above \(0.97\), never crossing the radix_topk or approx_radix_topk clouds, so its dominance is not an artifact of the chosen recall floor.

11.3.0.2 Choice of the recall threshold.

The operating point is bounded from both sides. (i) Lower bound from competing variants. At thresholds below \(\sim\) \(0.95\), plain approx_radix_topk with aggressive tolerate ratios (\(\tau \geq 0.25\)) becomes eligible and undercuts approx_radix_topk + remap in some cells (most visibly on lognormal at small prefill, large \(k\)), blurring the headline claim. The smallest threshold at which approx_radix_topk + remap is uncontested across (prefill, \(k\), distribution) is \(\sim\) \(0.96\). (ii) Upper bound from kernel feasibility. At thresholds above \(\sim\) \(0.99\), even approx_radix_topk + remap struggles to qualify in cells where the score histogram is intrinsically diffuse (notably small-\(k\) uniform cells, where the maximum reachable recall is \(\sim\) \(0.95\)). Operating closer to \(1.0\) would force fallback to radix_topk and eliminate the speedup.

12 Templates for AI Agents↩︎

In my_sparse_attention.py, AI agents will register my_sparse_attention by sub-classing vFlow. Vortex now supports bfloat16, fp8_e4m3 and fp8_e5m2.

13 Operators and the Cache/Indexer Flow↩︎

A vFlow program has two stages. In the cache stage, create_cache declares named auxiliary fields and forward_cache fills them from the KV cache; this is query-independent and runs per block, so cache-side operators act within a single block. In the indexer stage, forward_indexer consumes the current query together with the cached fields to produce a per-block routing score and ends in a selection op (topK); indexer-side operators act across pages and additionally provide cross-page reductions, normalization, cross-step persistence (Load/Save), and selection. 1 lists the operators and the stage(s) in which each is available.

14 AI-Agent-Proposed Algorithms↩︎

Each model proposed \(20\) sparse-attention flows in the one-shot setting of 6.1.1. We describe all of them below in words and notation rather than code.

14.0.0.1 The shared pipeline.

Every flow is a complete instance of the same four-stage computation that, at each decoding step, turns the query \(q\) and the KV cache \((K,V)\) into an attention output. (1) Build cache. The KV sequence is split into blocks of size \(b\); block \(j\) holds keys \(K_j\) and values \(V_j\) with token vectors \(k_t,v_t\), from which the flow precomputes (in create_cache/forward_cache) the per-block statistics it will need, such as the key centroid \(c_j=\tfrac1b\sum_{t\in j}k_t\), the value centroid \(\bar v_j\), the feature-wise key envelope \(k^{\max}_{j},k^{\min}_{j}\), the mean token value norm \(\overline{\lVert v\rVert}_j\), the mean key norm \(\overline{\lVert k\rVert}_j\), or a set of finer sub-centroids \(c_j^{(g)}\). (2) Form the query view. The decode query is reduced to a routing query, either the group-averaged query \(\bar q\) or a per-head query \(q_h\). (3) Score. The flow computes a scalar score \(s_j\) for every block from the query view and the cached statistics; this score is the only stage that differs across methods. (4) Select and attend. The top-\(k\) blocks by \(s_j\) are kept and the query runs exact softmax attention over only those blocks’ keys and values, producing the output. Flows that name an exponential moving average (EMA) or a previous-step quantity carry state across decoding steps through Load/Save; a few customize stage (4) (for example masking with \(-\infty\) instead of a fixed top-\(k\)), which we note inline.

For each algorithm we therefore give only what distinguishes it, namely the block statistics it caches and the score it computes; stages (2) and (4) are as above unless stated. Most ideas recombine a few recurring motifs, namely centroid alignment, the Quest key envelope, value-energy gating, multi-resolution sub-blocks, and temporal accumulation.

14.0.0.2 Claude Opus 4.7.

Cache: key centroid \(c_j\) and the previous step’s score \(s_j^{\mathrm{prev}}\) (carried via Save/Load). Score: \(s_j=\langle\bar q,c_j\rangle-\tfrac12 s_j^{\mathrm{prev}}\). Rewards blocks whose relevance is rising and discounts ones already heavily attended.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\). Score: \(s_j=\min_h\sum_d\max(q_h k^{\max}_{j,d},q_h k^{\min}_{j,d})\), evaluating the Quest envelope per head and keeping a block only if it looks relevant to every head.

Cache: several sub-centroids \(c_j^{(g)}\) per block. Score: \(s_j=\min_g\langle\bar q,c_j^{(g)}\rangle\), the least-aligned sub-region, so a block is kept only if it matches the query at every resolution.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\). Score: \(s_j=\max_h\sum_d\min(q_h k^{\max}_{j,d},q_h k^{\min}_{j,d})\), inverting Quest to a conservative lower bound on similarity.

Cache: key centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\overline{\lVert v\rVert}_j\), favoring blocks whose values carry more energy.

Cache: centroid \(c_j\) and a running mean and second moment of the block score (state). Score: \(s_j=\langle\bar q,c_j\rangle+\tfrac12\widehat{\mathrm{Var}}_j\), periodically re-examining blocks whose relevance fluctuates.

Cache: centroid \(c_j\) and an EMA of past scores (state). Score: \(s_j=\langle\bar q,c_j\rangle-0.4\,\mathrm{EMA}[s_j]\), discouraging repeated selection of the same blocks.

Cache: centroid \(c_j\). Score: the base alignment \(\langle\bar q,c_j\rangle\) convolved along the sequence axis with a \([0.25,0.5,0.25]\) kernel, damping isolated spikes and rewarding locally coherent regions.

Cache: centroid \(c_j\). Score: \(s_j=\sum_d\sigma(\bar q_d c_{j,d})\), passing each feature contribution through a sigmoid so no single dimension dominates.

Cache: mean key norm \(\overline{\lVert k\rVert}_j\). Score: \(s_j=\overline{\lVert k\rVert}_j\), a query-independent baseline that attends to high-energy regions regardless of the query.

Cache: value centroid \(\bar v_j\). Score: \(s_j=\langle\bar q,\bar v_j\rangle\), testing whether value content alone is a useful routing signal.

Cache: key centroid \(c_j\) and per-feature key norms \(\lVert k_{\cdot,d}\rVert_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\sum_d\bar q_d\lVert k_{\cdot,d}\rVert_j\), requiring both directional and magnitude agreement.

Cache: centroid \(c_j\). Score: the centroid alignment multiplied by an inverted softmax of itself, suppressing the largest scores so selection peaks at moderate rather than extreme relevance.

Cache: sub-centroids \(c_j^{(g)}\). Score: \(s_j=\min_{h,g}\langle q_h,c_j^{(g)}\rangle\) over all head-by-sub-centroid pairings (a Kronecker grid), demanding agreement across both heads and intra-block regions.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\). Score: \(s_j=\sum_d q_d(k^{\max}_{j,d}-k^{\min}_{j,d})\), preferring blocks with a wide per-feature key range.

Cache: centroid \(c_j\) and the previous routing query \(\bar q^{\mathrm{prev}}\) (state). Score: \(s_j=\langle 0.7\,\bar q^{\mathrm{prev}}+0.3\,\bar q,\,c_j\rangle\), stabilizing selection against per-step query noise.

Cache: centroid \(c_j\) and a running minimum of the block score (state). Score: \(s_j=\langle\bar q,c_j\rangle-\min\text{-hist}_j\), selecting blocks currently well above their own historical floor.

Cache: whole-block centroid \(c_j\) and half-block sub-centroids \(c_j^{(g)}\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\max_g\langle\bar q,c_j^{(g)}\rangle\), combining a coarse and a fine view of the block.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\langle\bar q,\bar v_j\rangle\), keeping blocks where both representations point toward the query.

Cache: centroid \(c_j\). Score: \(\langle\bar q,c_j\rangle\) convolved with a \([-0.5,1.5,-0.5]\) kernel, isolating sharp peaks and penalizing blocks surrounded by similarly high neighbors.

14.0.0.3 Claude Sonnet 4.6.

Cache: key centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\overline{\lVert v\rVert}_j\), favoring high-energy blocks.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=\min(\langle\bar q,c_j\rangle,\langle\bar q,\bar v_j\rangle)\), so a block must agree on both to score highly.

Cache: a value-norm-weighted key centroid \(c_j^{w}=\big(\sum_{t\in j}\lVert v_t\rVert\,k_t\big)/\big(\sum_{t\in j}\lVert v_t\rVert\big)\) instead of a plain mean. Score: \(s_j=\langle\bar q,c_j^{w}\rangle\), so keys paired with high-energy values dominate the block representation.

Cache: key centroid \(c_j\). Score: \(s_j=-\lVert\bar q-c_j\rVert_1\), selecting blocks closest under an absolute-difference metric.

Cache: mean absolute key \(\mathbb{E}[|k|]_j\). Score: \(s_j=\langle|\bar q|,\mathbb{E}[|k|]_j\rangle\), matching on activation strength rather than sign.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\). Score: \(s_j=\max_h\sum_d\mathrm{ReLU}(q_h k^{\max}_{j,d})\), summing only positive envelope evidence.

Cache: sub-centroids \(c_j^{(g)}\). Score: \(s_j=\max_{h,g}\langle q_h,c_j^{(g)}\rangle\), so any strongly matching head-region pair can select a block.

Cache: centroid \(c_j\). Score: \(s_j=\sum_d\mathrm{SiLU}(\bar q_d c_{j,d})\), a soft alternative to hard feature thresholding.

Cache: centroid \(c_j\) and its norm \(\lVert c_j\rVert\). Score: \(s_j=\langle\bar q,c_j\rangle+0.1\log\lVert c_j\rVert\), gently preferring blocks with larger key magnitude.

Cache: centroid \(c_j\). Score: \(\langle\bar q,c_j\rangle\), but select adaptively, keeping a block only when its score exceeds the per-step mean across blocks and masking the rest with \(-\infty\) rather than taking a fixed top-\(k\).

Cache: centroid \(c_j\). Score: \(\langle\bar q,c_j\rangle\) smoothed by a 3-tap kernel along the sequence axis, rewarding locally coherent regions.

Cache: centroid \(c_j\) and an EMA (rate \(0.95\)) of the block value norm (state). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\mathrm{EMA}[\overline{\lVert v\rVert}_j]\), a slowly varying value-importance weight.

Cache: centroid \(c_j\) and a fast and a slow EMA of the score (state). Score: the minimum of the two EMAs, keeping blocks relevant on both short and long horizons.

Cache: per-sub-block key envelopes \(k^{\max}_{j,g},k^{\min}_{j,g}\). Score: \(s_j=\max_g\langle\bar q,\,k^{\max}_{j,g}-k^{\min}_{j,g}\rangle\), combining the Quest envelope with sub-block resolution.

Cache: centroid \(c_j\). Score: \(s_j=p_j(-\log p_j)\) with \(p=\mathrm{softmax}_j\langle\bar q,c_j\rangle\), emphasizing mid-probability blocks over both certain and negligible ones.

Cache: centroid \(c_j\). Score: \(s_j=\langle\bar q,c_j\rangle+0.5\,\big|[-1,2,-1]*\langle\bar q,c\rangle\big|_j\), boosting blocks at local extrema of the score profile.

Cache: value centroid \(\bar v_j\). Score: \(s_j=\langle\bar q,\bar v_j\rangle\), a check on how much routing signal lives in the values.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=\mathrm{ReLU}(\langle\bar q,c_j\rangle)\cdot\mathrm{ReLU}(\langle\bar q,\bar v_j\rangle)\), a smooth logical AND of the two signals.

Cache: lower key envelope \(k^{\min}_j\). Score: \(s_j=\max_h\sum_d q_h k^{\min}_{j,d}\), a pessimistic counterpart to standard Quest.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\). Score: \(s_j=\max_h\sum_d\sigma\!\big(q_d(k^{\max}_{j,d}-k^{\min}_{j,d})\big)\max(q_d k^{\max}_{j,d},q_d k^{\min}_{j,d})\), down-weighting features whose envelope is narrow and thus uninformative.

14.0.0.4 GPT-5.

Cache: key centroid \(c_j\) and per-feature value norms \(\lVert v_{\cdot,d}\rVert_j\). Score: \(s_j=\sum_d(\bar q_d c_{j,d})\,\lVert v_{\cdot,d}\rVert_j\), letting value energy act as a per-feature temperature on key relevance.

Cache: key envelope \(k^{\max}_j,k^{\min}_j\) and per-feature value norms. Score: the head-maximum of the Quest envelope products with each feature scaled by \(\lVert v_{\cdot,d}\rVert_j\), combining envelope key matching with value importance.

Cache: centroid \(c_j\) and its absolute value \(|c_j|\). Score: \(s_j=\langle\bar q,c_j\rangle-0.25\,\langle\bar q,|c_j|\rangle\), penalizing blocks that score highly only through large-magnitude features.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=\langle\bar q,c_j\rangle+\mathbf{1}\!\left[\langle\bar q,\bar v_j\rangle>\overline{\langle\bar q,\bar v\rangle}\right]\), a binary value-side gate atop centroid routing.

Cache: \(c_j\), per-feature value norms, and an EMA accumulator \(r_j\) (state). Score: \(r_j\leftarrow0.7\,r_j+\sum_d(\bar q_d c_{j,d})\lVert v_{\cdot,d}\rVert_j\) with \(s_j=r_j\), smoothing the value-weighted signal over decoding steps.

Cache: four sub-centroids \(c_j^{(g)}\). Score: \(s_j=\max_g\langle\bar q,c_j^{(g)}\rangle\), so a strong match in any region selects the block.

Cache: four sub-centroids \(c_j^{(g)}\). Score: \(s_j=\tfrac14\sum_g\langle\bar q,c_j^{(g)}\rangle\), a smoother counterpart to the peak version.

Cache: the interleaved key envelope. Score: the maximum over a Kronecker pairing of the query with the interleaved envelope, exposing intra-block envelope structure to the score.

Cache: centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\langle\bar q,c_j\rangle+\mathbf{1}[\,\bar q\!\cdot\!\overline{\lVert v\rVert}_j>\text{mean}\,]\), promoting value outliers.

Cache: centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\mathrm{softmax}_j(\langle\bar q,c_j\rangle)\cdot(\bar q\!\cdot\!\overline{\lVert v\rVert}_j)\), sharpening relevant blocks while weighting by value content.

Cache: centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\sigma(-0.02\,\bar q\!\cdot\!\overline{\lVert v\rVert}_j)\), a soft saturating value modulation.

Cache: centroid \(c_j\) and lower key envelope \(k^{\min}_j\). Score: \(s_j=\langle\bar q,c_j\rangle-0.5\,\langle\bar q,k^{\min}_j\rangle\), penalizing blocks whose minimum keys already align with the query.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=0.75\,\langle\bar q,c_j\rangle+0.25\,\langle\bar q,\bar v_j\rangle\), a soft combination of the two signals.

Cache: key centroid \(c_j\) and value centroid \(\bar v_j\). Score: \(s_j=\langle\bar q,c_j\rangle\cdot\langle\bar q,\bar v_j\rangle\), requiring both to be high at once.

Cache: centroid \(c_j\) and an EMA accumulator \(r_j\) (state). Score: \(r_j\leftarrow0.6\,r_j+\big|\langle\bar q,c_j\rangle-\mathrm{mean}_{j'}\langle\bar q,c_{j'}\rangle\big|\) with \(s_j=r_j\), selecting blocks that are persistently atypical.

Cache: per-head key centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\max_h\langle q_h,c_j\rangle\cdot(\bar q\!\cdot\!\overline{\lVert v\rVert}_j)\), scaling the strongest head match by block value energy.

Cache: mean value norm \(\overline{\lVert v\rVert}_j\). Score: \(s_j=\bar q\!\cdot\!\overline{\lVert v\rVert}_j\), a baseline using no key statistics.

Cache: mean key norm \(\overline{\lVert k\rVert}_j\). Score: \(s_j=\bar q\!\cdot\!\overline{\lVert k\rVert}_j\), the key-side counterpart using no value statistics.

Cache: centroid \(c_j\). Score: \(\langle\bar q,c_j\rangle\) with a triangular 3-tap smoothing along the sequence axis.

Cache: centroid \(c_j\) and mean value norm \(\overline{\lVert v\rVert}_j\). Score: the product of a normalized centroid-relevance distribution and a normalized value-energy distribution, selecting blocks that rank highly on both.

References↩︎