From Static Inference to Dynamic Interaction:
A Survey of Streaming Large Language Models
March 04, 2026
Abstract
Standard Large Language Models (LLMs) are predominantly designed for static inference with pre-defined inputs, which limits their applicability in dynamic, real-time scenarios. To address this gap, the streaming LLM paradigm has emerged. However, existing definitions of streaming LLMs remain fragmented, conflating streaming generation, streaming inputs, and interactive streaming architectures, while a systematic taxonomy is still lacking. This paper provides a comprehensive overview and analysis of streaming LLMs. First, we establish a unified definition of streaming LLMs based on data flow and dynamic interaction to clarify existing ambiguities. Building on this definition, we propose a systematic taxonomy of current streaming LLMs and provide an in-depth discussion of their underlying methodologies across text, speech, and video streaming scenarios. Furthermore, we explore the applications of streaming LLMs in real-world scenarios and outline promising research directions to support ongoing advances in streaming intelligence. We maintain a continuously updated repository of relevant papers at https://github.com/EIT-NLP/Awesome-Streaming-LLMs.
1 Introduction↩︎
Large Language Models (LLMs) have shown remarkable efficacy across diverse domains, exhibiting strong reasoning, generation, and cross-modal capabilities [1]–[3]. However, LLMs are predominantly pre-trained on static and full-context corpora, following a “read-at-once” paradigm in which the complete input is provided before any output is generated. While effective for benchmark-style tasks, this paradigm fundamentally limits their applicability in real-world environments, where information arrives incrementally, accumulates over time, and may be unbounded in length.
Such dynamic conditions are ubiquitous in tasks like real-time translation, streaming video understanding, and interactive tool agents [4]–[6]. In these real-world applications, inputs such as speech and video data stream continuously, forcing systems to maintain an evolving understanding based on partial observations. In more complex scenarios, these signals may originate from multiple concurrent streams [7], while systems may also need to generate multiple outputs in parallel [8]. For instance, a robot may need to act, speak, and reason simultaneously [9], whereas an interactive assistant may coordinate speech, visual updates, and control commands [10]. Since the input is never fully available at any given moment, the system must dynamically decide when to respond, when to wait for more information, and when to terminate [11], [12]. These requirements expose a fundamental mismatch with the offline, full-context design of standard LLMs.
Figure 1: Illustration of three types of streaming large language models (LLMs). (Left) Output-streaming LLM performs streaming generation after static reading. (Middle) Sequential-streaming LLM performs streaming generation after streaming reading. (Right) Concurrent-streaming LLM performs streaming generation while streaming reading..
Adapting LLMs to these real-world streaming scenarios presents significant challenges. Beyond architectural modifications, there is a scarcity of large-scale pre-training data that supports real-time interaction, partial-input supervision, and fine-grained temporal alignment. Motivated by this gap, recent research has begun to investigate streaming LLMs [13]–[15]. However, the field currently suffers from terminological ambiguity. Existing studies often conflate distinct concepts, such as autoregressive decoding [16], incremental or chunk-wise encoding [17], and full-duplex interaction like GPT-4o [1], under a single “Streaming LLM” umbrella, obscuring meaningful comparisons.
In this work, we provide the first systematic review of streaming LLMs, proposing a unified definition based on data flow and interaction concurrency. As illustrated in 1, we categorize these models into three distinct levels: (1) Output-streaming LLMs, which retain static input processing but support streaming output generation. (2) Sequential-streaming LLMs, which process streaming inputs incrementally but generate with full input. (3) Concurrent-streaming LLMs, which enable full-duplex interaction by continuously receiving inputs and generating outputs.
This taxonomy captures both conceptual distinctions and a clear progression of technical challenges: Output streaming addresses challenges in streaming and low-latency generation; sequential streaming introduces incremental encoding and context management; and concurrent streaming builds upon both to address architecture adaptation and interaction strategies required for full-duplex processing. By disentangling these paradigms, the taxonomy clarifies which challenges are shared, which are incremental, and which are unique to each category, thereby providing a structured roadmap toward the ultimate goal of fully interactive streaming LLMs. Guided by this framework, we systematically review representative methods in each category, examine emerging applications such as streaming video understanding and real-time reasoning, and highlight open problems, including trade-offs between latency and performance, to inform future research.2
To summarize, our main contributions include:
To our knowledge, we are the first systematic survey of streaming LLMs.
We introduce a unified definition of streaming LLMs, clarifying the conceptual distinctions among existing paradigms.
We provide a systematic taxonomy and comprehensive technical analysis, disentangling the mechanisms of three streaming paradigms.
We discuss emerging applications and open research directions for real-time and interactive streaming scenarios.
Figure 2: Overview of streaming LLM paradigms and their key challenges. The figure contrasts Output-streaming, Sequential-streaming, and Concurrent-streaming LLMs, highlighting their core goals and corresponding research components. Concurrent-streaming builds on the first two and adds extra challenges in real-time streaming architecture adaptation and interaction policy learning..
2 Preliminaries↩︎
2.1 Background of Streaming LLMs↩︎
Current LLMs typically operate under a batch processing paradigm, where the model encodes the entire input sequence into the KV cache during the prefill phase and subsequently generates tokens autoregressively in the decoding phase. Consequently, from a data flow perspective, standard LLMs can be categorized as “streaming-output LLMs” that rely on static context availability. However, real-world data flows often exhibit dynamic and continuous characteristics (e.g., real-time speech transcription and content understanding), necessitating models capable of handling streaming inputs and executing timely output decisions; therefore, generalized streaming LLMs are defined to address such dynamic input and immediate response scenarios, aiming to transcend the limitations of static preprocessing and delayed response.
=[ rectangle, draw=hidden-draw, rounded corners, text opacity=1, minimum height=1.5em, minimum width=5em, inner sep=2pt, align=center, fill opacity=.5, line width=0.8pt, ] =[my-box, minimum height=1.5em, fill=hidden-pink!80, text=black, align=left,font=, inner xsep=2pt, inner ysep=4pt, line width=0.8pt, ]
2.2 Formal Definition↩︎
To rigorously unify the diverse landscape of streaming LLMs in 1, we formulate the modeling process as a conditional probability distribution \(P(Y|X)\), where \(X = (x_1, \dots, x_M)\) denotes the bounded input stream and \(Y = (y_1, \dots, y_N)\) denotes the output stream. This distribution can be factorized autoregressively using the chain rule: \[P(Y|X) = \prod_{t=1}^{N} P\big(y_t | y_{<t}, h_{1:\phi(t)}(X);\theta\big)\] where \(\theta\) denotes the LLM parameters, and \(h_{\phi(t)}(X)=llm(x_{\phi(t)})\) represents the encoded hidden states corresponding to the input prefix \(x_{\phi(t)}\). Here, \(\phi(t)\) is a decision function to determine the input stream visible at generation step \(t\). This general definition can be instantiated into three sub-types by applying varying operational constraints.
2.2.0.1 Output-streaming LLMs
This paradigm imposes a static constraint where the entire input must be processed before generation begins. Mathematically, the decision function is constant relative to the total input length \(M\), i.e., \(\phi(t) = M\) for all \(t \in \{1, \dots, N\}\). The hidden states are computed via a one-time global prefilling: \(h_{1:\phi(t)}(X) = h_{1:M}(X) = llm(X_{1:M}).\)
2.2.0.2 Sequential-streaming LLMs
This paradigm processes dynamic streaming inputs but generates based on a fixed input. While the decision function mirrors the above type (i.e., \(\phi(t) = M, \forall t\)), the hidden states are constrained by stepwise arrival: \(h_{1:M}(X) = \{ llm(x_1), \dots, llm(x_M) \}.\) This represents a sequential encoding process where the context is accumulated token-by-token (or chunk-by-chunk) before the generation phase begins.
2.2.0.3 Concurrent-streaming LLMs
This paradigm imposes the strictest temporal constraints, representing a dynamic process where streams unfold continuously. Mathematically, \(\phi(t)\) must satisfy monotonicity and partial visibility:\(1\le \dots \le \phi(t)\le \phi(t+1)\le \dots \le M.\) The hidden states of input stream are computed via a dynamic or interactive process: \(h_{\phi(t)}(X) = llm(X_{\phi(t)},y_{<t}).\)
The tripartite taxonomy defined above reflects a trajectory of escalating operational constraints and functional demands, shifting the paradigm from static processing to dynamic, real-time interaction.
\(1\le \dots \le \phi(t)\le \phi(t+1)\le \dots \le M.\)
2.3 Overview↩︎
This survey provides a systematical overview of research in streaming LLMs. Figure 2 illustrates the proposed taxonomy, detailing the primary research focuses and challenges within each category. Specifically, output-streaming emphasizes streaming generation mechanisms and efficient generation; sequential-streaming focuses on incremental encoding processing and context management for input streams; and concurrent-streaming integrates both tasks, additionally introducing architectural adaptations and the interactive management of simultaneous input and output streams. To navigate this comprehensive landscape, 3 outlines the taxonomy structure of this survey. Guided by this taxonomy, we begin with output-streaming in Section 3, expand to the dynamic input processing of sequential-streaming in Section 4, and culminate with the interactive dynamics of concurrent-streaming in Section 5. Beyond the technical part, Section 6 reviews downstream tasks and applications, and Section 7 discusses the future directions.
3 Output-Streaming LLMs: Generating with Progressive Revelation↩︎
3.1 Streaming Generation Mechanism↩︎
Output streaming enables progressive revelation by continuously emitting intermediate results rather than waiting for completion. Based on the generation granularity and update mechanism, we categorize existing methods into: (i) token-wise, (ii) block-wise, and (iii) refinement-based.
3.1.0.1 Token-wise
This represents the dominant generation paradigm for LLMs, employing token-wise autoregressive decoding [2], [3], [18]. For multimodal outputs, systems typically extend this paradigm by aligning non-text modalities to the textual space for autoregressive streaming [19], [20].
3.1.0.2 Block-wise
These methods expand the generation unit from single tokens to multi-token blocks, reducing serial depth while retaining the controllability of autoregressive modeling. We summarized them into two lines. (1) Semi-autoregressive relaxes intra-block dependencies to predict multiple tokens in parallel. [21]–[24]. For example, MTP [25] predicts multiple tokens simultaneously for each autoregressive block step. (2) Block-diffusion combines diffusion-style refinement with block-wise generation, iteratively denoising a block at a time and streaming blocks autoregressively [26]–[29].
3.1.0.3 Refinement-based
Unlike token-by-token sequential accumulation, this paradigm performs progressive refinement from coarse to fine, iteratively improving the semantic completeness of the entire sequence rather than merely extending its length. (1) Multi-scale approach decomposes generation into discrete scales [30]–[32]. Models like VAR [30] predict the next-scale autoregressively, enabling a blur-to-clear streaming effect. (2) Global-diffusion refinement formulates generation as multi-step denoising over the entire sequence, starting from noise or a coarse initialization and progressively refining to a complete output. This mechanism has been successfully adapted to both text [33]–[36] and multimodal generation [37], [38].
3.2 Efficient Streaming Generation↩︎
Given the extensive scope of LLM optimization, we narrow our focus strictly to the streaming process itself, analyzing decoding and memory efficiency.3 As token-wise decoding remains dominant, we focus on its optimization for efficient streaming.
3.2.0.1 Decoding Path Acceleration
To mitigate autoregressive latency, optimizations modify the execution trajectory along two dimensions. (1) Token-path methods generate parallel candidate chains to relax strict serial dependency, including multi-path and speculative decoding [39], [40]. For instance, speculative decoding [41]–[43] leverages a lightweight draft model to propose multiple candidate tokens in parallel, which are then verified and selectively accepted by a target model, reducing streaming latency. (2) Layer-depth methods adaptively shorten the network depth based on token difficulty [44], [45] . For instance, by employing layer skipping [46], models terminate the execution path prematurely.
3.2.0.2 Memory Efficiency
Since the KV cache grows linearly, optimizations aim to decouple memory cost from generated length. Dynamic KV compression methods limit the scope of attention targets during streaming decoding [47]–[50]. Representative implementations range from sink-aware windowing [17], which maintains a fixed budget for stability, to dynamic decision strategies [49] for KV cache management based on token importance.
Figure 4: Illustration of structural conflicts when adapting batch-oriented LLMs (left) to concurrent streaming (right), where (0,0) – (2.5em, 0); indicates the token generation direction, (0,0) – (2.5em, 0); denotes attention dependencies, (0,0) rectangle (2.0ex,2.0ex); blocks represent the input, and (0,0) rectangle (2.0ex,2.0ex); blocks represent the output. (1) Attention contention: Ambiguous causal dependency between the newly inserted streaming input and historical outputs. (2) Position-ID conflict: The new streaming input and generated output compete for the identical position ID..
4 Concurrent-Streaming LLMs: The Streaming of Real-Time Interaction↩︎
Concurrent-streaming represent a crucial step toward real-time interactive intelligence, requiring LLMs to simultaneously process streaming inputs and generate outputs. However, this dynamic paradigm diverges from standard static pre-training. First, regarding architecture adaptation, concurrent streaming introduce structural conflicts, as illustrated in 4. Second, synchronization control governs system interactivity by dynamically deciding when to alternate between reading and writing, balancing responsiveness and coherence, as illustrated in 5. Accordingly, we categorize existing research into architecture adaptation and interaction policy.
4.1 Architecture Adaptation↩︎
Architecture adaptation mitigates structural conflicts inherent in concurrent processing, including attention contention and positional conflicts (4). Attention contention arises when continuously arriving inputs interleave with generation, making attention dependency ordering ambiguous, while positional conflicts occur when asynchronously injected inputs overlap with output positions. Existing work redesigns input–output interaction mechanisms, which we categorize into four representative streaming paradigms.
4.1.0.1 Re-encoded streaming
The model re-encodes all historical caches whenever new input arrives [4], [51], [52]. By recomputing representations over the entire context, this approach eliminates attention contention and positional misalignment, preserving batch-equivalent attention dependencies. However, the resulting computational overhead limits its applicability to long-context and real-time settings [53], [54].
| Re-encoded streaming | Concatenated streaming | Interleaved streaming | Grouped streaming | |
|---|---|---|---|---|
| Re-encode all past caches when new input arrives to match pretraining. | Concatenate the input and output tokens into a composite token per step. | Interleave input and output tokens on the timeline. | Restrict attention within input and output groups to match pretraining. | |
| Reassign positions via full re-encoding. | Assign monotonic positions over concatenation. | Assign positions by interleaved time order. | Maintain separate positional spaces per group. |
2pt
4.1.0.2 Concatenated streaming
Concatenated streaming concatenates the newly arrived input tokens with the previously generated outputs and feeds them jointly into the model at each step [55]–[58]. This design resolves both conflicts by unifying attention and positional ordering, but incurs growing memory and latency and requires architectural changes and retraining [58].
4.1.0.3 Interleaved streaming
This paradigm interleaves input and output tokens within a shared sequence, assigning attention and positional encodings according to their temporal order [14], [15], [59]–[62]. It preserves the temporal flow of streaming interaction, enabling input and output to coexist with consistent ordering [61]. While balancing computational efficiency and real-time continuity, it requires synchronization mechanisms to prevent dependency leakage.
4.1.0.4 Grouped streaming
Group streaming partitions input and output tokens into separate groups, each with independent attention relations and position IDs [13], [63]–[65]. This design eliminates attention contention while maintaining isolated positional spaces, and empirical results show that grouped positional encoding preserves streaming performance and can improve parallelism and efficiency.
Figure 5: Illustration of interaction decision in concurrent streaming LLMs, where the model learns to dynamically schedule reading inputs and emitting outputs..
4.2 Interaction policy↩︎
Interaction policy governs read–write synchronization in concurrent LLMs, balancing latency and output quality. Existing strategies fall into three paradigms based on their optimization approach: rule-based, SFT-based, and RL-based policies.
4.2.0.1 Rule-based Interaction
Rule-based approaches rely on predetermined schedules or statistical thresholds, offering interpretability and control without requiring model parameter updates. (1) Pre-defined strategy enforce a rigid, content-agnostic read-write rhythm [13], [61], [66]. The most representative approach is the Wait-\(k\) policy [66]. In this strategy, the model always waits for \(k\) tokens or segments of input lag before generating the corresponding output. While efficient and easy to implement, pre-defined policies lack adaptability to varying input complexity and rate fluctuations. (2) Adaptive thresholding methods utilize real-time inference statistics as decision signals to improve flexibility [4], [67]. These policies trigger read/write actions based on metric thresholds (e.g., attention weights) rather than a fixed schedule. For instance, SimulS2S [4] monitors model confidence and pauses generation to read more context whenever uncertainty exceeds a safety margin, effectively adapting to the difficulty of the incoming stream.
4.2.0.2 SFT-based Interaction
Moving beyond manual rules, supervised approaches leverage labeled data to explicitly train the model to predict the optimal interaction timing. (1) In-context prediction paradigm integrates decision-making directly into the autoregressive
generation process [14], [68]. Here, the LLM is
fine-tuned to emit special control tokens (e.g., <EOS> or <WAIT> ) alongside standard text. This strategy unifies policy execution with language modeling, allowing the model to leverage its reasoning capabilities for
control. (2) Auxiliary decision employ auxiliary decision modules to decouple control from generation [62], [69], [70]. This typically involves training a lightweight classifier to output a binary decision. By
isolating the interaction signal, this approach allows for focused supervision on the decision boundary without interfering with the semantic distribution of the generated text.
| Streaming-In | Bound | Inc. | Cxt. | Example methods |
|---|---|---|---|---|
| Text | Memory | - | StreamingDialogue [71] | |
| Audio | Causal, Memory | WhisperStreaming [72] | ||
| Video | Memory | - | Timechat-online [73] |
3pt
| Task type | Level | Modality | Paradigm | Interaction policy | Example methods | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3-4 (lr)5-8 (lr)9-11 | In | Out | R. | C. | I. | G. | Rule | SFT | RL | |||||
| Translation | \(\mathcal{X} \to \mathcal{Y}\) | T/S | T/S | Seed LiveInterpret 2.0 [74] | ||||||||||
| Detection | \(\mathcal{X} \to \mathcal{Y}\) | T/S/V | T | - | - | - | - | - | FineHarm [75] | |||||
| ASR | \(\mathcal{X} \to \mathcal{Y}\) | S | T | - | - | - | ReaLLM [76], Llama-omni [77] | |||||||
| TTS | \(\mathcal{X} \to \mathcal{Y}\) | T | S | - | - | - | Cosyvoice [15], DSM [78] | |||||||
| QA | \(\mathcal{X} \to \mathcal{Y}\) | T/S/V | T/S | - | - | - | Qwen3-omni [55], VideoLLM-online [14] | |||||||
| Description | \(\mathcal{X} \to \mathcal{Y}\) | V | T | - | - | - | LiveCC [61], StreamMind [79] | |||||||
| VLA | \(\mathcal{X} \to \mathcal{Y}\) | V | T | - | - | - | - | - | StreamVLN [80], ActiveVLN [9] | |||||
| Reasoning | \(\mathcal{X} \to \mathcal{Z}\) | T/S/V | T/S | - | - | - | - | StreamingThinker [13] | ||||||
| \(\mathcal{Z} \to \mathcal{Y}\) | T/S/V | T/S | - | - | - | - | - | AsyncReasoning [81] | ||||||
| Tool usage | \(\mathcal{X} \to \mathcal{Z}\) | T/S/V | T | - | - | - | - | - | AViLA [10], StreamRAG [82] | |||||
| \(\mathcal{Z} \to \mathcal{Y}\) | T/S/V | T | - | - | - | - | - | Conveyor [83], AsyncLM [84] | ||||||
3pt
4.2.0.3 RL-based Interaction
RL-based policies model interaction control as sequential decision-making, where the LLMs selects read or write actions based on the current context [74], [85], [86]. Optimizing quality–latency rewards enables the discovery of non-trivial interaction patterns that are difficult to encode with static rules. For example, MMDuet2 [85] formulates proactive video interaction as an RL-driven control problem, enabling asynchronous perception and reaction under streaming video inputs.
5 Streaming Applications and Tasks↩︎
This section reviews the application-level tasks enabled by streaming LLMs, building upon the methodological taxonomy established in Sections 3–5. Notably, since output streaming is a universal property of LLM-based generation, we concentrate on task settings where streaming arises from incremental input, real-time interaction, or bidirectional coupling between input and output.
5.0.0.1 Sequential Streaming Tasks
Sequential streaming tasks target long, unbounded input streams that cannot be processed in a single pass due to resource limitations. For instance, streaming long video understanding [73], [87] requires incremental video encoding, followed by immediate decoding upon query arrival. As summarized in 2, different modalities emphasize distinct technical components.
5.0.0.2 Concurrent Streaming Tasks
Concurrent streaming covers multimodal tasks that require simultaneous input reception and output generation. Based on processing depth, these tasks can be divided into two levels. (1) Perception-Level (\(\mathcal{X} \rightarrow \mathcal{Y}\)): Models focus on direct cross-modal mappings with minimal latency, including streaming translation (e.g., Seed LiveInterpret 2.0 [74]), ASR/TTS (e.g., CosyVoice [15]), real-time video captioning (e.g., LiveCC [61]), and streaming QA (e.g., Qwen3-Omni [55]). (2) Cognition-Level (\(\mathcal{X} \rightarrow \mathcal{Z} \rightarrow \mathcal{Y}\)): Tasks require maintaining and updating a latent state \(\mathcal{Z}\) to support complex behaviors such as streaming reasoning (e.g., StreamingThinker [13]) and streaming tool usage (e.g., AViLA [10]). Here, the latent state decouples immediate perception from final output generation. We summarize the corresponding technical categories of these tasks in 3.
6 Future Directions↩︎
To provide a comprehensive roadmap, we categorize future research into two complementary perspectives: the technical level (i.e., how to build better streaming models) and the application level (i.e., how to apply streaming models).
6.0.0.1 Technical Level
(1) Efficient Streaming LLMs. Efficiency under strict latency and memory constraints remains a core challenge, involving incremental encoding, decoding acceleration, and long-term context management. (2) Alternative Concurrent Streaming Paradigms. Beyond interleaved and group-based strategies, more effective streaming paradigms remain to be explored. In particular, extending streaming interaction to semi-autoregressive or block-wise generation frameworks presents a promising yet underexplored direction. (3) Proactive Interaction Policies. Designing interaction policies that adaptively balance reading and generation is essential for real-time streaming performance. (4) Interpretability. The behavioral dynamics of LLMs in interactive streaming settings remain largely unexplored, calling for greater interpretability.
6.0.0.2 Application Level
(1) Expansion of Streaming Modalities. Current streaming LLMs primarily focus on text, audio, and basic video interactions. Extending streaming LLMs to additional modalities requires transcending these limitations toward complex, omni-modal continuous streams (e.g., parallel video-audio streams) to achieve real-time streaming multimodal understanding and generation in highly dynamic environments. (2) Expansion of Concurrency Levels. A promising direction is to expand current streaming LLMs from two-level perceptual concurrency (e.g., “listen-while-speaking” and “read-while-thinking”) to deeper, multi-level asynchronous processing. This includes 3-level streaming (introducing streaming “perceiving, reasoning, and generation”) and 4-level streaming (introducing concurrent “perceiving, reasoning, tool-using, and generation”) to achieve true multi-stream intelligence. (3) Expansion of Streaming Tasks. The application of streaming LLMs is expected to shift from simple, passive responses toward complex proactive interactions and long-context engagements. Advancing these capabilities involves empowering models to actively initiate interventions and maintain long-term memory, ultimately achieving brain-like streaming intelligence.
7 Conclusion↩︎
This survey presents a unified view of streaming LLMs by clarifying their definitions and organizing existing approaches into output-streaming, sequential-streaming, and concurrent-streaming paradigms based on data flow and interaction concurrency. We review representative methodologies and application scenarios, and discuss the fundamental challenges posed by real-time and interactive settings. We hope this work serves as a concise reference and a conceptual foundation for future research on streaming intelligence.
Limitations↩︎
This survey focuses on clarifying the conceptual landscape of Streaming Large Language Models through unified definitions, paradigms, and representative methods. As a result, it does not aim to provide an exhaustive comparison of all existing implementations or a comprehensive empirical evaluation across tasks and systems. Moreover, our discussion primarily centers on high-level design principles and paradigms, leaving detailed system-level optimizations and deployment-specific considerations for future studies.
8 Survey Scope and Positioning↩︎
8.1 Motivation and Necessity of This Survey↩︎
The motivation for this survey stems from three key observations regarding the current landscape of Large Language Models (LLMs): the paradigm shift to streaming scenarios, ambiguity in "streaming" terminology, and absence of comprehensive reviews in streaming LLMs domain.
8.1.0.1 The Paradigm Shift to Streaming Scenarios
While LLMs have demonstrated remarkable capabilities across various static inputs, real-world deployment increasingly demands streaming interaction. Applications such as digital human assistants, real-time simultaneous interpretation, and embodied robotics require models to process continuous input streams and generate low-latency responses. The transition from "static batch processing" to "dynamic streaming interaction" presents unique challenges in memory management, temporal coherency, and inference efficiency that traditional LLM research overlooks.
8.1.0.2 Ambiguity in "Streaming" Terminology
There is currently a significant semantic ambiguity in the usage of the term "streaming" within the community. It is often conflated across three distinct dimensions: streaming generation (token-by-token output), streaming processing (handling dynamic input context), and streaming interaction (dynamic generate with partial and dynamic input). This survey aims to disambiguate these concepts and provide a rigorous taxonomy.
8.1.0.3 Absence of Comprehensive Reviews
Despite the surge in related research, there is a notable lack of a systematic survey dedicated to Streaming LLMs.
8.2 Focus and Scope Delimitation↩︎
To ensure depth and coherence, we delineate the scope of this survey as follows: We primarily focus on decoder-only LLMs, and we structure the survey by tracing the evolution of streaming capabilities: from static input / streaming output (standard generation), to streaming Input / streaming output (infinite context processing), and finally to dynamic interaction (duplex/omni-streaming). We conducted a systematic literature review of top-tier venues in AI, NLP, CV, and speech, with a cutoff date of December 2025.
8.3 Comparison with Existing Surveys↩︎
While our survey establishes a unified taxonomy for Streaming LLMs centered on dynamic data flow and real-time interaction, it is crucial to delineate its scope from other prominent research directions in the LLM landscape. Below, we contrast our focus with three major categories of existing surveys: Efficient LLMs, Multimodal LLMs, and Long-Context LLMs.
8.3.0.1 Efficient LLMs.
The technologies surveyed under the field of Efficient LLMs, including model compression and KV cache management, are foundational technology for efficient, accurate and intelligent Streaming LLMs [88]–[90]. However, existing typical surveys in this category, such as the survey on KV cache management for acceleration [88], the comprehensive survey on efficient LLMs [89], and the review on compression techniques [90], predominantly analyze these methods from an offline and static perspective. Central questions of these surveys is mostly on how to reduce the computational or memory footprint of a model that is operating on a complete, existing context to acquire higher throughput or enable development on hardware with constrained resources. In comparison, this survey re-contextualizes these optimizations within a streaming paradigm. Techniques such as dynamic KV cache management and lightweight model adaptation under the overarching imperative of online, real-time interaction are unified. The key challenge shifts from static resource reduction to dynamic runtime budgeting under the strict latency constraints of streaming, where inputs are incrementally available, as is concurrent streaming defined, and outputs must also be generated incremenntally. Thus, while efficient LLM research only casts light on how can we run the model more efficiently, this research also asks how can it read, listen, see and respond efficiently as the world unfolds.
8.3.0.2 Multimodal LLMs.
The field of MLLMs [91] focuses on augmenting language models with the ability to process and generate content across diverse modalities like vision, audio, and video. Key challenges include cross-modal alignment, fusion strategies, and the design of modality-specific encoders and decoders. Although some MM-LLM applications (e.g., real-time video analysis or speech-to-speech translation) are inherently streaming, the primary goal of MM-LLM research is to achieve strong performance on multimodal understanding and generation benchmarks. Our survey, however, abstracts away from the specifics of any single modality. We treat the input and output as generic token streams and instead concentrate on the temporal dynamics of the interaction. A streaming LLM architecture, as defined in our work, can serve as the backbone for a multimodal system, but the core innovations we survey—such as concurrent perception-generation loops and infinite context processing—are orthogonal to the problem of modality grounding. Our focus is on how information flows over time, not what the information represents.
8.3.0.3 Long-Context LLMs.
Surveys in this category, such as [92] and [93], primarily focus on expanding the model’s static capacity to process extremely long, finite input sequences (e.g., long documents or multi-turn histories). Their core goal is to extend the usable context window and make inference over long sequences efficient, covering key technologies like positional encoding extrapolation, efficient attention architectures (e.g., sparse attention), and sophisticated KV-cache management. While these advances in long-context modeling provide a crucial foundational capability for processing extensive information, their perspective is largely centered on a "read-then-write" inference paradigm for offline, bounded inputs. In stark contrast, our survey on Streaming LLMs investigates the dynamic interaction paradigm required for unbounded, real-time token streams. We focus on the unique challenges of concurrent reading and writing, incremental processing of growing states, and online context/KV budgeting under strict latency constraints. Therefore, while long-context techniques are often essential enabling components, our work shifts the focus from merely enlarging a fixed context window to orchestrating continuous, low-latency reasoning and generation within an ever-flowing data stream.
| Typical Surveys | Primary Focus | Typical Technologies Covered | Differentiation in This Survey |
|---|---|---|---|
| [88] | |||
| [89] | |||
| [90] | |||
| [94] | Compression/adaptation and memory bottlenecks. | ) Compression: quantization, pruning, distillation, low-rank; and 2) KV-cache management: selection / eviction, cache compression, offloading, sliding-window / hierarchical cache. | Prior surveys treat compression and KV-cache optimization as separate threads; we unify them under streaming interaction, highlighting online constraints and dynamic runtime budgeting. |
| 1-4 | |||
| [91] | |||
| [95] | Architectures, training recipes, and benchmarks for MLLMs. | ) Encoder + Projector + LLM, alignment module, tokenizer; and 2) multimodal pretraining & instruction tuning. | Prior MLLM surveys assume fixed inputs and emphasize alignment and benchmarked capabilities. We focus on streaming interaction with token stream abstraction, concurrent IO, incremental perception, and online memory and budget control. |
| 1-4 | |||
| [93] | |||
| [92] | Long-context modeling: extending usable context windows and making long-sequence inference efficient. | ) Position extrapolation / interpolation; 2) efficient long-sequence attention and architectures; 3) KV-cache management (compression, eviction, and offloading); and 4) workflow-level augmentation (prompt compression, retrieval/external memory). | Prior surveys focus on enlarging a fixed context window for offline inputs or read then write inference. We study streaming token streams with concurrent read and write, incremental inputs, growing states, and online context and KV budgeting for unbounded streams. |
9 Supplementary Literature↩︎
Due to space limitations, we defer a broader collection of related work to this appendix. Following the taxonomy in Figure 3, we organize additional literature into three paradigms: (1) Output-streaming LLMs, (2) Sequential-streaming LLMs, and (3) Concurrent-streaming LLMs. This appendix complements the main text by summarizing representative yet less-discussed threads and implementations, rather than aiming for an exhaustive bibliography.
Table 5 presents additional methods for output-streaming LLMs, organized by streaming generation mechanisms and efficiency techniques.
| Token | Block | Refinement | ||
| - | - | T | GPT [1], Gemini [2], Qwen3 [96], DeepSeek-V3 [3], InternVL [97], ChatGLM [98], Gemma [18] | |
| - | - | S | AudioLM [99], SpeechGPT [19], AudioPaLM [100], FireRedTTS [101], Moshi [102], Llama-omni2 [103], Qwen3-Omni [55], StyLLE [104], Llmvox [58], SpeakStream [105] | |
| - | - | V | DALLE [106], VideoPoet [16], Chameleon [107], Emu3 [108], Anole [109], Lumina-mGPT2.0 [37], Infinity [110] | |
| - | - | T | SAT [21], SoT [23], CtrlDiff [111], PredSent [22], Falcon [24], SSD-LM [26], WeDLM [27], Next-Block [28], Block Diffusion [29] | |
| - | - | S | PALLE [112], SyncSpeech [113], DCAR [114], StreamFlow [115], TtT [116], DiTAR [117] | |
| - | - | V | show-o [118], XTRA [119], NTP [120], CausVid [121], BlockVid [122], NBP [120] | |
| - | - | T | Mask-Predict [123], LevT [124], Insertion-Deletion [125], Diffusion-LM [36], DiffuSeq [126], D3PM [127] | |
| - | - | S | SoundStorm [128], Voicebox [129], Specmaskgit [130], IMPACT [131], Maskgct [132], DDM-TASTE [133] | |
| - | - | V | MaskGIT [134], Muse [135], DiT [136], VAR [30], DetailFlow [137], DC-AR [138] | |
| Decode | Memory | |||
| - | T | Speculative Sampling [41], Medusa [42], EAGLE2 [43], BiLd [139], CTC-based Drafting [140], FLY [141], SkipDecode [45], SkipGPT [46], EESD [142], HiDrop [143], Visipruner [144] | ||
| - | S | LiveSpeech [145], MTP-SpecDec [146], SSD [147], VocalNet [148], VADUSA [149], PCG [150] | ||
| - | V | SJD [151], CSpD [152], GSD [153], VVS [154], FreqExit [155], SkipVAR [156], PAR [157], ADT-Tree [158], Lantern [159] | ||
| - | T | StreamingLLM [17], H2O [48], Scissorhands [47], Snapkv [160], Dynamickv [161], Chunkkv [162] | ||
| - | S | wu2024ts3 [163], LST [164], SpeechTokenPrediction [165] | ||
| - | V | HACK [166], ScaleKV [167], AMS-KV [168], LineAR [169] |
2.5pt
Table 6 summarizes methods for sequential-streaming LLMs, focusing on incremental encoding and streaming context management.
| Fragmented Encoding | Atomic Encoding | ||
| - | T | SimulMT [170], Moshi [102], Codec [171], dmel [172], Lightweight Audio Segmentation [173], Semantic VAD [174] | |
| - | S | Whisper-Streaming [71], SimulST [175], CTC [176], Speechtokenizer [177], Moshi [102], Codec [171], dmel [172], Lightweight Audio Segmentation [173], Semantic VAD [174] | |
| - | V | S-ViT [178] | |
| - | T | SaT [173], SegFree [179], WtP [180], subword regularization [181], SentencePiece [182], | |
| - | V | ViT [183], CLIP [184] | |
| Mem. | KV | Attn. | |
| - | - | StreamingTOM [185], MemoryBank [186], LongMem [187], VideoStreaming [188], Timechat-online [73], Prunevid [189], DyCoke [190], ProVideLLM [191], VideoLLaMB [192], STREAMMIND [79], VideoStreaming [188], StreamingAssistant [5], Focus [193], StreamForest [194], Flash-vstream [87] | |
| - | - | H2o [48], PyramidKV [195], SnapKV [160], StreamKV [196], STC [197], Streammem [198], AViLA [10], StreamingVLM [60], PyramidInfer [199], DynamicKV [161], PrefixKV [200], CAKE [201], SimLayerKV [202], AdaKV [203], CriticalKV [204], LeanKV [205], RazorAttention [206], HeadKV [207], DuoAttention [208] | |
| - | - | Attention Sink [17], Sirllm [209], GLA [210], DeltaNet [211], Lightning attention-2 [212], SAMPLEATTENTION [213], Lserve [214], DCA [215] |
3pt
| (lr)5-6 R. | C. | I. | G. | In | Out | |
| - | - | - | T | T | Simul-LLM [4], SiLLM [52], TransLLaMA [68], CAST [216], RALCP [170] | |
| - | - | - | S | T | CAST [216], TransLLaMA [68] | |
| - | - | - | T | S | LLMVoX [58], Mini-Omni [217] | |
| - | - | - | S | S | Mini-Omni [217] | |
| - | - | - | V | T | ViSpeak [218] | |
| - | - | - | T | T | EAST [219], Shanks [220] | |
| - | - | - | T | S | STITCH [221] | |
| - | - | - | S | T | EASiST [222], InfiniSST [223], SASST [67], StreamingASR [224] | |
| - | - | - | S | S | SALMONN-omni [225] | |
| - | - | - | V | T | Videollm-online [14], LiveCC [61], ProVideLLM [191], StreamBridge [226], LiveStar [227], SVBench [228], ProASIST [12] | |
| - | - | - | T | T | StreamingGPE [64], StreamingThinker [13], DST [53] | |
| - | - | - | S | T | StreamingGPE [64] | |
| - | - | - | V | T | StreamChat [63], Speak-While-Watching [65], TaYS [229] | |
| (lr)5-6 Rule | SFT | RL | In | Out | ||
| - | - | T | T | Simul-LLM [4], [54], StreamingGPE [64], STACL [66], AsyncReasoning [81], StreamingThinker [13], Conveyor [83], AsyncLM [84] | ||
| - | - | T | S | CosyVoice 2 [15], IST-LM [230], DSM [78] | ||
| - | - | S | T | MFLA [231], InfiniSST [223], LLM as Processor [64], SASST [67], SimulS2S-LLM [51], ReaLLM [76], Llama-omni [77] | ||
| - | - | S | S | StreamRAG [82] | ||
| - | - | V | T | LiveCC [61], StreamVLN [80], ActiveVLN [9], AViLA [10] | ||
| - | - | T | T | SiLLM [52], TransLLaMa [68], EAST [219], DrFrattn [69], FineHarm [75], PsFuture [232] | ||
| - | - | T | S | SimulMEGA [233], Cosyvoice [15], DSM [78] | ||
| - | - | S | T | Divergence [70], SimulMEGA [233], ReaLLM [76], Llama-omni [77] | ||
| - | - | S | S | StreamSpeech [234], EASiST [222], SimulMEGA [233] | ||
| - | - | V | T | Videollm-online [14], ProVideLLM [191], EyesWO [235], Streamo [236], ProASIST [12], Videollm-MOD [237], DisPider [62], Stream-VLM [11], Lion-FS [238], ProVideLLM [191], StreamBridge [226] | ||
| - | - | T | T | SeqPO-SiMT [86], Interleaved Reasoning [239] | ||
| - | - | T | S | Seed LiveInterpret 2.0 [74] | ||
| - | - | S | T | Seed LiveInterpret 2.0 [74] | ||
| - | - | S | S | Seed LiveInterpret 2.0 [74] | ||
| - | - | V | T | MMDuet2 [85] |
3pt