1 Introduction↩︎

Vision-Language-Action (VLA) models [1]–[10] extend Vision-Language Models [11]–[14] to embodied decision-making, offering a promising path toward generalist robot control. However, most pretrained VLAs remain reactive, conditioning only on the current observation and instruction [15]–[17]. This single-frame setup is insufficient for memory-dependent manipulation, where actions may depend on task progress, prior interactions, counts, occluded states, or failures, requiring action-relevant visual history.

To address this challenge, recent works have explored two main paradigms for memory-augmented VLAs. The first builds memory outside the policy, where a high-level VLM retrieves past keyframes and plans sparse subtasks for a low-level VLA controller [18]–[24]. While effective for extending temporal context, such systems often turn memory into text format. However, such subtask descriptions require costly annotations for training, and subtle details are difficult to faithfully encode in language only. Another line of work builds memory with freshly initialized modules inside the policy, through recurrent states [25], compressed histories [9], [26]–[29], or retrieval-augmented memory banks [30]–[35]. Since these newly introduced modules are unseen during policy pretraining, this paradigm not only increases the overall architectural complexity, but also introduces the risk of performance degradation of the pretrained VLA. More fundamentally, they face a compression dilemma: without sufficient compression, fine-grained temporal histories are too costly to retain in full; with excessive or poorly structured compression, memory may discard details critical for action. This dilemma raises a central question: how can a VLA retain long-horizon, fine-grained histories with minimal memory cost, while remaining compatible with the pretrained policy it builds upon?

We address this question with Native Memory Compression, which compresses history directly into the pretrained VLA’s own visual token space. Instead of introducing a separate memory architecture, NativeMEM repurposes the VLA’s own vision encoder to distill past observations into compact memory tokens. We push compression to its token-efficient extreme: each historical frame is represented by a single native memory token, as shown in Fig. 1 (a). Because these tokens are produced by the same visual pathway and injected into the original token sequence used by the pretrained policy, they can be interpreted as native visual evidence rather than external memory states, allowing the VLA to access long-horizon histories through its original attention mechanisms.

To learn such memory tokens so that they are both compact and aligned with the pretrained knowledge, we introduce a two-stage training pipeline that separates learning how to summarize memory from learning how to use it for a target task. In the first stage, we freeze the pretrained VLA and train a memory tokenizer derived from its native vision encoder. Given past observations, the tokenizer compresses each frame-view pair into a single visual summary token, which is appended to the original VLA token sequence. The tokenizer is optimized through the VLA’s native action prediction loss, making the summaries action-supervised rather than reconstruction-supervised. By training on a mixture of standard manipulation data and memory-demanding tasks, this stage learns a general memory tokenizer whose outputs remain aligned with the pretrained VLA’s token space. In the second stage, we freeze the learned tokenizer and perform task-specific full VLA finetuning with compact memory tokens, converting a pretrained single-frame VLA into a memory-augmented policy using limited task-specific demonstrations.

We extensively evaluate NativeMEM on manipulation tasks demanding long-horizon and adaptive memory understanding in both the simulation and the real world. Compared with pretrained memory-free VLA policies and prior memory-augmented baselines, NativeMEM achieves the highest task success rates while efficiently attending to minute-level histories with compact native memory, as shown in Fig. 1 (b) and (c). Notably, even on unseen real-robot memory tasks, NativeMEM converts a pretrained single-frame VLA into a memory-augmented policy using only 100 task-specific demonstrations and approximately 5 hours of finetuning. Beyond this conversion efficiency, NativeMEM also exhibits strong data efficiency, achieving competitive performance with prior state-of-the-art methods while using only 20% of their training data.

2 Related Work↩︎

Vision-Language-Action Models. Vision-Language-Action (VLA) models [3], [4], [6], [36]–[44] extend Vision-Language Models (VLMs) [11]–[13], [45] to embodied decision making by grounding language instructions in visual observations and predicting robot actions. RT-1 [46], RT-2 [1], OpenVLA [2], and \(\pi_0\) [3] have advanced the field through large-scale robot data, vision-language pretraining, open-source generalist policies, and continuous diffusion-based action generation. However, most VLA models condition on the current observation and short-term history. Autoregressive VLAs [2], [47]–[49] can encode previous visual, language, or action tokens as implicit memory, but such memory remains constrained by context length and lacks an explicit mechanism for preserving task-relevant information over long-horizon interactions.

VLM-Driven Memory with VLAs. To enhance memory in VLA models, recent methods often combine a high-level VLM-based memory or planning module with a low-level VLA controller [19]–[21], [50]. Representative examples include MemER [19], which retrieves task-relevant keyframes to guide a VLA policy via textual instructions; MEM [20], which combines short-term video memory with long-term language memory for progress tracking; and Mem-0 [21], which uses a VLM planner to generate subtasks for a diffusion-based executor. While effective, these methods rely on additional language-reasoning modules, increasing parameters and computation.

External Memory Modules in VLAs. To avoid the overhead of high-level memory or planning modules, recent works introduce external memory modules into VLA architectures [51]–[54], mainly through explicit memory-bank retrieval or compressed history modeling. In the first direction, MemoryVLA [51] stores visual details and semantic tokens in a perceptual-cognitive memory bank, while MAP-VLA [52] builds a demonstration memory library with task-stage soft prompts. However, such methods often store selective information, rely on reliable memory construction and subtask segmentation, and may suffer from ambiguous boundaries or accumulated retrieval errors. Compressed history methods summarize past observations into compact representations. HAMLET [26] encodes timesteps as moment tokens, ReMem-VLA [25] uses recurrent memory queries across frames and chunks, and MEM [20] compresses frame histories with a video encoder. Although these methods reduce the cost of full-history attention, they require architectural changes or dedicated design, limiting compatibility with pretrained VLAs and adding overhead as horizons grow. In contrast, NativeMEM encodes history using the pretrained VLA’s vision encoder, and reuses the policy’s existing attention with minimal overhead.

3 Method↩︎

3.1 Problem Formulation↩︎

We consider robotic manipulation in a partially observable environment. At control step \(t\), a pretrained single-frame VLA policy maps the current multi-view observation \(\mathbf{o}_t=\{o_t^v\}_{v=1}^{V}\) (with \(V\) camera views), proprioceptive state \(s_t\), and language instruction \(\ell\) to actions, \[\mathbf{a}_t \sim \pi_{\theta}(\cdot \mid \mathbf{o}_t, s_t, \ell),\] where \(\theta\) denotes the parameters of the pretrained VLA. However, such reactive policies are fundamentally limited for long-horizon manipulation, which is often non-Markovian: the correct action depends not only on the current input but on the interaction history \(\mathcal{H}_t\) available before step \(t\). Crucially, identical current observations can demand different actions depending on what happened before. Formally, there may exist two histories \(\mathcal{H}_t \neq \mathcal{H}'_t\) such that \[\begin{align} &\mathbf{o}_t = \mathbf{o}'_t,\quad s_t = s'_t,\quad \ell = \ell', \\ &\text{but}\quad \mathbf{a}^{\star}_t (\mathcal{H}_t,\mathbf{o}_t,s_t,\ell) \neq \mathbf{a}^{\star}_t (\mathcal{H}'_t,\mathbf{o}_t,s_t,\ell). \end{align}\] This is the case whenever the robot must recall previously manipulated objects, operation counts, intermediate progress, or failed attempts. In all of these, the missing ingredient is historical information, and a policy that ignores it is under-specified by construction. We therefore convert a pretrained single-frame VLA into a memory-augmented policy that additionally conditions on this history, \[\mathbf{a}_t \sim \pi_{\theta'}(\cdot \mid \mathbf{o}_t, s_t, \ell, \mathcal{M}_t),\] where \(\mathcal{M}_t\) is a compact memory built from the history \(\mathcal{H}_t\) and \(\theta'\) are the adapted parameters. The central challenge is to design \(\mathcal{M}_t\) to preserve action-relevant history while staying efficient and compatible with the pretrained VLA, motivating our native memory compression scheme.

3.2 Native Memory Compression via Action Supervision↩︎

Consider a policy that maintains a visual history over \(T\) seconds and updates memory at \(N\) frames per second. If each historical frame from each camera view is represented by \(M\) tokens, the memory length grows as \[|\mathcal{M}_t| = V \cdot T \cdot N \cdot M,\] where \(V\) is the number of camera views. This scaling reveals a fundamental memory fidelity and efficiency trade-off: increasing the temporal horizon \(T\) or the update frequency \(N\) improves the fidelity of historical context, but rapidly expands the VLA input sequence and its associated computation. Conversely, reducing the memory length improves efficiency, but risks discarding action-critical details. To retain fine-grained histories without overwhelming the VLA context, the per-frame token cost \(M\) must therefore be aggressively compressed.

Our key idea, Native Memory Compression, is to push this compression to an extreme by setting \(M=1\): each frame-view pair is summarized into a single memory token. However, compactness alone is insufficient. Since the memory tokens are consumed by a pretrained VLA, they must also be compatible with its native token space and action-generation prior. Arbitrary latent states from external modules may not be interpretable by the VLA, while summaries optimized for reconstruction or generic visual representation learning are not necessarily useful for action prediction. We therefore learn compact memory tokens directly through VLA’s original action objective.

To instantiate this idea, we derive a memory tokenizer from the VLA’s own visual encoder, as shown in Fig. 2. Given a frame-view observation \(o_\tau^v\), we initialize the memory encoder \(E_{\mathrm{mem}}\) from the pretrained VLA visual encoder and introduce a learnable memory query token \(q_{\mathrm{mem}}\). The query token attends to the visual patch-token sequence \(\mathbf{P}_\tau^v\) and aggregates it into a single summary token: \[[\hat{q}_\tau^v, \hat{\mathbf{P}}_\tau^v] = E_{\mathrm{mem}} \left( [q_{\mathrm{mem}}, \mathbf{P}_\tau^v] \right),\] where \(\hat{q}_\tau^v\) is the output memory summary. A linear memory projection maps this summary into the VLA token dimension by \[m_\tau^v = W_{\mathrm{mem}}\hat{q}_\tau^v .\] The resulting \(m_\tau^v\) can serve as a single native visual memory token for frame \(\tau\) and view \(v\).

Given a set of memory frame indices \(\mathcal{I}_t\), we concatenate memory tokens across time and views, preceded by a learnable memory beginning-of-sequence token \(b_{\mathrm{mem}}\): \[\mathcal{M}_t = \left[ b_{\mathrm{mem}}, \{m_\tau^v \mid \tau \in \mathcal{I}_t,\; \tau \leq t,\; v=1,\ldots,V\} \right]. \label{eq:memory95sequence}\tag{1}\] The memory sequence is appended to the original VLA input sequence, \[\mathbf{x}_t = \left[ \mathbf{x}^{\mathrm{obs}}_t, \mathbf{x}^{\mathrm{prompt}}(\ell, s_t), \mathcal{M}_t \right], \label{eq:augmented95input}\tag{2}\] where \(\mathbf{x}^{\mathrm{obs}}_t\) denotes the current observation tokens and \(\mathbf{x}^{\mathrm{prompt}}(\ell, s_t)\) denotes the language and proprioceptive conditioning tokens. This introduces historical context without changing the VLA architecture: the policy attends to memory through its existing token-processing pipeline.

We learn these native memory tokens in a first-stage alignment procedure. The pretrained VLA is frozen, and only the memory branch is trainable. Given the augmented token sequence \(\mathbf{x}_t\), the frozen VLA predicts the target action using its original action head, and the memory branch is optimized by the native VLA action loss: \[\min_{\phi_{\mathrm{mem}}, q_{\mathrm{mem}}, W_{\mathrm{mem}}, b_{\mathrm{mem}}} \mathbb{E}_{(\mathcal{H}_t,\mathbf{o}_t,s_t,\ell,\mathbf{a}_t)} \left[ \mathcal{L}_{\mathrm{act}} \left( \pi_{\theta}(\cdot \mid \mathbf{o}_t, s_t, \ell, \mathcal{M}_t), \mathbf{a}_t \right) \right],\] where \(\theta\) is fixed and \(\phi_{\mathrm{mem}}\) denotes the parameters of the memory encoder. Since gradients can only update the memory branch, the learned tokens are encouraged to encode information that is both action-relevant and aligned with the frozen VLA’s pretrained token space.

For training efficiency, we do not load the full visual history. Instead, for each training step, \(\mathcal{I}_t\) consists of the first frame of the episode and a recent history window. The first frame provides coarse task initialization context, while the recent window captures state changes and progress. We also include the current frame in \(\mathcal{I}_t\), so that the tokenizer learns a unified frame-level summarization behavior for both current and past observations. Our memory tokenizer is trained on a mixture of simulation and real-world demonstrations covering both standard manipulation data and memory-demanding tasks, and could be reused for downstream memory-augmented finetuning.

3.3 Task-Specific Finetuning and Real-Time Memory Inference↩︎

Task-specific finetuning. Given a target-task demonstration dataset, we first use the learned memory tokenizer to preprocess the visual history offline. For each episode, every frame from each camera view is converted into its corresponding memory summary token \(\hat{q}_\tau^v\). Since each frame-view pair is represented by only one token, this preprocessing introduces negligible storage and I/O overhead compared with storing dense visual token sequences.

For each training step \(t\), we retrieve the cached summary tokens from selected frame indices \(\mathcal{I}_t\) with \(\tau \leq t\), and form the memory sequence following Eq. 1 . The resulting memory sequence is appended to the standard VLA input as in Eq. 2 .

We initialize the VLA backbone from the pretrained single-frame policy and load \(W_{\mathrm{mem}}\) and \(b_{\mathrm{mem}}\) from the first-stage memory branch. The memory tokenizer encoder is kept fixed during this stage, while the VLA backbone, action head, and memory projection are finetuned on limited target-task demonstrations using the native action prediction loss: \[\min_{\theta', W_{\mathrm{mem}}, b_{\mathrm{mem}}} \mathbb{E}_{(\mathbf{o}_t,s_t,\ell,\mathcal{M}_t,\mathbf{a}_t)} \left[ \mathcal{L}_{\mathrm{act}} \left( \pi_{\theta'}(\cdot \mid \mathbf{o}_t, s_t, \ell, \mathcal{M}_t), \mathbf{a}_t \right) \right].\] Here, \(\theta'\) denotes the finetuned VLA parameters. This preserves the standard VLA finetuning workflow while extending the policy with compact historical context.

Real-time memory inference. During deployment, the memory tokenizer can operate independently from the VLA policy. As new observations arrive, the tokenizer converts them into memory tokens at a specified update frequency and maintains a compact memory queue. Since the tokenizer is derived from the original visual encoder and produces only one token per frame-view pair, memory updates can be performed with low overhead and do not require modifying the VLA inference pipeline. When the VLA is queried for an action, the current observation, proprioceptive state, and language instruction are processed as usual. The current memory queue is simply concatenated after the original input tokens, following Eq 2 . Thus, memory construction and policy inference are decoupled: the tokenizer can update historical memory at a high frequency, while the VLA consumes the latest compact memory sequence whenever action prediction is required.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Datasets and Tasks. We evaluate NativeMEM on memory-dependent manipulation tasks in both simulation and the real world. For real-world evaluation, we consider three tasks, as illustrated in Fig. 3. Notably, Grocery Checkout Scanning (unseen) is not included during first-stage memory-tokenizer training. This setting evaluates whether the learned native memory representation transfers to new forms of long-horizon task progress tracking. In simulation, we use three RMBench tasks [21] and two additional button-pressing tasks. The simulated Click Buttons task follows the same memory requirement as its real-world counterpart, while Click Buttons (Hard) removes color distinctions between buttons, forcing the policy to rely on spatial memory and interaction history.

Baselines. We compare NativeMEM with representative VLA policies that instantiate different memory-modeling paradigms. For VLM-driven memory, we include MemER [19] and Mem-0 [21]. For external memory modules, we include HAMLET [26] and MEM-short [20]. For a fair comparison, we implement both HAMLET and MEM-short on the same \(\pi_{0.5}\) backbone used by NativeMEM. Since HAMLET and MEM-short are not publicly released, we faithfully reproduce them following the official papers. MEM-short is originally designed to be retrained on large-scale data. To ensure a controlled comparison, we train it using the same demonstration trajectories available to all methods. Therefore mark them as HAMLET\(^\ast\) and MEM-short\(^\ast\) in Tab. 1 and  ¿tbl:tab:results95real?.

4.2 Main Results↩︎

Simulation Experiments. As shown in Tab. 1, NativeMEM achieves the best performance across all tasks, improving the average success rate to \(84.0\%\). The evaluated tasks cover two complementary memory requirements: long-range recall of early observations, as in Put Back Block and Observe and Pickup, and continuous progress tracking, as in Click Buttons, Click Buttons (Hard), and Swap Blocks. Existing VLM-driven or short-horizon memory methods struggle with one or both settings, while NativeMEM remains consistently effective, suggesting its native memory compression provides a unified representation for both recalling visual evidence and tracking task states.

Real-World Experiments. Tab. ¿tbl:tab:results95real? shows that NativeMEM also transfers effectively to real-robot manipulation, achieving the highest overall success rate across all tasks. While RTC improves over the \(\pi_{0.5}\) policy by exploiting temporal action continuity, rather than explicitly remembering past observations. Its performance still drops in later stages that require explicit historical recall. MEM-short\(^\ast\) becomes less stable under limited real-world data, occasionally degrading the pretrained VLA’s manipulation capability. In contrast, NativeMEM preserves the pretrained policy prior while adding compact history, achieving the strongest performance across all three tasks.

Inference Efficiency. Fig. 4 (a) compares inference latency and peak GPU memory under increasing history length. Our NativeMEM supports up to 5,000 historical frames within a 32 GB memory budget, and still attends to about 200 frames under a 100 ms real-time latency constraint, enabling substantially longer histories while preserving the real-time reactiveness.

Data-Efficiency. Fig. 4 (b) shows that NativeMEM adapts effectively with limited demonstrations. With only 10 demonstrations, it reaches \(60\%\) average success, \(3.5\times\) higher than Mem-0, and consistently outperforms Mem-0 with 25 and 50 demonstrations, reflecting the benefit of reusing pretrained VLA manipulation priors.

4.3 Qualitative Analysis: What Does Native Memory Capture?↩︎

To understand what is encoded into each memory token, we visualize the tokenizer’s spatial attention over historical observations. As shown in Fig. 5, the tokenizer consistently focuses on manipulation-relevant regions rather than background pixels. For Put Back Blocks, attention concentrates on the block and its corresponding pad. Notably, this behavior also generalizes to Grocery Checkout Scanning, which is unseen during tokenizer training: the tokenizer still attends to foreground objects, especially the item about to be grasped.

Action Attention over the Memory Sequence.

To examine how NativeMEM uses history during action generation, we visualize the attention from action tokens to memory tokens across inference time. Fig. 6 shows that the policy attends to task-relevant moments rather than simply the most recent observations. In Click Buttons, high-attention memories align with the three timesteps when individual buttons were pressed. In Put Back Block, attention concentrates on the moment when the block was lifted from its original pad, which is critical for deciding where to return it.

4.4 Ablation Study↩︎

Native Memory Compression. As shown in Tab. 2 (lines 1 and 2), when the VLA is unfrozen during memory-compression learning, performance drops to \(23.2\%\). The model reduces the action loss by directly adapting the pretrained VLA itself rather than forcing the memory branch. After skipping Native Memory Compression, mean-pooled vision-encoder features are used as memory tokens. The results show that such generic visual features can capture coarse historical context, but fail to preserve details required by tasks such as Put Back Block and Observe and Pickup. These results indicate that Native Memory Compression is essential for distilling task-relevant historical cues, enabling the policy to recover critical information needed for downstream action prediction.

Temporal Memory Coverage. Sparse update at \(0.5\)Hz broadly reduces success rates, showing that the policy needs fine-grained temporal evidence to track interaction states. Short horizons degrade tasks whose critical information falls outside the retained window, such as Swap Blocks and Put Back Block. Results in Tab. 2 (lines 4\(\sim\)​6) show that NativeMEM’s performance relies on both dense temporal coverage and long-horizon retention: it updates memory frequently enough to capture fine-grained interaction changes, while its compact native tokens make minute-level history retention scalable.

5 Conclusion↩︎

We presented NativeMEM, which enables pretrained single-frame VLAs to retain long-horizon, fine-grained visual histories through Native Memory Compression. By repurposing the VLA’s own vision encoder, NativeMEM achieves one-token-per-frame compression. We further introduced a two-stage training pipeline that first learns an action-supervised memory tokenizer aligned with the pretrained VLA’s visual-action priors, and then performs task-specific finetuning with limited demonstrations. Across simulation and real-world manipulation tasks, NativeMEM substantially improves average success rates from 32.4% to 84.0% in simulation and from 34.7% to 98.7% on real robots, while matching leading memory-designed methods with only 20% of the training data.

6 Limitations↩︎

While NativeMEM enables dense minute-level visual memory for long-horizon manipulation, it is not designed to maintain persistent memories over hours or days. Supporting such long-term continuity will likely require additional system-level memory infrastructure. In addition, our memory tokenizer is learned solely through action supervision. Although we have not observed clear failures in our preliminary exploration, more complex tasks may expose a semantic gap between low-level action losses and higher-level, abstract memory requirements. Exploring more direct and scalable objectives for learning the relationship between memory and action remains an important direction.

References↩︎