1 Introduction↩︎

Vision-language-action (VLA) models [1]–[4] have become a promising paradigm for general robotic manipulation. By combining the powerful capabilities of pretrained vision-language models [5]–[7] with policy learning [8]–[10] on robotic data [11]–[14], they map visual observations and language instructions into executable action chunks. Despite this progress, most existing VLA models [1], [2], [15] implicitly rely on a Markovian assumption, predicting actions primarily from the current observation without considering temporal dependencies. This simplification creates a temporal short-horizon bias: VLA models can react to the currently visible state, but fail to reason about previous state transitions, completed operation steps, and the current phase of a multi-step task. As a result, these models especially struggle with long-horizon manipulation tasks.

Recent efforts have sought to alleviate temporal short-horizon bias by augmenting VLA models [8], [16], [17] with historical context or memory mechanisms along two main axes. (i) One line of work incorporates short-horizon episode context by concatenating historical frames [18], [19] or extending the input into a video sequence [20]–[22]. Although such designs expose recent state changes, they incur computational and memory costs that grow with the context length, while the fixed temporal horizon imposes an inherent memory ceiling, causing potentially task-relevant evidence outside the window to be discarded. (ii) Another line of work [23]–[25] retrieves past trajectories or relevant historical tokens from an external memory bank to condition downstream action policies. Although these methods demonstrate the value of historical experience for long-horizon manipulation, they still suffer from an architectural limitation: the historical memory is stored outside the model’s native token space and consumed as auxiliary policy-side context after the VLA model reasoning. This rigid separation prevents memory from being fluidly interleaved with the internal reasoning where the VLA model jointly perceives the scene, interprets the instruction, and resolves action queries before action formation.

This limitation raises a more fundamental question for memory-dependent VLA models: whether historical experience can be represented as context-native robotic memory, stored, retrieved, and consumed in the same continuous space where the VLA model already perceives, reasons, and acts? Latent embedding space [26]–[32] offers a natural answer to this question. Modern VLA models already integrate visual observations and language instructions in a continuous token embedding space [1], [2], [15], [33]; therefore, robotic historical memory can be organized as machine-native latent memory tokens that are compatible with the internal reasoning process. Under this formulation, robotic memory becomes part of the model’s operating context rather than an auxiliary scaffold attached after multimodal reasoning. Long-horizon robotic manipulation also calls for two complementary forms of historical memory: short-term memory is visually dominant, preserving visually grounded evidence from the current episode, such as object locations and subtle state changes; long-term memory is semantically dominant, preserving task progress, contextual semantics, and action continuity across longer horizons. Notably, their distinction lies in provenance and function rather than in representation form: both are ultimately reconstructed as latent memory tokens that can be consumed by the VLA model in the latent space. This leads to our pivotal research question:

To answer this question, we propose LaMem-VLA, a novel native latent memory framework for robotic manipulation, which explicitly organizes robotic history into two complementary memory vaults, and weaves dual-scale memory into the model reasoning for memory-augmented action generation. At its core, LaMem-VLA closes the loop between latent memory weaving and action reasoning through four coordinated modules: First, a latent memory curator factorizes past robotic experience into two complementary vaults: a short-term memory vault for recent visual evidence and a long-term memory vault for semantic and action-continuity evidence. Second, during the action reasoning process, a latent memory seeker builds a context-aware query from the current multimodal cognition state (i.e., the visual and instruction tokens), and uses it to retrieve task-relevant historical evidence from dual memory vaults, grounding the present decision in past perceptual evidence and long-horizon semantic-action continuity. Third, a latent memory condenser compresses these potentially redundant retrieved evidence into fixed-length short-term and long-term latent memory tokens that are compatible with the VLA embedding space. Finally, a latent memory weaver stitches these condensed memory tokens directly into the action reasoning sequence before action query tokens are resolved, allowing historical memory to participate in the same latent reasoning process as the current image, instruction, and action queries. The resulting memory-grounded action queries condition a diffusion-based action expert [8], [34] to generate temporally aware robotic action sequences.

We conduct comprehensive evaluations across two simulators (i.e., LIBERO [35] and SimplerEnv-Bridge [36]). On LIBERO, LaMem-VLA reaches an average success rate of 97.6%, outperforming MemoryVLA [23] by 1.1 points and our baseline CogACT [15] by 4.4 points, while improving over \(\pi_0\) [1] by 3.5 points on the first four suites. On SimplerEnv-Bridge, LaMem-VLA achieves 73.9% average success, surpassing our baseline CogACT [15] by 16.6 points and \(\pi_0\) [1] by 4.7 points. These results indicate that weaving dual-scale latent robotic memory into VLA reasoning boosts the robustness of VLA models beyond policy-side memory conditioning, especially when action generation depends on task progress and historical cues. One limitation of the current version is that the empirical validation is conducted in simulated environments. We are currently extending LaMem-VLA to real-world robot platforms, and the corresponding real-world experiments will be included in the next version.

In summary, our main contributions are as follows:

• We introduce a new paradigm for robotic memory in VLA models: historical experience is treated as context-native latent memory, which is stored, retrieved, and consumed in the model embedding space, supporting scene perception, instruction understanding, and action-intent formation within the same latent reasoning process.

• We propose LaMem-VLA, a dual latent memory framework for robotic manipulation, which explicitly organizes robotic history into two complementary memory vaults, and weaves dual-scale memory into the model reasoning for memory-augmented action generation.

• We design a latent memory condensing mechanism, which transforms retrieved historical evidence into fixed-length latent short-term and long-term memory tokens that are compatible with the VLA model embedding space.

2 Related Work↩︎

Vision-Language-Action (VLA) Models. Driven by advances in pretrained vision-language models (VLMs), VLA models [1], [37]–[39] have demonstrated promising performance in robotic control via mapping visual observations and language instructions to robot actions. According to their action policy design, VLA models can be broadly classified into two paradigms: single-stream architecture and hierarchical architectures. (i) Single-stream models [40]–[43] directly discretize continuous actions [2], [33] within a unified vision-language backbone, and autoregressively predict action tokens in a language-like manner. To meet the high control-frequency requirements in robotic manipulation, many works focus on inference efficiency optimization, e.g., parallel or speculative decoding [44], [45], dynamic LLM layer activation [46], and parameter quantization [47]. (ii) Hierarchical models [3], [48], [49] typically employ VLMs for high-level reasoning and planning, and adopt a separate generative policy, such as diffusion-based [1], [15] and flow-matching-based [9], [50] policies, to synthesize smooth and high-quality action trajectories. This separation reduces response latency and facilitates smooth real-world deployment, promoting robust high-level planning alongside high-frequency control for robotic manipulation. Despite these advances, these methods still predict actions primarily from the current observation and instruction without leveraging extended historical context, limiting their robustness in long-horizon manipulation tasks. In contrast, our work aims to equip VLA models with explicit progress awareness via memory mechanisms, addressing memory-dependent tasks that current approaches fail to handle.

Memory Mechanisms for Robotic Control. Mainstream VLA policies are formulated under a Markovian assumption, predicting actions primarily from the current observation and therefore lacking explicit awareness of task progress. Existing attempts to incorporate historical context can be broadly categorized into three groups. (i) Temporal-context expansion: This line of work directly exposes the policy to temporally extended observations by interleaving historical frames with language tokens [18], [19] or aggregating multi-frame features [20]–[22] within a predefined context window. Although straightforward, these methods incur computational and memory costs that grow with the context length, while the fixed temporal horizon imposes an inherent memory ceiling, causing potentially task-relevant evidence outside the window to be discarded. (ii) Sparse history abstraction: Another paradigm compresses past interactions into lightweight proxy representations, such as recurrent latent states [51], [52], motion-centric cues [53], action summaries [54], or a sparse set of representative observations [55]. By avoiding direct processing of the full observation history, this paradigm improves temporal awareness with relatively low computational overhead. However, aggressive abstraction of historical experience can remove fine-grained perceptual details and high-level semantic context that are critical for long-horizon manipulation tasks. (iii) External memory conditioning: A third direction [23]–[25] stores historical observations and action tokens in an auxiliary memory bank, and retrieves a compact set of task-relevant evidence from this bank to condition current action prediction. Since memory access is driven primarily by the current query, retrieval may become unreliable when the present observation provides weak cues or contains visually similar distractors. More fundamentally, the retrieved memory is typically consumed as auxiliary policy-side context rather than integrated into the VLA model’s native token-level reasoning process, limiting its ability to directly shape action formation. In contrast, LaMem-VLA formulates historical experience as context-native latent memory. It maintains complementary short-term visual and long-term semantic memory vaults. Decision-relevant evidence is then retrieved and distilled into compact latent memory tokens. These latent memory tokens are directly interwoven with current visual, language, and action tokens before action formation, enabling fine-grained perceptual evidence and long-range task progress to jointly shape the VLA model’s native reasoning process without incurring growing context.

3 Method↩︎

3.1 Problem Formulation↩︎

We formulate the robotic manipulation task in VLA models as a language-conditioned Markovian decision-making problem. At each timestep \(t\), the VLA policy [2], [15], [45], \(\boldsymbol{\Pi}_\theta\) takes a natural-language instruction \(\boldsymbol{I}\) and the current visual observation \(\boldsymbol{o}_t\) as input, and predicts a chunk of future actions for executing the specified task: \[\label{eq:problem95formulation} \boldsymbol{a}_{t:t+H-1} = \boldsymbol{\Pi}_\theta(\boldsymbol{o}_t, \boldsymbol{I}),\tag{1}\] where \(H\) denotes the action horizon, and each action \(a_t \in {\mathbb{R}}^{7}\) is a 7-DoF end-effector control vector, consisting of 3-DoF relative translation, 3-DoF relative rotation, and a 1-DoF gripper state.

3.2 LaMem-VLA Model Architecture↩︎

Motivation. The Markovian formulation above in VLA models makes action prediction primarily depend on the instantaneous observation and instruction. Although effective for short-horizon behaviors, this paradigm can induce a temporal short-horizon bias in long-horizon and temporally dependent manipulation tasks, where historical state transitions, completed subtasks, and task-progress cues are essential for reliable control. Existing works attempt to augment VLA or visuomotor policies with memory through incorporating explicit episode context into the input [20], [56], [57] or externalized memory bank retrieval [23], [24]. However, they store the historical memory outside the model’s native token space and consume it as auxiliary policy-side context for action decoding, preventing memory from being fluidly interleaved with the internal reasoning process that jointly perceives, reasons, and forms action intent. Thus, our LaMem-VLA reformulates robotic memory as context-native latent memory to bridge this gap.

Overview. LaMem-VLA is an end-to-end native latent memory framework for robotic manipulation that directly weaves dual-scale historical experience into VLA reasoning to refine action generation, as shown in Fig. 2. At each timestep \(t\), given the current visual observation \(\boldsymbol{o}_t\) and instruction \(\boldsymbol{I}\), the vision-language backbone embeds them into the visual tokens \(\boldsymbol{X}_t\) and instruction tokens \(\boldsymbol{I}\). Learnable action queries are appended to the token sequence to obtain manipulation-relevant latent action representations. LaMem-VLA closes the loop between latent memory reconstruction and action reasoning through four coordinated modules. First, a latent memory curator dynamically establishes and updates two complementary memory vaults: the short-term memory vault \(\mathcal{M}^\text{short}\) stores visual tokens that preserve recent perceptual evidence, while the long-term memory vault \(\mathcal{M}^\text{long}\) stores action tokens that preserve semantic and action-continuity evidence across longer horizons. Second, a latent memory seeker constructs a context-aware query \(\boldsymbol{Q}_t^\text{con}\) from the current multimodal cognition representation and uses it to retrieve decision-relevant history from the two vaults. Third, a latent memory condenser distills the retrieved raw memory content and learnable memory tokens into compact latent short-term memory tokens \(\boldsymbol{M}^\text{short}\) and latent long-term memory tokens \(\boldsymbol{M}^\text{long}\), respectively. Finally, a latent memory weaver prepends these two sources of latent memory tokens to the current image and language tokens, forming a memory-augmented VLA input sequence \(\boldsymbol{S}_t\): \[\label{eq:vlm95sequence95overview} \boldsymbol{S}_t = [\boldsymbol{M}^\text{short}; \boldsymbol{M}^\text{long}; \boldsymbol{X}_t; \boldsymbol{I}; \boldsymbol{Q}^\text{action}],\tag{2}\] where \(\boldsymbol{Q}^\text{action}\) denotes learnable action tokens. The resulting action tokens are then fed into a diffusion-based action expert [34] to generate action sequences.

3.3 Latent Memory Curator↩︎

The latent memory curator maintains the historical evidence that will later be retrieved, condensed, and woven into the VLA reasoning sequence. LaMem-VLA adopts a 7B-parameter Prismatic vision-language model [58] as the backbone, which is further pretrained on the large-scale cross-embodiment real robotic dataset Open-X Embodiment [11]. For the current RGB observation, the backbone first extracts visual tokens with its vision encoder [59], [60] and projects them into the language embedding space to obtain final visual tokens \(\boldsymbol{X}_t\!\in\!\mathbb{R}^{N_\text{i}\times C}\), where \(N_\text{i}\) is the sequence length and \(C\) is the hidden dimension. These visual tokens are concatenated with the tokenized instruction and processed by LLaMA-7B [61]. We append learnable action queries \(\boldsymbol{Q}^\text{action}\!\in\!\mathbb{R}^{N_\text{a}\times C}\) to the sequence, and use their output hidden states \(\boldsymbol{H}^\text{action}\) as compact action representations for downstream action prediction. The curator factorizes the historical evidence into two memory vaults: a short-term memory vault \(\mathcal{M}^\text{short}\) and a long-term memory vault \(\mathcal{M}^\text{long}\).

Short-term Memory Vault. The short-term memory vault \(\mathcal{M}^\text{short}\) stores visual perceptual evidence from the current episode. The resulting initialized vault is denoted as \(\{\boldsymbol{m}_\text{s}^i\}_{i=1}^L\), where \(L\) specifies its initial capacity. Each short-term memory unit is represented as a key-value pair \(\boldsymbol{m}_\text{s}^i=(\boldsymbol{k}_\text{s},\boldsymbol{v}_\text{s})\): the key \(\boldsymbol{k}_\text{s}\) provides a concise retrieval summary of visual evidence, while the value \(\boldsymbol{v}_\text{s}\) stores the latent short-term memory content. Specifically, at each timestep, a learnable compression module distills the current visual tokens \(\boldsymbol{X}_t\) into a compact set of short-term memory tokens, and their mean-pooled representation is used as the retrieval key: \[\label{eq:visual95memory95bank} \begin{align} \boldsymbol{v}_\text{s} &= \mathcal{C}_\text{s}(\boldsymbol{X}_t) \in \mathbb{R}^{N_\text{s} \times C}, & \boldsymbol{k}_\text{s} &= \mathrm{MeanPool}(\boldsymbol{v}_\text{s}) \in \mathbb{R}^{C}, \end{align}\tag{3}\] where \(\mathcal{C}_\text{s}\) is an SE-bottleneck compression module [62], and the unit \(\boldsymbol{m}_\text{s}\) is then appended to \(\mathcal{M}^\text{short}\).

Long-term Memory Vault. The long-term memory vault \(\mathcal{M}^\text{long}\) stores the action hidden states that track task progress and action continuity across long horizons. At each timestep, the curator directly writes the action hidden states \(\boldsymbol{H}^\text{action}\) into the vault as one long-term memory unit \(\boldsymbol{m}_\text{l}\). Unlike the short-term vault, \(\mathcal{M}^\text{long}\) is not a key-value bank: each long-term unit preserves the action hidden state at each timestep, allowing the vault to accumulate task-progress and action-continuity information across the trajectory.

Memory Vault Updating Strategy. After a new memory unit is written into its corresponding vault, the memory curator applies a compression strategy only when the number of stored units exceeds the capacity \(L\). For the short-term visual stream, let \(\mathcal{M}^\text{short}=\{\boldsymbol{m}_\text{s}^i=(\boldsymbol{k}_\text{s}^i,\boldsymbol{v}_\text{s}^i)\}_{i=1}^{n_\text{s}}\) after insertion. If \(n_\text{s}>L\), we compute the cosine similarity between temporally adjacent keys and select the most redundant adjacent pair: \[\label{eq:short95memory95capacity} i_\text{s}^* = \arg\max_{1 \le i < n_\text{s}} \mathrm{cos}(\boldsymbol{k}_\text{s}^i,\boldsymbol{k}_\text{s}^{i+1}).\tag{4}\] The selected pair is consolidated by averaging both its key and value tokens: \[\label{eq:short95memory95merge} \begin{align} \tilde{\boldsymbol{k}}_\text{s} &= \frac{1}{2}(\boldsymbol{k}_\text{s}^{i_\text{s}^*}+\boldsymbol{k}_\text{s}^{i_\text{s}^*+1}), \qquad \tilde{\boldsymbol{v}}_\text{s} = \frac{1}{2}(\boldsymbol{v}_\text{s}^{i_\text{s}^*}+\boldsymbol{v}_\text{s}^{i_\text{s}^*+1}), \qquad \tilde{\boldsymbol{m}}_\text{s} &= (\tilde{\boldsymbol{k}}_\text{s},\tilde{\boldsymbol{v}}_\text{s}). \end{align}\tag{5}\] The two adjacent units are replaced by \(\tilde{\boldsymbol{m}}_\text{s}\), reducing redundancy while preserving their shared visual evidence. LaMem-VLA applies the same memory updating strategy to the long-term memory vault. See more details in Appendix.

3.4 Learning to synthesize latent memory with Dual-scale Vault↩︎

LaMem-VLA does not expose memory vaults to the action policy as raw auxiliary context. Instead, it treats the two vaults as latent evidence substrates: given the current multimodal cognition representation, LaMem-VLA first retrieves relevant evidence from the two vaults and then reconstructs it into compact memory tokens that are native to the VLA embedding space. This design decouples memory storage from memory consumption, allowing historical experience to remain flexible in the vaults while entering action reasoning through a bounded latent interface.

Latent Memory Seeker. The latent memory seeker retrieves evidence from the memory vaults according to the current multimodal cognition context rather than the visual observation or language instruction alone. Given the current visual tokens \(\boldsymbol{X}_t\) and instruction tokens \(\boldsymbol{I}\), the VLA backbone produces context-aware query \(\boldsymbol{Q}_t^\text{con}\) that encode the current visual-linguistic cognition. The seeker then appends learnable query slots \(\boldsymbol{Q}^\text{init}\in\mathbb{R}^{K_\text{q}\times C}\) to \(\boldsymbol{Q}_t^\text{con}\) and updates only the query slots with a lightweight query builder: \[\label{eq:latent95memory95query} \boldsymbol{Q}_t = \mathcal{B}([\boldsymbol{Q}_t^\text{con}; \boldsymbol{Q}^\text{init}])[-K_\text{q}:] \in \mathbb{R}^{K_\text{q}\times C},\tag{6}\] where \(\mathcal{B}\) is a transformer-based query builder (more details in Appendix) and \(\boldsymbol{Q}_t\) serves as the shared memory hook for both vaults. We apply masked attention inside \(\mathcal{B}\) such that the appended query slots read from \(\boldsymbol{Q}_t^\text{con}\), while the original multimodal hidden states are not perturbed by the query slots. The mean-pooled query \(\boldsymbol{q}_t\!=\!\mathrm{MeanPool}(\boldsymbol{Q}_t)\in\mathbb{R}^{C}\) is used as the global retrieval vector.

The memory seeker then uses \(\boldsymbol{q}_t\) to retrieve context-relevant units from the short-term vault \(\mathcal{M}^\text{short}=\{(\boldsymbol{k}_\text{s}^i,\boldsymbol{v}_\text{s}^i)\}_{i=1}^{|\mathcal{M}^\text{short}|}\) by cosine similarity: \[\label{eq:short95memory95retrieval} \begin{align} \mathcal{I}_\text{s} &= \mathrm{Top\!-\!}K \left(\{\mathrm{cos}(\boldsymbol{q}_t,\boldsymbol{k}_\text{s}^i)\}_{i=1}^{|\mathcal{M}^\text{short}|}\right), & \boldsymbol{Z}^\text{short} &= \mathrm{Concat}_{i\in\mathcal{I}_\text{s}}(\boldsymbol{v}_\text{s}^i) \in \mathbb{R}^{KN_\text{s}\times C}, \end{align}\tag{7}\] where \(K\) is the number of retrieved short-memory units, and the discrete \(\mathrm{Top\!-\!}K\) operation is not optimized by gradient descent. For long-term memory retrieval, the seeker similarly ranks the units in \(\mathcal{M}^\text{long}=\{\boldsymbol{m}_\text{l}^i\}_{i=1}^{|\mathcal{M}^\text{long}|}\) via comparing the mean-pooled long-term units with the mean-pooled query \(\boldsymbol{q}_t\). It then selects the \(\mathrm{Top\!-\!}K\) long-term memory units and stacks them into \(\boldsymbol{Z}^\text{long}\in \mathbb{R}^{KN_\text{l}\times C}\). The retrieved sets \(\boldsymbol{Z}^\text{short}\) and \(\boldsymbol{Z}^\text{long}\) are passed to the memory condenser below as short-term visual evidence and long-term semantic evidence, respectively, instead of being inserted verbatim into the VLA sequence.

Latent Memory Condenser. The retrieved short-term visual evidence \(\boldsymbol{Z}^\text{short}\in\mathbb{R}^{KN_\text{s}\times C}\) and long-term semantic evidence \(\boldsymbol{Z}^\text{long}\in\mathbb{R}^{KN_\text{l}\times C}\) often contain redundant historical evidence that is not fully aligned with the current context. Additionally, directly inserting these lengthy retrieved evidence sequences into the VLA sequence would expand the reasoning context and introduce redundant historical tokens, so the latent memory condenser reconstructs them into fixed-length latent memory tokens and maps them into the VLA reasoning embedding space. Specifically, we introduce learnable short-term visual memory slots \(\boldsymbol{T}_\text{s}\in\mathbb{R}^{L_\text{s}\times C}\) and long-term semantic memory slots \(\boldsymbol{T}_\text{l}\in\mathbb{R}^{L_\text{l} \times C}\), and update them with lightweight memory formers conditioned on the context query tokens \(\boldsymbol{Q}_t\in\mathbb{R}^{K_\text{q}\times C}\) and the retrieved evidence: \[\label{eq:memory95condensation} \begin{align} \boldsymbol{M}^\text{short} &= \mathcal{F}_\text{v}([\boldsymbol{Q}_t; \boldsymbol{Z}^\text{short}; \boldsymbol{T}_\text{s}])[-L_\text{s}:], & \boldsymbol{M}^\text{long} &= \mathcal{F}_\text{c}([\boldsymbol{Q}_t; \boldsymbol{Z}^\text{long}; \boldsymbol{T}_\text{l}])[-L_\text{l}:], \end{align}\tag{8}\] where \(\mathcal{F}_\text{v}\) and \(\mathcal{F}_\text{c}\) denote lightweight transformer-style memory formers similar to \(\mathcal{B}\) for the short-term visual and long-term semantic memory, respectively. The resulting \(\boldsymbol{M}^\text{short}\) and \(\boldsymbol{M}^\text{long}\) are query-conditioned latent short-term and long-term memory tokens in the same \(C\)-dimensional embedding space used by VLA reasoning. This fixed-length property makes the injected memory independent of the retrieval size while preserving evidence from the two vaults.

3.5 Interweaving Memory into Action Reasoning in Latent Space↩︎

Latent Memory Weaver for Guidance Generation. Rather than passing the condensed memory tokens to a policy head as external conditioning alone, the latent memory weaver injects the synthesized memory into the VLA reasoning sequence before action queries are resolved. Although the two types of latent memory differ in provenance, they share the same latent token interface. Given the condensed short-term memory tokens \(\boldsymbol{M}^\text{short}\), long-term memory tokens \(\boldsymbol{M}^\text{long}\), the weaver constructs the memory-augmented VLA input sequence \(\boldsymbol{S}_t\): \[\label{eq:memory95weaver} \begin{align} \boldsymbol{S}_t &= [\boldsymbol{M}^\text{short} + \mathbf{1}_{L_\text{s}}\boldsymbol{b}_\text{s}^\top; \boldsymbol{M}^\text{long} + \mathbf{1}_{L_\text{l}}\boldsymbol{b}_\text{l}^\top; \boldsymbol{X}_t; \boldsymbol{I}; \boldsymbol{Q}^\text{action}], & \boldsymbol{Z}^\text{action}= \mathrm{VLM}(\boldsymbol{S}_t)[-N_\text{a}:], \end{align}\tag{9}\] where \(\boldsymbol{b}_\text{s},\boldsymbol{b}_\text{l}\in\mathbb{R}^{C}\) are learnable source embeddings for the two memory streams, and \(\mathbf{1}_{L_\text{s}},\mathbf{1}_{L_\text{l}}\) are all-one column vectors that broadcast them over the short-term and long-term memory tokens. Because the two sources of memory tokens are part of the model input sequence, they participate in self-attention with the current observation, language instruction, and action queries. Thus, the resulting \(\boldsymbol{Z}^\text{action}\) inside the model reasoning process is formed as memory-grounded action tokens rather than by external policy-side fusion.

Diffusion-based Action Expert. After the vision-language backbone produces the memory-grounded action tokens \(\boldsymbol{Z}^\text{action}\), the diffusion-based action expert decodes them into a continuous action chunk. Following the diffusion policy [8], we formulate action generation as conditional denoising over an action chunk. Let \(\boldsymbol{a}_{t:t+H-1}^{0}\) denote the clean expert action chunk and \(\boldsymbol{a}_{t:t+H-1}^{n}\) denote its noisy version at diffusion step \(n\). At each denoising step, the noisy action tokens are injected with the diffusion timestep embedding and conditioned on action tokens \(\boldsymbol{Z}^\text{action}\).

The diffusion expert \(\epsilon_\theta\) is trained with mean squared error (MSE) loss to predict the injected noise under these action and memory conditions: \[\label{eq:diffusion95loss} \mathcal{L}_\text{action} = \mathbb{E}_{n,\epsilon} \left[ \left\| \epsilon - \epsilon_\theta(\boldsymbol{a}_{t:t+H-1}^{n}, n, \boldsymbol{Z}^\text{action}) \right\|_2^2 \right].\tag{10}\] During inference, DDIM sampling [63] iteratively denoises the action chunk under the same conditions, producing history-aware continuous \(7\)-DoF control actions.

4 Experiment↩︎

4.1 Implementation Details↩︎

We instantiate LaMem-VLA with a 7B-parameter Prismatic VLM backbone and a diffusion action expert with approximately 300M parameters. LaMem-VLA receives a single third-person RGB observation resized to \(224\times224\) together with the language instruction, and predicts continuous 7-DoF end-effector actions with an action chunk size of 16. We train the model on 8 NVIDIA H800 GPUs using PyTorch FSDP. Each GPU processes 32 samples, resulting in a global batch size of 256, and the learning rate is set to \(2\times10^{-5}\). Both the short-term and long-term memory vaults keep a maximum capacity of \(L=16\) units. The latent memory seeker retrieves the top \(K\!=\!8\) units from each vault, and the latent memory condenser reconstructs the retrieved evidence into \(L_\text{s}\!=\!8\) short-term memory tokens and \(L_\text{l}\!=\!4\) long-term memory tokens. The query builder \(\mathcal{B}\) and the two memory formers \(\mathcal{F}_\text{v}\) and \(\mathcal{F}_\text{c}\) are implemented as two-layer transformer blocks with masked attention (more details in Appendix). During inference, the diffusion action expert generates actions with DDIM [63] sampling using 10 denoising steps.

4.2 Simulated Evaluation on SimplerEnv-Bridge↩︎

Training and Evaluation Protocol. SimplerEnv evaluates real-to-sim generalization of robot manipulation policies trained on real-world data. We evaluate LaMem-VLA on the SimplerEnv-Bridge [36] suite under the standard Bridge protocol. The policy is trained on the Bridge v2 dataset [13] for 50k optimization steps, with validation performed every 2.5k steps. The final results are reported using the checkpoint that achieves the best validation performance. For evaluation, each task is tested over 24 trials, and we report the average success rate across trials.

Evaluation Results. As shown in Table ¿tbl:tab:simplerenv95bridge?, LaMem-VLA achieves the highest average success rate of 73.9% on SimplerEnv-Bridge, yielding a 16.6 gain over the CogACT [15] baseline and surpassing recent state-of-the-art VLAs such as \(\pi_0\) [1] and SemanticVLA [64]. Per task, LaMem-VLA obtains 83.3% on Put Spoon on Towel, 75.0% on Put Carrot on Plate, 41.7% on Stack Cube, and 95.8% on Put Eggplant in Basket. These results indicate that injecting dual latent memory tokens into the VLA reasoning sequence provides effective historical context for action prediction, improving manipulation performance and maintaining strong robustness across diverse task settings.

4.3 Simulated Evaluation on LIBERO↩︎

Training and Evaluation Protocol. We evaluate LaMem-VLA on the LIBERO [35] benchmark using a Franka robot across five suites: spatial awareness (Spatial), object manipulation (Object), goal completion (Goal), and long-horizon reasoning (Long-10 and Long-90). The first four suites comprise 10 tasks each, while Long-90 contains 90 tasks. Following the OpenVLA protocol [2], we use 50 demonstrations per task. We train separate models for Spatial, Object, and Goal for 20k optimization steps, and jointly train on Long-10 and Long-90 for 40k steps. During training, we evaluate checkpoints at 1k-step intervals and select the best-performing one according to validation success. We report success rates for each suite and the overall average over 50 rollouts per task.

Evaluation Results. As shown in Table ¿tbl:tab:libero?, LaMem-VLA achieves the best overall performance on LIBERO, reaching an average success rate of 97.6% across the five suites. It improves over the strongest reported memory-augmented method MemoryVLA [23], by 1.1 points, and surpasses strong VLA baselines such as CogACT [15] by 4.4 points. Compared with \(\pi_0\) [1], LaMem-VLA obtains a first-four-suite average of 97.7%, yielding a 3.5 point improvement. Notably, these gains are achieved without the additional proprioceptive and wrist-camera inputs used by starred methods. Across individual suites, LaMem-VLA consistently attains the highest success rates, with 98.8% on Spatial, 99.0% on Object, 97.2% on Goal, 95.8% on Long-10, and 97.0% on Long-90. The improvements on the long-horizon suites are particularly important: LaMem-VLA outperforms MemoryVLA by 2.4 points on Long-10 and 1.4 points on Long-90. These results suggest that weaving short-term visual evidence and long-term semantic memory directly into the VLA reasoning sequence helps the policy resolve temporally ambiguous states, preserve task progress, and generate more reliable actions for both general manipulation and long-horizon instruction following.

4.4 Ablation Study↩︎

Effectiveness of Dual-scale Latent Memory. We evaluate the contribution of each memory source by selectively removing it from the latent VLA input sequence. We consider four settings: the full LaMem-VLA with both short-term and long-term latent memory, w/o Short-term Memory that removes the visual-dominant memory tokens and only keeps long-term semantic memory, w/o Long-term Memory that removes the long-term semantic memory tokens and only keeps short-term visual memory, and w/o Dual-scale Memory that removes both memory streams. Table ¿tbl:tab:ablation95dual95memory? shows that the full dual-memory design achieves the best performance on both benchmarks, reaching 73.9% on SimplerEnv [36] and 97.0% on LIBERO-90 [35]. Removing both memory streams causes the largest degradation, dropping performance to 57.3% and 92.1%, respectively. These results suggest the complementary roles of the two vaults: short-term memory preserves current-episode visual evidence such as object states and transient perceptual cues, while long-term memory maintains task progress and action-continuity evidence over longer horizons. Therefore, removing either stream should weaken temporally grounded action reasoning, and removing both streams should further reduce LaMem-VLA to a memory-free VLA policy.

Latent-native Memory vs. Policy-side Conditioning. We isolate the effect of memory consumption path in Table [tbl:tab:ablation95latent95native95memory], where Baseline denotes the memory-free action policy without short-term and long-term memory. The baseline reaches only 57.3% on SimplerEnv [36] and 92.1% on LIBERO-90 [35], while adding memory as an external policy-side condition improves performance to 71.9% and 94.8%, respectively. Directly feeding raw retrieved evidence to the action policy also improves over the baseline, but remains below the full model LaMem-VLA, suggesting that uncompressed retrieval can introduce redundant historical tokens. By prepending compressed memory tokens as a memory-augmented VLA input sequence, LaMem-VLA enables action tokens to attend to latent memory, observation, and instruction in the same embedding space, achieving the best results of 73.9% and 97.0%. These results confirm that the gain comes from latent-native memory integration rather than merely adding memory as external context.

r0.43

The Number of Retrieved Memory Units \(K\). We further study how the retrieval budget of the memory seeker affects manipulation performance by varying the number of retrieved memory units per vault (Eq. 7 ). Specifically, we evaluate \(K \in \{2,4,8,12\}\) to examine how much historical evidence should be exposed to the condenser. As shown in Table ¿tbl:tab:ablation95retrieved95k?, increasing the retrieval budget from \(K\!=\!2\) to \(K\!=\!4\) improves SR from 66.7% to 70.8% on SimplerEnv [36] and from 94.4% to 95.9% on LIBERO-90 [35]. Further increasing the budget to \(K\!=\!8\) yields the best performance, reaching 73.9% on SimplerEnv and 97.0% on LIBERO-90. This indicates that a very small retrieval budget provides insufficient historical evidence for recovering task progress and transient visual changes. However, using \(K\!=\!12\) slightly reduces performance to 71.8% and 96.2%. These results suggest that retrieving more memory units is beneficial to a moderate budget, while excessive retrieval can introduce redundant or weakly related evidence and increase the compression burden of the memory condenser.

The Number of Latent Memory Tokens \(L_\text{s}\) and \(L_\text{l}\). We next investigate the impact of the short-term and long-term latent memory token number (Eq. 8 ) in Fig. 3. For short-term memory, increasing \(L_\text{s}\) from \(2\) to \(16\) raises SimplerEnv [36] SR from 61.4% to 65.6%, suggesting that more short-term latent memory tokens help preserve fine-grained perceptual evidence; the gain saturates when \(L_\text{s}=32\). For long-term memory, fixing \(L_\text{s}=8\) and increasing long-term latent memory token number \(L_\text{l}\) provides stronger task-progress and action-continuity cues, reaching up to 73.9% on SimplerEnv [36] and 97.0% on LIBERO-90 [35]. Although larger latent memory token budgets can yield strong performance, they also lengthen the VLA self-attention context and increase computational costs; therefore, we use \((L_\text{s},L_\text{l})=(8,4)\) as a balanced default between performance and efficiency.

5 Conclusion↩︎

In this paper, we propose LaMem-VLA, a dual latent memory framework for reducing the temporal short-horizon bias of vision-language-action models. The core idea is to make robotic history part of the model’s native latent context, rather than an external condition attached after multimodal reasoning. To this end, LaMem-VLA uses complementary short-term visual memory and long-term semantic memory, and represents the retrieved history as compact latent memory tokens inside the VLA input sequence. This allows past visual evidence and task-progress cues to interact with the current observation, language instruction, and action queries through the same embedding space. Extensive experiments in two simulation platforms demonstrate the superior performance of LaMem-VLA. These results suggest that context-native latent memory is a practical direction for building VLA systems with stronger temporal awareness.

References↩︎