1 Introduction↩︎

A robot that reasons about its body from video alone reasons about a body it cannot fully see. Force, contact, and joint torque leave little trace in a single RGB frame, and we confirm that force is nearly unreadable from vision alone (raw-feature \(R^2\) at or below \(0.10\) on every robot; Appendix 8, Table 9). The common recipe (encode observations with a vision-only backbone and append robot state downstream as raw numbers [4]) inherits this blind spot. Yet on a robot, vision and state are not independent. On the internet an image is disembodied and could depict anything; on a robot, the images the camera produces are a narrow subset of all possible images, and which subset is fixed by joint configuration, gripper aperture, and contact. Vision and state are two views of one underlying body state, so encoding them jointly should let each inform the other.

We present Kepler-Encoder-v0.1, a robot-first multimodal encoder that maps vision and robot state (proprioception, force/torque, gripper) into a single shared latent. Because the two share a latent, the encoder partially recovers body state that lies outside the frame, and the representation carries force and proprioception rather than appearance alone. We train it without task labels by masked cross-modal prediction, holding one modality out and predicting its representation from the rest, under the LeJEPA/SIGReg objective [1]. Following JEPA [5], the target is a latent rather than pixels, which keeps the code focused on world-relevant structure instead of texture, the structure a robot needs to anticipate the consequences of its actions.

A robot’s sensor set is variable in number, sampling rate, and composition across embodiments. We represent each reading as a token and compress the set with a single learned-query cross-attention block [6]. Cost grows linearly in the number of tokens, and the output is a fixed-size latent regardless of how many sensors enter. One encoder therefore ingests a 6-DOF UR5 or a 7-DOF KUKA, with or without a force sensor, by masking absent inputs rather than maintaining a separate architecture per robot.

Our study is organized around one question, is vision enough? That is, does a vision-only representation (or a compression of it) already encode the robot’s physical state, so that fusing in force and proprioception buys nothing recoverable at test time? For the state the camera cannot see, it is not enough. There, the fused vision-only latent carries state that raw features and controlled baselines do not, while on camera-visible motor state it is statistically tied with the strongest vision baselines; one such encoder spans four robots; and the frozen latent is directly useful downstream, most notably as a training-free safety monitor. This report covers the encoder itself, the first step; native-rate temporal fusion (Appendix 13) and an action predictor built on the frozen latent are the natural next ones.

Our contributions are:

• A robot-first multimodal encoder. We treat video, proprioception, and force/torque as first-class tokens, fuse them with a learned-query cross-attention layer, and train by masked cross-modal latent prediction under LeJEPA/SIGReg, with a robot-agnostic state representation that keeps every joint via masking rather than truncation (Section 3).

• The vision-only latent carries body state the camera cannot see. It reads end-effector state, and force in particular, out of pixels significantly better than raw frozen-ViT features on every sensored robot, and stays statistically tied with the strongest vision baselines on motor state, which the camera largely sees. Two controls, a compute-matched vision-only encoder and a PCA 256 compression of the pretrained ViT features, isolate the end-effector gain as cross-modal learning rather than in-domain training or dimensionality reduction, with force the clearest and most carefully bounded signal (Section 5.1.4, Section 5.2).

• One embodiment-agnostic encoder across four robots. A single encoder matches per-robot specialists on their own robot and, for robots in the training mix, stays strong where specialists collapse; a data-budget-matched control shows the breadth reflects embodiment diversity rather than data volume, while on-diagonal parity partly reflects the larger training set (Section 5.2).

• The frozen latent is directly useful downstream. Its cross-modal prediction error is a training-free invalid-state / safety monitor (AUROC \(0.90\) out-of-range, \(0.69\) scene-swapped); and a latent-conditioned diffusion decoder (PixNerd) reconstructs the camera frame from the vision-only latent, confirming the spatial compression preserves world-state (Section 5.3).

2 Related Work↩︎

2.0.0.1 Joint-embedding and predictive representations.

Joint-embedding predictive architectures learn representations by predicting latent targets rather than reconstructing inputs, avoiding the pixel-memorization trap of autoencoders [5], [7]. The paradigm has been applied to video [8], functional brain signals [9], multivariate time series [10], and robotic motor control [11]. Our training objective follows LeJEPA [1], whose SIGReg regularizer replaces the architectural anti-collapse heuristics (stop-gradient, predictor asymmetry, teacher momentum) with a distributional constraint. On the multimodal side, masked cross-modal prediction appears in M3-JEPA [12], 4M [13], MultiMAE [14], and data2vec [7]; we adopt the predict-one-modality-from-the-rest structure but in latent space with SIGReg, over robot sensor streams rather than images/text.

2.0.0.2 Robot multi-sensor representation.

Vision-language-action models [4] produce a vision-centric representation and treat proprioception as an appended input; heterogeneous pre-trained transformers align proprioceptive-visual streams across embodiments [15]. A separate line learns tactile and multisensory touch representations [16]–[19]. The closest work, MSDP [20], pre-trains a multi-sensory dynamics model but assumes synchronized observations, precisely the multi-rate assumption our tokenization is designed to remove in the native-rate extension (Appendix 13). Action tokenization for sequence models [21]–[24] is complementary, compressing actions for policies, whereas we compress observations into a reusable state latent. Beyond robotics, a wearable-health foundation model [25] fuses heterogeneous physiological streams at population scale, evidence that heterogeneous-sensor pretraining generalizes beyond robots.

2.0.0.3 Latent-query cross-attention.

Our fuser summarizes a variable-size token set with a fixed number of learned queries, compressing \(N\) tokens into \(M\) latents at \(O(NM)\) rather than the \(O(N^2)\) of self-attention. The pattern originates with the Set Transformer’s induced set-attention block [26] and was generalized by the Perceiver family [6], [27], [28]. The same learned-query mechanism recurs across detection and segmentation [29], [30], resamplers that condense visual features for frozen language models [31], [32], audio-visual fusion [33], and speaker verification [34].

2.0.0.4 Continuous-time and irregular multimodal fusion.

The continuous-time embedding of each token’s timestamp borrows from the irregular time-series literature, including mTAN [35], Time2Vec [36], SeFT [37], and ContiFormer [38]. COPER couples a continuous-time embedding with a Perceiver [39], and FuseMoE [40] and UTDE [41] fuse irregular multimodal streams, but all in the electronic-health-record domain. The planned extension (Appendix 13) transplants the continuous-time-token idea into robot sensor fusion, where the rate mismatch (100 Hz force vs. Hz vision) is the defining difficulty.

2.0.0.5 Image and video autoencoders; compression.

Compression-oriented and diffusion-based video autoencoders [42]–[46] and neural codecs repurposed as tokenizers [47] inform the (optional) decoder and the compression framing, but not the training signal, since reconstruction is deliberately kept out of the objective.

2.0.0.6 Robot interaction datasets.

We train on RH20T [2], and situate our data choices relative to humanoid [48], large-scale manipulation [49], mobile manipulation [50], and contact-rich force-aware collections [51].

2.0.0.7 Downstream validation target.

Auxiliary latent-prediction objectives improve VLA policies when the target encoder is strong. FLARE [52] predicts the latent of a future observation and shows the target encoder’s quality is the deciding factor. This is a natural downstream slot for a multimodal encoder such as ours, and for the native-rate extension it is built toward.

3 Method↩︎

3.1 Problem formulation↩︎

At time \(t\) a robot emits a set of heterogeneous sensor streams, namely an RGB frame \(x_v\), proprioceptive state \(x_m\) (joint angles, velocities, gripper aperture), and end-effector signals \(x_e\) (force/torque, TCP pose). These streams are partial views of one underlying body state. We seek an encoder \(f_\theta\) that fuses the available streams into a single latent \(z\) (a unified representation of the robot and its perceivable environment) and, at evaluation, recovers that latent from vision alone: \[\label{eq:encoder} z = f_\theta(x_v, x_m, x_e), \qquad z_v = f_\theta(x_v, \emptyset, \emptyset),\tag{1}\] where \(x_v, x_m, x_e\) are the vision, motor, and end-effector streams and \(\emptyset\) marks a dropped stream, one for which no input is provided at evaluation. Because \(z_v\) is the exported evaluation embedding, the representation must be genuinely cross-modal; we adapt the spatial masking of I-JEPA [5] to the modality dimension. Writing \(z^{(\backslash k)}\) for the fused latent with modality \(k\) held out, and \(\bar f_{\bar\theta}\) for an exponential-moving-average target copy of the encoder, the learning objective is \[\label{eq:objective} \mathcal{L}_{\text{pred}} = \sum_{k \in \{v,\,m,\,e\}} \big\lVert\, h_k\!\big(z^{(\backslash k)}\big) - \operatorname{sg}\!\big[\bar f_{\bar\theta}(x_k)\big] \,\big\rVert_2^2,\tag{2}\] with \(\operatorname{sg}[\cdot]\) the stop-gradient and \(h_k\) a per-modality predictor head that maps the fused latent back to modality \(k\)’s target embedding. The full training loss adds anti-collapse regularization to 2 , detailed in Section 3.4. The encoder must therefore (i) retain the force and proprioceptive structure the camera cannot reveal, (ii) keep \(z_v\) informative of body state despite dropping \(x_m, x_e\) at test time, and (iii) share one architecture across embodiments, tolerating hardware-absent sensors by masking.

Which images the camera produces is determined by joint configuration, gripper aperture, and contact, and force leaves essentially no trace in a single frame (\(R^2 \approx 0\); Appendix 8). Crucially, this force is a directly sensed instantaneous state variable, read from the wrist force/torque sensor at each tick rather than differentiated from motion, so a single frame is a well-posed target for it, exactly as for joint angle or gripper aperture. Velocity and acceleration are the opposite case, genuine time-derivatives of position that a single frame cannot determine and that lie outside the single-timestep model of this report (Appendix 13). The stream count, native sampling rates, and the sensor suite itself vary across embodiments, so no fixed state vector spans the corpus. Appendix 10 collects every symbol used in this section.

3.2 Token representation↩︎

Every sensor reading is embedded as a token that carries its tokenized value, a learned modality embedding, a spatial position, and a continuous-time embedding of its timestamp (constant in the single-timestep model of this report; Appendix 13).

• Vision. Patch tokens from a frozen LeJEPA ViT-B/16 (196 patches \(\times\) 768-d); we write \(F_v\) for these patch features of the frame \(x_v\). The backbone is frozen on the evidence of Appendix 8 (finetuning on robot video degrades the encoder and adds nothing robot-relevant), not by assumption. Only the fusion head trains (\(\sim\)​2M parameters).

• Proprioception (motor). A robot-agnostic \(8\times3\) grid, where rows 0–6 are joints and row 7 is the gripper. Channels use scale-respecting encodings, namely \([\sin q, \cos q, \mathrm{symlog}\,\dot{q}]\) for joints and symlog width for the gripper. Angles use \(\sin/\cos\) to remove the \(2\pi\) wrap, unbounded quantities use symlog, and rotations use the continuous 6-D representation [53].

• End-effector (ee). A \(13\times15\) block of native-rate samples in the interval \([\text{tick}_k, \text{tick}_{k+1})\), comprising symlog force/torque (6), symlog TCP translation (3), and TCP 6-D rotation (6), zero-padded to a fixed window.

3.2.0.1 Robot-agnostic state via masking rather than truncation.

Different robots expose different joint counts (6 for UR5, 7 for Flexiv, Franka, and KUKA) and different sensor suites (Franka has no physical force/torque sensor). Rather than define a per-robot state vector, we fix the \(8\times3\) motor grid and the \(13\times15\) ee block for all robots and mask the rows and channels a given robot does not populate. This choice has three consequences. One architecture spans every embodiment; joints remain first-class tokens rather than being truncated away; and a hardware-absent sensor enters through the same validity masks as any other invalid reading, zero-padded values with validity bits 3 . That data-level masking is a different operation from the attention-level hiding the training objective uses (Section 3.4), but it removes the stream from fusion with the same effect, so Franka’s hardware-absent force sensor provides a built-in robustness test rather than a special case.

3.3 Architecture: the cross-attention fuser↩︎

3.3.0.1 Forward pass.

The fuser is a single asymmetric cross-attention block in which a fixed set of learned queries reads from the token set (Section 2), and the right panel of Figure 2 traces it end to end. The three projected streams are concatenated into a single context set of \(N = N_v + N_m + N_e\) tokens (\(N_v\) vision, \(N_m\) motor, and \(N_e\) ee), heterogeneous in origin but homogeneous in width \(d\). A set of \(M\) learned query vectors cross-attends over this set, and in each of \(L\) depth blocks the queries attend to the context and pass through an FFN (pre-norm residuals), after which they are mean-pooled into a single fixed-size latent \(z\in\mathbb{R}^{d}\). The per-stream token counts, the latent width \(d\), the query count \(M\), and the depth \(L\) are hyperparameters, set to the values reported in the experiments (Section 5). Attention masks serve two roles, encoding padding and validity as well as the modality masking of the training objective. The two mask types use opposite sign conventions, since chunk masks mark valid entries as True, whereas cross-attention masks mark blocked entries as True, so one is inverted before use.

3.3.0.2 Formal specification.

The fuser realizes the encoder \(f_\theta\) of 1 after tokenization. Abusing notation, \(f_\theta(x_v, x_m, x_e) = \mathrm{Fuse}(C, \mathcal{M}_{\mathrm{valid}})\) when every stream is present, and a stream dropped at evaluation corresponds to hiding its tokens, as in 8 below. The motor stream first zeroes the channels a robot does not populate and appends the validity bits as extra features, so the projection sees both the value and the fact that it is present, \[\label{eq:motorfeat} \phi_m(x_m, \mathbf{v}_m) = \big[\, x_m \odot \mathbf{v}_m \;\Vert\; \mathbf{v}_m \,\big],\tag{3}\] where \(\mathbf{v}_m\) is the motor validity mask, matching \(x_m\) in shape, and \(\Vert\) is channel concatenation. Each stream is then linearly projected to the shared width \(d\) and tagged with a learned modality embedding \(\mathbf{e}_{(\cdot)}\) and, for the structured streams, a learned position embedding \(\mathbf{p}_{(\cdot)}\), \[\begin{align} \label{eq:proj} T_v &= W_v\, F_v + \mathbf{e}_v, & T_m &= W_m\, \phi_m(x_m,\mathbf{v}_m) + \mathbf{e}_m + \mathbf{p}_m, & T_e &= W_e\, x_e + \mathbf{e}_e + \mathbf{p}_e, \end{align}\tag{4}\] producing \(N_v\), \(N_m\), and \(N_e\) tokens respectively, each of width \(d\). The context set is their concatenation, \[\label{eq:context} C = \big[\, T_v \;\Vert\; T_m \;\Vert\; T_e \,\big] \in \mathbb{R}^{N\times d}, \qquad N = N_v + N_m + N_e.\tag{5}\] The fuser \(\mathrm{Fuse}(C, \mathcal{M})\) carries \(M\) learned queries \(Q^{(0)}=\{\xi_i\}_{i=1}^{M}\) through \(L\) pre-norm blocks that cross-attend to \(C\) under a boolean block-mask \(\mathcal{M}\) (entries set to \(-\infty\) in the attention logits), \[\begin{align} \label{eq:fuse} \tilde{Q}^{(\ell)} &= Q^{(\ell-1)} + \mathrm{CrossAttn}\big(\mathrm{LN}(Q^{(\ell-1)}),\, C,\, \mathcal{M}\big), & Q^{(\ell)} &= \tilde{Q}^{(\ell)} + \mathrm{FFN}\big(\mathrm{LN}(\tilde{Q}^{(\ell)})\big), \end{align}\tag{6}\] for \(\ell = 1,\dots,L\), and the fused latent is the mean over queries, \[\label{eq:pool} \mathrm{Fuse}(C,\mathcal{M}) = \frac{1}{M}\sum_{i=1}^{M} Q^{(L)}_i \in \mathbb{R}^{d}.\tag{7}\] Writing \(\mathcal{M}_{\mathrm{valid}}\) for the mask that blocks only invalid and padding tokens, and \(\mathcal{M}_{\text{hide}(S)}\) for the mask that additionally blocks the tokens of every modality in \(S\), the training-time fused latent uses the full context and the held-out-modality latents hide one stream, \[\label{eq:latents} z = \mathrm{Fuse}\big(C, \mathcal{M}_{\mathrm{valid}}\big), \qquad z^{(\backslash k)} = \mathrm{Fuse}\big(C, \mathcal{M}_{\text{hide}(\{k\})}\big), \qquad z_v = \mathrm{Fuse}\big(C, \mathcal{M}_{\text{hide}(\{m,e\})}\big),\tag{8}\] where \(z_v\), the vision-only latent obtained by hiding motor and ee, is the exported evaluation embedding. The held-out latents \(z^{(\backslash k)}\) feed the predictor heads of 2 .

3.3.0.3 Why a latent-query bottleneck.

Three properties suit this block to heterogeneous robot streams, over a full self-attention transformer or late concatenation of per-modality encoders. (i) Compute decoupled from input size: cross-attention from \(M\) queries to \(N\) tokens costs \(O(NM)\) (Figure 1), so adding a sensor or more patches grows the context linearly and leaves the latent width unchanged. (ii) Fixed-size latent under variable input: \(z\) has the same shape whether a robot exposes 6 or 7 joints, carries a force sensor or not, or (at evaluation) sees vision alone, so masking rather than per-robot truncation carries embodiment-agnosticism; a hardware-absent sensor amounts to blocked columns, avoiding the per-embodiment “stems” HPT needs [15]. (iii) Set-structured fusion: position and modality live in the token embeddings rather than a fixed layout, so one block ingests any token set, and every query reads all modalities at once [6].

3.4 Training objective↩︎

Training combines a cross-modal predictive signal with two anti-collapse regularizers. Let \(z\) denote the fused latent and \(z^{(\backslash k)}\) the fused latent when modality \(k\) is held out, following 8 .

1. Masked cross-modal latent prediction (the learning signal). Hold out one modality (vision, motor, or ee) and predict its exponential-moving-average target latent from the other two. The prediction is made by the per-modality head \(h_k\) from the held-out-modality latent \(z^{(\backslash k)}\), and its target is the frozen EMA encoder’s embedding \(\operatorname{sg}[\bar f_{\bar\theta}(x_k)]\), not the raw input \(x_k\). This applies JEPA’s predict-do-not-equate principle [5], [7] across modalities, in the manner of masked multimodal prediction [12]–[14] but carried out in latent space. It forces each stream to encode structure relevant to the others (vision learns force-relevant cues because it is trained to predict force), avoids intersection collapse, and confers robustness to dropped or hardware-absent streams by construction. A shared encoder without this signal underperforms single-modality baselines, so the cross-modal loss is necessary rather than optional. The ee prediction term is evaluated only over samples whose robot carries a force sensor, since embodiments without one (for example Franka) supply no ee target to predict.

2. Per-modality SIGReg. A SIGReg (SlicedEppsPulley) penalty applied to each modality’s embeddings before fusion. It prevents per-modality collapse and acts as a magnitude standardizer, making modalities commensurate before they are fused, a standardization that naive concatenation of separately-trained encoders omits. As with the prediction term, the ee penalty is applied only when a batch holds enough force-sensored samples, so vision and motor are always regularized while ee is regularized conditionally.

3. Joint SIGReg. The same penalty applied to the fused latent \(z\), with the aim of keeping it high-rank and expressive rather than collapsed. At the current scale its empirical effect is small (Appendix 12); we retain it as insurance against collapse at wider bottlenecks and larger data, where anti-collapse pressure on \(z\) matters more.

The EMA target encoder is a slow copy of the online encoder, the momentum-target mechanism that stabilizes predictive SSL [7]; SIGReg is the SlicedEppsPulley objective from LeJEPA [1], the explicit distributional anti-collapse constraint described in Section 2, and collapse is monitored directly with RankMe [54]. Only the fusion head and the per-modality projections train; the vision backbone stays frozen.

A decoder is deliberately excluded from this objective, because training on reconstruction would pull the latent back toward pixel memorization, the autoencoder failure mode this design is intended to avoid. Decoders reappear only at evaluation, as probes of what the frozen latent stores (Section 5.1.5, Section 5.1.6).

4 Data↩︎

4.0.0.1 Corpus.

We train on RH20T [2], a large real-robot manipulation dataset with 7 configurations spanning 4 embodiments with different degrees of freedom, grippers, and sensor suites (Table 1). Each scene provides multi-view RGB with per-frame timestamps and robot state (joint position/velocity, TCP pose, force/torque, gripper). The sharded corpus is \(\sim\)​54.3M frames (\(\sim\)​4 TB) across 12,776 scenes.

4.0.0.2 The sensor streams in plain terms.

Each scene delivers four kinds of measurement, and the rest of the paper leans on all of them, so we spell out what each one physically is.

• RGB frames. What the external camera sees, one timestamped color image per tick. This is the only stream available at evaluation.

• Joint state. A robot arm is a chain of motorized rotary joints. The joint angle \(q_i\) says how far joint \(i\) is currently rotated, and the joint velocity \(\dot{q}_i\) says how fast it is rotating; together the angles fix the arm’s shape in space. The gripper aperture is the distance between the gripper’s fingers, from fully open to fully closed.

• TCP pose. The tool center point (TCP) is the working reference point of the end-effector, conventionally placed between the gripper fingertips. Its pose is that point’s 3-D position plus its 3-D orientation in the workspace, in plain terms where the hand is and which way it points. Unlike joint state, which describes the arm’s internal configuration, the TCP pose describes the hand’s placement in the world.

• Force/torque. A sensor in the wrist reports six numbers, the force pushing along each of three axes and the torque twisting about each of them. This is the robot’s sense of touch and effort at the wrist, the signal that distinguishes pressing on a surface from hovering above it. Franka (cfg5) has no such sensor.

4.0.0.3 Timing.

The shipped state streams are camera-aligned (their timestamps match the color frames), so the released data runs at an effective \(\sim\)​7–15 Hz (varying by config and scene) rather than the raw sensors’ native rate. A separate high-frequency stream (wrist force/torque and TCP) runs at 100–125 Hz, absent for cfg5 and empty for a third of cfg3. This irregular, multi-rate timing is why the model cannot assume a fixed clock. We split scenes at recording gaps \(>500\) ms before chunking.

4.0.0.4 Chunk packet.

We convert scenes into tick-anchored chunks (native samples only, no interpolation), each a robot-agnostic packet (Table 2): frozen ViT patch tokens, a masked \(8\times3\) motor grid, a masked \(13\times15\) end-effector block, a robot id, and a tick timestamp (cached, so the temporal extension needs no re-preprocessing). Chunks are precomputed to per-config caches.

4.0.0.5 Reading the encodings.

The packet stores transforms of the raw values, not the values themselves. The \(8\times3\) motor grid has one row per actuator, with joints in rows 0–6 (the seventh joint row masked on 6-DOF robots) and the gripper in row 7. Its 3 columns are not three physical quantities. A joint row holds \([\sin q, \cos q, \mathrm{symlog}\,\dot{q}]\), the angle in the first two columns through its encoding and the velocity in the third. The gripper row carries only its symlog width, in the first column. Joint torque, which the KUKA vector ships, is deliberately excluded. The \(13\times15\) ee block holds the wrist stream instead, one row per native-rate sample landing between the current camera tick and the next (zero-padded, with ee_mask marking the filled rows). Each ee row’s 15 features are the six symlog force/torque readings, the three symlog TCP position coordinates, and the TCP orientation’s six rotation numbers. Each transform fixes a specific numerical pathology of its quantity, as follows.

• Joint angles \(\rightarrow [\sin q, \cos q]\). Here \(q\) is one joint’s rotation angle in radians, read from the shipped joint-position vector (joint.npy), and every joint of the arm contributes its own pair to the motor grid. The raw angle wraps around, so \(359^\circ\) and \(1^\circ\) are physical neighbors yet numerically far apart, and a regression target with that jump is needlessly hard. Storing the pair \((\sin q, \cos q)\), the coordinates of the tip of a clock hand pointing at angle \(q\), places wrapped neighbors close together and removes the discontinuity. The raw angle is recovered exactly as \(q = \mathrm{atan2}(\sin q, \cos q)\), which is how the decoded joint angles of Figure 10 are produced.

• Joint velocities, gripper width, force/torque, TCP position \(\rightarrow \mathrm{symlog}\). These raw values (the joint’s angular velocity \(\dot{q}\) where the robot reports one, the commanded finger-opening width from gripper.npy, the six wrist force and torque readings, and the TCP’s position coordinates from tcp_base.npy) share a different pathology. They are unbounded and heavy-tailed, mostly small values with occasional large spikes, for example at contact. Each is stored as \(\mathrm{symlog}(x) = \mathrm{sign}(x)\,\log(1+|x|)\), a log scale that also accepts negative values; near zero it is almost the identity, so small readings stay distinguishable, while spikes are compressed so they do not dominate the loss. The raw value is recovered as \(\mathrm{sign}(y)(e^{|y|}-1)\). One caveat carries over from the hardware, since the three gripper models report widths on their own scales, raw widths are not comparable across configs.

• TCP orientation \(\rightarrow\) 6-D rotation. The corpus ships the TCP orientation as a quaternion (the last four entries of the 7-D pose in tcp_base.npy). A 3-D rotation suffers the same wrap-around problem as a single angle, in three dimensions at once, and quaternions additionally represent every rotation twice. We therefore convert to the first two columns of the rotation matrix, six numbers that vary continuously with the physical orientation [53].

4.0.0.6 Splits and filtering.

One user repeating a task yields \(\sim\)​10 near-duplicate scenes, so we hold out by (config, task, user) group (stratified per config, \(\sim\)​30% held out, frozen in a committed CSV) and evaluate only on held-out groups, so no near-copy leaks into training. We use one deterministic external camera per scene (the wrist camera is excluded; multi-view is future work, framed as cross-view prediction). Two documented traps are filtered out, namely the \(543\) _human_2 scenes, which carry no robot state, and the \(57\) scenes (\(\sim\)​0.4%) missing joint.npy.

5 Experiments↩︎

Our experiments test a single hypothesis. The vision-sufficiency null asserts that a vision-only representation, or a low-dimensional compression of one, already captures the robot’s physical state, so that fusing proprioception and force during training adds nothing recoverable from vision at test time. Force makes the null non-trivial because it leaves almost no trace in a single frame (raw-feature \(R^2\) at or below \(0.10\) on every robot; Table 9). We reject it in three steps. Section 5.1 presents the headline comparison, scoring the encoder against the vision-only baselines of Section 5.1.3 with every result averaged across embodiments. Section 5.2 breaks the comparison out by embodiment and isolates each ingredient of the result, covering the cross-embodiment transfer matrix, a compute-matched vision-only control, the separation of force from pose, and a data-budget-matched control. Section 5.3 then shows that the frozen encoder is directly useful downstream with no retraining.

5.1 Training results↩︎

Unless noted otherwise, every result in this section concerns Kepler-Encoder-v0.1, a single encoder trained on all seven RH20T configurations (Section 4), abbreviated KEv0.1 in tables. The instantiation sets \(d=256\), \(M=8\) queries, and \(L=2\) cross-attention blocks (8 heads) over \(N_v=196\), \(N_m=8\), and \(N_e=13\) tokens, and trains for 40 epochs at batch 256 with AdamW (learning rate \(10^{-3}\), weight decay \(10^{-4}\)), repeated over 5 seeds. Only the fusion head and the per-modality projections train (\(\sim\)​2M parameters); the ViT backbone stays frozen. Evaluation uses the committed group-held-out splits (Section 4), probes are fit on train rows and scored on held-out rows only, and error bars are \(\pm\)std over seeds. Per-robot specialist encoders, trained identically on single embodiments, appear in the ablation study (Section 5.2).

5.1.1 Training loss↩︎

The training loss sums the masked cross-modal prediction term and the SIGReg penalties (Section 3.4). Figure 3 shows its per-epoch mean for the five seeds of Kepler-Encoder-v0.1, logged every ten epochs. Optimization is stable and consistent across seeds, with most of the decrease arriving in the first ten epochs (\(\approx\)​4.3 to \(\approx\)​3.1) and a slow drift to 2.7–2.9 thereafter. The exception is instructive. Seed 3 tracks the other seeds through epoch 30 (\(2.89\), mid-range) and destabilizes only in the final epochs, ending at \(12.57\). The spike does mark real damage of one kind, since that seed’s RankMe drops from the \(\approx\)​185 band of the other seeds to \(\approx\)​110, but its linear-probe accuracy is essentially unchanged (ur5 motor \(R^2\) \(0.334\), against \(0.326\)–\(0.365\) for the healthy seeds). The three signals disagree because they measure different things. The loss reflects the joint state of the encoder and its predictor heads, RankMe the spectrum of the latent, and the probe the state recoverable from it; the loss is the least informative of the three about the exported encoder, which is why we evaluate with the encoder-only metrics defined next.

5.1.2 Predictor-free performance metrics↩︎

Training relies on predictor heads that map the fused latent to each held-out modality’s target embedding, so the objective value measures the encoder and the predictors jointly. A collapsed latent paired with a predictor that has learned the collapse can score a near-perfect loss. We therefore evaluate the encoder with a family of measurements that involve no trained predictor. No single number in this family is treated as sufficient, because rank-style statistics alone can order models differently from downstream performance [55]; every table pairs them with probes.

5.1.2.1 Latent health (RankMe).

RankMe is the effective rank of the latent, the exponential of the entropy of the normalized singular-value spectrum [54], bounded by \([1,\dim]\). It is a label-free collapse detector, and it catches a collapsed latent that a probe alone can miss. Because it scales with dimensionality, we report it alongside each feature’s width and compare it only within a width family.

5.1.2.2 Latent organization (distance correlation and R_LDA).

RankMe sees only the spectrum, so we add two measurements of how the latent is arranged. Distance correlation samples pairs of held-out frames and reports the Spearman correlation between their latent distance and their true robot-state distance; a feature can be decodable by a probe yet carry no state structure in its geometry, and this statistic separates the two. R_LDA [55] is an LDA-style effective rank, the exponential entropy of the eigenvalues of the within-view versus between-sample covariance ratio, computed here with same-scene temporal neighbors as the views; it measures how many latent directions separate samples stably rather than how many carry variance, and it shares RankMe’s dimensionality caveat.

5.1.2.3 Robot-state recovery (linear probe).

With the encoder frozen, we ask how much of the robot’s true state can be read off a feature by a single linear map. The probe is ridge regression, that is, least-squares linear regression with an \(\ell_2\) weight penalty (features standardized, \(\alpha=10\)), which stabilizes the fit when feature dimensions are correlated, as they are here. The probe is deliberately the weakest read-out available. If a linear map recovers a state variable, the information is not only present in the feature but arranged so that it is directly accessible, so the credit belongs to the representation rather than to the decoder; a high-capacity decoder would blur that attribution, and Section 5.3 measures separately what a nonlinear read-out adds. We fit the probe on train rows and report \(R^2\uparrow\) on held-out rows, where \(1.0\) is perfect recovery, \(0\) matches always predicting the mean, and negative values are worse than the mean. We score two target groups separately, motor (joint angles and gripper) and end-effector (force/torque and TCP pose). Unless noted, the probed feature is the vision-only fused latent \(z_v\) with the state modalities masked at evaluation, so no robot state leaks into the probe input and the probe measures what the encoder learned to read out of pixels.

5.1.3 Baselines↩︎

We compare the vision-only latent \(z_v\) against three vision-only features, all probed under the identical protocol.

1. Pretrained ViT (PT-ViT). Mean-pooled patch tokens (768-d) of the frozen LeJEPA ViT-B/16 pretrained on ImageNet-1k [1], the same backbone that feeds our encoder and the bar that fusion must beat.

2. Pretrained ViT (PCA 256). The top 256 principal directions of the pretrained features, fit on train rows only. This is a pure compression control at the same width as \(z_v\), so beating it rules out dimensionality reduction as the source of any gain.

3. Pretrained ViT \(+\) finetuned linear head. The frozen backbone with a trained \(768\!\rightarrow\!256\) projection, trained on our training split with the vision-only LeJEPA objective. This controls for lightweight in-domain adaptation at the same output width as \(z_v\).

The strictest control is a second encoder of identical architecture and size, trained on the same data with the state streams permanently masked. In the notation of 1 its exported feature is \(z_v^{\mathrm{vo}} = f_{\theta_{\mathrm{vo}}}(x_v, \emptyset, \emptyset)\), where the parameters \(\theta_{\mathrm{vo}}\) never observe \(x_m\) or \(x_e\) during training; \(z_v\) has the same functional form but its parameters \(\theta\) were trained with all modalities present. This vision-only control appears in the main table and is analyzed per embodiment in the ablation study (Section 5.2). One further baseline, a full finetune of the backbone, lowers every probe and changes no conclusion; it is described and reported in Appendix 9.

5.1.4 Quantitative results↩︎

Table 3 reports the 256-d features. The two 768-d baselines (pretrained ViT and finetuned ViT) are tabulated in Appendix 9 (Table 13) under the identical protocol and are quoted below where relevant. Each cell averages over the three embodiments that carry both target groups (flexiv, ur5, kuka). Franka has no force/torque sensor and is degenerate for every feature (motor \(R^2\) from \(-0.35\) to \(-6.71\)), so it is reported in the per-embodiment break-out (Table 6) rather than folded into an average it would dominate.

First, training with state raises what the encoder reads out of pixels. The fused \(z_v\) and the vision-only control share architecture, data, and compute, and the fused latent probes higher on both target groups (\(+0.106\) motor, \(+0.140\) end-effector). The difference between the two runs is only whether state was present during training, so that is what the gap measures.

Second, the gain is specific to end-effector state, and within it, to force. On motor, the fused latent is statistically indistinguishable from both PCA 256 (paired cluster bootstrap over held-out groups, difference \(-0.004\), 95% CI \([-0.015, +0.009]\), \(p=0.52\)) and PT-ViT (\(-0.008\), \(p=0.37\)); joint pose is largely visible to the camera, so vision-derived features already recover it. The motor tie is itself an average of per-embodiment differences in both directions, with the fused latent above PCA 256 on flexiv (\(+0.033\), \(p<0.001\)) and below it on kuka (\(-0.064\), \(p<0.001\)). On end-effector, the fused latent is above every baseline, and the gap is seed-robust and significant (\(+0.049\) over PCA 256, 95% CI \([+0.039, +0.060]\); \(+0.062\) over PT-ViT; \(p<0.001\) for every seed), holding on each sensored robot individually (Table 6). Isolating the force dimensions (Table 9, Section 5.2), every vision-only feature reads force at or near zero (PT-ViT \(-0.005\)/\(-0.115\)/\(0.103\) on flexiv/ur5/kuka), and the fused latent is above PT-ViT and the control on all three sensored robots (paired \(t\), \(p\le0.012\)). Absolute force recovery is nonetheless modest (\(0.05\)/\(-0.00\)/\(0.19\)), consistent with a single timestep bounding what is recoverable (Appendix 13).

Third, in-domain adaptation of the backbone does not substitute for state, and rank does not predict probe accuracy. The finetuned linear head probes below PCA 256 on both targets while compressing to RankMe 53, the full finetune lowers both probes further (Table 13), and force \(R^2\approx0\) for every finetuned checkpoint (Appendix 8). A 5-seed retrain of the head reproduces this picture with negligible seed variance and slightly lower values (motor \(0.258\pm0.002\), end-effector \(0.190\pm0.001\), RankMe \(44.2\pm0.0\)), so the tabulated single run is the head’s best case and its deficit is not a seed artifact. RankMe of \(z_v\) sits near two thirds of the 256 ceiling, so the latent does not collapse; the vision-only control is higher-rank with weaker probes, so we report rank and probe accuracy together and read neither alone.

Fourth, the fused latent is organized by state, not merely decodable to it, and the organization comes from cross-modal training specifically. Table 4 reports the two organization measurements of Section 5.1. The distance correlation between latent distances and true state distances is \(0.190\) (motor) and \(0.221\) (end-effector) for the fused latent, against \(|\rho|\le0.03\) for every frozen vision baseline and a near-flat \(0.050\)/\(0.015\) for the vision-only control, whose single nonzero cell is kuka motor at \(0.131\), half the fused latent’s \(0.241\) there. The fused latent is therefore the only feature whose geometry consistently tracks state, including on motor, where its probe accuracy is merely tied. Because the control shares the architecture, data, compute, and SIGReg regularization and differs only in never seeing state, the geometry is attributable to cross-modal training rather than to the architecture or the regularizer; the control is in fact the best-spread feature (RankMe \(190\), Table 3) while carrying the least separable structure (R_LDA \(22.2\), below even PCA 256). R_LDA gives the same ordering overall, with the fused latent highest (\(75.6\)) even against the 768-d PT-ViT (\(34.9\)), whose larger width favors it under this metric. Probes measure what a decoder can extract; these two measure how the space is arranged, and the baselines decode motor state without arranging their geometry by it.

The relative strength of PCA 256 has a simple reading. It is not a trained model but the variance-optimal linear compression of the PT-ViT features, so under a linear probe it inherits nearly all of their decodable content (\(0.308\) vs.\(0.315\) on motor). The two trained 256-d features are instead optimized for objectives other than preserving that content. The linear head is trained for multi-crop invariance under SIGReg; tracked across its training epochs, it sits at motor \(0.275\)–\(0.280\) throughout, above a randomly initialized \(768\!\rightarrow\!256\) projection (\(0.251\)) but never approaching PCA 256, while its spectrum collapses to RankMe 29 in the first epoch and recovers only to 53. Invariance training therefore recovers some decodable content over an arbitrary projection but preserves less of it than the variance-optimal one. Kepler-Encoder-v0.1 is trained for cross-modal prediction, which reshapes the code toward state-predictive structure and away from pixel-level appearance detail. On motor, where joint pose is visible in the frame and appearance detail is exactly what the probe uses, this leaves it tied with PCA 256 (\(p=0.52\)); on end-effector, where the pixels lack the information and preservation cannot supply it, the same trade puts the fused latent \(0.049\) above (\(p<0.001\)). A feature that merely preserves the backbone is therefore the strongest 256-d baseline wherever the backbone already suffices, and fusion pays where it does not.

In sum, fusing state into training yields a moderate but consistent improvement in state recovery from pixels, and the improvement is concentrated where the camera carries the least information.

5.1.5 Storage check: the latent retains its inputs↩︎

The design excludes decoders from training and uses them as probes of what the latent stores (Section 3.4); this section is the quantitative half of that check. We encode the full-context latent \(z\) (all modalities visible, as in training) and probe it for the same state targets on held-out rows. Table 5 shows the result. With state visible at encode time, a ridge probe alone reads back motor \(R^2\) \(0.57\) and end-effector \(0.67\) (up to \(0.84\) on kuka), roughly double the vision-only ceiling of Table 3, and the MLP raises end-effector to \(0.73\). The bottleneck therefore retains a large share of what enters it after fusing three modalities into 256 dimensions, though not all of it. The generative decodes of Section 5.1.6 are the qualitative face of the same conclusion, recovering the scene from the vision path and arm pose from the state path.

5.1.6 Qualitative results↩︎

The latent organizes by continuous world-state. Within ur5, a PCA of the vision-only \(z_v\) traces a smooth open-to-closed gripper gradient (Figure 4), so the latent encodes graded state rather than only robot or scene identity.

Whereas a probe reports that information exists in the latent, a generative decoder makes explicit what the latent represents, and decoding is one-to-many because the latent intentionally discards pixel nuisance (Section 3.4). A small latent-conditioned diffusion decoder (PixNerd [3]) regenerates the camera frame from the frozen vision-only \(z_v\) (Figure 5a). Reconstructions are recognizable in scene layout, surfaces, colors, and rough arm and object placement, but blurry by design, because the 256-d latent keeps world-state and drops fine texture; quality tracks scene complexity, clean on simple scenes and degrading on clutter. The mirror experiment decodes from a state-only latent (vision hidden, motor and ee only) and recovers arm pose from proprioception but not the surrounding scene (Figure 5b). These reconstructions illustrate what the latent retains; the fusion claim rests on the quantitative comparison above and the ablations below.

Two further probes ask how the vision path is reshaped to carry robot-relevant structure. The renderings are illustrative, and each is paired with the quantitative diagnostic that constrains its reading.

5.1.6.1 Patch-feature PCA.

We compare three per-patch feature spaces over the same test-split frame at a matched \(14\!\times\!14\) patch grid (224px input for every column), so no panel benefits from a finer rendering. Column 1 shows the frozen pretrained ViT (LeJEPA ViT-B/16 [1]) patch tokens. Column 2 shows the same frozen backbone with a single trained linear head (proj_v, \(768\!\rightarrow\!256\); the pretrained ViT \(+\) finetuned linear head of Section 5.1.3). Column 3 shows Kepler-Encoder-v0.1’s own vision projection \(\text{proj\_v}(\text{patch})+\text{mod}_v\), the last point in the pipeline where a spatial patch grid still exists, taken before the fuser pools the patches into the non-spatial bottleneck, so it is the fair spatial comparison point. Each panel is reduced by the same recipe, its top three principal components with robust 2–98 percentile normalization, bicubic upsampled to the image, following the DINO/DINOv2 patch-PCA visualization convention [56]; the differing feature widths (768 vs.) rule out sharing one set of PCA axes across columns. Figure 6 shows the result across four embodiments.

An earlier rendering of this comparison drew the frozen-ViT columns at a finer \(28\!\times\!28\) grid and made the encoder projection appear smoother; at the matched grid that difference reverses, so the apparent smoothness was a resolution artifact. Quantitatively, the encoder projection has the sharpest patch-to-patch transitions of the three (mean adjacent-patch distance in the normalized PCA image \(0.325\), against \(0.246\) for the frozen ViT and \(0.281\) for the linear head, averaged over the four scenes). What distinguishes the encoder is instead object alignment. The frozen ViT and the linear head render regions whose colors follow object silhouettes, with arm, workpiece, and background separating cleanly, while the encoder’s principal-component colors fragment across those silhouettes, so its per-patch code is no longer organized primarily by local appearance. This reorganization is consistent with a code shaped for masked latent prediction rather than pixel fidelity (Section 3.4, [5]), though that link remains interpretive.

5.1.6.2 Attention maps.

The complementary question is spatial, asking where on the image each model’s attention lands. We overlay three attention maps on the same frame sequence (Figure 7). For the frozen pretrained ViT and a full-finetuned ViT (whole backbone trained on robot video), we take last-layer CLS-to-patch self-attention, averaged over heads (DINO-style [56]). For Kepler-Encoder-v0.1 we take the fuser’s final cross-attention, the \(\text{softmax}(qk^\top)\) weight of its bottleneck queries over the 196 vision patches (averaged over heads and queries), i.e.what the fused bottleneck reads from the image. We note the standard caveat that raw attention weight is not a faithful importance attribution [57]; we use these maps as an illustrative complement to the probe results, not as proof.

The pretrained ViT’s attention is diffuse, spread broadly over the scene with the generic, texture-driven saliency of backbones not adapted to the domain. Kepler-Encoder-v0.1’s cross-attention is markedly more concentrated, collapsing onto a small number of compact regions, most of them on or near the manipulated object and the gripper, the regions where the manipulation occurs. Read alongside the cross-embodiment study (Section 5.2.1), this is the qualitative face of the same finding, in which the bottleneck has learned to select task-relevant image regions rather than average over the frame.

A few activations fall on regions where nothing is visibly happening. Two diagnostics constrain how to read them. First, they are largely not the known alternative in which ViTs park global information in a few high-norm background patches (register-token / attention-sink behavior [58]). Over 96 held-out frames spanning 12 scenes, the cross-attention weights correlate only weakly with patch-token norms (mean Spearman \(0.15\)), and the top-10 attention patches overlap the top-10 high-norm patches at barely twice chance (\(0.12\) against \(0.05\)). Second, the attention tracks scene content rather than fixed positions. Attention maps correlate at \(0.36\) across frames within a scene but only \(0.12\) across different scenes of the same config, whereas the high-norm patch positions themselves are strongly position-fixed across scenes (\(0.45\)). The off-action activations are therefore scene-dependent and not norm-driven, which is necessary for the anchoring reading, in which the encoder attends to reference structures (table edges, fixtures, the robot base) that fix the geometric frame. Whether the attended regions are semantically meaningful landmarks remains unverified, and the moderate within-scene consistency shows the attention also moves as the manipulation progresses, so anchoring should be read as a tendency rather than a fixed set of points per scene.

5.2 Ablation study↩︎

The main table averages over embodiments and pools force with pose. This section breaks both aggregations apart and isolates each ingredient of the result. All studies use 5 seeds unless noted.

5.2.1 Cross-embodiment transfer↩︎

Using the robot-agnostic chunk packet (Section 4), we first ask whether the per-embodiment picture supports the averaged one, and whether one encoder trained on every robot matches per-robot specialists. Alongside Kepler-Encoder-v0.1 we train four specialists (flexiv \(=\) cfg1\(+\)​2, ur5 \(=\) cfg3\(+\)​4, franka \(=\) cfg5, kuka \(=\) cfg6\(+\)​7), each for 5 seeds and 40 epochs, and probe every encoder on every embodiment’s held-out groups. This yields a \(5\times4\) transfer matrix; the probed feature is always the vision-only latent \(z_v\) (state masked at eval), scored on the same two target groups as before. A single-embodiment proof-of-concept that preceded this study appears in Appendix 11. Table 6 breaks the encoder against the two feature baselines out by embodiment, and Table ¿tbl:tab:phase1matrix? gives the specialists’ full transfer matrix.

5.2.1.1 The per-embodiment picture matches the average.

On the end-effector probe the Kepler-Encoder-v0.1 latent beats PT-ViT on every sensored robot (\(+0.06\)/\(+0.08\)/\(+0.04\); Table 6). On motor the gain is partial. The latent beats PT-ViT on flexiv and ur5 but not on kuka, where joint pose is already visible to the camera, so we expect the motor payoff to come with the temporal extension (Appendix 13). Franka is the hardest embodiment, negative for every feature, but fusion stabilizes it (\(-0.38\) against PT-ViT’s \(-6.71\)).

5.2.1.2 One encoder covers four robots.

Kepler-Encoder-v0.1 matches per-robot specialists on their own robot (Table 6 against the diagonal of Table ¿tbl:tab:phase1matrix?) and stays strong on every robot, while specialists degrade sharply off-diagonal (the franka specialist to \(0.07\)–\(0.15\), even the strong kuka specialist to \(0.11\)–\(0.22\); Table ¿tbl:tab:phase1matrix?). Two controls locate the gain. The pretrained ViT (PCA 256) stays within \(0.01\)–\(0.04\) of PT-ViT, so the coverage is not a compression effect, and the data-budget-matched encoder below still beats every specialist off its training robot, so the coverage reflects embodiment diversity rather than data volume.

5.2.1.3 Per-embodiment latent health.

RankMe of \(z_v\) holds a healthy \(\sim\)​165–180 band on every test embodiment, about two thirds of the \(256\) ceiling (Table 7). Isolated low-rank seeds, including the loss-spike seed of Section 5.1, pull some means down without moving the probes; these warrant a re-run rather than indicating a failure of the recipe.

5.2.2 Cross-modal gain under matched compute↩︎

The vision-only control \(z_v^{\mathrm{vo}}\) of Table 3 is an identically-sized 256-d cross-attention encoder trained with the state streams permanently masked (same head, data, and compute; Section 5.1.3), which isolates cross-modal fusion from the alternative that we merely trained an in-domain encoder. Table 8 breaks the comparison out on each robot’s own held-out groups, and Figure 8 plots the same comparison.

Fusion adds \(+0.06\)–\(0.14\) (motor) and \(+0.10\)–\(0.17\) (end-effector) on the three data-rich robots. Because architecture and compute are identical, the gain is cross-modal learning rather than in-domain training; together with the pretrained ViT (PCA 256) control, this rules out dimensionality reduction as well. Franka is inconclusive, a small force-blind config where both variants are near-degenerate.

5.2.3 Force isolated from TCP pose↩︎

The end-effector probe mixes force/torque with TCP pose, so we split it (Kepler-Encoder-v0.1, 5-seed), probing force (\([0{:}6]\)) and pose (\([6{:}15]\)) separately. Pose-only \(R^2\) is high (\(0.41\)/\(0.31\)/\(0.53\) on flexiv/ur5/kuka) and dominates the aggregate, while force-only \(R^2\) is modest (\(0.05\)/\(-0.00\)/\(0.19\)). The signal is nonetheless real and cross-modal. PT-ViT reads force at or near zero everywhere, whereas the fused latent is above it, and above the vision-only control \(z_v^{\mathrm{vo}}\), on all three robots, with both gaps significant under a 5-seed paired \(t\)-test (Table 9; \(p\le0.012\)). Partialling pose out of the force target leaves a near-zero residual for the fused latent (\(+0.01\)/\(-0.03\)/\(+0.01\)) and a negative one for PT-ViT, so the force recoverable at a single timestep is largely the component entangled with joint configuration and contact geometry; the pose-independent part is out of reach without time, motivating the temporal extension (Appendix 13).

5.2.4 Data-budget-matched Kepler-Encoder-v0.1↩︎

To separate embodiment diversity from data volume, we retrain Kepler-Encoder-v0.1 on a random \(32{,}870\)-frame subset (the mean specialist’s training size; same \(40\) epochs, hence a matched gradient-step budget), 5 seeds. Matched to a single specialist’s budget, the encoder falls \(0.02\)–\(0.06\) below each specialist on that specialist’s own robot (Table 10), so the full model’s on-diagonal parity partly reflects its \(4\times\) larger training set. The embodiment-agnostic result, however, is not a volume artifact. At the same total budget the shared encoder covers all four robots (motor \(R^2\) \(0.22\)/\(0.30\)/\(0.31\) on flexiv/ur5/kuka) and beats every specialist evaluated off its training robot (\(0.07\)–\(0.25\); Table ¿tbl:tab:phase1matrix?). Distributing a fixed data budget across embodiments buys broad competence that concentrating it on one does not.

5.3 Downstream capabilities of the frozen encoder↩︎

Beyond probes, we ask what the frozen encoder is directly useful for, with no retraining. These results show the latent supports concrete downstream use; the fusion claim itself rests on the quantitative comparison and the ablation study above, not on the demonstrations here.

5.3.0.1 Invalid-state and safety detection.

Because the encoder is trained by cross-modal prediction, its per-sample prediction error is a built-in surprise signal, low when the robot state is consistent with vision and high when it is not. On ur5 held-out, valid vs.corrupted state separates at AUROC \(0.90\) (out-of-range state) and \(0.69\) (state swapped from another scene), with no additional training (Figure 9), giving a robot-safety anomaly detector on the encoder we already have.

5.3.0.2 State read-out.

The linear probe of Section 5.1 lower-bounds what a feature contains, because it can only read information that is linearly arranged. To measure the headroom above that bound we fit MLP read-outs to held-out ur5 motor state, identically for every feature, and the decoder’s own training budget turns out to matter. An under-trained MLP (two hidden layers of 512, early stopping, at most 200 iterations) shows no gain on \(z_v\) (\(0.328\pm0.020\) against ridge \(0.339\pm0.014\)); the same architecture trained to convergence (300 epochs, no early stopping) lifts \(z_v\) to \(0.441\pm0.020\). Three conclusions follow at full decoder strength. First, the 256-d latent stores motor state beyond the linear bound (\(+0.10\) over ridge). Second, the fusion gain persists, since the vision-only control reaches only \(0.315\pm0.005\) under the same decoder. Third, preservation still wins on motor, with PT-ViT at \(0.523\) and PCA 256 at \(0.459\), so compression keeps its motor-state price at high decoder capacity, consistent with the motor tie of Section 5.1.4. Figure 10 shows the decode itself, with joint angles recovered from the ridge probe’s predicted \(\sin\)/\(\cos\) channels on each robot’s held-out rows, the paper’s standard read-out. Gripper aperture behaves differently. Its distribution is near-discrete, mostly fully open or fully closed, so we score it as classification; \(z_v\) gives \(78\%\) accuracy and \(0.87\) AUROC against \(77\%\) and \(0.86\) for PT-ViT, a near-tie consistent with gripper state being visible in pixels.

5.3.0.3 Read-outs are configuration-specific.

A natural deployment hope is to fit one read-out on the fully-sensored latent and reuse it when sensors drop out. Table 11 shows this does not work here. The same ridge probe that reads the full-context latent \(z\) at \(0.57\)/\(0.67\) scores below zero when applied unchanged to \(z_v\), worse than predicting the mean on every embodiment and target, while refitting on \(z_v\) recovers the Table 3 performance; the MLP read-out transfers no better. The two latents therefore encode state along different directions. This is consistent with the training objective, which predicts the held-out modality’s embedding through a per-modality head (Section 3.4) and never aligns \(z_v\) with \(z\) itself. The practical rule for a downstream consumer at a single timestep is to fit its read-out on the latent of the sensor configuration it will run with; what stays invariant across configurations is the encoder, not the probe. The finding also motivates the temporal extension (Appendix 13). In deployment, sensor dropout is usually transient, and a sensor that vanishes mid-stream remains constrained by its own recent past. An encoder that fuses across time can therefore hold the latent in one coordinate frame through the dropout, and whether that restores read-out transfer is a directly testable prediction for the temporal model.

6 Discussion and Limitations↩︎

6.0.0.1 An information-flow reading of the results.

Three numbers summarize what the encoder does with state. State that enters is largely kept, since a ridge probe reads motor \(R^2\) \(0.57\) and end-effector \(0.67\) back out of the full-context latent (Section 5.1.5). State that is absent at evaluation is partially inferred from pixels, at \(0.30\) and \(0.28\) from the vision-only latent, with the gain over vision baselines significant only where the camera is weakest, on end-effector state and force in particular (Section 5.1.4). And the two access paths are geometrically distinct, since a probe fit on one latent fails on the other (Section 5.3). Read alongside the organization results (Table 4), where the fused latent is the only feature whose distances track state, the picture is of an encoder that reorganizes the vision path around body state rather than one that appends state to appearance features.

6.0.0.2 Single-timestep.

Every result here fuses modalities at a single tick, so the encoder captures structure across modalities but not across time. True time-derivatives (velocity, acceleration) and contact dynamics are out of reach, which also bounds the modest force recovery; the native-rate temporal extension (Appendix 13) is where we expect the motor and force payoff.

6.0.0.3 Correlational, not causal.

Same-time alignment shows the latent contains cross-modal structure, not that it models dynamics; action-conditioned forward prediction, the causal step, is future work (Appendix 13). The evaluation is probe-centric and aligned with the training target, which is favorable to the method, but three safeguards bound the risk. The controls of Section 5.2 rule out in-domain training and compression, the paired cluster bootstrap of Section 5.1.4 marks the motor comparison as a tie rather than a win, and the predictor-free measurements of Section 5.1 (RankMe, distance correlation, R_LDA) involve no fitted decoder at all.

6.0.0.4 No zero-shot transfer to unseen robots.

Embodiment-agnosticism holds for robots in the training mix (Section 5.2.1), not for unseen ones. In a leave-one-robot-out test the zero-shot latent trails PT-ViT on the held-out robot’s motor probe by \(0.03\)–\(0.13\) (Table 12), as the fused vision path tunes to the training embodiments’ visual domains. It is, however, far more stable than PT-ViT on the degenerate franka config and beats it on the ur5 end-effector probe. Few-shot adaptation and a broader embodiment set are future work.

6.0.0.5 Read-outs do not transfer across sensor configurations.

A probe fit on the full-context latent does not apply to the vision-only latent (Table 11), so a consumer must fit its read-out for the sensor configuration it deploys with. The objective never asks the two latents to agree, only to predict each other’s modality embeddings (Section 3.4), so this is a designed property with a real cost. Two extensions could recover single-read-out deployment. A latent-consistency term aligning \(z_v\) with \(z\) is the direct route, and its interaction with the predict-do-not-equate principle is an open design question. The temporal extension (Appendix 13) is the indirect route, since real sensor dropout is transient and a temporally fused latent remains constrained by the dropped sensor’s recent past; whether temporal fusion restores read-out transfer is a testable prediction of that model. Relatedly, compression has a price at high decoder capacity, since an MLP extracts more motor state from the raw 768-d features than from the 256-d latent (Section 5.3).

6.0.0.6 Other caveats.

A single external camera is used (multi-view is future work); RH20T’s shipped streams are camera-aligned rather than native-rate, so the multi-rate design is exercised against the raw high-frequency logs; and cfg5’s force is a robot-internal estimate rather than a physical sensor reading.

7 Conclusion↩︎

Kepler-Encoder-v0.1 fuses video, proprioception, and force/torque into a single latent with a learned-query cross-attention layer, trained by masked cross-modal latent prediction under LeJEPA/SIGReg, non-collapsing, prediction-based, and robust to dropped or hardware-absent sensors by construction. Our question was whether this buys anything over vision alone. On RH20T the answer is yes where the camera is weakest. The vision-only latent carries end-effector state, and force in particular, that raw frozen-ViT features do not, and it is the only feature whose latent geometry tracks state; on motor state, which the camera largely sees, it is statistically tied with the strongest vision baselines, and two controls rule out in-domain training and mere compression. One embodiment-agnostic encoder covers the four training robots, competitive with per-robot specialists and, at a matched data budget, still covering robots specialists cannot, so its breadth reflects embodiment diversity, not data volume; zero-shot transfer to unseen robots remains open. The frozen latent is directly useful. Its own cross-modal prediction error is a training-free monitor for invalid robot states, a natural hook for latent-level safety intervention, and a diffusion decoder reconstructs the camera frame from it, confirming the spatial compression preserves world-state. The clearest limitation is that everything here is single-timestep. The natural next steps are to fuse streams across time at their native rates (Appendix 13), which would also separate this encoder from robot self-supervised methods that assume synchronized observations and may restore read-out transfer under sensor dropout, and to build an action predictor on the frozen latent, the direction we expect to be most useful.

8 Preliminary: freezing the vision backbone↩︎

Before the main study, one preliminary experiment fixes a design choice, whether to finetune the vision backbone on robot video. It does not help. Continuing LeJEPA on cfg3 video has no effect. A high learning rate (2e-4) collapses the encoder (RankMe \(300\!\rightarrow\!158\)); a gentle rate (2e-5) restores health (RankMe \(\sim\)​285) but leaves robot-relevant probes flat (contact \(0.69\!\rightarrow\!0.67\)), and force \(R^2\approx0\) for every checkpoint. The apparent drop in a task-id probe (\(0.91\!\rightarrow\!0.74\)) is a saturated metric tracking ImageNet appearance, not a real regression, a first instance of over-reliance on a single metric. The lasting finding motivates the whole model. A single frame does not contain force, so vision alone cannot encode it. Hence the backbone is frozen (warm-started from the pretrained checkpoint) and force must enter through a different modality.

9 Frozen and finetuned ViT baselines↩︎

Table 13 reports the two 768-d ViT baselines, evaluated under the protocol of Section 5.1.4 (same splits, targets, and ridge probes; averages over the flexiv, ur5, and kuka held-out groups). PT-ViT is the frozen pretrained backbone of Section 5.1.3. The finetuned ViT is a full finetune of that backbone on our training split with the vision-only LeJEPA objective, at the highest learning rate that does not collapse the encoder (\(2\times10^{-5}\); Appendix 8), evaluated as a single run. The pretrained features set the strongest motor baseline (\(0.315\)). The full finetune lowers both probes relative to the frozen backbone; this result, together with the head-only finetune and the PCA 256 compression of Table 3, shows that in-domain adaptation of the backbone does not improve state recovery, in line with the freezing decision of Appendix 8.

10 Notation↩︎

Table ¿tbl:tab:notation? lists every symbol used in the Method (Section 3), together with the evaluation-specific symbols introduced in Section 5. Symbols are chosen so that no glyph carries two meanings. Two conventions resolve near-collisions. Case is significant, so \(M\) (query count) and \(m\) (the motor modality) are distinct, as are \(N\) (a token count) and any lowercase index. Weight is significant, so a plain \(e\) (the ee modality label) and a boldface \(\mathbf{e}_{(\cdot)}\) (a learned modality embedding) are distinct. The generic held-out modality is always indexed by \(k\), never by \(m\), so \(m\) refers only to the motor stream.

11 Single-embodiment fusion: cfg3 POC and cfg3\(+\)cfg4 scale-up↩︎

Before scaling to the full transfer matrix (Section 5.2.1), we fuse video \(+\) robot state at a single timestep on UR5 data and ask whether the fused latent beats each modality alone, and whether the gain is cross-modal rather than mere compression. A cross-modal signal gate confirms the premise (frozen vision \(\rightarrow\) state \(R^2=0.43\), TCP \(0.73\)), justifying the fusion encoder. We train the cross-attention fuser with masked cross-modal latent prediction \(+\) per-modal \(+\) joint SIGReg, and at evaluation feed vision only, probing to robot state. Table 14 reports two runs, a cfg3 proof-of-concept (\(\sim\)​24k frames) and a cfg3\(+\)cfg4 scale-up (86,430 frames / 2,881 scenes, \(3.6\times\) the data).

\(z_v\) beats raw vision by \(+0.294\pm0.078\) (POC) / \(+0.137\pm0.007\) (scale) and the PCA control by \(+0.417\pm0.050\) / \(+0.235\pm0.010\), positive on every seed. Two facts make this meaningful. First, beating the PCA control shows the gain is cross-modal, not compression, because a latent \(3\times\) smaller than raw vision predicts the robot better, while PCA 256, the same compression with no cross-modal training, loses. Second, only vision enters at eval, so the encoder has learned to read robot-relevant structure out of pixels by having been trained alongside state. This is exactly what a late-fusion recipe (train each modality’s encoder separately, then concatenate) cannot produce, because its vision features were never shaped by force or joints.

11.0.0.1 Why every baseline rises from cfg3 to cfg3\(+\)cfg4.

The margin over raw narrows in absolute terms (\(+0.294\!\rightarrow\!+0.137\)) but stays decisive over the compression control (\(+0.417\!\rightarrow\!+0.235\)). The baselines rise largely for reasons that are not “raw vision got better at decoding within-robot state.” (i) \(R^2\)’s denominator changed, since pooling two UR5 setups injects easy, pixel-readable between-config variance (different mounts/backgrounds), which any encoder captures before decoding any pose. (ii) The POC numbers are noisy/under-fit (\(\pm0.075\!\rightarrow\!\pm0.010\), \(7\times\) tighter). (iii) PCA 256 improved the most (\(+0.284\)), diagnostic of the new between-config variance landing in its top components. (iv) \(z_v\) rose the least (\(+0.102\)), since it was already near saturation, which is why the margin over raw narrows while the margin over compression stays large. The comparison carrying the cross-modal claim is \(z_v\) vs.the compression control, decisive in both runs.

12 Secondary ablations↩︎

Two further ablations, run on the ur5 specialist configuration (5-seed), included for completeness.

12.0.0.1 Bottleneck size and joint-SIGReg.

On ur5 the latent width has an optimum, where \(d{=}128\) gives \(0.356/0.147\) (motor/ee), \(d{=}256\) gives \(0.324/0.156\) (the ur5 specialist of Table ¿tbl:tab:phase1matrix?), and \(d{=}512\) degrades to \(0.255/0.122\) with unstable RankMe (some seeds collapse), so \(d{\in}[128,256]\) is preferred over a wider bottleneck. Removing the joint-SIGReg term slightly improves the ur5 probes (\(0.357/0.173\) vs.\(0.324/0.156\) with it), consistent with its role as insurance rather than a learning signal and with the healthy RankMe band (Section 5.2.1) showing \(z\) does not collapse at this scale; as noted in Section 3.4 we keep the term against collapse at wider bottlenecks, where the \(d{=}512\) instability above shows the pressure begins to matter.

12.0.0.2 Latent geometry (triplet accuracy and alignment/uniformity).

A triplet probe (positive \(=\) same scene, nearby tick; negatives laddered from temporally-distant same-scene to different-task) saturates for every feature at the cache’s \(\sim\)​1.5–3 s chunk spacing. Kepler-Encoder-v0.1 scores \(0.95\)–\(0.99\) across tiers (5 seeds) and every frozen vision baseline scores \(0.98\)–\(1.00\), since pixel-faithful features pass temporal-metadata tiers trivially, so the probe does not separate abstraction quality and the discriminating geometry measurement is the distance correlation of Section 5.1.4. Alignment and uniformity [59] describe the trade behind this. PT-ViT places temporal neighbors nearly on top of each other (alignment \(0.007\)) because all of its embeddings are similar (uniformity \(-0.68\)), while the fused latent pays a higher alignment (\(0.231\)) for far better spread (\(-2.04\)). A probe-based state-retrieval test (rank held-out candidates by predicted end-effector state) places the finetuned head last on every embodiment but mixes PT-ViT and the fused latent at the top with low absolute recall (R@10 \(\le 0.09\) against pools of 10–27k), so we do not use it as a headline metric.

13 Continuous-time, multi-rate extension (planned)↩︎

The single-timestep model of this report validates modality fusion; the central architectural hypothesis, left to future work, is fusing streams across time at their native rates. The data side already supports it (tick-anchored native-rate windows with per-token timestamps, Section 4); the remaining work is model-side (Figure 11). Beyond the motor and force payoff expected from time-derivatives, the read-out transfer failure of Table 11 gives this extension a second concrete target, restoring read-out transfer under transient sensor dropout (Section 6).

The continuous-time embedding encodes each token’s timestamp. We embed each token’s real timestamp with a Fourier / Time2Vec feature map [36] using fixed, log-spaced frequencies, upgradable to learned frequencies (mTAN proper [35]) behind the same interface. We deliberately take only mTAN’s continuous-time embedding and not the whole network, because the Perceiver already supplies mTAN’s other half. The learned queries are the reference points and cross-attention is attention-to-observations, a point COPER makes architecturally explicit [39]. The decisive knob is the frequency band, which must span \(\sim\)ms (100 Hz force) to seconds (multi-second episodes).

Tokenization varies by stream density. Dense force/torque (\(\sim\)​100–125 Hz) is tokenized by a 1-D CNN over time, which is locally near-regular within a chunk, with the validity mask carried as extra channels so that variable-DOF masking is preserved. Sparse streams such as vision and joints at \(\sim\)​10 Hz remain native tokens with continuous-time embeddings. Context windows are selected by \(\Delta t\) from cached timestamps, and masking spans (modality \(\times\) time) to predict held-out-time or future ee latents. Causality is a property of the attention mask, causal for the prediction head and any streaming deployment, with causal-versus-bidirectional tested for the representation objective, rather than something baked into the tokenizer.

13.0.0.1 SIGReg under time.

One caveat is specific to this extension. SIGReg’s Epps–Pulley statistic is a distributional test over a pool of samples that pushes the marginal embedding distribution toward isotropy, and applied to a time-pooled latent its isotropy constraint could wash out structured temporal variance. This does not threaten the design, because SIGReg is an anti-collapse regularizer, not the learning signal; temporal structure is carried by the prediction objective (masking over modality \(\times\) time, future-latent prediction) and the recurrent carry-forward, not by the regularizer. To keep the statistic from pooling away time we apply SIGReg to the per-timestep latent (each instant’s marginal), never to a time-pooled one, and treat its placement and strength as an on/off ablation.

A decisive ablation follows from this design. We compare continuous-time embedding with no resampling against resample-everything-then-1D-CNN, the classical DSP approach. This test validates or rejects the central claim, and we build both and adopt the simpler one if it wins.

References↩︎