1 Introduction↩︎

3D human motion prediction (HMP) forecasts future joint trajectories from an observed motion sequence. It is a core component for virtual characters, mixed reality avatars, assistive robots, and embodied agents that must react before a full human action has unfolded. The dominant progress in HMP has come from modeling motion itself: early recurrent and structural recurrent models [1]–[3], graph and attention networks [4]–[10], MLP-Mixer style predictors [11], and recent graph, transformer, and frequency-domain systems such as SimpliHuMoN and KHMP [12], [13]. Recent studies continue to refine this motion-only line through active perception, human-like inference, one-step formulations, interpretable graphs, and multi-resolution temporal modeling [14]–[22].

Pure motion sequences, however, are a narrow description of human behavior. Many future motions are easier to interpret once the observed motion is described at a semantic level: a person may be walking forward, turning, sitting down, or preparing to reach. Modern vision-language models (VLMs) can produce such descriptions from simple stick-figure renders, turning each motion clip into a lightweight natural-language prior. Text-to-motion generation has already shown the value of language as a motion-level control signal [23]–[28]. In prediction, the role of text is different: the observed motion remains the primary evidence, while language should quietly supply high-level context only when it helps.

This distinction motivates a design principle: text conditioning for HMP should be useful but hard to overfit. The observed motion sequence already carries most of the predictive signal, so language should serve as a complementary cue rather than a replacement for motion dynamics. For a well-tuned transformer predictor, it is natural to ask whether semantic captions can be used through a small and transparent interface.

We propose ZGL, a text-guided zero-gated cross-attention recipe for 3D HMP. Starting from a DCT-based spatial-temporal Transformer backbone, we encode a VLM-generated caption with a frozen plain CLIP-L text tower, project the caption embedding into a small set of conditioning tokens, and insert a cross-attention adapter after each motion encoder block. The critical choice is zero-gating: every adapter output is multiplied by a learnable gate initialised to zero [29], [30]. At initialization, the full model is numerically identical to the no-text baseline. During training, the adapter can learn to use text only where the caption supplies predictive information.

The contributions of this paper are:

• We introduce ZGL, a compact text-conditioning module for 3D human motion prediction that adds zero-gated cross-attention adapters to a strong DCT-based spatial-temporal Transformer backbone.

• We show that VLM-generated captions can improve a competitive Human3.6M predictor with a simple frozen CLIP-L text tower and a small set of conditioning tokens.

• We validate the same recipe on CMU-Mocap, showing that the method remains competitive when transferred to a second benchmark.

2 Method↩︎

2.1 Backbone↩︎

We take a DCT-based spatial-temporal Transformer baseline as our motion backbone [9], [11]. Given an observed window \(x \in \mathbb{R}^{B \times T_{\text{in}} \times J \times 3}\) of \(T_{\text{in}}\!=\!10\) subsampled frames and \(J\) joints (22 for Human3.6M and 25 for CMU), the input is first transformed into the discrete cosine transform (DCT) domain along the time axis following siMLPe [11]. A linear joint embedding lifts the per-joint coordinates to \(C\!=\!128\) channels and a learnable positional embedding is added.

The backbone is an \(N\!=\!6\)-block stack. Each block runs a parallel spatial attention and temporal attention, fused by a learned soft mixture. We re-use the published hyperparameters: 8 heads, MLP ratio 4, drop-path 0.2, LayerScale init \(10^{-5}\), learning rate \(5\!\times\!10^{-4}\) with LR decay 0.999, batch size 64, and 58 epochs of training. The output head is a two-layer MLP that maps the encoded sequence back to joint coordinates, followed by inverse DCT and a residual on the last input frame as in [11].

2.2 Text encoder and projection↩︎

Each input window is described by a one-sentence caption generated by Qwen2.5-VL applied to a stick-figure rendering of the observed motion only; future frames are not used for caption generation. The caption is encoded by a frozen CLIP-ViT-L/14 text tower into a single \(d_\text{text}\!=\!768\)-dimensional vector \(t \in \mathbb{R}^{d_\text{text}}\). We use the standard 77-token CLIP encoder; the impact of this choice relative to BGE-M3 and Long-CLIP-L is measured in Section 3.3. The text tower is never fine-tuned. This keeps the caption pathway separate from motion learning and avoids introducing a second large optimization problem into the motion predictor. It also makes the conditioning signal cheap to cache: each training window only needs one caption embedding, while all temporal reasoning remains inside the motion backbone.

A small projection head maps \(t\) into \(M\) conditioning tokens \[\mathbf{t}_{\text{kv}} = \texttt{LN}\bigl(\texttt{GELU}(W\,t + b)\bigr) \in \mathbb{R}^{M \times C},\] with a learnable null token \(t_\emptyset \in \mathbb{R}^{M \times C}\) initialised to zero. Following classifier-free guidance [35], during training we replace \(\mathbf{t}_{\text{kv}}\) by \(t_\emptyset\) for a fraction \(p\) of samples (we use \(p\!=\!0.1\)), so the model also learns an unconditional mode and cannot collapse onto the text channel alone.

2.3 Zero-gated cross-attention adapter↩︎

Our method, ZGL, inserts a text-guided zero-gated cross-attention adapter after every encoder block \(\ell\!\in\!\{0,\ldots,N{-}1\}\). The intermediate representation \(x_\ell \in \mathbb{R}^{B \times T \times J \times C}\) is augmented by an additive cross-attention adapter: \[\begin{align} q &= \texttt{LN}_q(x_\ell) \in \mathbb{R}^{B \times TJ \times C},\\ k,v &= \texttt{LN}_{kv}(\mathbf{t}_{\text{kv}}) \in \mathbb{R}^{B \times M \times C},\\ a &= \texttt{MHA}(q, k, v),\\ x_{\ell+1} &= x_\ell + \gamma \odot \texttt{Proj}(a), \label{eq:adapter} \end{align}\tag{1}\] where \(\texttt{MHA}\) is multi-head attention with 8 heads, \(\texttt{Proj}\) is a \(C \to C\) linear with dropout, and \(\gamma \in \mathbb{R}^{C}\) is a per-channel learnable gate initialised to zero [29], [30], [36], [37]. At initialisation \(\gamma = \mathbf{0}\) so the adapter is an exact identity and the model is numerically equivalent to the no-text Transformer baseline (Figure 1). The adapter has \(\approx 200\text{k}\) parameters per layer, increasing the model size from 4.5 M to 5.7 M parameters – less than a 27 % addition on top of the baseline. This identity start is important in HMP because the pose stream is already highly predictive. A non-zero text branch can disturb a well-trained motion prior before the caption features become useful. In contrast, the zero gate lets the optimizer first recover the pose-only solution and then open individual adapter channels only when cross-attention consistently reduces prediction error.

The set of layers that get an adapter is a hyperparameter, \(\mathcal{I}\!\subseteq\!\{0,\ldots,N{-}1\}\); we treat \(\mathcal{I}\!=\!\{0,\ldots,N{-}1\}\) (every layer) as our default and ablate over five subsets in Section 3.3.

2.4 Training objective↩︎

The training loss is the joint position MPJPE on the output sequence of length \(T_{\text{out}}\!=\!10\) plus three auxiliary terms from siMLPe. These auxiliary terms are the 3D velocity, bone-length, and mutual-information losses, with weights as published in [11]. We also keep mirror augmentation, time-reversal augmentation, and sample rate 2 identical to the no-text baseline.

3 Experiments↩︎

3.1 Datasets and evaluation protocol↩︎

3.1.0.1 Datasets.

We evaluate on Human3.6M [38] and CMU Mocap. We follow the standard Human3.6M split with S1, S6, S7, S8, and S9 for training, S11 for validation, and S5 for testing on 22 joints. CMU uses the common 25-joint evaluation setting adopted by recent HMP studies.

3.1.0.2 Metric.

We observe \(T_{\text{in}}\!=\!10\) frames and report MPJPE at the dataset-standard horizons: 80, 160, 320, 400, 560, and 1000 ms on Human3.6M and CMU Mocap. We also report short-AVG, long-AVG, and Total-AVG where applicable. We select the epoch with the best validation short-AVG.

3.1.0.3 Text features.

Qwen2.5-VL captions from stick-figure renders are encoded by frozen BGE-M3 [39], Long-CLIP-L [40], or plain CLIP-ViT-L/14 [41]; plain CLIP-L is default.

3.2 State-of-the-art comparisons↩︎

Table 1 compares ZGL with representative Human3.6M state-of-the-art methods using plain CLIP-L captions, \(M\!=\!4\), and every-layer injection. We select our model by validation short-AVG. ZGL achieves the lowest short-AVG, the lowest 560 ms error, and the best overall average in this comparison. The gains are concentrated in the short and middle horizons, where the observed motion still constrains the future and semantic intent can disambiguate similar local dynamics. At 1000 ms the method remains competitive, which is consistent with captions acting as high-level context rather than replacing long-range motion dynamics.

Table ¿tbl:tab:sota95cmu? compares ZGL with representative CMU-Mocap methods at 80, 160, 320, 400, and 1000 ms. For CMU, we reuse the Human3.6M architecture and hyperparameter setting: plain CLIP-L captions, \(M\!=\!4\), every-layer adapters, and dropout probability \(p\!=\!0.1\). Under this setting, ZGL obtains the best average MPJPE and the lowest error at the 80–400 ms horizons, while SPGSN remains stronger at 1000 ms.

3.3 Ablation studies↩︎

We vary one design axis at a time around the Table 1 recipe: plain CLIP-L, \(M\!=\!4\), every-layer injection, and dropout \(p\!=\!0.1\).

3.3.0.1 Conditioning source.

Table ¿tbl:tab:main? compares alternative real-text encoders with deterministic random and all-zero controls. The SOTA plain CLIP-L setting gives the best short-AVG, while random vectors degrade performance, indicating that the gain is not explained by extra adapter parameters alone. The all-zero row is close to the no-text baseline, as expected from the zero-gated initialization: without semantic variation in the text stream, the adapter has little useful signal to exploit. The random row is worse, showing that arbitrary conditioning vectors can make the adapter attend to non-semantic noise.

3.3.0.2 Injection placement.

Table [tbl:tab:abl95layer] tests whether text should be fused late by restricting adapters to different encoder-block subsets. The prefix scan keeps all other settings fixed and gradually exposes more motion blocks to the caption tokens. This isolates whether the caption should act only after motion features are deep, or whether earlier spatial-temporal representations can also benefit from the semantic prior. The all-layer setting performs best, but the prefix variants remain close, suggesting that text does not need a deep specialized branch but should be available throughout the motion stack.

3.3.0.3 Token capacity.

Table [tbl:tab:abl95M] varies the number of conditioning tokens \(M\!\in\!\{1,2,4,8\}\). The adapter uses these tokens only as keys and values, so increasing \(M\) expands the text memory seen by motion queries without changing the motion-token length. The best setting is \(M\!=\!4\), suggesting that one global caption vector is too compressed, while a larger token budget does not add useful information for the short captions used here.

4 Conclusion↩︎

We presented ZGL, a simple recipe for text-conditioned 3D human motion prediction: a zero-gated cross-attention adapter after every Transformer encoder block, conditioned by tokens projected from a frozen plain CLIP-L caption embedding. The method improves the no-text baseline on Human3.6M with a lightweight adapter pathway. The main finding is that language does not need to dominate the motion model to be useful. When captions are generated from the observed motion only and injected through zero-initialized gates, the backbone can keep its original dynamics at the start of training and gradually learn where semantic context helps. The ablations support this view: real caption features outperform random or zero controls, all-layer injection is more effective than late-only fusion, and a small set of conditioning tokens is sufficient for concise motion descriptions.

These results suggest a practical direction for language-assisted HMP. Instead of building a separate text-heavy prediction system, a strong DCT-based spatial-temporal Transformer can be extended with a compact semantic interface that is easy to train and inexpensive at inference once captions are precomputed. Future work can study richer caption generation, online visual observations, and more explicit handling of ambiguous future motion while keeping the motion backbone as the central predictor.

References↩︎