Enhancing Human-Likeness in Reinforcement Learning Agents via Hierarchical Macro Action Quantization
May 29, 2026
Abstract
Human-like agents are a long-standing goal of artificial intelligence. Despite strong performance, most reinforcement learning (RL) agents remain reward-driven and often exhibit behaviors that differ from humans, limiting interpretability and reliability. In this work, we introduce a novel human-like RL framework that predicts action sequences closely aligned with human behaviors while maximizing rewards. Specifically, we encode human demonstrations into macro actions using a hierarchical macro action quantization approach (termed HiMAQ) consisting of two successive levels of vector quantization. The lower quantization level maps input actions to fine-grained subaction clusters, while the higher quantization level aggregates these subaction clusters into action clusters. Extensive evaluations on the D4RL benchmarks show that our hierarchical approach outperforms the non-hierarchical baseline (MAQ), achieving better human-likeness scores while maintaining comparable or better success rates than previous RL agents. The improvements generalize across integrations with various RL algorithms, namely IQL, SAC, and RLPD.
1 Introduction↩︎
As robots take on increasingly complex tasks alongside humans [1]–[6], a key expectation emerges: motion that is not merely successful, but recognizably human-like. Conventional reward-driven RL policies routinely produce unnatural, jerky motions [7]–[14] that erode trust and fail to transfer the rich knowledge encoded in human demonstrations.
Recent work has begun to close this gap by grounding RL within the space of human demonstrations. Human-like receding horizon control (HRC) [15] constrains trajectory search to human-generated segments, producing more structured motion. MAQ [16] extends this by encoding human trajectory segments into a discrete codebook via a conditional vector quantized variational autoencoder (VQ-VAE) [17], allowing a policy to select from a compact set of learned macro actions. However, MAQ’s single-level codebook treats each macro action as a single atomic prototype, limiting its ability to capture the full variation within each motion segment. A reaching motion, for instance, spans a broad approach, fine grasp alignment, and controlled contact — collapsing all of this into one prototype loses the subtle dynamics that make motion feel human.
We propose Hierarchical Macro Action Quantization (HiMAQ), inspired by MAQ [16] and hierarchical VQ-VAE [18], a two-level vector quantization framework that captures human demonstrations at both the macro action level and the fine-grained subaction level. The key insight is that instead of quantizing an encoded trajectory directly to an action prototype, HiMAQ first maps it to a fine-grained subaction prototype, which is then mapped to a coarser action prototype, preserving both coarse structure and fine-grained detail. Evaluated on four dexterous manipulation tasks from D4RL’s Adroit benchmark [5] across three RL algorithms (IQL [19], SAC [20], and RLPD [6]), HiMAQ outperforms MAQ [16] on trajectory similarity metrics, while yielding comparable or better success rates. In a human study with 22 evaluators, HiMAQ+RLPD achieves a Turing test win rate of 43%, and HiMAQ agents outperform all competing methods in the human-likeness ranking test.
In summary, our contributions are:
HiMAQ, a hierarchical macro action quantization framework that encodes human demonstrations through two levels of vector quantization — subaction prototypes capturing fine-grained structure, and action prototypes capturing coarse motion patterns — enabling richer human motion representation than single-level approaches.
Consistent human-likeness gains across three RL algorithms. HiMAQ improves trajectory similarity (DTW and Wasserstein distance) over MAQ [16] under IQL [19], SAC [20], and RLPD [6], while matching or improving task success rates — including a significant \(0.00 \to 0.87\) success rate gain on the Hammer task under IQL.
Perceptual validation via human study. With 22 human evaluators across Turing and ranking tests, HiMAQ variants outperform MAQ [16] counterparts, confirming that quantitative trajectory improvements translate to perceptually recognizable human-likeness.
2 Related Work↩︎
Human-Likeness in Reinforcement Learning. RL often prioritizes reward optimization over realistic behavior generation. Human-like RL approaches can be classified into two groups. One line of work introduces constraints or penalties to suppress unnatural behaviors, e.g., Fujii et al. [21] penalize non-human actions, while Ho et al. [22] design penalties for unrealistic movements. Despite great performance, the above methods rely heavily on manually designed rules. Another direction learns from human demonstrations collected in domains such as gaming [23], autonomous driving [24], and robotics [5]. Techniques such as behavior cloning and inverse RL [25], [26] can transfer human behavioral traits to agents, but are often limited by dataset quality and diversity. Recent work models behavior through learned macro actions, e.g., MAQ [16] learns discrete action chunks from human trajectories using VQ-VAE [17]. In this work, we propose a hierarchical macro action quantization framework, yielding better human-likeness scores without sacrificing task performance.
Macro Action Representation Learning. To enable temporally extended decision-making, MAQ [16] adopts Semi-Markov Decision Processes (SMDPs). Formally, at state \(\boldsymbol{s}_t\), the agent executes a macro action \(\mathbf{m}_t=(\mathbf{a}_t,\mathbf{a}_{t+1},\dots,\mathbf{a}_{t+H-1})\) over \(H\) time steps. SMDP is defined by \((\mathcal{S},\mathcal{M},R,P,\gamma)\), where \(\mathcal{S}\) is the state space and \(\mathcal{M}\) is the set of macro actions. Executing \(\boldsymbol{m}_t\) from state \(\boldsymbol{s}_t\) yields an aggregate reward collected across the multi-step execution interval, specified as \(R(\mathbf{s}_t,\mathbf{m}_t)=\sum_{k=t}^{t+H-1} r(\mathbf{s}_k,\mathbf{a}_k)\) [27]. The policy objective is to maximize the expected discounted return over these macro actions: \(J(\pi_\theta)=\mathbb{E}_{(\mathbf{s}_t,\mathbf{m}_t)\sim\pi_\theta}\left[\sum_{t=0}^{\infty}\gamma^t R(\mathbf{s}_t,\mathbf{m}_t)\right]\). By replacing fine-grained actions with macro actions, SMDPs enable long-horizon planning. This reduces decision frequency while preserving long-term behavioral structure. Our work follows this line of research.
Vector Quantized Variational Autoencoders. Building on the classical variational autoencoder (VAE) [28], VQ-VAE [17] introduces a discrete latent codebook. An encoder maps inputs into continuous embeddings, which are quantized to the nearest codebook vectors, while a decoder reconstructs the input from these discrete tokens. Such representations yield compact, structured features suitable for sequential modeling and reinforcement learning. For example, Ozair et al. [29] and Antonoglou et al. [30] deploy conditional VQ-VAEs to model stochastic environments, while Luo et al. [31] project continuous action spaces into discrete, reusable action primitives to facilitate policy optimization. Hierarchical variants further capture behaviors at multiple abstraction levels; for example, Spurio et al. [18] separate low-level subactions from high-level strategies. Inspired by the success of [18] in temporal action segmentation, we develop a hierarchical macro action quantization framework to improve human-likeness in RL agents.
3 Our Approach↩︎
3.1 Human-Like Trajectory Optimization↩︎
Robot manipulation tasks require agents to generate smooth, human-like trajectories by learning from human demonstrations, where trajectory optimization provides a principled framework for this purpose.Trajectory optimization is defined under a Markov Decision Process (MDP) \((\mathcal{S}, \mathcal{A}, \mathcal{P}, R, \gamma, T, p_0)\), where \(\mathcal{S}\) and \(\mathcal{A}\) denote the continuous state and action spaces, respectively. The transition dynamics are given by the policy \(\mathcal{P} : \mathcal{S} \times \mathcal{A} \rightarrow \Delta(\mathcal{S})\), with reward function \(R : \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}\), discount factor \(\gamma \in [0,1)\), episode length \(T\), and initial state distribution \(p_0\). The goal is to find an action sequence that maximizes the expected cumulative discounted return over the trajectory. The human-like receding horizon control (HRC) framework [15] is used for trajectory optimization over sequences of actions, where at each timestep \(t\), a sequence of \(H\) actions (\(H\) is much smaller than \(T\)) is optimized jointly. Replanning is performed every \(j\) steps, with \(1 \leq j \leq H\), after executing the first \(j\) actions of \(\boldsymbol{a}^*_{t:t+H-1}\). Human-likeness is enforced by restricting the search space to human-generated trajectory segments \(\mathcal{H}\) sampled from a dataset \(\mathcal{D} = \{\tau^{(i)}\}_{i=1}^{N}\): \[\boldsymbol{a}_{t:t+H-1}^{*} = {\arg\max}_{\boldsymbol{a}_{t:t+H-1}\in\mathcal{H}} \mathbb{E} \bigg[ \sum_{i=t}^{t+H-1}\gamma^{i-t}R(\boldsymbol{s}_i,\boldsymbol{a}_i) \Big\rvert \boldsymbol{s}_t; \boldsymbol{a}_{t:t+H-1} \bigg]. \label{eq:HRC}\tag{1}\] Replanning after every \(j\) steps is computationally expensive due to repeated searching over all human trajectory segments. Furthermore, the current state \(\boldsymbol{s}_t\) rarely exactly matches the starting state of stored demonstration segments, making direct reuse of continuous action trajectories unreliable. MAQ [16] addresses these limitations by representing behavior as fixed-length \(H\)-step action sequences \((\boldsymbol{a}_t, \boldsymbol{a}_{t+1}, \ldots, \boldsymbol{a}_{t+H-1})\), which are encoded into discrete latent codes using a conditional VQ-VAE [17]. Given a macro action segment, the encoder maps the trajectory into a latent embedding, which is then quantized to the nearest codebook vector \(\boldsymbol{e}_t\). The corresponding discrete code, together with the current state \(\boldsymbol{s}_t\), is passed to the decoder to reconstruct the macro action sequence \(\widetilde{\boldsymbol{m}}_t\). Despite its effectiveness, MAQ [16] employs a single-level discrete codebook over entire \(H\)-step trajectories, which limits its ability to capture fine-grained subactions within macro actions. This coarse discretization can result in loss of subtle variations in complex behaviors. HiMAQ introduces a subaction codebook to capture finer structures within each macro action segment.
3.2 Human Demonstration Encoding with HiMAQ↩︎
Inspired by the success of HVQ [18] in temporal action segmentation, HiMAQ represents macro actions from offline human demonstrations using a two-level hierarchical codebook structure, \(\boldsymbol{Z} = \{\boldsymbol{z}_j\}_{j=1}^{\alpha K}\) for subactions and \(\boldsymbol{A} = \{\boldsymbol{a}_i\}_{i=1}^{K}\) for actions, as illustrated in Fig. 1 (a). Here, \(K\) denotes the number of action codebook entries and \(\alpha\) is a ratio parameter controlling the size of the subaction codebook, where \(\boldsymbol{A}\) contains \(K\) action prototypes and \(\boldsymbol{Z}\) contains \(\alpha K\) subaction prototypes. The encoder takes the state \(\boldsymbol{s}_t\) and macro action \(\boldsymbol{a}_{t:t+H-1}\) as input and generates the latent embedding \(\boldsymbol{e}_t\).
The first level of quantization maps the encoder embedding \(\boldsymbol{e}_t\) to its nearest prototype \(\boldsymbol{z}_{j^*} \in \boldsymbol{Z}\) using Euclidean distance, producing the quantized subaction representation \(\boldsymbol{q}^\boldsymbol{Z}_t\) as: \[\boldsymbol{q}^\boldsymbol{Z}_t = \boldsymbol{z}_{j^*}, \quad j^* = \arg\min_{j} \|\boldsymbol{e}_t - \boldsymbol{z}_j\|^2.\] The quantized subaction representation \(\boldsymbol{q}^\boldsymbol{Z}_t\) is then passed to the second level, where the nearest prototype \(\boldsymbol{a}_{i^*} \in \boldsymbol{A}\) is computed using Euclidean distance, producing the action representation \(\boldsymbol{q}^\boldsymbol{A}_t\) as: \[\boldsymbol{q}^\boldsymbol{A}_t = \boldsymbol{a}_{i^*}, \quad i^* = \arg\min_{i} \|\boldsymbol{q}^\boldsymbol{Z}_t - \boldsymbol{a}_i\|^2.\] Finally, \(\boldsymbol{q}^\boldsymbol{A}_t\) along with the state \(\boldsymbol{s}_t\) is passed to the decoder to reconstruct the macro action \(\widetilde{\boldsymbol{m}}_t\).
To train HiMAQ, we use a combination of reconstruction loss, commitment losses, and codebook losses, which are defined as follows: \[\mathcal{L}_{recon} = \|\boldsymbol{m}_t - \widetilde{\boldsymbol{m}}_t\|^2,\] \[\mathcal{L}_{commit_Z} = \|\boldsymbol{e}_t - sg[\boldsymbol{q}^\boldsymbol{Z}_t]\|^2, \qquad \mathcal{L}_{commit_A} = \|\boldsymbol{q}^\boldsymbol{Z}_t - sg[\boldsymbol{q}^\boldsymbol{A}_t]\|^2,\] \[\mathcal{L}_{codebook_Z} = \|sg[\boldsymbol{e}_t] - \boldsymbol{q}^\boldsymbol{Z}_t\|^2, \qquad \mathcal{L}_{codebook_A} = \|sg[\boldsymbol{q}^\boldsymbol{Z}_t] - \boldsymbol{q}^\boldsymbol{A}_t\|^2.\] Here, the reconstruction loss \(\mathcal{L}_{recon}\) measures the discrepancy between the original macro action \(\boldsymbol{m}_t\) and reconstructed macro action \(\widetilde{\boldsymbol{m}}_t\). Next, the commitment loss \(\mathcal{L}_{commit_Z}\) pushes the encoder embedding \(\boldsymbol{e}_t\) towards the closest prototype in codebook \(\boldsymbol{Z}\), while the commitment loss \(\mathcal{L}_{commit_A}\) pushes the quantized representation \(\boldsymbol{q}^\boldsymbol{Z}_t\) towards the nearest prototype in codebook \(\boldsymbol{A}\). Lastly, the codebook losses \(\mathcal{L}_{codebook_Z}\) and \(\mathcal{L}_{codebook_A}\) update codebooks \(\boldsymbol{Z}\) and \(\boldsymbol{A}\) by pulling \(\boldsymbol{q}^\boldsymbol{Z}_t\) towards \(\boldsymbol{e}_t\) and \(\boldsymbol{q}^\boldsymbol{A}_t\) towards \(\boldsymbol{q}^\boldsymbol{Z}_t\) respectively. The stop-gradient operator \(sg[\cdot]\) ensures that only codebook parameters are updated, preventing gradient flow back to the encoder or the previous quantization level.
The total training loss combines the reconstruction loss \(\mathcal{L}_{recon}\), the commitment losses weighted by \(\lambda_{commit}\), and the codebook losses weighted by \(\lambda_{codebook}\), across both quantization levels as: \[\mathcal{L} = \mathcal{L}_{recon} + \lambda_{commit}(\mathcal{L}_{commit_Z} + \mathcal{L}_{commit_A}) + \lambda_{codebook}(\mathcal{L}_{codebook_Z} + \mathcal{L}_{codebook_A}).\] For our experiments, we set \(\lambda_{commit} = 0.25\) and \(\lambda_{codebook} = 1.0\), following MAQ [16]. As we will show in Sec. 4, HiMAQ achieves higher human-likeness scores while maintaining similar or higher success rates than prior works. Imposing trajectory smoothness (e.g., Lipschitz regularization [32]) may further improve human-likeness, which remains part of our future work.
3.3 Reinforcement Learning with HiMAQ↩︎
We train a policy \(\pi_\theta\) to select an index from the subaction codebook \(\boldsymbol{Z}\), producing \(\boldsymbol{q}^\boldsymbol{Z}_t\). The policy \(\pi_\theta\) takes the current state \(\boldsymbol{s}_t\) as input and produces a distribution over \(\alpha K\) logits, each corresponding to one subaction prototype in \(\boldsymbol{Z}\). At each state \(\boldsymbol{s}_t\), \(\pi_\theta\) samples an index from this distribution, retrieving the corresponding \(\boldsymbol{q}^\boldsymbol{Z}_t\). This subaction code \(\boldsymbol{q}^\boldsymbol{Z}_t\) is then used to retrieve the action prototype \(\boldsymbol{q}^\boldsymbol{A}_t\) from codebook \(\boldsymbol{A}\). With \(\boldsymbol{q}^\boldsymbol{A}_t\) and \(\boldsymbol{s}_t\), the decoder reconstructs the macro action \(\widetilde{\boldsymbol{m}}_t\) to interact with the environment, as illustrated in Fig. 1 (b). The policy \(\pi_\theta\) is optimized to find the macro action that maximizes the expected cumulative reward within the human manifold: \[\boldsymbol{m}^*_t = {\arg\max}_{\boldsymbol{m}_t \in \mathcal{H}} \; \mathbb{E}\left[\sum_{i=t}^{t+H-1} \gamma^{i-t} R(\boldsymbol{s}_i, \boldsymbol{a}_i) \;\bigg|\; \boldsymbol{s}_t;\; \boldsymbol{m}_t\right].\] Unlike MAQ [16], which encodes segments from \(\mathcal{H}\) using a single-level codebook, HiMAQ encodes them through a two-level hierarchical quantization, enabling finer-grained human-like behavior.
4 Experiments↩︎
We evaluate HiMAQ on four Adroit tasks [5], showing consistent human-likeness improvements over MAQ [16] across three RL algorithms. For fair comparisons, our experiments share the same tasks, datasets, evaluation metrics, and competing methods with MAQ [16].
Implementation Details. We evaluate on four Adroit tasks from D4RL [5] using three RL algorithms: IQL [19], SAC [20], and RLPD [6], each augmented with MAQ [16] and HiMAQ to yield six human-like agents. A behavioral cloning (BC) [33] agent serves as an imitation-learning reference. Of the ten agents evaluated, seven — BC, IQL, SAC, RLPD, MAQ+IQL, MAQ+SAC, and MAQ+RLPD — are taken directly from Guo et al. [16]; the three HiMAQ variants (HiMAQ+IQL, HiMAQ+SAC, HiMAQ+RLPD) are our contributions. All agents are trained for \(10^6\) steps; IQL additionally receives \(10^6\) offline steps. HiMAQ uses a hierarchical VQ-VAE with two codebooks, while MAQ uses a single-level one. SAC variants use discrete SAC (DSAC) [34] to accommodate the discrete action space. Full hyperparameter settings are provided in the supplementary material.
Datasets. We evaluate on four Adroit dexterous manipulation tasks from D4RL [5], which provide both task environments and human demonstration datasets collected via teleoperation [35]. Each task is controlled by a 24-DOF simulated hand and poses distinct manipulation challenges: Door requires the agent to grasp a handle and swing open a hinged door; Hammer requires the agent to pick up a hammer and drive a nail into a board; Pen requires the agent to reorient a pen in-hand to match a target orientation; and Relocate requires the agent to pick up a ball and move it to a target location. Each task includes 25 human demonstration trajectories, which are split into training and testing sets with a 9:1 ratio. All methods that rely on human demonstrations are trained on the same training split, and trajectory similarity is evaluated on the test split, following Guo et al. [16].
Evaluation Metrics. We follow Guo et al. [16] to adopt the same two trajectory similarity metrics to enable direct comparisons. Dynamic Time Warping (DTW) measures sequential alignment between agent and human trajectories, evaluated separately on states (\(\text{DTW}_s\)) and actions (\(\text{DTW}_a\)). Wasserstein Distance (WD) measures distributional similarity between agent and human trajectories, likewise split into state-based (\(\text{WD}_s\)) and action-based (\(\text{WD}_a\)) variants. For all four metrics, higher normalized scores indicate more human-like behavior. Full metric definitions and computation details are provided in the supplementary material.
4.1 Quantitative Trajectory Evaluations↩︎
4.1.1 State-of-the-Art Comparisons↩︎
Tabs. 1, 2, and 3 report results across Adroit tasks under IQL [19], SAC [20], and RLPD [6] baselines. Each table reports results for BC [33], base RL agent, and MAQ [16] and HiMAQ variants. HiMAQ outperforms MAQ [16] on trajectory similarity metrics, while improving success rates. For IQL baseline (Tab. 1), HiMAQ+IQL improves average \(\text{DTW}_s\) from 0.63 to 0.65 and success rate from 0.39 to 0.56, with a particularly large gain on Hammer (success rate from 0.00 to 0.87). Next, for SAC baseline (Tab. 2), HiMAQ+SAC improves average \(\text{DTW}_s\) from 0.56 to 0.63 and success rate from 0.28 to 0.29. Lastly, for RLPD baseline (Tab. 3), HiMAQ+RLPD improves average \(\text{DTW}_s\) from 0.56 to 0.62 and success rate from 0.52 to 0.56. Although the unconstrained RLPD policy achieves a higher success rate (0.68) than HiMAQ+RLPD (0.56) by not restricting actions to the human manifold, HiMAQ+RLPD (0.56) outperforms the closest competitor MAQ+RLPD (0.52) while improving trajectory similarity metrics. The above results validate the benefits of modeling subtle variations within macro actions via our hierarchical macro action quantization framework.
| Tasks | BC | IQL | MAQ+IQL | HiMAQ+IQL | |
|---|---|---|---|---|---|
| Door | \(\text{DTW}_s(\uparrow)\) | 0.18 \(\pm\) 0.09 | 0.43 \(\pm\) 0.06 | 0.84 \(\pm\) 0.06 | 0.87 \(\pm\) 0.08 |
| \(\text{DTW}_a(\uparrow)\) | 0.42 \(\pm\) 0.13 | 0.61 \(\pm\) 0.04 | 0.95 \(\pm\) 0.01 | 0.96 \(\pm\) 0.01 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.05 | 0.48 \(\pm\) 0.05 | 0.75 \(\pm\) 0.05 | 0.77 \(\pm\) 0.05 | |
| \(\text{WD}_a(\uparrow)\) | 0.41 \(\pm\) 0.08 | 0.50 \(\pm\) 0.02 | 0.81 \(\pm\) 0.03 | 0.82 \(\pm\) 0.03 | |
| \(\text{Success}(\uparrow)\) | 0.02 \(\pm\) 0.01 | 0.16 \(\pm\) 0.06 | 0.93 \(\pm\) 0.04 | 0.95 \(\pm\) 0.04 | |
| Hammer | \(\text{DTW}_s(\uparrow)\) | -0.16 \(\pm\) 0.07 | -0.14 \(\pm\) 0.34 | 0.64 \(\pm\) 0.17 | 0.68 \(\pm\) 0.12 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.02 | 0.45 \(\pm\) 0.20 | 0.92 \(\pm\) 0.06 | 0.94 \(\pm\) 0.05 | |
| \(\text{WD}_s(\uparrow)\) | 0.11 \(\pm\) 0.06 | 0.12 \(\pm\) 0.13 | 0.75 \(\pm\) 0.03 | 0.73 \(\pm\) 0.04 | |
| \(\text{WD}_a(\uparrow)\) | 0.30 \(\pm\) 0.03 | 0.30 \(\pm\) 0.10 | 0.84 \(\pm\) 0.02 | 0.87 \(\pm\) 0.02 | |
| \(\text{Success}(\uparrow)\) | 0.00 \(\pm\) 0.00 | 0.01 \(\pm\) 0.01 | 0.00 \(\pm\) 0.00 | 0.87 \(\pm\) 0.06 | |
| Pen | \(\text{DTW}_s(\uparrow)\) | 0.53 \(\pm\) 0.13 | 0.34 \(\pm\) 0.09 | 0.55 \(\pm\) 0.17 | 0.57 \(\pm\) 0.11 |
| \(\text{DTW}_a(\uparrow)\) | 0.58 \(\pm\) 0.05 | 0.51 \(\pm\) 0.05 | 0.58 \(\pm\) 0.09 | 0.60 \(\pm\) 0.06 | |
| \(\text{WD}_s(\uparrow)\) | 0.59 \(\pm\) 0.12 | 0.54 \(\pm\) 0.11 | 0.59 \(\pm\) 0.13 | 0.61 \(\pm\) 0.10 | |
| \(\text{WD}_a(\uparrow)\) | 0.65 \(\pm\) 0.14 | 0.63 \(\pm\) 0.12 | 0.66 \(\pm\) 0.15 | 0.68 \(\pm\) 0.11 | |
| \(\text{Success}(\uparrow)\) | 0.40 \(\pm\) 0.03 | 0.40 \(\pm\) 0.05 | 0.42 \(\pm\) 0.07 | 0.66 \(\pm\) 0.02 | |
| Relocate | \(\text{DTW}_s(\uparrow)\) | 0.09 \(\pm\) 0.14 | 0.20 \(\pm\) 0.20 | 0.52 \(\pm\) 0.06 | 0.52 \(\pm\) 0.04 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.15 | 0.51 \(\pm\) 0.11 | 0.82 \(\pm\) 0.01 | 0.84 \(\pm\) 0.04 | |
| \(\text{WD}_s(\uparrow)\) | 0.27 \(\pm\) 0.15 | 0.36 \(\pm\) 0.06 | 0.47 \(\pm\) 0.07 | 0.46 \(\pm\) 0.05 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.12 | 0.50 \(\pm\) 0.04 | 0.65 \(\pm\) 0.03 | 0.65 \(\pm\) 0.02 | |
| \(\text{Success}(\uparrow)\) | 0.01 \(\pm\) 0.02 | 0.00 \(\pm\) 0.00 | 0.20 \(\pm\) 0.10 | 0.07 \(\pm\) 0.05 | |
| Average | \(\text{DTW}_s(\uparrow)\) | 0.16 \(\pm\) 0.29 | 0.21 \(\pm\) 0.25 | 0.63 \(\pm\) 0.14 | 0.65 \(\pm\) 0.16 |
| \(\text{DTW}_a(\uparrow)\) | 0.49 \(\pm\) 0.07 | 0.52 \(\pm\) 0.06 | 0.82 \(\pm\) 0.17 | 0.83 \(\pm\) 0.17 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.20 | 0.38 \(\pm\) 0.18 | 0.64 \(\pm\) 0.14 | 0.63 \(\pm\) 0.14 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.15 | 0.48 \(\pm\) 0.14 | 0.74 \(\pm\) 0.10 | 0.75 \(\pm\) 0.11 | |
| \(\text{Success}(\uparrow)\) | 0.11 \(\pm\) 0.19 | 0.14 \(\pm\) 0.19 | 0.39 \(\pm\) 0.40 | 0.56 \(\pm\) 0.36 |
| Tasks | BC | SAC | MAQ+SAC | HiMAQ+SAC | |
|---|---|---|---|---|---|
| Door | \(\text{DTW}_s(\uparrow)\) | 0.18 \(\pm\) 0.09 | -0.39 \(\pm\) 0.10 | 0.80 \(\pm\) 0.08 | 0.83 \(\pm\) 0.03 |
| \(\text{DTW}_a(\uparrow)\) | 0.42 \(\pm\) 0.13 | -0.25 \(\pm\) 0.04 | 0.91 \(\pm\) 0.03 | 0.92 \(\pm\) 0.02 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.05 | -0.28 \(\pm\) 0.02 | 0.71 \(\pm\) 0.08 | 0.75 \(\pm\) 0.03 | |
| \(\text{WD}_a(\uparrow)\) | 0.41 \(\pm\) 0.08 | -0.15 \(\pm\) 0.02 | 0.77 \(\pm\) 0.07 | 0.80 \(\pm\) 0.02 | |
| \(\text{Success}(\uparrow)\) | 0.02 \(\pm\) 0.01 | 0.43 \(\pm\) 0.23 | 0.56 \(\pm\) 0.50 | 0.50 \(\pm\) 0.29 | |
| Hammer | \(\text{DTW}_s(\uparrow)\) | -0.16 \(\pm\) 0.07 | -1.10 \(\pm\) 0.35 | 0.61 \(\pm\) 0.21 | 0.64 \(\pm\) 0.14 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.02 | -0.33 \(\pm\) 0.11 | 0.91 \(\pm\) 0.10 | 0.92 \(\pm\) 0.07 | |
| \(\text{WD}_s(\uparrow)\) | 0.11 \(\pm\) 0.06 | -0.44 \(\pm\) 0.09 | 0.64 \(\pm\) 0.12 | 0.67 \(\pm\) 0.07 | |
| \(\text{WD}_a(\uparrow)\) | 0.30 \(\pm\) 0.03 | -0.19 \(\pm\) 0.04 | 0.78 \(\pm\) 0.11 | 0.82 \(\pm\) 0.02 | |
| \(\text{Success}(\uparrow)\) | 0.00 \(\pm\) 0.00 | 0.01 \(\pm\) 0.01 | 0.00 \(\pm\) 0.00 | 0.00 \(\pm\) 0.00 | |
| Pen | \(\text{DTW}_s(\uparrow)\) | 0.53 \(\pm\) 0.13 | 0.06 \(\pm\) 0.20 | 0.58 \(\pm\) 0.17 | 0.58 \(\pm\) 0.12 |
| \(\text{DTW}_a(\uparrow)\) | 0.58 \(\pm\) 0.05 | -0.34 \(\pm\) 0.16 | 0.58 \(\pm\) 0.11 | 0.58 \(\pm\) 0.05 | |
| \(\text{WD}_s(\uparrow)\) | 0.59 \(\pm\) 0.12 | 0.29 \(\pm\) 0.10 | 0.61 \(\pm\) 0.12 | 0.61 \(\pm\) 0.10 | |
| \(\text{WD}_a(\uparrow)\) | 0.65 \(\pm\) 0.14 | 0.22 \(\pm\) 0.15 | 0.67 \(\pm\) 0.14 | 0.67 \(\pm\) 0.10 | |
| \(\text{Success}(\uparrow)\) | 0.40 \(\pm\) 0.03 | 0.32 \(\pm\) 0.09 | 0.41 \(\pm\) 0.01 | 0.51 \(\pm\) 0.04 | |
| Relocate | \(\text{DTW}_s(\uparrow)\) | 0.09 \(\pm\) 0.14 | -0.55 \(\pm\) 0.20 | 0.25 \(\pm\) 0.19 | 0.46 \(\pm\) 0.07 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.15 | -0.10 \(\pm\) 0.16 | 0.66 \(\pm\) 0.09 | 0.80 \(\pm\) 0.05 | |
| \(\text{WD}_s(\uparrow)\) | 0.27 \(\pm\) 0.15 | -0.22 \(\pm\) 0.06 | 0.40 \(\pm\) 0.07 | 0.47 \(\pm\) 0.08 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.12 | -0.05 \(\pm\) 0.03 | 0.61 \(\pm\) 0.03 | 0.66 \(\pm\) 0.05 | |
| \(\text{Success}(\uparrow)\) | 0.01 \(\pm\) 0.02 | 0.00 \(\pm\) 0.00 | 0.14 \(\pm\) 0.07 | 0.16 \(\pm\) 0.08 | |
| Average | \(\text{DTW}_s(\uparrow)\) | 0.16 \(\pm\) 0.29 | -0.49 \(\pm\) 0.48 | 0.56 \(\pm\) 0.23 | 0.63 \(\pm\) 0.15 |
| \(\text{DTW}_a(\uparrow)\) | 0.49 \(\pm\) 0.07 | -0.26 \(\pm\) 0.11 | 0.76 \(\pm\) 0.17 | 0.81 \(\pm\) 0.16 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.20 | -0.17 \(\pm\) 0.32 | 0.59 \(\pm\) 0.14 | 0.63 \(\pm\) 0.12 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.15 | -0.04 \(\pm\) 0.18 | 0.71 \(\pm\) 0.08 | 0.74 \(\pm\) 0.08 | |
| \(\text{Success}(\uparrow)\) | 0.11 \(\pm\) 0.19 | 0.19 \(\pm\) 0.22 | 0.28 \(\pm\) 0.25 | 0.29 \(\pm\) 0.25 |
| Tasks | BC | RLPD | MAQ+RLPD | HiMAQ+RLPD | |
|---|---|---|---|---|---|
| Door | \(\text{DTW}_s(\uparrow)\) | 0.18 \(\pm\) 0.09 | -0.06 \(\pm\) 0.04 | 0.76 \(\pm\) 0.04 | 0.83 \(\pm\) 0.07 |
| \(\text{DTW}_a(\uparrow)\) | 0.42 \(\pm\) 0.13 | 0.28 \(\pm\) 0.08 | 0.91 \(\pm\) 0.05 | 0.93 \(\pm\) 0.01 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.05 | -0.14 \(\pm\) 0.04 | 0.71 \(\pm\) 0.03 | 0.76 \(\pm\) 0.04 | |
| \(\text{WD}_a(\uparrow)\) | 0.41 \(\pm\) 0.08 | 0.10 \(\pm\) 0.02 | 0.76 \(\pm\) 0.03 | 0.79 \(\pm\) 0.03 | |
| \(\text{Success}(\uparrow)\) | 0.02 \(\pm\) 0.01 | 0.96 \(\pm\) 0.07 | 0.93 \(\pm\) 0.05 | 0.93 \(\pm\) 0.10 | |
| Hammer | \(\text{DTW}_s(\uparrow)\) | -0.16 \(\pm\) 0.07 | -0.03 \(\pm\) 0.14 | 0.68 \(\pm\) 0.17 | 0.67 \(\pm\) 0.12 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.02 | 0.37 \(\pm\) 0.08 | 0.94 \(\pm\) 0.07 | 0.94 \(\pm\) 0.05 | |
| \(\text{WD}_s(\uparrow)\) | 0.11 \(\pm\) 0.06 | -0.03 \(\pm\) 0.08 | 0.76 \(\pm\) 0.04 | 0.79 \(\pm\) 0.04 | |
| \(\text{WD}_a(\uparrow)\) | 0.30 \(\pm\) 0.03 | 0.20 \(\pm\) 0.03 | 0.85 \(\pm\) 0.03 | 0.88 \(\pm\) 0.04 | |
| \(\text{Success}(\uparrow)\) | 0.00 \(\pm\) 0.00 | 1.00 \(\pm\) 0.00 | 0.56 \(\pm\) 0.37 | 0.58 \(\pm\) 0.03 | |
| Pen | \(\text{DTW}_s(\uparrow)\) | 0.53 \(\pm\) 0.13 | 0.48 \(\pm\) 0.24 | 0.54 \(\pm\) 0.18 | 0.61 \(\pm\) 0.11 |
| \(\text{DTW}_a(\uparrow)\) | 0.58 \(\pm\) 0.05 | 0.40 \(\pm\) 0.17 | 0.59 \(\pm\) 0.13 | 0.64 \(\pm\) 0.08 | |
| \(\text{WD}_s(\uparrow)\) | 0.59 \(\pm\) 0.12 | 0.49 \(\pm\) 0.08 | 0.59 \(\pm\) 0.12 | 0.62 \(\pm\) 0.06 | |
| \(\text{WD}_a(\uparrow)\) | 0.65 \(\pm\) 0.14 | 0.44 \(\pm\) 0.12 | 0.66 \(\pm\) 0.14 | 0.69 \(\pm\) 0.08 | |
| \(\text{Success}(\uparrow)\) | 0.40 \(\pm\) 0.03 | 0.62 \(\pm\) 0.09 | 0.42 \(\pm\) 0.05 | 0.61 \(\pm\) 0.06 | |
| Relocate | \(\text{DTW}_s(\uparrow)\) | 0.09 \(\pm\) 0.14 | 0.03 \(\pm\) 0.13 | 0.27 \(\pm\) 0.14 | 0.37 \(\pm\) 0.07 |
| \(\text{DTW}_a(\uparrow)\) | 0.47 \(\pm\) 0.15 | 0.32 \(\pm\) 0.13 | 0.69 \(\pm\) 0.10 | 0.74 \(\pm\) 0.01 | |
| \(\text{WD}_s(\uparrow)\) | 0.27 \(\pm\) 0.15 | 0.02 \(\pm\) 0.07 | 0.38 \(\pm\) 0.09 | 0.42 \(\pm\) 0.07 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.12 | 0.20 \(\pm\) 0.03 | 0.55 \(\pm\) 0.08 | 0.60 \(\pm\) 0.05 | |
| \(\text{Success}(\uparrow)\) | 0.01 \(\pm\) 0.02 | 0.14 \(\pm\) 0.03 | 0.17 \(\pm\) 0.10 | 0.13 \(\pm\) 0.05 | |
| Average | \(\text{DTW}_s(\uparrow)\) | 0.16 \(\pm\) 0.29 | 0.10 \(\pm\) 0.25 | 0.56 \(\pm\) 0.21 | 0.62 \(\pm\) 0.19 |
| \(\text{DTW}_a(\uparrow)\) | 0.49 \(\pm\) 0.07 | 0.34 \(\pm\) 0.05 | 0.78 \(\pm\) 0.17 | 0.81 \(\pm\) 0.15 | |
| \(\text{WD}_s(\uparrow)\) | 0.32 \(\pm\) 0.20 | 0.08 \(\pm\) 0.28 | 0.61 \(\pm\) 0.17 | 0.65 \(\pm\) 0.17 | |
| \(\text{WD}_a(\uparrow)\) | 0.45 \(\pm\) 0.15 | 0.24 \(\pm\) 0.15 | 0.70 \(\pm\) 0.13 | 0.74 \(\pm\) 0.12 | |
| \(\text{Success}(\uparrow)\) | 0.11 \(\pm\) 0.19 | 0.68 \(\pm\) 0.40 | 0.52 \(\pm\) 0.32 | 0.56 \(\pm\) 0.33 |
4.1.2 Impacts of Action Lengths↩︎
Fig. 2 shows how macro action length \(H\) affects HiMAQ+RLPD performance, averaged over all four Adroit tasks. \(H{=}9\) achieves the best overall balance, with the highest success rate (0.56) and strong similarity scores (\(\text{DTW}_s{=}0.62\), \(\text{DTW}_a{=}0.81\), \(\text{WD}_s{=}0.65\), \(\text{WD}_a{=}0.74\)). This optimal value of \(H{=}9\) for RLPD matches the value used by MAQ+RLPD [16]; more generally, HiMAQ inherits MAQ’s per-algorithm optimal macro action length (\(H{=}9\) for IQL and RLPD, \(H{=}8\) for SAC; see supplementary material), confirming that the hierarchical extension does not alter the optimal macro action length. Performance improves from \(H{=}1\) to \(9\), then plateaus or declines — success drops to 0.53 at \(H{=}12\) — indicating diminishing returns for excessively long chunks.
4.2 Human Evaluations↩︎
To verify if trajectory-level alignment is perceptually recognized as human-like, we conducted human evaluations with 22 evaluators across Turing test and human-likeness ranking test.
4.2.1 Turing Tests↩︎
Turing test measures whether human evaluators can distinguish agent behavior from real human demonstrations. Each evaluator is shown 24 trials, each presenting two side-by-side video clips — one from a human demonstrator and one from a trained agent — and asked to identify which is human. A higher win rate (i.e., the fraction of trials in which evaluators are fooled) indicates more human-like behavior. This test directly probes perceptual indistinguishability, complementing the quantitative trajectory metrics. Fig. 3 shows the Turing Test win rates. MAQ [16] and HiMAQ-based agents substantially outperform base RL agents, with HiMAQ variants topping the ranking: HiMAQ+RLPD (43%) \(>\) HiMAQ+IQL (38%) \(>\) MAQ+RLPD (37%) \(>\) HiMAQ+SAC (34%) \(>\) MAQ+SAC (33%) \(>\) MAQ+IQL (32%) \(>\) RLPD (25%) \(>\) BC (24%) \(>\) SAC (18%) \(>\) IQL (13%).
4.2.2 Ranking Tests↩︎
While the Turing test compares agents against humans, the ranking test directly compares agents against each other. Each evaluator is shown 21 pairs of videos, both from trained agents, and asked to choose which appears more human-like. This produces pairwise win rates that reveal the relative human-likeness ordering across all ten agents in a single unified ranking. Fig. 4 shows the pairwise win rates: each cell gives the fraction of trials in which Agent A (row) was judged more human-like than Agent B (column), with the Avg column summarizing each agent’s overall score. The ranking by average win rate is: Human (71%) \(>\) HiMAQ+RLPD (54%) \(>\) HiMAQ+SAC (52%) \(>\) HiMAQ+IQL (51%) \(>\) MAQ+RLPD (48%) \(>\) MAQ+SAC (38%) \(>\) MAQ+IQL (36%). All three HiMAQ variants consistently rank above their respective MAQ [16] counterparts, and HiMAQ+RLPD is the closest to the human baseline, achieving a win rate of 54% compared to the human’s 71%.
Supplementary Material. Due to space constraints, please refer to our supplementary material for additional details, results, and discussions.
5 Conclusion↩︎
We present a human-like RL framework that produces action trajectories consistent with human behaviors while optimizing rewards. Our HiMAQ method hierarchically discretizes action sequences using a two-stage vector quantization hierarchy: the first stage discovers low-level subactions, while the second stage composes them into higher-level actions. Experiment results on the D4RL benchmarks demonstrate clear improvements over the non-hierarchical MAQ baseline, yielding more human-like behavior without sacrificing task performance. These benefits remain consistent across different RL algorithms such as IQL, SAC, and RLPD. Our future work will focus on improving trajectory smoothness [32], which may further boost human-likeness.