1 Introduction↩︎

Recent advances in unsupervised reinforcement learning (URL) have enabled the discovery of semantically distinct behaviors (“skills”) from state transitions, without access to external reward signals [1]–[5]. This paradigm aims to create a general-purpose foundation for control rather than just learning individual skills. It provides a pre-trained policy that readily adapts to diverse downstream tasks with minimal fine-tuning [6], [7]. Achieving scalability in URL requires both efficiency during pre-training and robustness during transfer, ensuring the learned policy generalizes to varied downstream tasks.

Despite recent progress in URL, we argue that current state-of-the-art methods face two critical bottlenecks that hinder this scalability: (1) sample inefficiency arising from the overlooked non-stationary skill semantics, and (2) brittle generalization caused by overfitting to global context.

First, real-world interactions are costly, making data efficiency important. To maximize data efficiency, modern URL methods rely on off-policy algorithms that reuse past experiences stored in a replay buffer. However, this introduces a fatal flaw: semantic drift. In off-policy settings, each trajectory is collected under a randomly sampled skill \(z\) and \(z\)-conditioned skill policy [8], [9]. The resulting \(z\)–trajectory pair is stored in a replay buffer and reused throughout training [10]–[13]. As learning progresses, the behavioral trajectories induced by the same skill \(z\) can change, because skill policy evolves across off-policy learning. The time-varying semantics of \(z\) induce destabilization of the skill policy by providing stale samples.

Second, for a skill policy to be a scalable foundation for various downstream tasks, it must be robust to distribution shifts. A skill such as “walking forward" should be executable regardless of the global contextual information, such as environmental factors. However, in many prior works [8], [12], [13], the observations used for training skill policies include global contextual information that is not essential for executing skills. When skill policies are conditioned on these global signals, the same skill may exhibit drastically different behaviors under slight shifts in coordinate distributions or become overfitted to specific global contexts. This brittleness limits the reusability of skills in downstream tasks and leads to fundamental generalization failures.

To address these challenges:

• We formalize two overlooked failure modes and provide theoretical and empirical evidence of their impact.

• We introduce skill relabeling to mitigate the semantic drift in the previous off-policy URL. We also propose a novel regularizer to ensure skill diversity under the mutual information objective, effectively differentiating our skill relabeling framework from prior literature in other fields [14].

• We propose a Complementary Information Bottleneck (CIB), a drop-in module compatible with an unsupervised framework. CIB learns an embedding that prevents the policy from exploiting global contextual information, improving skill execution consistency under distribution shifts.

We evaluate our approach on a diverse set of state and pixel benchmarks, demonstrating superior data efficiency and skill-policy generalization compared to prior methods. Notably, we also consider high-dimensional state environments, where existing approaches fail to discover meaningful skills, while our method successfully learns coherent and reusable skill policies.

2 Preliminaries↩︎

2.0.0.1 Unsupervised skill discovery

We follow the settings of prior work [12], a controlled Markov process without a reward function. MDP is defined as \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mu, p)\). \(\mathcal{S}\) represents the state space, \(\mathcal{A}\) represents the action space, \({\mu \text{: } \Delta(\mathcal{S})}\) represents the initial state distribution, and \(p \text{: } \mathcal{S} \times\mathcal{A} \rightarrow \Delta (\mathcal{S})\) represents the transition dynamics kernel. We also consider a discrete or continuous set of latent vectors \(z \in \mathcal{Z}\) with a latent-conditioned skill policy \(\pi(a\mid s, z)\).

In the terminology of unsupervised skill discovery, the latent skill vector \(z\) is sampled from the prior distribution \(p(z)\). Then skill policy \(\pi(a\mid s, z)\) rolls out a trajectory \(\tau=(s_0,a_0,s_1,a_1, ..., s_T)\) with a fixed \(z\) for the entire episode. In this setting, the main goal is to learn diverse and useful behaviors \(\pi(a\mid s,z)\) from scratch without guidance (e.g., data, prior knowledge, and supervision).

We follow the representation objective of the metric-aware unsupervised skill discovery [12] with latent mapping function \(\phi(s)\) to inject semantic meaning in the skill vector \(z \in \mathcal{Z}\): \[\begin{align} \sup_{\pi, \phi} \mathbb{E}_p(\tau,z) \bigg[\sum^{T-1}_{t=0} (\phi(s') - \phi(s))^\top z\bigg] \label{eq:metra95obj} \\ \text{s.t.} \|\phi(s) - \phi(s')\|_2 \le 1, \forall(s, s') \in \mathcal{S}_{adj} \end{align}\tag{1}\]

2.0.0.2 Downstream application with pre-trained skill policy

A pre-trained skill policy \(\pi\) can be adapted to downstream tasks. A hierarchical controller \(\pi^h(z|s)\) samples from the set of learned skills and delivers \(z\) to the (frozen) skill policy to maximize the downstream reward.

3 Problem Statements↩︎

We formally identify the two core challenges preventing off-policy URL from achieving scalability.

3.1 Sample Inefficiency from Semantically Ungrounded Skill Rollouts↩︎

In the off-policy URL setting, the agent collects a trajectory \(\tau\) conditioned on a sampled skill \(z \sim p(z)\). The transition tuple, including the skill label \((\tau, z_{roll})\), is stored in a replay buffer \(\mathcal{B}\). The critical issue lies in the dual role of \(z\): it serves as both the input skill for the policy \(\pi(a|s, z)\) and the target label for the representation \(\phi(s)\).

Unlike standard RL, where the reward function is static, in URL, the semantic meaning of \(z\) is defined by the representation \(\phi\), which is continuously evolving. Let \(\phi_t\) denote the representation at training step \(t\). A trajectory \(\tau\) generated at time \(t_{old}\) with skill \(z\) satisfied the relationship \(z \approx \phi_{t_{old}}(\tau)\). However, at the current step \(t_{curr}\), the updated representation \(\phi_{t_{curr}}\) may map this trajectory to a completely different point in the latent space.

Most existing methods ignore this drift, updating the current policy using the stale pair \((\tau, z)\) from the buffer. This creates a semantic misalignment: the policy is penalized for failing to generate \(z\) according to the current \(\phi_{t_{curr}}\), even though the trajectory was correct under \(\phi_{t_{old}}\). Consequently, no meaningful correction occurs until the agent gathers enough “fresh" samples to effectively displace or outweigh the stale data. This acts as a source of high-variance noise in the gradient estimation. Consequently, increasing the sample reuse rate (i.e., high UTD ratio)—which typically improves efficiency in off-policy RL—ironically worsens instability in URL by overfitting to this persistent label noise.

Figure 1 illustrates that the same skill z can be associated with different trajectories depending on the training stage of \(\phi\) and \(\pi\), highlighting a potentially critical failure mode in the off-policy URL framework.

3.2 Lack of Skill-Policy Generalization under Downstream Tasks↩︎

A second limitation concerns the limited generalization capability of the learned skill policy. In URL, skill policies are typically intended to function as consistent local behavioral primitives that respond to local state changes, directed by the given skill’s guidance. This property is a key prerequisite for reusing skill policy across diverse initial conditions and downstream environments.

In most prior works, however, skill policies are often conditioned directly on the full state observation. Such observations include global contextual information (e.g., absolute positions or visual background cues) that is not essential for executing local behaviors. When the policy is conditioned on this information, skill execution becomes entangled with the global context, weakening the intended locality of the learned behaviors.

This entanglement makes the policy fragile under distribution shifts. Changes in the global context between pre-training and deployment cause the same latent \(z\) to produce unreliable behaviors, significantly hindering downstream performance.

In Figure 2, we show a failure mode of the prior algorithm where conditioning on full state observation fails to execute reliably on varying global contexts. This failure highlights that maximizing state coverage during pre-training is a necessary but insufficient condition for scalability; the learned skills must also be disentangled from the specific global context in which they were discovered.

4 Method↩︎

We propose GenDa (Generalizable Data-efficient Agent), a unified framework designed to train a scalable skill foundation for control. Unlike prior works that treat skill discovery as a static optimization problem, GenDa addresses the dynamic nature of off-policy learning and structural requirements for generalization. We introduce (a) skill relabeling to dynamically align past experiences with the evolving latent space, and (b) Complementary Information Bottleneck (CIB) to structurally encourage the policy to learn ego-centric, reusable behaviors. Our approach is based on the previous state-of-the-art algorithm [12].

4.1 Relabeling for Skill Representation Learning↩︎

4.1.1 Relabeling with the current latent interpretation.↩︎

Standard off-policy URL methods suffer from semantic drift, where the skill labels \(z\) stored in the replay buffer become stale as the representation \(\phi\) evolves. Updating the policy with these stale labels injects high-variance noise into the learning process, thereby limiting the data efficiency in the off-policy setting. To overcome this, we introduce skill relabeling, which treats past trajectories not as fixed (\(\tau,z\)) pairs, but as flexible experiences that can be re-interpreted.

4.1.1.1 Relabeling (\(z\)-step)

To correct the semantic inconsistency caused by fixed rollout-time \(z_{roll}\) in the replay buffer, we need to define a new label \(z\) that is consistent with the current \(\phi\). To define that \(z\), we revisit the telescoping sum in Equation 1 . For an episode \(\tau=(s_0,a_0,s_1,...,s_T)\), the telescoping sum can be written as \((\phi(s_T) - \phi(s_0))^\top z\), which means \(\sum\limits_{t=0}^{T-1}{(\phi(s_{t+1})-\phi(s_t))}\). We can define this telescoping sum as delivering the most appropriate direction vector to explain \(\tau\) with on-policy \(\phi\).

Instead of using \(z_{roll}\), we compute \(z_{\text{relab}}\) from the current latent function: \[\begin{align} z_{\text{relab}}(\tau) = \mathrm{unit}\!\left(\phi(s_T) - \phi(s_0)\right), \\ \mathrm{unit}(x) := \frac{x}{\text{max}(\|x\|_2, \epsilon)}. \label{eq:z95relab} \end{align}\tag{2}\] where \(\epsilon\) is a very small positive scalar.

4.1.1.2 Optimization (\(\phi\)-step)

We then learn with the objective in equation 1 by replacing \(z\) with \(z_{\text{relab}}\) while stopping gradients: \[\begin{align} \sup_{\pi, \phi} \mathbb{E}_p(\tau,z_{\text{relab}}) \bigg[\sum^{T-1}_{t=0} (\phi(s') - \phi(s))^\top z_{\text{relab}}\bigg] \label{eq:obj95with95relab} \\ \text{s.t.} \|\phi(s) - \phi(s')\|_2 \le 1, \forall(s, s') \in \mathcal{S}_{adj} \end{align}\tag{3}\]

Iterating (\(z\)-step \(\rightarrow\) \(\phi\)-step) mitigates the gradient conflicts caused by the fixed \(z_\text{roll}\), thereby resolving the aforementioned semantic drift. We demonstrate that our iterative update is mathematically robust and structurally stable at each step (Details in Appendix 8). \[\begin{align} z_{relab} = \mathrm{unit}\!\left(\phi_{\text{tgt}}(s_T) - \phi_{\text{tgt}}(s_0)\right). \label{eq:z95relab95ema} \end{align}\tag{4}\]

4.1.1.3 Implementation details of skill relabeling

In practice, to reduce training instability, we compute the relabeled targets using an exponential moving average (EMA) network  [15], as shown in Equation 4 .

4.1.2 Preventing collapse via a uniformity regularizer.↩︎

In prior works utilizing a fixed \(z_{roll}\) label, the distribution of \(z_{roll}\) is naturally guaranteed to follow a random distribution \(p(z)\), which is fixed after sampling. Consequently, the entropy term \(H(z)\) becomes a trivial constant in the objective for \(\phi\).

While relabeling stabilizes learning, naively maximizing alignment with re-assigned labels can lead to representation collapse, where \(\phi\) maps all states to a narrow region of the latent space to trivially satisfy the objective (Equation 3 ). Since the distribution of the relabeled skill \(z_{relab}\) is not guaranteed to follow \(p(z)\), to prevent such collapse, we introduce a contrastive-style uniformity regularizer to explicitly maximize \(H(z_{relab})\).

Let \(v_i = \mathrm{unit}\!\left(\phi(s_T^{(i)}) - \phi(s_0^{(i)})\right)\) be the episodic direction for episode \(i\) in a mini-batch. We define \[\begin{align} \mathcal{L}_{\text{unif}} = \frac{1}{B}\sum_{i=1}^{B} \log \sum_{\substack{j=1 \\ j\neq i}}^{B} \exp\!\left(v_i^\top v_j\right), \label{eq:uniformity} \end{align}\tag{5}\] where \(B\) is the number of samples in a mini-batch. We minimize \(\mathcal{L}_{\text{unif}}\), which improves the lower bound of \(H(z_{relab})\) [16]. By encouraging the skill vectors to be uniformly distributed on the hypersphere, we ensure that the agent discovers a diverse repertoire of behaviors even while continuously redefining what those behaviors are.

4.1.3 Final representation objective.↩︎

Let \(\mathcal{L}_{\text{base}}\) denote the representation loss for the objective in Equation 3 . Our final representation learning loss is \[\begin{align} \mathcal{L}_{\phi} = \mathcal{L}_{\text{base}} + \beta \, \mathcal{L}_{\text{unif}} \label{eq:rep95final} \end{align}\tag{6}\] where \(\beta\) is a scalar coefficient (we use \(\beta=1.0\) for all environments). Figure 9 (in Appendix) depicts that our uniformity term can handle the induced bias well. In Appendix 9.1, we provide a theoretical analysis demonstrating that relabeling stabilizes the representation learning process in an off-policy setting.

4.2 Relabeling for Skill Policy Learning↩︎

In the policy learning phase, the intrinsic reward function \[\begin{align} r(\phi, z) = (\phi(s') - \phi(s))^\top z \label{align:intrinsic95reward} \end{align}\tag{7}\] evaluates how well a transition \(s \rightarrow s'\) follows a conditioned \(z\) in the latent space based on the magnitude and direction of \(\Delta\phi\). Prior methods use only fixed \(z_{\text{roll}}\) as a condition of the intrinsic reward function, which limits the privilege of the off-policy setting.

We instead relabel each transition with multiple targets derived from diverse state pairs within the same trajectory to enrich the off-policy reward signal by exploiting the intrinsic reward in Equation 7 .

For a transition at time \(t\), we sample a horizon \(c>0\) and define: \[\begin{align} z_{c} = \mathrm{unit}\!\left(\phi(s_{t+c}) - \phi(s_t)\right). \label{eq:policy95relabel} \end{align}\tag{8}\] We mix \(z_c\) with the \(z_{\text{relab}}\) and the original rollout label \(z_{\text{roll}}\) (details in Appendix 10) as commonly done in relabeling to preserve on-policy semantics in practice [14]. This enables the policy/value function to learn from a richer set of \((s, s', z)\) configurations than those directly experienced, improving bootstrapping.

4.3 Complementary Information Bottleneck (CIB)↩︎

A pre-trained foundational skill policy \(\pi(a \mid s, z)\) is desired to execute pre-trained skills for diverse applications, where distributional shifts in global contextual information frequently occur. However, previous algorithms use the full state as an observation that leads to unintended overfitting on the global context. To disentangle this connection, we propose the Complementary Information Bottleneck (CIB).

4.3.1 Key idea.↩︎

Metric-aware skill discovery methods encourage the latent space \(\phi\) to capture temporal distances. Often, the global contextual information is desired for the \(\phi\) latent space, because this objective promotes the agent to capture the extensive “temporal" manifolds in the state space [12]. We desire to make the policy rely on the skill without entangling global contextual information in the raw state. To address this, we propose substituting the raw policy observations with a learned embedding that is complementary to \(\phi(s)\).

We train an encoder \(q_{\psi}(\ell \mid s )\) that produces an embedding \(\ell\), together with a decoder \(p_{\omega}(\hat{s} \mid \ell,\phi(s))\), by maximizing a variational information bottleneck [17] style objective: \[\begin{align} \mathcal{J}_{\text{CIB}} = I(\ell ; s) - I(\ell;\phi(s)) \label{eq:cve95obj} \end{align}\tag{9}\]

Intuitively, the decoder can use \(\phi(s)\) to reconstruct whatever information \(\phi\) already captures; thus, \(q_{\psi}(\ell \mid s)\) is encouraged to preserve the remaining information needed to reconstruct \(s\), making \(\ell\) complementary to \(\phi(s)\).

4.3.2 Skill Policy with CIB.↩︎

We replace the raw state input of the skill policy with the CIB embedding \(\ell\): \[\begin{align} \pi(a \mid s, z) \;\;\longrightarrow\;\; \pi(a \mid\ell, z), \qquad \ell \sim q_{\psi}(\ell \mid s). \label{eq:policy95cve} \end{align}\tag{10}\] The CIB is used consistently during data collection, training, and downstream deployment. The CIB encoder only changes the observation channel the policy conditions on, improving robustness to shifts in global contextual information.

5 Experiments↩︎

We design our experiments to verify whether GenDa successfully overcomes the scalability bottlenecks identified in Sec. 3. Specifically, we investigate the following questions:

1. Data Efficiency Does skill relabeling enhance the data efficiency of the pre-training phase by mitigating the non-stationarity of off-policy learning?

2. Generalization Does disentangling global context via CIB produce robust skills that generalize to unseen initial conditions and downstream tasks?

3. Scalability Can GenDa serve as a better foundation policy for challenging, high-dimensional control problems compared to state-of-the-art baselines?

As baselines, we include METRA [12] and CSF [13], which are state-of-the-art metric-aware URL methods, as well as DIAYN [8] and CIC [11], which represent alternative approaches to URL. For evaluation, we use numeric and pixel-based locomotion environments from the DeepMind Control Suite (DMC) [18], along with manipulation environments in MuJoCo [19]–[21]. Detailed descriptions of the environments and experimental settings for each algorithm are provided in the appendix (see Appendix 11).

5.1 Main Results↩︎

5.1.0.1 Skill pre-training.

In Figure 3, we evaluate the skill policy’s state and task coverage. Coverage is measured by the number of unique coordinate bins (or unique tasks for Kitchen) visited across \(k=48\) evaluation episodes using randomly sampled skills. We set the bin size to 1 for the x and y dimensions in general environments and 0.01 for the x, y, and z dimensions in Fish.

In a diverse set of environments, GenDa achieves higher state coverage with substantially fewer environment interactions than prior methods. Moreover, in Dog—an environment with high-dimensional state (226 dims) and action (38 dims) spaces where existing algorithms often fail to learn meaningful skills—our method succeeds in learning skills. These results indicate that our relabeling approach can enhance the data efficiency in pre-training, which is also scalable to challenging tasks.

5.1.0.2 Downstream tasks.

To investigate the generality and scalability of the pre-trained policies, we propose diverse downstream tasks.

We further evaluate on downstream tasks using the pre-trained skill policy within a hierarchical framework. Specifically, the skill policy is frozen, and a high-level task policy outputs an action \(z\); the skill policy then executes the corresponding skill for a fixed number of timesteps. Each episode has a goal. If the agent reaches the given goal, it receives a +1 success reward and 0 reward for the other states. The meaning of each suffix is described in Figure 4.

(FS, RG) is the easiest setting, as it provides the global information distribution similar to that of the pre-trained policy. The (RS, RG) setting tests whether the skill policy can execute the learned skill under a shift in global information. The Maze benchmark is the most challenging task in our suite; solving it requires strong generalization and exploration capabilities to perform a given skill over a wider range of coordinates. As shown in Table ¿tbl:table:downstream?, our GenDa achieves the highest average success rate across downstream tasks.

\begin{table}[h]

\vskip

-0.1in \end{table}

Crucially, GenDa achieves comparable performance to the asymptotic limit of baselines with 5\(\times\) fewer environment interactions in Table ¿tbl:table:genda2m10m?. This superior data efficiency positions GenDa as a practical foundation for real-world robotic learning where interaction is costly.

5.2 GenDa ablation study↩︎

In this section, we evaluate the contribution of each component in GenDa and provide further analyses of our algorithm.

5.2.0.1 Component-wise Evaluation.

We conduct ablation studies to quantify the contribution of each module in GenDa. Figure 5 demonstrates that each component addresses a distinct challenge in the URL pipeline. The w/o Relab variant fails to acquire meaningful skills efficiently during the pre-training phase. These empirical results confirm that our skill relabeling is essential to address off-policy non-stationarity and maintain robust data efficiency. In contrast, w/o CIB achieves high state coverage during pre-training but struggles in the (RS, RG) downstream task. This discrepancy highlights that while standard objectives may increase coverage, they often lead to context-dependent policies. The CIB is therefore essential for disentangling the policy from the global context to ensure generalizability. Collectively, these results confirm that our components are complementary: Relabeling enables efficient learning, while CIB ensures the learned skills are robust and scalable.

5.2.0.2 \(\beta\) Analysis.

We conducted additional skill pre-training experiments across diverse environments using various values of \(\beta\) to verify its robustness. The results in Table 1 highlight that while performance varies depending on \(\beta\), the model is not hyper-sensitive to the point where slight variations trigger an objective collapse in either numeric or pixel domains. This provides evidence that our proposed objective, driven by skill relabeling and the uniformity regularizer, can be deployed across varied environments without requiring exhaustive per-environment tuning.

5.3 GenDa analysis↩︎

5.3.0.1 Update-to-Data (UTD) Analysis.

Figure 6 serves as empirical evidence for the non-stationary hypothesis. While baselines suffer from instability at high UTD ratios due to semantic drift, GenDa effectively utilizes frequent updates. This confirms that our skill relabeling turns stale \(z\) of the replay buffer into a consistent source of supervision, enhancing the potential of off-policy learning. This result confirms our theoretical hypothesis in Appendix 9.2: Variance of semantic drift in replay buffer via relabeling is the key factor enabling data-efficient off-policy learning.

5.3.0.2 Global information robustness.

To assess our research question 2, we investigate how robust GenDa is to the generalization issue induced by the global context discussed in Section 3.2. The qualitative results in Figure 7 show GenDa has better generalizability by using CIB that encourages the skill policy to rely on the given skill and its semantic meaning rather than global context.

5.3.0.3 Complementary encoder analysis.

To investigate whether our CIB has a complementary relationship with \(\phi\), we decode the remaining information from each representation. Figure 8 shows that \(\phi\) mainly retains background color while losing recognizable proprioception, whereas CIB retains proprioception while losing background color.

We also tested the skill policy’s robustness by applying unseen visual shifts to the background in pixel-based environment. Consequently, the results in Table 2 demonstrate that the CIB provides relative robustness compared to the w/o CIB in pixel environments, but its effectiveness is degraded under the visual shifts. We abstained from using any image augmentation techniques during visual training to ensure a fair comparison with the baselines. This highlights a highly promising direction: because the CIB module structurally disentangles redundant context, incorporating standard visual generalization techniques [22] during CIB training would seamlessly bridge this gap and yield a model highly robust to visual characteristics.

6 Related Work↩︎

Unsupervised Skill Discovery Unsupervised reinforcement learning (URL) aims to learn skill representations, policies, or dynamics models through interaction with the environment without any external task reward [8], [23]–[25], and then reuse them for downstream learning [11], [26], [27]. Prior unsupervised skill discovery methods train a skill-conditioned policy from states or trajectories by maximizing mutual information objectives [8], [11], [28]. They enable the learning of diverse and distinguishable behaviors even without external rewards. Some approaches further leverage dynamics models to capture mutual information at the state-transition level [9], [29]. However, these methods can prioritize discriminability over how far behaviors spread in the environment. As a result, in high-dimensional state spaces, they have been reported to discover redundant or locally confined skills [12], [13], [30]. To mitigate this, more recent advances in URL leverage explicit state-space metrics to discover skills with Wasserstein dependency measure [31], [32] and InfoNCE [16], [33]–[35] that transfer better to downstream tasks [12], [13]. Despite the promise of URL, prior methods face two limitations: (1) data efficiency and (2) policy generalization.

Disentanglement in unsupervised skill discovery. Some prior work has explored inducing disentanglement by explicitly structuring the latent space via factorization [36], [37]. Another line of work proposes masking the policy input to encourage structured skill usage [38]. However, these methods tend to rely on assumptions such as factorized factors or periodic structure. In contrast, our complementary variational encoder (CIB) improves skill reusability by modulating the skill policy’s input conditioning, without a hand-crafted structure.

7 Conclusion↩︎

7.1 Implication for Unsupervised Skill Learning↩︎

We identify two fundamental challenges in off-policy unsupervised reinforcement learning that limit the scalability of skill discovery. Our skill relabeling mitigates the semantic drift in prior URL methods together with a compatible regularization term. It significantly improves the data efficiency in the pre-training phase. We also point out a failure mode of the skill policy in downstream tasks, which is caused by global contextual information. With our drop-in CIB module, the generalizability of the skill policy is improved in various downstream tasks.

7.2 Open Challenges and Future Directions↩︎

7.2.0.1 Skill relabeling

Our skill relabeling still inherits a one-skill-per-episode assumption that can be violated when a single episode contains heterogeneous behaviors (e.g., exploratory “moving around” segments interleaved with skill-consistent transitions). Our relabeling target is derived from the telescoping sum; thus, it can be generalized to any arbitrary time segment. So, our skill relabeling framework is structurally and mathematically equipped to handle intra-episode skill changes. We will explore extending relabeling to assigning time-varying skill labels, which may better capture abrupt changes in intent, such as sharp direction switches or rapid reactive adjustments.

7.2.0.2 Generalization

CIB is designed to disentangle the global context from the skill policy; its effectiveness depends on the learned metric-aware latent function \(\phi\); if one wishes to apply the idea to non-metric-aware skill discovery algorithms, this dependency can limit applicability. We consider it a feasible future direction to use dynamics-aware objectives rather than a particular embedding metric.

7.2.0.3 Manipulation task

Our metric-aware objective, which maximizes temporal state distances, inherently encourages large-scale movements. This objective hinders the acquisition of the fine-grained, small-scale behaviors required for precise manipulation. Addressing this will require incorporating magnitude-conditioned skill mechanisms, such as PSD [38], to balance large explorations with delicate object manipulation.

Impact Statement↩︎

This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.

Acknowledgements↩︎

This work was partly supported by the National Research Foundation of Korea (NRF) (No. RS-2024-00348376), by the Institute of Information \(\&\) Communications Technology Planning \(\&\) Evaluation (IITP) (No. RS-2024-00438686, No. RS-2022-II221045, No. RS-2025-02218768, No. RS-2019-II190421) grant funded by the Korea government (MSIT), and by the Industrial Technology Innovation Program (No. RS-2025-25448266) funded by the Ministry of Trade, Industry \(\&\) Energy (MOTIE) and Korea Evaluation Institute of Industrial Technology (KEIT).

8 Relabel-and-Optimize Procedure↩︎

8.0.0.1 Setting.

Let \(B\) be a fixed replay buffer containing either transitions \((s,s',z)\) or trajectories \(\tau\) with endpoints \((s_0(\tau), s_T(\tau))\). Let \(V \subseteq \mathcal{S}\) denote the finite set of states that appear in the dataset (tabular setting), and restrict the optimization variable to \[\phi: V \to \mathbb{R}^d.\] Hence \(\phi\) can be identified with a vector in the finite-dimensional space \(\mathbb{R}^{d|V|}\).

We encode the Lipschitz constraints via an undirected graph \(G=(V,E)\): \[\{u,v\}\in E \iff \text{a transition } u\to v \text{ or } v\to u \text{ is observed in the dataset}.\] We fix the gauge by choosing an anchor state \(s_0\in V\) and enforcing \(\phi(s_0)=0\).

8.0.0.2 Relabeling map.

For \(\varepsilon>0\), define \[g(\phi; a,b) \;=\; \frac{\phi(b)-\phi(a)}{\max(\|\phi(b)-\phi(a)\|,\;\varepsilon)}.\] Then \(g\) is well-defined for all \((a,b)\) and continuous, with \(\|g(\phi;a,b)\|\le 1\). While \(g\) may not be differentiable on the boundary \(\|\phi(b)-\phi(a)\|=\varepsilon\), differentiability is not required for the existence results below.

8.0.0.3 Relabel-and-optimize procedure.

Given the current \(\phi\),

1. (z-step; relabel) For each trajectory \(\tau\), set \[z_\tau \leftarrow g(\phi;\;s_0(\tau),\;s_T(\tau)).\]

2. (\(\phi\)-step; optimize) Treat \(\{z_\tau\}\) as constants during the \(\phi\)-update (i.e., apply a stop-gradient through \(z_\tau\)) and solve \[\begin{align} \max_{\phi}\quad & \sum_{\tau} \left\langle \phi\big(s_T(\tau)\big)-\phi\big(s_0(\tau)\big),\;z_\tau \right\rangle \tag{11}\\ \text{s.t.}\quad & \|\phi(u)-\phi(v)\| \le 1 \quad \forall \{u,v\}\in E, \tag{12}\\ & \phi(s_0)=0. \tag{13} \end{align}\]

We emphasize that we do not claim global monotonic improvement or convergence of the full iteration without additional structure; instead, we establish that each step is well-defined and that the \(\phi\)-step admits an optimizer at every iteration.

9 Relabeling under Non-Stationary Label Noise↩︎

9.1 Non-Stationary with a fixed z label.↩︎

9.1.0.1 Noise model

For each trajectory \(\tau\) with endpoints \((s_0(\tau), s_T(\tau))\), suppose the original label \(z_\tau\) is noisy: \[\mathbb{E}[z_\tau \mid s_0,s_T] = \mu(s_0,s_T),\qquad \mathrm{Var}(z_\tau \mid s_0,s_T) = \Sigma(s_0,s_T).\] In non-stationary settings, the conditional variance \(\Sigma(s_0,s_T)\) may vary across endpoint pairs, and typically \(\Sigma(s_0,s_T)\neq 0\) on parts of the support.

9.1.0.2 Alignment scalar and conditional variability.

For a fixed \(\phi\), define the per-trajectory alignment contribution \[Y_\tau(\phi) \;=\; \left\langle \phi\big(s_T(\tau)\big)-\phi\big(s_0(\tau)\big),\;z_\tau \right\rangle.\] When \(\Sigma(s_0,s_T)\neq 0\), the conditional variance of \(Y_\tau(\phi)\) generally remains positive: \[\mathrm{Var}\big(Y_\tau(\phi)\mid s_0,s_T,\phi\big) > 0,\] reflecting that even for the same endpoint pair, label noise induces variability in the alignment term.

9.2 \(z\) relabeling and label noise.↩︎

9.2.0.1 Relabeling removes conditional label-noise variability (from the \(\phi\)-step perspective).

Define the relabeled signal deterministically via \[\tilde{z}_\tau \;=\; g(\phi;\;s_0(\tau), s_T(\tau)),\] and in the \(\phi\)-step treat \(\tilde{z}_\tau\) as a constant (stop-gradient). Then the relabeled alignment term \[\tilde{Y}_\tau(\phi) \;=\; \left\langle \phi\big(s_T(\tau)\big)-\phi\big(s_0(\tau)\big),\;\tilde{z}_\tau \right\rangle\] satisfies \[\mathrm{Var}\big(\tilde{Y}_\tau(\phi)\mid s_0,s_T,\phi\big)=0.\] Thus, from the perspective of the \(\phi\)-step update, conditional variability due to label noise is removed.

9.2.0.2 Implication: potential reduction in stochastic-gradient variance (with possible bias).

Because the per-sample gradient of 11 is affine (here linear) in the label \(z\), the law of total variance suggests decomposing stochastic-gradient variance into (i) a component attributable to conditional label noise and (ii) a component attributable to sampling. When using the original noisy labels, the former component persists whenever \(\Sigma(s_0,s_T)\neq 0\). When using relabeling (and treating \(\tilde{z}_\tau\) as constant in the \(\phi\)-step), this conditional label-noise component can drop to zero, potentially reducing gradient variance and improving update stability in non-stationary regimes.

However, since \(\tilde{z}_\tau\) depends on \(\phi\), relabeling can introduce bias relative to updates driven by the original noisy labels. Hence relabeling reflects a bias–variance trade-off: it may improve stability when label noise dominates, provided the induced bias remains controlled.

10 Policy Relabeling Analysis.↩︎

To investigate the proper \(z\) conditions for the intrinsic reward function, we evaluate multiple relabeling strategies that can be used for selecting target \(z\) and report their comparative results in Figure 10.

For Mix(c,\(\tau\)), we use a 0.5:0.5 ratio. For Ours, we compute the probability of \(z_{\text{roll}}\) using a count-based estimator with respect to the \(z_{\text{relab}}\) directions of the episodes stored in the replay buffer. This estimator is updated like an exponential moving average (EMA) each time a \(z_{\text{roll}}\) is sampled from the buffer for learning. If the resulting probability is lower than threshold \(\epsilon_{z}=0.4\), we keep the corresponding \(z_{\text{roll}}\); otherwise, we relabel it using \(\mathrm{Mix}(c,\tau)\). We also test multiple settings for the c-\(\tau\) mixture ratio and the threshold \(\epsilon_{z}\) in Table ¿tbl:tab:ablation95ratio95threshold?, but observe only marginal differences in performance.

11 Implementation details↩︎

We implement our algorithm and baselines with JAX for faster training. Our code is based on the official PyTorch implementation and paper of each baseline. For stable learning, we use the LayerNorm in the critic networks of both our method and the baselines. For CIC and DIAYN, we use the hyperparameters used in the official CSF implementation. For METRA, CSF and Ours, we use same hyperparameters except high-level controllers in downstream tasks that are represented in Table ¿tbl:table:hyperparams?.

11.1 Environment settings↩︎

\begin{table}[h]

\vskip

-0.1in \end{table}

11.1.0.1 Downstream tasks.

To evaluate performance on downstream tasks, we utilize 6 domains, each containing 1 to 4 specific tasks. High-level controllers \(\pi^{h}(z|s,s^{task})\) selects a skill every K=25 environment steps, where \(s^{task}\) denotes the task information (like desired goal).

Maze tasks. The agent navigates a \(7 \times 7\) maze to reach a goal. During training, both the agent and the goal are initialized at random feasible positions, with a minimum separation distance of 3. During evaluation, the agent starts from a fixed location \((1,0)\). The goal is placed at \((5,6)\) for Easy and at \((1,6)\) for Hard. For the Quadruped agent, we scale the maze dimensions by a factor of \(1.5\) to account for its larger body size. We set the episode horizon to 400 environment steps for Easy and 500 steps for Hard.

(RS, RG) tasks (Quadruped, Humanoid, Dog-Numeric). Both the agent and the goal are initialized uniformly at random in \([-3.5, 3.5]^2\), with a minimum separation distance of 3. As in the Maze tasks, we increase the environment scale by a factor of \(1.5\) for the Quadruped agent.

(FS, RG) tasks. The agent starts from the origin \((0,0)\), while the goal is sampled randomly. Specifically, the goal is sampled from \([-3.5, 3.5]^2\) for Humanoid-Numeric and from \([-5, 5]^2\) for Humanoid and Quadruped-Pixels. In the Fish environment, the agent starts from a fixed position with a random orientation (quaternion), and the goal is placed 0.5 units ahead along the agent’s forward direction. As in the Maze tasks, we increase the environment scale by a factor of \(1.5\) for the Quadruped agent.

For (FS, RG) and (RS, RG), we set the episode horizon to 200 environment steps for Quadruped (Num/Pix), Humanoid (Num/Pix), and Fish, and to 400 steps for Dog-Numeric.

\begin{table}[h]

\vskip

-0.1in \end{table}

References↩︎