1 INTRODUCTION↩︎

The use of reinforcement learning approaches for robotic control has skyrocketed in recent years, mainly thanks to advances in computational efficiency and simulation fidelity. However, for systems such as legged robots which have complex, nonlinear dynamics, learning effective policies is still a challenging and active research area. Although powerful, purely data-driven model-free RL approaches rely on costly trial and error. Conversely, more traditional model-based control approaches exploit the known physics of the system, but typically require expert knowledge and extensive manual tuning. Fusing model-based control with RL holds promise for overcoming their respective limitations and devising more efficient and effective robot control methods.

Model-based control methods employ a complete or reduced description of robot dynamics. Using the complete dynamics model requires explicit design and precise modeling, but produces a high-fidelity result. Instead, reduced or simplified model-based approaches, e.g., with inverted pendulum [1] or centroidal dynamics [2], [3] models, improve computational complexity while sacrificing fidelity.

Reinforcement learning, conversely, can be model-free thanks to the parallelized, high-fidelity simulators that are nowadays available, and recent works show its success in learning legged robot tasks [4], [5]. Nonetheless, model-free strategies suffer from low sample efficiency, lack of generalization, and convergence to suboptimal policies. In recent years, injecting physics priors into the RL training process has emerged as a strategy to mitigate the drawbacks of the two separate approaches. Here, we utilize two specific types of priors: symmetry information and linearized dynamics in the Koopman-lifted space [6].

The use of symmetries as priors was first introduced for supervised deep learning, namely through the imposition of symmetry constraints on learning architectures [7]–[9]. In robot learning, a handful of works incorporate symmetries into the RL process via data augmentation [10], [11]. The RL framework proposed in [11] constrains the learned policy to be equivariant. Encoding symmetry information in this way reduces the number of samples required for training and yields more generalizable policies, but introducing such constraints can negatively affect performance.

The other type of prior we consider is based on Koopman theory [6], which states that for any nonlinear system there exists an infinite-dimensional function space in which the system dynamics are globally, not just locally, linear. Early data-driven methods found finite approximations of the Koopman operator, such as via dynamic mode decomposition (DMD) [12] and DMD with control (DMDc), which extends the theory to controlled systems to produce a controlled Koopman model [13]. More recent works approximate both the lifting functions and the linear embedding with kernel methods or autoencoders [14], [15]. It has been shown that the controlled Koopman system can be effectively regulated with linear model predictive control (MPC) [16], [17].

Some recent works also utilize Koopman priors for RL. For instance, [18], [19] use the Koopman operator to facilitate learning optimal controllers. [20] presents two strategies in which the value function constitutes a linear mapping of the Koopman observables, and shows these methods to have state-of-the-art policy-learning performance on benchmark systems such as the Lorenz model [21]. Another approach, proposed in [22], trains the Koopman autoencoder in parallel with Proximal Policy Optimization (PPO) and learns the policy directly from the encoded Koopman state, achieving modest performance improvements on RL benchmark tasks. Although the results are relevant from a theoretical perspective, they are validated only on well-defined and low-dimensional systems.

In robotics, high-dimensional state spaces can be lifted and linearized at relatively low cost with an autoencoder, bypassing the hand-design of lifting functions. Many of the Koopman-based approaches for robotics specifically focus on characterizing the dynamics [23] or tailor the linearization of the system for use in optimal control frameworks such as MPC [24]. Some approaches use the Koopman operator with RL to learn the optimal control tuning parameters rather than the actions directly, e.g., [25]. However, none of these robotic control approaches directly inject the Koopman prediction into the policy learning process.

A key motivation for our work is that Koopman theory can facilitate the identification and exploitation of symmetries. For example, [26] improves the sample efficiency of an offline RL algorithm by using the Koopman linearization to more effectively enforce symmetries in the system via data augmentation, validating the approach on RL benchmark tasks. In [23], a system’s morphological symmetries are predefined and embedded in an equivariant Koopman autoencoder to more efficiently learn a global linear model. The authors present a harmonic analysis of quadruped locomotion modes, but do not address the control of the robot.

In this work, we propose SKooP (Symmetric Koopman Predictions) an approach which combines the advantages of a controlled Koopman embedding and morphological symmetry priors to enable faster initial policy convergence and improve learned behavior quality and generalization. By learning a symmetric linearized latent space concurrently with the policy, we ensure that the lifting function is trained on a relevant state space. Then, by providing the critic with information about the future in the form of Koopman-predicted latent states, we enable it to learn higher-quality motions. Specifically, our contributions are as follows:

• A control- and symmetry-informed Koopman autoencoder to predict the future state, trained concurrently with the RL policy.

• A modular, symmetric Koopman-based RL architecture, SKooP, which learns the Koopman embedding and employs the lifted state prediction as a prior for learning the value function.

• Detailed ablation analysis and performance validation on multiple challenging two-legged locomotion tasks for a quadruped robot, in two different simulation environments.

Our results show that SKooP improves initial reward convergence rate and symmetry of learned motions compared with standard PPO, and that it generalizes better than a state-of-the-art symmetric RL method to unseen, mirrored tasks.

2 BACKGROUND↩︎

2.1 Notation↩︎

• \(g \in \mathbb{G}\) denotes a group element within a group representation. Specifically, \(\mathbb{C}_2 := \{e, g_s | g_s^2 = e\}\) is a reflection group; \(g_s\), \(e\) are the reflection and identity transformations.

• \(\rho_{\mathcal{X}}, \rho_{\mathcal{U}}, \rho_{\mathcal{Y}}\) are matrix representations of a symmetry group acting on the state space \(\mathcal{X} \subset \mathbb{R}^{n'}\), the control space \(\mathcal{U} \subset \mathbb{R}^m\), and the output space \(\mathcal{Y} \subset \mathbb{R}^{n' \times n}\), respectively.

• \(\mathcal{K}\) is the Koopman operator, which acts on observables \(z := \psi(x) := [\psi_1(x), \dots, \psi_n(x)]\), where \(\psi_i(x) \in \mathcal{F}(\mathcal{X})\), to predict the system dynamics \(F(x_k, u_k)\) in the lifted space.

• \(A \in \mathbb{R}^{n\times n}\), \(B \in \mathbb{R}^{n\times m}\), \(C \in \mathbb{R}^{n'\times n}\) are linear system matrices and \(H \in \mathbb{N}\) is the look-ahead horizon length.

• \(q, \dot{q}\) are joint positions and velocities.

• \(\theta\), \(\xi\), \(\phi\) represent learned network parameters.

• \(\mathcal{L}_{sr}\), \(\mathcal{L}_{lp}\), \(\mathcal{L}_{sp}\) are the standard autoencoder losses: state reconstruction, latent state prediction, and state prediction, respectively.

2.2 Markov Decision Processes and Bellman Equation↩︎

Markov Decision Processes (MDPs) are used in optimal control and RL for modeling transition-based decisions. An MDP is represented by the tuple \((\mathcal{X}, \mathcal{U}, \mathcal{P}, r, \gamma)\) where \(\mathcal{P} : \mathcal{X} \times \mathcal{U} \times \mathcal{X} \mapsto [0, 1]\) is the transition probability, \(r: \mathcal{X} \times \mathcal{U} \times \mathcal{X} \mapsto \mathbb{R}\) is the reward function, and \(\gamma \in [0, 1]\) is the discount factor. RL aims to find an optimal policy \(\pi^*\) that maximizes the Bellman equation: \[V_{\pi}(x) = \sum_{x' \in \mathcal{X}} \mathcal{P}(x'|x, \pi(x)) \left[r(x, \pi(x), x') + \gamma V_{\pi}(x')\right], \label{eq:value}\tag{1}\] where \(V_{\pi}(x)\) is the value of state \(x\) using policy \(\pi\).

2.3 Morphological Symmetries↩︎

In this work, we take advantage of the inherent symmetries of the state and action spaces of the robot. Most legged robots possess \(\mathbb{C}_2\) symmetry over the sagittal plane, allowing information such as joint positions and poses to be mirrored over this plane. A symmetry acting on a point \(x \in \mathcal{X}\) can be interpreted as a matrix-vector multiplication with \(g \triangleright x := \rho_{\mathcal{X}}(g) x \in \mathcal{X}\).

A map \(f\) is considered group invariant, or \(\mathbb{G}\)-invariant, if its output does not change under any group transformation of the input, i.e., \(f(\rho_{\mathcal{X}}(g) x) = f(x), \; \forall g \in \mathbb{G}\), while it is \(\mathbb{G}\)-equivariant if applying a group transformation to the input and then evaluating \(f\) yields the same result as applying the corresponding group transformation to the output, i.e., \(f(\rho_{\mathcal{X}}(g) x) = \rho_{\mathcal{Y}}(g) f(x), \; \forall g \in \mathbb{G}\).

In the context of MDPs, the state and action group representations \(\rho_{\mathcal{X}}\) and \(\rho_{\mathcal{U}}\) can be defined as symmetric representations given the morphological symmetries of the system. This definition proves useful in the context of RL, where it makes the policy \(\pi : \mathcal{X} \mapsto \mathcal{U}\) \(\mathbb{G}\)-equivariant and the reward function \(r : \mathcal{X} \times \mathcal{U} \mapsto \mathbb{R}\) \(\mathbb{G}\)-invariant. Thus, the value function \(V_{\pi} : \mathcal{X} \mapsto \mathbb{R}\) is also \(\mathbb{G}\)-invariant [27].

An unknown target function to approximate via a neural network can be expressed as \(f_{\phi} \in \mathcal{F} : \mathcal{X} \mapsto \mathcal{Y}\). To impose symmetry constraints, we can require that the learned function \(f_{\phi}\) be \(\mathbb{G}\)-invariant or \(\mathbb{G}\)-equivariant: \[\begin{align} \mathcal{F}_{\mathbb{G}}^{inv} &:= \{f_{\phi} \in \mathcal{F} | y = f_{\phi}(g \triangleright x), \forall g \in \mathbb{G} \}, \\ \mathcal{F}_{\mathbb{G}}^{eq} &:= \{f_{\phi} \in \mathcal{F} | g \triangleright y = f_{\phi}(g \triangleright x), \forall g \in \mathbb{G} \}. \end{align}\]

2.4 Koopman Operator↩︎

The Koopman operator offers a linear representation of the nonlinear system dynamics \(x_{k+1} = F(x_k)\): \[\begin{align} \mathcal{K} \psi({x_k}) &:= \psi( F( x_k)) = \psi(x_{k+1}), \\ z_k &:= \psi(x_k). \end{align}\] A key challenge in finding a Koopman model is to design the lifting function \(\psi(x)\). Given that we must approximate an infinite-dimensional Koopman operator, the latent space \(\mathcal{F}(\mathcal{X})\) is much higher-dimensional than the state space \(\mathcal{X}\) [28]. Moreover, [16] proposes an extension of the Koopman operator approximation for controlled systems \(x_{k+1} = F(x_k, u_k)\): \[\begin{align} z_{k+1} &= A z_k + B u_k, \\ x_k &= C z_k. \end{align} \label{eq:controlled95koopman}\tag{2}\] For a controlled system, the linear representation is supported on the policy control set. As such, finding an approximation of the Koopman operator \(\mathcal{K}\) amounts to finding \([A, B]\). Extending Koopman theory to the controlled case opens up new possibilities for its use in optimal control and RL to provide predictive priors.

3 METHODS↩︎

In this Section, we present our novel, modular, and online SKooP approach that exploits symmetries and Koopman theory to enhance the sample efficiency and improve the performance of actor-critic RL algorithms. Our approach utilizes a symmetric Koopman dynamics autoencoder to learn a lifted space in which the dynamics are linear and system symmetries are respected. The inclusion of a Koopman prediction as part of the critic input in an actor-critic framework enables the critic to more easily learn a precise value function. This is thanks to the linear representation of the system dynamics that the Koopman model provides, which simplifies the critic’s learning problem of predicting the expected future return. The overall scheme of the proposed approach is displayed in 2.

3.1 Equivariant Networks↩︎

Our approach implements symmetry priors in two ways. The first way is via the structure of the actor and critic networks. We constrain the actor to be \(\mathbb{G}\)-equivariant and the critic to be \(\mathbb{G}\)-invariant, following the assertions made for MDPs in 2.3. We do so by predefining the symmetric state and action representations \(\rho_{\mathcal{X}}\) and \(\rho_{\mathcal{U}}\). For example, in the robot’s joint space, due to the \(\mathbb{C}_2\) symmetry, a state \((q, \dot{q})\) can be reflected to obtain the symmetric state \((g_s \triangleright q, g_s \triangleright \dot{q})\) [27]. This actor-critic architecture is similar to that proposed in [11], but we extend it by defining the symmetries for the privileged latent-state observations.

We also apply symmetry priors to the latent state. The Koopman autoencoder learns to encode and decode the Koopman state using the symmetry information for the state \(x_k\) and action \(u_k\). This is made possible by constraining the encoder \(f_{\phi}\) and decoder \(f_{\phi}^{-1}\) to be \(\mathbb{G}\)-equivariant. Using the notation from 2.3, the symmetry rules for the actor, critic, encoder, and decoder are summarized as: \[\begin{align} \pi_{\theta}(g \triangleright x) &= g \triangleright \pi(x), \\ V_{\xi}(g \triangleright x) &= V_{\xi}(x), \\ f_{\phi}(g \triangleright x) &= g \triangleright z, \\ f_{\phi}^{-1}(g \triangleright z) &= g \triangleright x. \end{align}\] Structurally, all these networks are termed equivariant multilayer perceptrons, or EMLPs.

3.2 Koopman Autoencoder with Symmetries↩︎

To learn a lifting function, we train an equivariant autoencoder concurrently with the agent. This simultaneous learning strategy avoids pretraining and ensures training data are relevant to the state and action space regions explored by the agent since the control policy changes over the course of the training. We adapt the Equivariant Dynamics Auto-Encoder (eDAE) structure proposed in [23], which is based on the original DAE [15], to learn the dynamics of a controlled system, which we term an Equivariant Controlled Dynamics Auto-Encoder (ecDAE).

Thus, the linearized dynamics take the form of 2 . We now represent the lifting function \(\psi(x)\) as \(f_{\phi}\) and the state reconstruction matrix \(C\) as \(f_{\phi}^{-1}\) to show that they are parameterized as neural networks in the autoencoder framework. We also employ a multi-step Koopman prediction, so we can adapt 2 to reflect these specifics of our approach, as follows: \[\begin{align} z_{k+H} &= A^{H} f_{\phi}(x_k) + \sum_{i=0}^{H-1} A^{H - 1 - i} B u_{k+i}, \\ x_k &= f_{\phi}^{-1} (z_k). \end{align} \label{eq:ae95koopman95dyn}\tag{3}\]

We use standard losses for training the autoencoders: \[\begin{align} \mathcal{L}_{sr} &:= \| x_k - f_{\phi}^{-1}\left(f_{\phi}(x_k)\right)\|^2_2, \\ \mathcal{L}_{lp} &:= \| f_{\phi}(x_{k+H}) - \mathcal{K}^{H} f_{\phi}(x_k)\|^2_2, \\ \mathcal{L}_{sp} &:= \| x_{k+H} - f_{\phi}^{-1}\left(\mathcal{K}^{H} f_{\phi}(x_k)\right)\|^2_2. \end{align} \label{eq:ae95loss}\tag{4}\]

3 shows how the control input fits into the autoencoder structure, but does not address the symmetries. \(f_{\phi}(x) = z\) and \(\pi_{\theta}(x) = u\) are constrained to be \(\mathbb{G}\)-equivariant by ensuring that the basis functions are \(\mathbb{G}\)-equivariant. Thus \(\mathcal{F}(\mathcal{X})\) becomes a symmetric function space, which can be decomposed into isotypic subspaces \({\mathcal{F}(\mathcal{X}) := \bigoplus^p_{i=1} \mathcal{F}_i(\mathcal{X})}\), where \(p\) is the number of distinct irreducible representations of \(\mathbb{G}\) [27]. This means that \(A\) and \(B\) are block-diagonal with \(p\) blocks. Essentially, 4 learns a Koopman operator \(\mathcal{K}_i\) for each of these subspaces, and symmetry constraints are preserved throughout.

3.3 Latent State Privileged Observations↩︎

In standard actor-critic frameworks, the state \(x\) is used as input to both the actor and the critic. Our actor-critic method is asymmetric (e.g., [29]), so the critic receives a single privileged Koopman state prediction \(z_{k+1} = A f_{\phi}(x_k) + B u_k\) learned by the ecDAE. Since the actor only requires \(x_k\) as input, this architectural choice informs training without increasing computational overhead at deployment time. The autoencoder is trained to predict the linear dynamics of the state \(z\) over an \(H\)-step horizon. This property means it provides the critic with an informative predictive prior.

3.4 Online DAE Training↩︎

To avoid costly pretraining and to ensure that the explored state space is represented in the learned Koopman model, we train the ecDAE concurrently with the RL policy. At the beginning of training, rollouts of random actions are performed to fill a replay buffer with training data for the DAE. This buffer is distinct from the buffer used for training the actor and critic in PPO. At each RL training step, an ecDAE training step is also executed. We choose to store and select training data from a Prioritized Experience Replay (PER) [30] buffer \(\mathcal{D}(x_k, u_k, x_{k+1})\) for each iteration. PER prioritizes keeping buffer data which results in high loss.

4 EXPERIMENTAL SETUP↩︎

We now show that SKooP is modular and succeeds in improving performance, sample efficiency, and generalization to unseen scenarios. We present the results of training SKooP on multiple challenging bipedal locomotion tasks to empirically evaluate its capabilities. In our experiments, we adopt vanilla PPO as an actor-critic algorithm due to its reliability and widespread use. However, note that SKooP is compatible with any actor-critic algorithm. We test on multiple bipedal locomotion tasks inspired by  [31] for the Cyberdog 2 quadruped [32]. We consider 3 tasks, all of which require the quadruped to transition into an upright, bipedal position and walk on its hind legs: stand dance, walk slope, and push door. Details on task parameters are available in 6 (Appendix). We employ the massively parallel Isaac Gym [33] physics simulator and execute our experiments on an NVIDIA A100 GPU. Averaged over 100 iterations, a single training iteration of SKooP, PPO, and PPOeqic respectively for the stand dance task takes 2.651 s, 1.978 s, and 2.245 s. PPOeqic takes 13.5% longer than baseline PPO, and SKooP takes 34.0% longer.

To apply our symmetry and Koopman techniques, we utilize the open source libraries DynamicsHarmonicsAnalysis [23], MorphoSymm [27], and ESCNN [34]. We employ a horizon of \(H=5\) for our ecDAE training. We use a PER buffer with 10000 data points to train the autoencoder. Each task is trained with 8192 parallel environments, over 5 random seeds. The dimensionality ratio of the latent state \(z\) to the state \(x\) is 3. The stand dance and walk slope tasks are trained for 30000 iterations and have a state dimension of 47, while push door is trained for 18000 iterations and has a state dimension of 43. All tasks use an action space of size 12, equal to the number of controllable joints. For further details on the task, refer to [11].

5 RESULTS↩︎

We define and adopt several standard metrics to evaluate our approach, including the mean episode return, training value loss, success rate (SR), symmetry index (SI), stability, and controllability. We also evaluate SR and SI when starting in out-of-distribution (OOD) initial states. Additionally, we provide an ablation study to isolate the effects of the symmetry constraints, latent state critic input, and Koopman prediction prior. Examples of SKooP policy performance on the considered tasks are available in the accompanying video.

The nomenclature we use for the ablation is as follows: PPO refers to vanilla PPO, PPOeqic denotes equivariant actor and invariant critic without using the autoencoder (presented in [11]), and SKooP denotes our complete approach. SKooP-NoSym-NoPred is our approach without symmetries (using the controlled autoencoder, cDAE) and using \(z_k\) instead of the prediction \(z_{k+1}\), SKooP-NoPred also uses \(z_k\), but is with symmetries, while SKooP-NoSym uses the prediction \(z_{k+1}\) and no symmetries.

5.1 Policy Symmetry Evaluation↩︎

We select the push door task as a means of evaluating the generalization of the policy learned by our method. To do this, we collect the SR, SI, OOD SR, and OOD SI metrics over 10000 episodes, averaged over 5 seeds per setting. We define a successful episode as one in which the agent manages to open the door to at least \(60^{\circ}\) and walk through it. The SR is the percentage of episodes in which the agent succeeds. The SI is defined using the SR. Specifically, \(\text{SI} := \frac{2|X_R-X_L|}{X_R+X_L} \times 100\%\), where \(X_R\) is the SR for a scenario where the door opens on the right, while \(X_L\) is the SR for the mirrored task which differs only in that the door opens on the left. To evaluate OOD performance, the initial base orientation of the robot is sampled uniformly at random within \(\pm 15^{\circ}\) around each axis compared to the trained initial orientation. These metrics are similar to those used in [11]. All models are trained only in the scenario with the right-opening door, while they are tested with both right- and left-opening doors to evaluate generalization to symmetric tasks. The results are reported in 1.

5.1.1 Success Rate (SR) and Symmetry Index (SI)↩︎

We find that all approaches that do not employ symmetry constraints, i.e., PPO, SKooP-NoSym-NoPred, and SKooP-NoSym, exhibit a very high SI, meaning that they do not generalize at all to the mirrored task (left-opening door). SKooP-NoSym and SKooP-NoSym-NoPred display the highest average SR for the right-side training task, although not significantly higher than SKooP. However, they systematically fail on the left-side task. Conversely, PPOeqic, which constrains the actor and critic with symmetry priors, achieves a diminished right-side SR compared to the baseline PPO, but generalizes well to the mirrored task. Both SKooP-NoPred and SKooP enforce symmetry priors on the actor and critic in the same way that PPOeqic does. SKooP-NoPred and SKooP additionally enforce the symmetry constraints in the ecDAE, leading to a latent space with embedded symmetries. The results show that the use of symmetry priors in the privileged \(z_k\) or \(z_{k+1}\) input enables significantly higher SRs on both the right and left sides, and lower SI, with respect to PPOeqic. Compared to SKooP-NoPred, SKooP yields stronger performance across the board and similarly low SI, demonstrating the benefit of the Koopman prediction \(z_{k+1}\). We note that the results for SKooP also exhibit lower variance compared to the other methods. 1 visually compares the SKooP method on the original and mirrored door-opening tasks, showing similar behavior in both.

The takeaways from these results are that, by providing a Koopman state prediction to the critic, and by constraining the actor, critic, and autoencoder with symmetry information, we enable the agent to learn not just a more successful policy, but also one that can generalize to a previously unseen, mirrored task.

5.1.2 OOD Performance↩︎

When tested on the OOD initial states that we define, we observe that while SKooP-NoSym-NoPred and SKooP-NoSym still perform comparably with PPO on the right-hand opening task, they also similarly fail on the mirrored task. In contrast, PPOeqic, SKooP-NoPred, and SKooP consistently perform highly symmetrically even on OOD tasks, as revealed by the low SI values. Notably, in the OOD case SKooP significantly outperforms PPOeqic for both SR and SI, also achieving stronger mean performance and lower variance with respect to SKoop-NoPred.

5.1.3 Joint Trajectory Analysis↩︎

We also examine a sample of joint trajectories for the rear legs in the stand dance task (see 4). Symmetric motion of the rear legs indicates greater stability of the robot. In the plots, the motion is symmetric if the right and left joint trajectories oscillate in similar ranges. The results show that SKooP produces the most symmetric behavior. Instead, PPO learns to command larger angles for the right rear knee than for the left.

2 reports the phase mean absolute error (MAE), the mean step amplitude \(\Delta \theta\), and the amplitude difference between left and right joints \(\Delta \theta_{\text{diff}}\), for the considered methods. The phase MAE is defined as \(\frac{1}{T} \sum_{k=0}^{T} |q_L(k) - q_R(k+ \tau)|\), where \(T\) is the total number of timesteps, \(q_L, q_R\) are the left and right joint positions of a given joint, and \(\tau=1/(2f_d \Delta t)\) is the expected phase offset assuming that the ideal is a \(180^{\circ}\) offset, given the desired gait frequency \(f_d=2.5 \si{\hertz}\) and timestep \(\Delta t\). SKooP-NoSym learns significantly different ranges for the left and right rear hip pitch joints, which is expected since it does not have access to symmetry priors. 2 shows that SKooP rectifies this with the use of symmetry priors, reducing the mean phase MAE by 96.7% and \(\Delta \theta_{\text{diff}}\) by 42.3% in comparison. While PPOeqic learns more visually symmetric right and left joint ranges, the left rear knee consistently has a larger maximum angle than the right one. Furthermore, SKooP learns joint trajectories that oscillate within the same range for both the left and right joints. Finally, 2 also shows that SKooP achieves larger and more consistent \(\Delta \theta\) across seeds than PPOeqic.

5.2 Training Performance↩︎

Across the three tasks, 3 shows that the SKooP, SKooP-NoPred, and PPOeqic approaches incorporating symmetries have steeper initial convergence rates than competitors. We note that across the tasks, the -NoPred methods which learn with \(z_k\) converge slightly faster than their counterparts which learn with the Koopman prediction \(z_{k+1}\). This is explained by the fact that during the initial learning phase the autoencoder is exposed to a wide variety of states and actions and has not yet converged. Using only the lifting function to get \(z_k\) introduces less error in this phase than using both the lifting function and the Koopman model prediction to get \(z_{k+1}\). For the walk slope task, incorporating the latent state (predicted or not) is sufficient to improve the convergence rate without the use of symmetries, while symmetries provide further improvement.

Overall, SKooP achieves higher returns than PPO and PPOeqic. The easiest task to learn, stand dance, exhibits comparably small improvements from the use of Koopman prediction and symmetry priors. Instead, on the most challenging, asymmetric task, push door, PPOeqic provides faster initial convergence than PPO while achieving similar reward. Interestingly, for push door, SKooP-NoSym and SKooP-NoSym-NoPred learn to maximize rewards and become highly specialized. However, as discussed in 5.1.1, they fail to generalize to the mirrored task. On push door, SKooP significantly outperforms PPO and PPOeqic in terms of reward values and converges faster than SKooP-NoPred.

5.3 Stability and Controllability of the Koopman Model↩︎

A discrete-time linear system with eigenvalues \(\lambda(A)\) closer to the center of the unit circle has dynamics that quickly decay to an equilibrium point. Conversely, at the edge of the unit circle, eigenvalues are marginally stable, meaning the system has dynamics that are time invariant.

Remarkably, 3 and 5 show that the use of symmetry constraints always produces a larger \(|\lambda(A)|_{max}\), with low variance across seeds. In fact, in two out of three tasks, the ecDAE learns exactly one eigenvalue \(|\lambda(A)| \approx 1.0\). 3 further reports numerical values related to the controllability, which measures how well the controller can reach goal states in the state space of the Koopman model. This can be evaluated using the singular values \(\sigma\) of the controllability matrix \(\mathcal{C} = \begin{bmatrix} B & AB & \cdots & A^{n-1} B \end{bmatrix}\). Across all tasks and seeds, the ecDAE has a larger condition number \(\kappa(\mathcal{C})=\sigma_{max}(\mathcal{C})/\sigma_{min}(\mathcal{C})\) and lower controllability rank \(\text{rank}(\mathcal{C})\) than its cDAE counterpart. This suggests that the ecDAE learns a more compact representation, by recognizing that symmetric modes cannot be controlled independently. The larger condition number shows that the ecDAE is forcing redundant control directions towards 0.

In order to better understand the performance and behavior of the cDAE vs. the ecDAE, 4 reports the average autoencoder state reconstruction, 1-step latent prediction, and \(H\)-step state prediction errors for a horizon \(H=5\) (4 ) for cDAE and ecDAE trained for push door. The table also shows the value of the invariant mode (if any) and compares the state prediction error on the trained (right) and mirrored task (left) both with the invariant mode enabled, disabled, and with only the invariant mode enabled. This analysis shows that while the symmetry constraints may have no significant benefit for latent state prediction, they improve state reconstruction and state prediction. The use of symmetry priors also enables the ecDAE to achieve almost identical state prediction on the mirrored task as it does on the trained task, whereas the cDAE’s state prediction is around 50% worse on the mirrored task. Furthermore, removing the ecDAE’s invariant mode significantly increases \(\mathcal{L}_{sp}\) MSE, as does disabling all the other modes except for the invariant one. Since only the ecDAE learns this \(|\lambda| \approx 1\), it must help represent the invariance of the system dynamics across symmetry groups. Unlike the other tasks, stand dance does not require significant base displacement, which may explain why for that task the ecDAE does not identify an invariant mode to represent the symmetry.

5.4 Sim-to-sim↩︎

To validate the robustness of the learned SKooP policies, we evaluate sim-to-sim transfer from the training simulator, Isaac Gym, to Mujoco. Unlike Isaac Gym, which trades some physical accuracy for massively parallelized training efficiency, Mujoco’s high-fidelity solver better models real-world joint and contact physics. 5 compares the command tracking MSE for the stand dance task across both simulators. Because this task is governed by linear \(x, y\) and angular \(z\) velocity commands, we report the linear \(v_{xy}\) and angular \(\omega_z\) tracking errors. To test robustness, we evaluate conditions at the edge of the agent’s training distribution. The training ranges are: friction coefficient \([1.0, 3.0]\), added mass \([-0.5, 0.5]\) kg, velocity pushes \([0.0, 0.2]\) m/s, and rotor inertia \(0.0\) kg m². Unless stated otherwise, the default test settings are: friction 1.0, added mass \(0.0\) kg, velocity push \(0.0\) m/s, and rotor inertia \(0.005\) kg m². Linear velocity commands \(v_{xy}\) are set to \(0.0\). We note that Isaac Gym by default assigns the rotor inertia (armature) to \(0.0\) kg m² during training, and in Mujoco this causes failure.

None of the rollouts summarized in 5 result in failure. However, they show that while the policy produces similar linear velocity MSE across test conditions in both simulators, it suffers in the yaw velocity tracking. These results demonstrate that SKooP is robust to perturbations and transferable across simulated settings.

6 CONCLUSIONS↩︎

We introduce SKooP, a unified learning approach which accelerates convergence and improves reward performance. Imposing symmetry constraints enables SKooP to generalize beyond training scenarios, while ecDAE-trained Koopman predictions for value function learning improves total return and qualitative learned behavior. We also demonstrate that SKooP is modular and produces consistent advantages across tasks. Future work will address alternative Koopman learning strategies and transfer to other robots and tasks.

Appendix↩︎

Here, we include a table listing some of the parameter settings we use for the quadruped training tasks we consider.

References↩︎