1 Introduction↩︎

Markov decision processes (MDPs) [1] have found a wide range of applications in different domains, ranging from optimal stopping [2], inventory management [3], or queueing control [4]. MDPs also play a central role in reinforcement learning (RL) [5], where the agent-environment interaction is typically modelled within the MDP framework. In recent years, RL has achieved remarkable success across diverse domains [6]–[8].

However, many relevant objectives cannot be easily expressed within the standard MDP framework [9]. Examples include imitation learning [10], [11], pure exploration problems [12], risk-averse RL [13], diverse skills discovery [14], [15], and constrained MDPs [16], [17]. To address these limitations, the framework of general-utility Markov decision processes (GUMDPs) has been proposed as a more expressive formalism capable of modelling such objectives [18]. In GUMDPs, the agent’s objective is encoded as a function of the occupancy induced by a given policy, i.e., as a function of the frequency of visitation of state-action pairs induced by a given policy. GUMDPs generalize MDPs by allowing the objective function to be a non-linear function of occupancies.

Previous studies have been focused on studying policy optimization in GUMDPs by considering risk-neutral objectives, where the performance of a given policy is evaluated in terms of its induced expected frequency of visitation of states [19]–[22]. However, these approaches overlook the variability in the objective: for a fixed policy, the stochastic nature of the environment can lead to a distribution over possible state-action visitation frequencies, resulting in a range of potential objective values. Consequently, optimizing only for the expectation may fail to capture important aspects of the policy’s performance. Given the stochasticity of the possible outcomes, it is thus natural to consider optimizing a specific risk measure [23] over the distribution of objective values, enabling the learned behavior to be tuned toward risk-seeking or risk-averse preferences. The entropic risk measure (ERM) [24] is one of the most common risk measures, which has received considerable attention across various domains such as finance and sequential decision-making [25]–[30]. As a risk-aware criterion, the ERM enables agents/decision-makers to trade off between expected performance and risk aversion, providing robustness against adverse (worst-case) outcomes. To further motivate the relevance of trading off between risk-neutral and risk-averse behaviors in the context of GUMDPs, we present the following illustrative example.

1.0.0.1 Motivating example

We consider the illustrative environment in [fig:motivating95example:a], where a robot aims to explore three rooms as uniformly as possible (pure exploration task [12]). Transitioning from room \(1\) to room \(0\) incurs no danger to the robot. However, attempting to transition from room \(1\) to room \(2\) may cause irreparable damage to the robot, rendering it unable to explore the environment further. The dynamics of this environment can be abstracted by the MDP in [fig:motivating95example:b], where each state corresponds to one of the rooms, the actions encode the transitions between rooms, and \(\epsilon \in (0,1)\) is the probability of irreparable damage to the robot when attempting to transition between rooms \(1\) and \(2\). We can resort to a GUMDP to encode the agent’s objective by setting the objective function to be (minus) the entropy of the occupancy induced by any policy. Now, a risk-averse agent will not be interested in trying to explore state 2 since there exists a probability that it will get absorbed into state 3 and, thus, not being able to explore the environment further. On the other hand, a risk-seeker agent will try to visit state 2 since it may be successful, hence exploring all three rooms. In [fig:motivating95example:c], we compare two policies:

The left plot shows the density of the objective values for the policies. Both policies yield very different distributions, with the risk-seeking policy \(\pi_1\) attaining lower (better) objective values at the cost of sometimes obtaining higher (worse) objective values. On the other hand, the risk-averse policy \(\pi_2\) always attains the same intermediate objective value due to its deterministic behavior. The plot on the right shows the \(\text{ERM}_\beta\) of both policies as a function of \(\beta \in (0, \infty)\). Higher \(\beta\) values correspond to more risk-averse preferences, while \(\beta\) approaching zero reflects risk neutrality. As shown, \(\pi_2\) achieves a lower \(\text{ERM}_\beta\) value for higher \(\beta\), confirming its risk-averse behavior. Conversely, as \(\beta\) decreases, \(\pi_1\) becomes the superior policy.

1.0.0.2 Contributions

We study risk-aware GUMDPs with ERM objectives. As a risk-aware criterion, the ERM enables agents/decision-makers to trade off between expected performance and risk aversion, providing robustness against adverse (worst-case) outcomes. Our contributions are threefold. First, we motivate, propose, and formalize risk-aware GUMDPs, which enable agents to trade off between expected performance and risk aversion while benefiting from the rich set of objectives that can be cast under the framework of GUMDPs. We focus our attention on the ERM, as it has been commonly used by previous works in the context of MDPs [26]–[30]. Second, we show how we can solve risk-aware GUMDPs with ERM objectives (ERM-GUMDP) by resorting to online planning techniques. In particular, we show that any ERM-GUMDP can be reformulated as a specific MDP — referred to as an occupancy MDP — with an ERM objective. This reformulation enables the use of online planning techniques to compute approximately optimal policies for arbitrary ERM-GUMDPs. We propose an MCTS-based approach to provably solve risk-aware GUMDPs up to any desired accuracy. Third, we provide experimental results showing that our approach successfully optimizes for a spectrum of risk-aware behaviors in the context of GUMDPs under diverse tasks (standard MDPs, maximum state entropy exploration, imitation learning, and multi-objective MDPs).

2 Background↩︎

2.1 Markov decision processes↩︎

MDPs [1] can be defined by the tuple \(\mathcal{M} = (\mathcal{S}, \mathcal{A}, \{{\boldsymbol{P}}^a : a \in \mathcal{A} \}, {\boldsymbol{p}_0}, c)\) where: \(\mathcal{S}\) is the discrete state space; \(\mathcal{A}\) is the discrete action space; \(\{{\boldsymbol{P}}^a : a \in \mathcal{A} \}\) is a set of transition probability matrices \({\boldsymbol{P}}^a\) for each action \(a \in \mathcal{A}\); \({\boldsymbol{p}_0} \in \Delta(\mathcal{S})\) is the distribution of initial states; and \(c: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}\) is a cost function. The interaction between the agent and its environment, as modeled by the MDP framework, is as follows:

Depending on the action chosen by the agent, the MDP evolves to a new state \(\mathrm{s}_{t+1} \in \mathcal{S}\) with probability given by \({\boldsymbol{P}}^{\mathrm{a}_t}(\mathrm{s}_t, \cdot)\), and the agent receives a cost \(c(\mathrm{s}_t, \mathrm{a}_t)\); and (iii) the interaction repeats infinitely.

A decision rule \(\pi_t\) specifies the action-selection mechanism at timestep \(t\). A non-Markovian decision rule maps the history of states and actions to a probability distribution over actions, i.e., \(\pi_t : \mathcal{S} \times (\mathcal{S} \times \mathcal{A})^t \rightarrow \Delta(\mathcal{A})\). In contrast, a Markovian decision rule depends only on the current state and maps it to a distribution over actions, i.e., \(\pi_t : \mathcal{S} \rightarrow \Delta(\mathcal{A})\). Both types of decision rules can be deterministic if they map directly to actions rather than distributions: for non-Markovian rules, \(\pi_t : \mathcal{S} \times (\mathcal{S} \times \mathcal{A})^t \rightarrow \mathcal{A}\), and for Markovian rules, \(\pi_t : \mathcal{S} \rightarrow \mathcal{A}\). A policy \(\pi = (\pi_0, \pi_1, \ldots)\) is a sequence of decision rules, one for each timestep. A policy is Markovian or non-Markovian if all its decision rules are Markovian or non-Markovian, respectively. Likewise, a policy is deterministic or stochastic if all its decision rules are deterministic or stochastic. We denote the set of non-Markovian policies by \(\Pi_{\text{NM}}\), Markovian policies by \(\Pi_{\text{M}}\), non-Markovian deterministic policies by \(\Pi^\text{D}_{\text{NM}}\), and Markovian deterministic policies by \(\Pi^\text{D}_{\text{M}}\). A policy is stationary if it uses the same decision rule at every timestep. We denote the set of stationary policies by \(\Pi_{\text{S}}\) and the set of stationary deterministic policies by \(\Pi_{\text{S}}^\text{D}\). Among the relationships between these classes, we highlight the following:

For a given policy \(\pi \in \Pi_\text{NM}\), the interaction between the agent and the environment induces a random process \((\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots)\). We denote by \(\mathrm{h}_t = (\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots, \mathrm{s}_t)\) the random history up to and including timestep \(t\), and by \(h_t = (s_0, a_0, s_1, a_1, \ldots, s_t) \in \mathcal{S}\times (\mathcal{S}\times \mathcal{A})^t\) a particular realization of such a history. The random process \((\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots)\) satisfies the following conditions:

Let \((\Omega, \mathcal{F}, \mathbb{P}_\pi)\) be the probability space over trajectories \((\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots)\) satisfying (i)–(iii), as defined in [31]. We denote a sample trajectory by \(\omega \in \Omega\), where \(\omega = (s_0, a_0, s_1, a_1, \ldots)\). We also let \(\mathbb{P}_\pi[\mathrm{s}_t = s, \mathrm{a}_t = a]\) be the probability of observing the state-action pair \((s, a)\) at timestep \(t\) under the MDP while following policy \(\pi\).

The expected discounted cumulative cost objective is \(J_{\gamma}^\pi = \mathbb{E}\left[\sum_{t=0}^\infty \gamma^t c(\mathrm{s}_t, \mathrm{a}_t) \right],\) where \(\gamma \in (0,1)\) is the discount factor and the expectation is taken over the random trajectory of state-action pairs. The discounted state-action occupancy for policy \(\pi\) is defined as \[\begin{align} \label{eq:discounted95state95action95occupancy} d_{\pi}(s,a) &= (1-\gamma) \sum_{t=0}^\infty \gamma^t \mathbb{P}_\pi\left[\mathrm{s}_t = s, \mathrm{a}_t = a\right]. \end{align}\tag{1}\] The expected discounted cumulative cost of policy \(\pi\) can be rewritten as \(J_{\gamma}^\pi = {\boldsymbol{c}}^\top {\boldsymbol{d}}_{\pi},\) where \({\boldsymbol{d}}_{\pi} = [d_{\pi}(s_0,a_0), \ldots, d_{\pi}(s_{|\mathcal{S}|},a_{|\mathcal{A}|})]^\top, \; {\boldsymbol{c}} = [c(s_0,a_0), \ldots, c(s_{|\mathcal{S}|},a_{|\mathcal{A}|})]^\top.\) We aim to find \(\pi^* = \mathop{\mathrm{arg\,min}}_{\pi \in \Pi_\text{S}} J_{\gamma}^\pi\), which can be formulated as a linear program [1].

2.2 Risk-aware MDPs↩︎

Risk-aware MDPs [32] generalize the classical MDP framework by considering the associated variability of the discounted cumulative costs. This is done by considering \(J_{\gamma, \rho}^\pi = \rho\left(\sum_{t=0}^\infty \gamma^t c(\mathrm{s}_t, \mathrm{a}_t) \right),\) where \(\rho\) is a risk measure [23], computed over the random trajectory. Risk measures quantify an agent’s attitude toward uncertainty and variability in outcomes. We note that for \(\rho(\cdot)=\mathbb{E}[\cdot]\) we have the standard discounted MDP objective. The risk-aware objective allows the agent to reason over the entire distribution instead of just its expectation.

In this work, we consider risk-aware MDPs with an ERM objective [24]. The ERM of a random variable \(\mathrm{z}\) is defined as \(\text{ERM}_\beta(\mathrm{z}) = \frac{1}{\beta} \ln\left( \mathbb{E}\left[\exp(\beta \mathrm{z})\right]\right)\), where parameter \(\beta \in (0, \infty)\) trades off between expected performance as \(\beta \rightarrow 0\) and risk-aversion as \(\beta\) increases. Also, the \(\text{ERM}_\beta\) belongs to the family of optimized certainty equivalents (OCEs) [33] since it can be equivalently defined as \(\text{ERM}_\beta(\mathrm{z}) = \min_{\lambda \in \mathbb{R}} \left\{ \lambda + \mathbb{E}\left[ u(\mathrm{z} - \lambda) \right]\right\},\) where \(u(x) = \frac{1}{\beta} \left( \exp(\beta x) - 1\right)\). In this work, we focus our attention on the \(\text{ERM}_\beta\) and, thus, we implicitly refer to the \(\text{ERM}_\beta\) whenever we write \(\rho\).

2.3 General-utility MDPs↩︎

GUMDPs generalize utility specification by allowing the agent’s objective to be written in terms of the frequency of visitation of state-action pairs. This is in contrast to the standard MDPs framework, where the objective of the agent is encoded by the cost function. We define an infinite-horizon discounted GUMDP as a tuple \(\mathcal{M}_f = (\mathcal{S}, \mathcal{A}, \{{\boldsymbol{P}}^a : a \in \mathcal{A} \}, {\boldsymbol{p}_0}, f)\), where \(\mathcal{S}\), \(\mathcal{A}\), \(\{{\boldsymbol{P}}^a : a \in \mathcal{A} \}\), and \({\boldsymbol{p}_0}\) are defined in a similar way to the standard MDP formulation, as introduced in 2.1. The function \(f: \Delta(\mathcal{S} \times \mathcal{A}) \rightarrow \mathbb{R}\) encodes the objective of the agent, which depends on a discounted occupancy \({\boldsymbol{d}}\), as defined in 1 . We then aim to find \(\pi^* \in \mathop{\mathrm{arg\,min}}_{\pi \in \Pi_\text{S}} f({\boldsymbol{d}}_\pi).\) If \(f\) is a linear function, then we are under the standard MDP setting. If \(f\) is convex, then we are under the convex MDP/RL setting [21].

In this work, we consider four tasks, each associated with a particular objective function:

Nevertheless, we highlight that our results apply to any task that can be modeled using the GUMDPs framework. We refer to [21] for a comprehensive list of the different objectives considered by previous works.

3 Towards Risk-Aware GUMDPs↩︎

We now introduce the framework of risk-aware GUMDPs. Because risk-awareness requires reasoning about the underlying distribution of objective values, we begin by adopting a distributional perspective on GUMDPs. Similarly to distributional RL [36], we employ the following assumption to ensure that all random variables defined below are bounded and well-defined.

3.1 A distributional perspective on GUMDPs↩︎

For a given fixed policy \(\pi \in \Pi_\text{NM}\), let \(\mathbf{d}^\pi : \Omega \rightarrow \Delta(\mathcal{S}\times \mathcal{A})\) be the random vector defined on the probability space \((\Omega, \mathcal{F}, \mathbb{P}_\pi)\) with entries \[\mathrm{d}^\pi_{s,a}(\omega) = (1-\gamma) \sum_{t=0}^\infty \gamma^t \mathbf{1}(s_t=s,a_t=a). \label{eq:empirical95occupancy}\tag{2}\]

The random vector above is well-defined since the mapping \(\mathbf{d}^\pi\) is \(\mathcal{F}/\mathcal{B}(\Delta(\mathcal{S}\times \mathcal{A}))\) measurable, where \(\mathcal{B}(\Delta(\mathcal{S}\times \mathcal{A}))\) denotes the Borel \(\sigma\)-algebra of \(\Delta(\mathcal{S}\times \mathcal{A})\). Let also \(\mathrm{f}^\pi : \Omega \rightarrow \mathcal{Y}\) be the random variable with support in \(\mathcal{Y}\) such that \[\mathrm{f}^\pi(\omega) = (f \circ \mathbf{d}^\pi) (\omega) = f(\mathbf{d}^\pi(\omega)),\] satisfying \[\mathbb{P}\left[ \mathrm{f}^\pi \le \eta \right] = \mathbb{P}_\pi\left[\left\{\omega \in \Omega : f(\mathbf{d}^\pi(\omega)) \le \eta \right\}\right], \quad \forall \eta \in \mathcal{Y},\] where we assume \(f\) is a Borel measurable function. Essentially, the random variable \(\mathrm{f}^\pi\) is defined as the composition of two measurable mappings. If \(f\) is continuous, then \(f\) is Borel measurable.

We refer to [fig:motivating95example:c] for an illustration of the density associated with the random variable \(\mathrm{f}^\pi\) for two arbitrary policies. As observed, the underlying distributions induced by different policies can differ substantially and, therefore, depending on the exact preferences of the decision-maker, one policy may be preferable over the other. Risk measures, such as the \(\text{ERM}_\beta\), allow for the comparison between distributions induced by different policies.

3.2 Risk-aware GUMDPs↩︎

In the context of GUMDPs, we denote \(\mathcal{L}\) the set of possible random variables with support in \(\mathcal{Y}\) associated with each of the probability spaces \((\Omega, \mathcal{F}, \mathbb{P}_\pi)\), for all \(\pi \in \Pi_\text{NM}\). Note that \(\mathrm{f}^\pi \in \mathcal{L}\) for any \(\pi \in \Pi_\text{NM}\). Then, we consider risk measures \(\rho: \mathcal{L}\rightarrow\mathbb{R}\) that associate any random variable in \(\mathcal{L}\) with a scalar value. In this work, we are particularly interested in the \(\text{ERM}_\beta\), as introduced in 2.2. We aim to find \[\pi^* = \mathop{\mathrm{arg\,min}}_{\pi \in \Pi_\text{NM}} \rho(\mathrm{f}^\pi). \label{eq:risk95aware95gumdp95objective}\tag{3}\] For any policy \(\pi \in \Pi_\text{NM}\), we define its optimality gap as \(\mathrm{OptGap}(\pi) = \rho\left(\mathrm{f}^\pi\right) - \min_{\pi' \in \Pi_{\text{NM}}} \rho\left(\mathrm{f}^{\pi'}\right).\) Essentially, \(\mathrm{OptGap}(\pi)\) measures how suboptimal a given policy \(\pi\) is compared to the best policy.

3.2.0.1 Policy optimization in risk-aware GUMDPs

While in the context of risk-aware MDPs with ERM objectives the class of deterministic and Markovian policies, \(\Pi_\text{M}^\text{D}\), suffices for optimality [28], this is no longer the case in the context of risk-aware GUMDPs with ERM objectives. As shown in [20] in the context of GUMDPs, if \(\rho(\cdot) = \mathbb{E}[\cdot]\) then the class of Markovian policies is strictly dominated by the class of non-Markovian policies. Since, in the context of risk-aware GUMDPs, as \(\beta \rightarrow 0\) the \(\text{ERM}_\beta\) approaches \(\mathbb{E}[\cdot]\), non-Markovian policies are also needed, in general, in the context of solving GUMDPs with ERM objectives.

4 Solving Risk-Aware GUMDPs with Online Planning↩︎

In this section, we study how to solve risk-aware GUMDPs using online planning techniques. We begin by establishing a close connection between risk-aware GUMDPs and a particular risk-aware MDP in which the agent tracks the accrued occupancy at each timestep. Leveraging this formulation, we show that solving the risk-aware MDP with online planning techniques yields approximately optimal action selection for the original risk-aware GUMDP.

We refer to 8 for the derivation of \(L_f\)-Lipschitz constants for the objective functions considered.

4.1 Computing optimal policies by resorting to finite-horizon GUMDPs↩︎

We start by showing that we can compute an approximately optimal policy for our infinite-horizon discounted risk-aware objective 3 by resorting to a finite-horizon formulation of the same problem. For any policy \(\pi \in \Pi_\text{NM}\), let \(\mathrm{f}^\pi_H : \Omega \rightarrow \mathcal{Y}\) be the random variable with support in \(\mathcal{Y}\) such that \[\begin{align} \mathrm{f}^\pi_H(\omega) &= (f \circ \mathbf{d}^{\pi,H}) (\omega) = f(\mathbf{d}^{\pi,H}(\omega)) \tag{4} \\ \mathrm{d}^{\pi,H}_{s,a}(\omega) &= \frac{1-\gamma}{1-\gamma^H} \sum_{t=0}^{H-1} \gamma^t \mathbf{1}(s_t=s,a_t=a)\tag{5}. \end{align}\] Essentially, random variable \(\mathrm{f}^\pi_H\) is defined in a similar fashion to the random variable \(\mathrm{f}^\pi\), but the former considers a random occupancy \(\mathbf{d}^{\pi,H}\) that only takes into account the state-action pairs visited up to a given horizon \(H \in \mathbb{N}\). We state the following result (proof in 9).

The result above shows that we can resort to the truncated objective to compute approximately optimal policies, up to any desired accuracy, for the infinite-horizon risk-aware objective 3 . Therefore, we focus our attention on solving \(\pi^* = \mathop{\mathrm{arg\,min}}_{\pi \in \Pi_\text{NM}} \rho(\mathrm{f}^\pi_H),\) where \(\mathrm{f}^\pi_H\) is defined in 4 . Since we now focus on the case of finite-horizon trajectories, we assume that, for a given policy \(\pi \in \Pi_\text{NM}\), the interaction between the agent and the environment gives rise to a random process \((\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots, \mathrm{s}_{H-1}, \mathrm{a}_{H-1})\) such that:

We thus let from now on \((\Omega, \mathcal{F}, \mathbb{P}_\pi)\) be the probability space over the sequence of random variables \((\mathrm{s}_0, \mathrm{a}_0, \mathrm{s}_1, \mathrm{a}_1, \ldots, \mathrm{s}_{H-1}, \mathrm{a}_{H-1})\) that satisfies conditions (i)-(iii) above. We write specific trajectories as \(\omega \in \Omega\), with \(\omega = (s_0, a_0, s_1, a_1, \ldots, s_{H-1}, a_{H-1})\). We highlight that the probability of a given trajectory \(\omega \in \Omega\) under policy \(\pi \in \Pi_\text{NM}\) can be calculated as \(\mathbb{P}_{\pi}\left[\omega\right] = p_0(s_0) \cdot \pi(a_0|h_0) \cdot P^{a_0}(s_0, s_1) \cdot \pi(a_1|h_1) \cdot P^{a_1}(s_1, s_2) \ldots P^{a_{H-2}}(s_{H-2}, s_{H-1}) \cdot \pi(a_{H-1}|h_{H-1})\).

4.2 The occupancy MDP↩︎

To derive our planning algorithms for solving risk-aware GUMDPs, we make use of a finite-horizon MDP derived from the original GUMDP formulation. In particular, we consider the occupancy MDP [20], defined by the tuple \(\mathcal{M}_{\text{O}} = \{\mathcal{S}_\text{O}, \mathcal{A}_\text{O}, \{{\boldsymbol{P}}_\text{O}^a : a \in \mathcal{A} \}, {\boldsymbol{p}}_{0,\text{O}}, c_\text{O}, H\}\), where \(\mathcal{S}_\text{O} = \mathcal{S} \times \mathcal{O}\) is the discrete state space and \(\mathcal{O}\) is the set of occupancies up to length \(H-1\) in the original GUMDP. We let \(\{s,{\boldsymbol{o}}\}\) be a state of the occupancy MDP such that \(s \in \mathcal{S}\) is a state from the original GUMDP and \({\boldsymbol{o}} \in \mathcal{O}\) is a \(|\mathcal{S}||\mathcal{A}|\)-dimensional vector that keeps track of the running occupancy of the agent up to a given timestep. Then, \(\mathcal{A}_\text{O} = \mathcal{A}\) is the action space and \({\boldsymbol{p}}_{0,\text{O}}\) is such that \(p_{0,\text{O}}(\{s,{\boldsymbol{o}}\}) = p_0(s)\) if \({\boldsymbol{o}} = [0,\ldots,0]\) and zero otherwise. The dynamics are as follows: (i) component \(\mathrm{s}_{t+1} \sim {\boldsymbol{P}}^{\mathrm{a}_t}(\cdot | \mathrm{s}_t)\) evolves according to the dynamics of the original GUMDP; and (ii) the running occupancy evolves deterministically as \(o_{t+1}(s,a) = \gamma^t + o_t(s,a)\) if \(s = \mathrm{s}_t\) and \(a = \mathrm{a}_t\), and \(o_{t+1}(s,a) = o_t(s,a)\) otherwise. Finally, \(H \in \mathbb{N}\) denotes the horizon and the cost function \(c_\text{O} : \mathcal{S}\times \mathcal{O}\rightarrow \mathbb{R}\) is defined as \[c_\text{O}(\{s,{\boldsymbol{o}}\}) = \begin{cases} 0 & \text{if t < H }, \\ f\left(\frac{1-\gamma}{1-\gamma^H} {\boldsymbol{o}}\right) & \text{if t = H }. \end{cases}\]

Stationary policies \(\pi_\text{O} \in \Pi_\text{S}\) for \(\mathcal{M}_\text{O}\) are mappings of the type \(\pi_\text{O} : \mathcal{S}\times \mathcal{O}\rightarrow \Delta(\mathcal{A})\). For a given policy \(\pi_\text{O} \in \Pi_\text{S}\), the interaction between the agent and the occupancy MDP gives rise to a random process \((\{\mathrm{s}_0,\mathbf{o}_0\}, \mathrm{a}_0, \{\mathrm{s}_1,\mathbf{o}_1\}, \mathrm{a}_1, \ldots, \{\mathrm{s}_{H}, \mathbf{o}_H\})\) associated with the probability space \((\Omega_\text{O}, \mathcal{F}_\text{O}, \mathbb{P}_{\pi_\text{O}}^O)\). We write specific trajectories as \(\omega_\text{O} \in \Omega_\text{O}\), with \(\omega_\text{O} = (\{s_0,{\boldsymbol{o}}_0\}, a_0, \{s_1,{\boldsymbol{o}}_1\}, a_1, \ldots, \{s_{H}, {\boldsymbol{o}}_H\})\).

For any policy \(\pi_\text{O} \in \Pi_\text{S}\), we let \(\mathrm{J}_\text{O}^{\pi_\text{O}}: \Omega_\text{O} \rightarrow \mathcal{Y}\) be the random variable with support in \(\mathcal{Y}\) associated with the probability space \((\Omega_\text{O}, \mathcal{F}_\text{O}, \mathbb{P}_{\pi_\text{O}}^\text{O})\) such that \(\mathrm{J}_\text{O}^{\pi_\text{O}}(\omega_\text{O}) = \sum_{t=0}^H c_\text{O}(\{s_t, {\boldsymbol{o}}_t\}).\) We reproduce the following result from [20].

An important conclusion that can be derived from the result above is that there exists a one-to-one mapping between non-Markovian policies for \(\mathcal{M}_f\) and stationary policies for \(\mathcal{M}_\text{O}\). This is because every state in \(\mathcal{M}_\text{O}\) is uniquely associated with a particular history in \(\mathcal{M}_f\) (and vice versa), as the result above shows. With this in mind, we now state the following results (proofs in 9).

The result above shows that it suffices to search for an (approximately) stationary optimal policy for \(\mathcal{M}_\text{O}\), since such a policy corresponds to a non-Markovian policy that is (approximately) optimal for \(\mathcal{M}_f\). In particular, such an (approximately) optimal policy for \(\mathcal{M}_\text{O}\) can be seen as a non-Markovian policy for \(\mathcal{M}_f\) that compresses the history up to any timestep into a running occupancy. The result above shows that it suffices to keep track of the running occupancy up to any timestep in order to attain optimal behavior when solving risk-aware GUMDPs.

In light of 2, we consider risk-aware planning algorithms to solve the occupancy MDP. Unfortunately, solving the risk-aware occupancy MDP poses some challenges. One of the key challenges is due to the fact that the size of the state space of the occupancy MDP grows combinatorially with \(H\) since every state in the occupancy MDP is associated with a possible history in \(\mathcal{M}_f\). Consequently, the state space of the occupancy MDP is typically very large, which precludes the use of offline planning methods. In the next section, we therefore turn to online planning approaches to implicitly compute approximately risk-aware optimal policies for the occupancy MDP.

4.3 Solving risk-aware occupancy MDPs via online planning↩︎

We aim to propose a practical algorithm to compute an approximately optimal risk-aware policy for the occupancy MDP \(\mathcal{M}_O\), as introduced in 4.2. Since the occupancy MDP is a standard undiscounted finite-horizon MDP, we can resort to the risk-aware Monte Carlo tree search (MCTS) algorithm put forth by [37], ERM-MCTS, to solve the risk-aware occupancy MDP with the ERM objective. ERM-MCTS works in a similar fashion to standard MCTS in the sense that the algorithm iteratively builds a search tree that alternates between decision nodes, where actions are selected, and chance nodes corresponding to random next states sampled from the MDP. At each iteration, ERM-MCTS refines the search tree by simulating a random trajectory in the occupancy MDP. Action-selection at each decision node \(\mathrm{s}_t \in \mathcal{S}_{O}\) is given by \[\mathrm{a}_t \in \mathop{\mathrm{arg\,min}}_{a \in \mathcal{A}} \Bigg\{ \frac{1}{\beta} \ln\left(\frac{1}{N(\mathrm{s}_t,a)} \sum_{i=1}^{N(\mathrm{s}_t,a)} \exp \left(\beta \mathrm{x}_i^{(\mathrm{s}_t,a)} \right)\right) - \theta_t \sqrt{\frac{\sqrt{N(\mathrm{s}_t)}}{N(\mathrm{s}_t,a)}}\Bigg\},\] where \(N(\mathrm{s}_t)\) is the number of times \(\mathrm{s}_t\) has been visited, \(N(\mathrm{s}_t,a)\) the number of times action \(a\) has been selected while in \(\mathrm{s}_t\), \(\mathrm{x}_i^{(\mathrm{s}_t,a)}\) is the \(i\)-th sampled terminal cost starting from \((\mathrm{s}_t,a)\) (given that the occupancy MDP only has non-zero costs at \(t=H\)), and \(\theta_t\) is an exploration constant. In case \(N(\mathrm{s}_t,a) = 0\) for some \(a \in \mathcal{A}\), then \(a\) is selected. We refer to 10 for the full pseudocode of the ERM-MCTS algorithm. Without loss of generality, assume we fix an initial state \(s_0 \in \mathcal{S}\). We state the following result, which is a consequence of Theo. 6 in [37].

5 Experimental Results↩︎

We empirically assess the performance of ERM-MCTS for solving risk-aware GUMDPs, investigating how ERM-MCTS trades off risk-neutral and risk-averse behavior. We describe our experimental methodology and refer to 11 for a complete description of our experiments (environments, baselines, hyperparameters, etc.).

We consider four tasks:

The definition of the objective function for each task is given in 2.3. We consider two sets of environments: (i) illustrative environments that consist of low-dimensional GUMDPs; and (ii) high-dimensional GUMDPs consisting of grid-based environments showcasing different sources of stochasticity in order to encourage distinct types of policy behaviors. To our knowledge, we are the first work to propose an algorithm to solve risk-aware GUMDPs and, hence, there are no baselines that we can use to compare the performance of our method against. However, for the particular case of linear \(f\) (MDP), we use an oracle baseline ERM-BI to validate the performance of our ERM-MCTS algorithm. ERM-BI computes the optimal risk-aware policy for the underlying occupancy MDP using a dynamic programming approach by exploiting the dynamic decomposition of the \(\text{ERM}_\beta\) put forth by [28]. All our plots are computed by aggregating the experimental results of, at least, 100 independent runs, and we refer to [tab:appendix:illustrative-gumdps-hyperparams,tab:appendix:grid-gumdps-hyperparams] for the detailed list of our experimental hyperparameters across all environments.

5.1 Illustrative environments↩︎

We display in 2 our illustrative environments. The MDP (2 (a)) consists of a four-state MDP where the agent needs to tradeoff at the initial state (\(s_0\)) between: (i) a risky action (\(a_0\)) that can lead to a low-cost state (\(s_2\)), but with some probability the agent ends in a high-cost state (\(s_3\)); and (ii) a safe action (\(a_1\)) that deterministically leads to a medium-cost state (\(s_1\)). The agent resets to the initial state with \(10\%\) probability while not in the initial state. For the IL task (2 (a)), the agent aims to imitate the empirical occupancy induced by the trajectory of an agent that selected twice the risky action (\(a_0\)) and then selected the safe action (\(a_1\)) for all the remainder timesteps. The trajectory to imitate had a rather “lucky” outcome, as it never ended up in the absorbing state (\(s_3\)). Hence, the agent needs to trade off between imitating the behavior policy in states (\(s_0\), \(s_1\), \(s_2\)) and risking being absorbed into \(s_3\), or only imitating the behavior policy in the “less risky” states (\(s_0\), \(s_1\)). The MSEE task (2 (b)) is similar to the motivating example from 1, where the agent aims to explore an environment as uniformly as possible, but there is a chance that the agent transitions to an absorbing state (\(s_4\)). Finally, the MO-MDP (2 (c)) is inspired by the FishWood environment [38], where the agent needs to tradeoff between two cost functions. Each cost function penalizes different behaviors. We consider three utility functions [39]: (i) a weighted combination of the discounted cumulative costs; (ii) the maximum of the discounted cumulative costs; and (iii) the minimum of the discounted cumulative costs.

In  3, we display the box plots computed for the empirical distributions of costs obtained by ERM-MCTS under different tasks and \(\beta\) values. To validate our algorithm, we include a cost-minimization task corresponding to a standard MDP. As seen, for the MDP, the box plot obtained by ERM-MCTS closely matches that obtained by the ERM-BI oracle baseline across the tested \(\beta\) values, providing empirical validation of ERM-MCTS. Furthermore, across all tasks, ERM-MCTS successfully trades off risk-neutral and risk-averse behavior as a function of \(\beta\). As \(\beta\) increases and ERM-MCTS progressively focuses on optimizing worst-case outcomes, upper whiskers and outliers (i.e., the worst outcomes) of the box plots decrease. This indicates that worst-case outcomes become less probable and that ERM-MCTS effectively computes increasingly risk-averse policies. Naturally, optimizing for worst-case outcomes typically degrades best-case or expected outcomes, as reflected by the increase in median values and in the lower whiskers and outliers as \(\beta\) increases.

5.2 Grid environments↩︎

5.2.1 Maximum State Entropy Exploration↩︎

The MSEE task integrates a \(10\times10\) grid that comprises two different types of squares:

The agent can move in all four directions, ending up in the corresponding adjacent square (if moving out-of-bounds, the agent remains in the same square). When the agent is inside a difficult terrain square, there is a \(p_\text{stuck}\) probability that the agent gets stuck, which translates in canceling the selected (moving) action and thus remaining trapped in the current square. Upon getting stuck, there is an additional \(p_\text{unstuck}\) probability that untraps the agent allowing it to move freely again. As expected, inside normal terrain, the agent’s selected moving action is always guaranteed to occur. Due to the environment’s stochastic nature, the agent needs to consider if there are benefits in exploring certain difficult terrain squares at the expense of getting stuck, which can have a negative effect on the discounted costs received from that point onward. In 4 (a), we report the box plots of the empirical costs obtained by ERM-MCTS with \(\beta\in\{0.001,1,1000\}\). When \(\beta\in\{0.001,1\}\), ERM-MCTS outputs similar risk-seeking policies, as both display identical Inner Fence Interval (IFI) ranges, inside \([0.1, 0.2]\). Additionally, the expectation value is lower when \(\beta=0.001\), having a value of \(0.167\), whereas the expectation for \(\beta=1\) is \(0.177\). Furthermore, both policies exhibit a wide range of outliers on the upper-half of the distribution, implying that in some runs the agent got trapped for long periods of time when covering difficult terrain squares, further alluding to the potential disadvantages of having a risk-seeking behavior. Additionally, the worst outlier is higher for the policy with \(\beta=0.001\), which is expected since this policy is effectively closer to the risk-seeking (or risk-neutral) boundary in the risk-awareness spectrum. When comparing the policy obtained when \(\beta=1000\) to the the previous two (\(\beta\in\{0.001,1\}\)), the range of the objective distribution becomes more concentrated as there are fewer outliers. This shows the effectiveness of the risk-averse behavior in avoiding the worst outcomes, as the highest cost is only \(0.48\) when setting \(\beta=1000\), vs. \(0.93\) using \(\beta=0.001\). The drawback of this policy manifests by inducing an average case that performs worse than policies that are more risk-seeking, where the average of the objective distributions are \(0.30\) and \(0.167\), for the policies using \(\beta=1000\) and \(\beta=0.001\), respectively. Finally, we further validate our analysis with 5, which exposes the average number of timesteps spent in each square over all runs. We observe that when \(\beta=1000\), the policy navigates towards the lower-right corner to avoid difficult terrain squares altogether. As \(\beta\) decreases, the heatmap becomes more uniformly distributed showcasing that the policy tries to visit all squares (normal and difficult terrain) more often.

5.2.2 Imitation Learning↩︎

We reuse the MSEE environment for our IL task. Additionally, we carefully design a behavior policy that changes from risk-seeking and risk-aware behaviors at well-defined intervals. In high-level terms, the behavior policy starts by exploring a zone containing both types of squares, followed by a period where only normal squares are visited. This procedure repeats a second time to make sure all grid squares are visited once. Furthermore, the behavior policy will not be subjected to the environment’s stochasticity, i.e., it will never get trapped when visiting difficult terrain. In this manner, we guarantee that achieving a complete match will be extremely unlikely, further leading the ERM-MCTS policy to consider when and what risks to take.

To analyze the IL task, we compare ERM-MCTS with three different beta values, \(\beta\in\{0.02,1,500\}\). As shown in 4 (b), we clearly observe the same pattern reported in the MSEE task (5.2.1): as \(\beta\) increases, the maximum cost obtained decreases while the expectation and median increase. Here, the expectation obtains the values \(0.106\), \(0.109\), and \(0.192\), while the median acquires \(0.08\), \(0.09\), and \(0.18\) for the policies with \(\beta=0.02\), \(\beta=1\), and \(\beta=500\), respectively. Conversely, and for the same policies, the uppermost outliers are \(0.81\), \(0.65\), and \(0.44\), respectively. To get a closer look at how behaviors differ as \(\beta\) changes, we plot, in 6, the average number of runs visiting difficult terrain squares in each timestep. Interestingly, the policy with the highest \(\beta\) emphasizes matching the occupancy of normal squares during the first 40 time steps, as there is a negligible number of runs stepping inside difficult terrain. For the remaining time, the policy focuses more on equalizing the occupancy at difficult terrain squares, as outlined by the increased number of runs exploring such squares. We argue that such behavior is due to discounting, since it is more forgiving to get trapped closer to the end of the episode, considering that it has less impact on the discounted return. Contrastingly, the other policies (\(\beta\in\{0.02,1\}\)) start visiting difficult terrain squares from the beginning of the episode. Consequently, the likelihood of getting trapped sooner and for longer periods of time increases, as hinted in 4 (b) by the wide range of outliers on the upper half of the cost distribution.

5.2.3 Multi-Objective↩︎

For the MO task, we extend the resource-gathering environment [40], aiming at introducing challenging probabilistic dynamics that allow the modulation of policies demonstrating behaviors with different degrees of risk-awareness. In the modified environment, the agent’s objective consists in retrieving two different resources (\(R_1\) and \(R_2\)) back to its starting location while avoiding enemies present at specific squares. Additionally, we set a slippery dynamic, where with probability \(p_\text{slip}\) the agent will move perpendicularly to the selected direction. All agent objectives are modulated using the following reward functions:

Finally, we combine the three independent reward functions using a non-linear mapping that prioritizes collecting \(R_1\) over the other ones. We refer to 11.5.1 for a complete description of the dynamics, cost functions, and hyperparameters used.

We simulate ERM-MCTS with \(\beta\in\{0.125,1,8\}\). As shown in 4 (c), the cost distributions obtained by the policies where \(\beta\in\{0.125,1\}\) are highly identical. We again observe the risk-seeking behavior of these policies, since, on average, the costs obtained remain concentrated near the lower bound. For instance, the expectation is \(0.16\) and \(0.18\) for \(\beta=0.125\) and \(\beta=1\), respectively. The IFI is \([0.10,0.14]\) in both cases. Furthermore, since during the optimization of the ERM’s objective, the expectation outweighs the upper tail of the distribution, eventually some runs conclude at the worst possible outcome, in which the agent gets defeated by an enemy. Contrastingly, the policy generated for \(\beta=8\) clearly eliminates this possible outcome. In this case, the agent never gets the cost of \(1\) by inducing a risk-aware behavior that most of the time prefers not to collect \(R_1\) due to its proximity to enemies and, as such, the likelihood of defeat. Therefore, the median centers around \(0.87\), indicating the agent never retrieved any resources to its initial position. Finally, in [fig:grid95envs95mo95outcome], we show in more detail the frequency of outcomes that each policy achieves. As hinted previously, when \(\beta\in\{0.125,1\}\) we observe a small fraction of runs where the agent gets defeated. On the other hand, the risk-averse policy (\(\beta=8\)) disallows this outcome entirely by, most of the time (82%), either avoiding gathering any resources or just retrieving \(R_2\).

6 Conclusion↩︎

We motivate, propose and formalize risk-aware GUMDPs, which allow to take advantage of the flexibility of the GUMDPs framework with respect to objective specification, while trading off expected performance and risk-aversion. To solve risk-aware GUMDPs, we first explore a connection between risk-aware GUMDPs and solving a particular risk-aware MDP, named occupancy MDP, in which the agent keeps track of the empirical frequency of visitation of state-action pairs up to the current timestep. We propose a provably correct online planning approach based on an MCTS algorithm to solve the risk-aware occupancy MDP, effectively solving the original risk-aware GUMDP problem up to any desired accuracy. We provide a set of experimental results under a set of diverse tasks showing that our approach successfully trades off risk-neutral and risk-averse behavior.

Future work could investigate whether a similar approach to ours can be used to solve GUMDPs with CVaR objectives [32], [41]. Other interesting direction is to investigate whether our approach can be extended to deal with very large/inherently continuous state spaces, e.g., by borrowing ideas from successor features [42], [43].

Acknowledgments↩︎

This work was supported by Portuguese national funds through the Portuguese Fundação para a Ciência e a Tecnologia (FCT) under projects UID/50021/2025 and UID/PRR/50021/2025 (INESC-ID multi-annual funding), as well as AI-PackBot (project number 14935, LISBOA2030-FEDER-00854700). Pedro P. Santos acknowledges the FCT PhD grant 2021.04684.BD and Fábio Vital acknowledges the FCT PhD grant 2022.14163.BD. Alberto Sardinha acknowledges the CNPq Research Productivity Fellowship (PQ), with reference 312699/2025-5. The authors thank the lab managers at GAIPS for the support provided when running the computational experiments of this work. The authors also thank Jacopo Silvestrin for discussions on earlier versions of this work and Zita Marinho for feedback on the manuscript.

7 Extended related work discussion↩︎

Previous works studied GUMDPs with risk-neutral objectives [21], [22]. [21] reformulate GUMDPs as a two-player game involving a policy and a cost (negative reward) player, using Fenchel duality. [22] connect GUMDPs and mean-field games. Still in the risk-neutral setting, subsequent works identified a key implicit assumption underlying the GUMDPs framework: the performance of a given policy may depend on the number of trials/trajectories drawn to evaluate its performance [18], [19]. In fact, the previous works show that the standard formulation of GUMDPs implicitly assumes the performance of a given policy is evaluated under an infinite number of trials/trajectories, an assumption that may be violated under many interesting application domains. To address this gap, [19], [20] introduce finite trials formulations for GUMDPs, allowing to find optimal risk-neutral policies with respect to a finite number of trials/trajectories. None of the works above study risk-aware settings, being focused on learning risk-neutral policies. As pointed out in the main text, some of the results in our paper build on top of results put forth by the aforementioned studies. As an example, our work relies on the occupancy MDP formulation, as described in Sec. 4.2, which was introduced in [20].

In the context of risk-aware MDPs, previous works studied the optimization of risk-aware objectives such as the conditional value-at-risk [32] or the ERM [28]–[30]. In the context of our work, we are focused on more general objective functions of the occupancies induced by the different policies, not only being focused on the case of linear objectives (MDPs). Nevertheless, some of the techniques we employ resemble those adopted in the context of risk-aware MDPs. As an example, it is common in the context of risk-aware MDPs to extend the state space to accommodate sufficient statistics since the optimal policy depends on the history only through a certain kind of “sufficient statistic” [44]. This resembles the construction of our occupancy MDP; however, the states in the occupancy MDP do not correspond to sufficient statistics of the history but instead correspond directly to histories (Lemma 1). This is required since, under more general settings, optimal policies may depend arbitrarily on the history. Other works, such as [28], exploit discounting and consider finite-horizon objectives to compute approximately optimal policies for infinite-horizon objectives - this technique is common in the field, and we followed a similar approach in 1.

Finally, in the context of MDPs, [45] propose a new definition of the risk of a policy (called caution) as a function of the occupancy induced by the policy. Then, the authors formulate a caution-sensitive policy optimization problem by adding the caution risk as a penalty function to the dual objective of the linear programming formulation of MDPs. The formulation proposed by the authors can itself be seen as solving a risk-neutral GUMDP, where the objective function contains a term that captures the risk of a given policy. In our work, we are instead focused on solving risk-aware GUMDPs.

8 Lipschitz constants↩︎

8.0.0.1 Standard MDP (\(f({\boldsymbol{d}}) = {\boldsymbol{c}}^\top {\boldsymbol{d}}\))

It holds that \[|f({\boldsymbol{d}}_1) - f({\boldsymbol{d}}_2)| = |{\boldsymbol{c}}^\top ({\boldsymbol{d}}_1 - {\boldsymbol{d}}_2)| \le \sum_{s,a} |c(s,a)| |d_1(s,a) - d_2(s,a)| \le \max_{s,a} |c(s,a)| \| {\boldsymbol{d}}_1 - {\boldsymbol{d}}_2\|_1.\]

8.0.0.2 Maximum state entropy exploration (\(f({\boldsymbol{d}}) = {\boldsymbol{d}}^\top \log({\boldsymbol{d}})\))

We assume \({\boldsymbol{d}}\) is lower bounded by \(\epsilon\), i.e., \(d(s,a) \ge\epsilon\) with \(0 < \epsilon < e^{-2}\) for all \(s \in \mathcal{S}, a \in \mathcal{A}\). We let \(f({\boldsymbol{d}}) = \sum_{s,a} g(d(s,a))\), for \(g(x) = x \log(x)\). We note that, \(g'(x) = \log(x) + 1\) and it holds for any \(x \in [\epsilon,1]\) that \(|g'(x)| \le |\log(\epsilon) + 1|\). Thus, for any \(x_1, x_2 \in [\epsilon,1]\) we have that

\[\begin{align} | g(x_1) - g(x_2)| &= \left| \int_{x_2}^{x_1} g'(x) dx \right| = \left| \int_{\min\{x_1,x_2\}}^{\max\{x_1,x_2\}} g'(x) dx \right|\\ &\le \int_{\min\{x_1,x_2\}}^{\max\{x_1,x_2\}} \left|g'(x) \right| dx \le \int_{\min\{x_1,x_2\}}^{\max\{x_1,x_2\}} \left|\log(\epsilon) + 1 \right| dx \\ &= \left|\log(\epsilon) + 1 \right| |x_1 - x_2|. \end{align}\]

Thus, for any \({\boldsymbol{d}}_1, {\boldsymbol{d}}_2 \in \Delta(\mathcal{S}\times \mathcal{A})\) lower bounded by \(0 < \epsilon < e^{-2}\), it holds that \[\begin{align} |f({\boldsymbol{d}}_1) - f({\boldsymbol{d}}_1)| &= \left|\sum_{s,a} \left( g(d_1(s,a)) -g(d_2(s,a)) \right) \right| \\ &\overset{(a)}{\le} \sum_{s,a} \left| g(d_1(s,a)) -g(d_2(s,a)) \right| \\ &\le \sum_{s,a} \left|\log(\epsilon) + 1 \right| |d_1(s,a) - d_2(s,a)| \\ &= \left|\log(\epsilon) + 1 \right| \| {\boldsymbol{d}}_1 - {\boldsymbol{d}}_2 \|_1 \end{align}\] were (a) follows from the triangular inequality.

8.0.0.3 Imitation learning (\(f({\boldsymbol{d}}) = \| {\boldsymbol{d}} - {\boldsymbol{d}}_{\pi_b}\|_2^2\))

It holds that \(\nabla f({\boldsymbol{d}}) = 2 ({\boldsymbol{d}}- {\boldsymbol{d}}_{\pi_b})\). Now, \[\max_{{\boldsymbol{d}} \in \Delta(\mathcal{S}\times \mathcal{A}) } \| \nabla f({\boldsymbol{d}}) \|_1 = 2 \max_{{\boldsymbol{d}} \in \Delta(\mathcal{S}\times \mathcal{A}) } \| {\boldsymbol{d}} - {\boldsymbol{d}}_{\pi_b} \|_1 \le 2 \max_{{\boldsymbol{d}}_1, {\boldsymbol{d}}_2 \in \Delta(\mathcal{S}\times \mathcal{A}) } \| {\boldsymbol{d}}_1 - {\boldsymbol{d}}_2 \|_1 = 4.\] Since the function \(f\) is continuous and differentiable over the simplex, which is compact, it holds that \(L = 4\) is a valid Lipschitz constant as it corresponds to an upper bound on the maximum magnitude of the gradient of \(f\) over \(\Delta(\mathcal{S}\times \mathcal{A})\).

8.0.0.4 Multi-objective MDP, weighted (\(f({\boldsymbol{d}}) = n_1 {\boldsymbol{d}}^\top {\boldsymbol{c}}_1 + \ldots + n_k{\boldsymbol{d}}^\top {\boldsymbol{c}}_k\))

We note that \(f\) can be equivalently rewritten as \(f(d) = {\boldsymbol{d}}^\top \tilde{{\boldsymbol{c}}}\), where \(\tilde{{\boldsymbol{c}}} = n_1 {\boldsymbol{c}_1} + \ldots + n_k {\boldsymbol{c}_k}\). Therefore \[|f({\boldsymbol{d}}_1) - f({\boldsymbol{d}}_2)| \le \max_{s,a} |\tilde{c}(s,a)| \| {\boldsymbol{d}}_1 - {\boldsymbol{d}}_2\|_1.\]

8.0.0.5 Multi-objective MDP, max (\(f({\boldsymbol{d}}) = \max_{i \in \{1, \ldots, k\}} {\boldsymbol{d}}^\top {\boldsymbol{c}}_i\))

The result follows by noting that letting \(L = \max_{i \in \{1, \ldots, k\}} L_i\) yields a valid Lipschiz constant, where \(L_i = \max_{s,a} |c_i(s,a)|\), since we are taking the maximum of Lipschitz functions with Lipschitz constants \(L_1, \ldots, L_k\).

8.0.0.6 Multi-objective MDP, min (\(f({\boldsymbol{d}}) = \min_{i \in \{1, \ldots, k\}} {\boldsymbol{d}}^\top {\boldsymbol{c}}_i\))

It holds that \[|f({\boldsymbol{d}}_1) - f({\boldsymbol{d}}_2)| = \left|\min_{i \in \{1, \ldots, k\}} {\boldsymbol{c}}_i^\top {\boldsymbol{d}}_1 - \min_{i \in \{1, \ldots, k\}} {\boldsymbol{c}}_i^\top {\boldsymbol{d}}_2 \right| \le \max_{i \in \{1, \ldots, k\}} |{\boldsymbol{c}}_i^\top {\boldsymbol{d}}_1 - {\boldsymbol{c}}_i^\top {\boldsymbol{d}}_2| \le L \| {\boldsymbol{d}}_1 - {\boldsymbol{d}}_2 \|_1,\] where \(L = \max_{i \in \{1, \ldots, k\}} L_i\) and \(L_i = \max_{s,a} |c_i(s,a)|\).

9 Supplementary materials for Sec. 4↩︎

9.1 Proof of Proposition 1↩︎

9.1.0.1 Proof of Proposition 1

9.2 Proof of Lemma 2↩︎

9.3 Proof of Theorem 2↩︎

10 ERM-MCTS pseudocode↩︎

We display in 7 the pseudocode for the ERM-MCTS algorithm.

11 Experimental Results↩︎

Our code is available at https://github.com/gh0stwin/risk-aware-gumdp.

11.1 Baselines↩︎

In the context of standard MDPs, the ERM-BI baseline exploits the dynamic programming decompositions of the ERM put forth by [28]. In particular, [28] show that the optimal value function \(V^* = \{V^*_{t}\}_{t \in \{0, \ldots, H\}}\) and the optimal policy \(\pi^* = \{\pi^*_t\}_{t \in \{0, \ldots, H-1\}}\) satisfy, for all \(s \in \mathcal{S}\), \[\begin{align} V_t^*(s) &= \min_{a \in \mathcal{A}}\left\{\text{ERM}_{\beta \gamma^t}\left( c_t(s,a) + \gamma V_{t+1}^*(\mathrm{s}')\right) \right\}, \; \forall t \in \{0, \ldots, H-1\}, \; V_{H}^*(s) = c_H(s) \label{eq:erm95bellman95optimality32equations}, \\ \pi^*_t(s) &= \mathop{\mathrm{arg\,min}}_{a \in \mathcal{A}}\left\{\text{ERM}_{\beta \gamma^t}\left( c_t(s,a) + \gamma V_{t+1}^*(\mathrm{s}')\right) \right\}, \; \forall t \in \{0, \ldots, H-1\}. \end{align}\tag{7}\]

In the context of standard MDPs, we perform backward induction from timestep \(t=H\) backwards until timestep \(t=0\) to extract the optimal risk-aware policy. We also note that the occupancy MDP is undiscounted (i.e., \(\gamma = 1\)) and, hence, \(\beta \gamma^t = \beta\) for all timesteps.

11.2 Hyperparameters↩︎

For implementation purposes, we let the ERM-MCTS parameters \(\theta_t = \theta\) for all \(t \in \{0, \ldots, H-1\}\), see 4.3. We display in [tab:appendix:illustrative-gumdps-hyperparams,tab:appendix:grid-gumdps-hyperparams] the hyperparameters used in our experiments, where \(N\) denotes the number of independent runs of ERM-MCTS, \(n\) is the number of MCTS iterations per timestep for ERM-MCTS, and \(\theta\) is the ERM-MCTS exploration constant. Under MDPs, we also run \(N\) independent runs of the oracle baseline ERM-BI.

11.3 Illustrative environments↩︎

We display in Fig. 2 an overview of the illustrative environments. In the remainder of this section, we detail the GUMDPs used for each task, regarding their state space, action space, dynamics, and cost function used.

11.3.1 Standard MDP (Fig. 2 (a))↩︎

We employ a four-state MDP where the agent needs to tradeoff at the initial state (\(s_0\)) between:

The agent resets to the initial state with \(10\%\) probability while not in the initial state. The cost function is given by \({\boldsymbol{c}} = [c(s_0),c(s_1),c(s_2),c(s_3)]\) where \(c(s_0) = 0, \; c(s_1) = \frac{1}{4}, \; c(s_2) = \frac{1}{20}, \; c(s_3) = 1\). We let \(f({\boldsymbol{d}}) = {\boldsymbol{d}}^\top {\boldsymbol{c}}\).

11.3.2 Maximum State Entropy Exploration (Fig. 2 (b))↩︎

The task is similar to the motivating example from 1, where the agent aims to explore an environment as uniformly as possible but there is chance that the agent transitions to an absorbing state (\(s_4\)). The agent starts at state \(s_1\). We let \(f({\boldsymbol{d}}) = {\boldsymbol{d}}^\top\log({\boldsymbol{d}})\).

11.3.3 Imitation Learning (Fig. 2 (a))↩︎

The dynamics of this environment are the same as the standard MDP (11.3.1). The agent aims to imitate the empirical occupancy induced by the trajectory of an agent that selected twice the risky action (\(a_0\)) under the standard MDP and then selected the safe action (\(a_1\)) for all the remainder timesteps. The trajectory to imitate had a rather lucky outcome, as it never ended up in the absorbing state (\(s_3\)). Hence, the agent needs to trade off between imitating the behavior policy in states (\(s_0\), \(s_1\), \(s_2\)) and risking being absorbed into \(s_3\), or only imitating the behavior policy in the less risky states (\(s_0\), \(s_1\)). To be precise, the empirical occupancy to imitate is \(d_{\pi_b}(s_0, a_0)= 0.20605099\), \(d_{\pi_b}(s_0, a_1)= 0.30175732\), \(d_{\pi_b}(s_1, a_0)= 0.17054104\), \(d_{\pi_b}(s_1, a_1)= 0.15004508\), \(d_{\pi_b}(s_2, a_0)= 0.10245629\), \(d_{\pi_b}(s_2, a_1)= 0.0829896\), \(d_{\pi_b}(s_3, a_0)= 0.0\), \(d_{\pi_b}(s_3, a_1)= 0.0\). We let \(f({\boldsymbol{d}}) = \| {\boldsymbol{d}} - {\boldsymbol{d}}_{\pi_b}\|_2^2\).

11.3.4 Multi-Objective MDP (Fig. 2 (c))↩︎

The environment is inspired by the FishWood environment [38], where the agent needs to trade off between two cost functions. Each cost function penalizes different behaviors. The cost function \({\boldsymbol{c}_1}\) is defined as \(c_1(s_0) = 0\), \(c_1(s_1) = -1\), \(c_1(s_2) = 0.5\), \(c_1(s_3) = 0\), \(c_1(s_4) = 0\). The cost function \({\boldsymbol{c}_2}\) is defined as \(c_2(s_0) = 0\), \(c_2(s_1) = 0\), \(c_2(s_2) = 0\), \(c_2(s_3) = -0.2\), \(c_2(s_4) = 0.2\). The agent starts at state \(s_0\). Any action at states \(s_1\), \(s_2\), \(s_3\), and \(s_4\) takes the agent deterministically back to the initial state. We consider three utility functions:

11.4 Grid Environments↩︎

11.4.1 Maximum State Entropy Exploration↩︎

4 contains the hyperparameters used in the environment employed in the MSEE task. The environment dynamics are described in 5.2.1.

11.5 Imitation Learning↩︎

We reuse the environment and corresponding hyperparameters defined in the MSEE task (11.4.1). Additionally, we compute the occupancy of the behavioral policy \({\boldsymbol{d}}_{\pi_b}\) after the agent performing the sequence of actions depicted in 8. As already mentioned in 5.2.2, we ensure that the behavior policy never gets trapped inside difficult terrain squares. Under this approach we make sure that ERM-MCTS can model different behaviors (with distinct risk awareness) by considering which difficult terrain squares to visit, if any.

11.5.1 Multi-Objective↩︎

As stated in 5.2.3, we use the resource-gathering environment [40] as our multi-objective task. Here, the agent has three objectives to maximize. The first two objectives address picking and delivering resources \(R_1\) and \(R_2\) to the home location, respectively. The last objective attends to the possibility of the agent getting defeated when stepping inside squares that contain enemies. Aiming to model the necessary information that is required for the reward functions, we define the environment state as a 5-tuple, \(s=\left((x,y),h_{R_1},h_{R_2},h_\text{start},h_\text{enemy}\right)\). Where \((x,y)\in\mathbb{N}^2\) holds the agent’s position, \(h_{R_1},h_{R_2}\in\{0,1\}\) assign if the agent is carrying the corresponding resource, \(h_\text{start}\in\{0,1\}\) indicates whether the agent left the initial position, and \(h_\text{enemy}\in\{0,1\}\) indicates if a clash with an enemy occurred. Regarding environment dynamics, the agent can move in all directions, plus we define a slippery condition where, with \(p_\text{slip}\) probability, the agent moves instead perpendicularly to the selected direction. Additionally, resources are collected when the agent enters the resource’s square, setting \(h_r=1\), where \(r\in\{R_1,R_2\}\). Furthermore, when the agent steps inside a square that contains an enemy, with \(p_\text{defeat}\) probability, \(h_\text{enemy}\) becomes \(1\). In the case when \(h_\text{enemy}=1\), the agent loses immediately and transitions to an absorbing state until the end of the episode. Regarding reward modeling, we define three different reward functions \({\boldsymbol{r}}_1,\ldots,{\boldsymbol{r}}_3\in\mathbb{R}^{|S||A|}\), where all entries are \(0\) except in specific states:

1. For \({\boldsymbol{r}}_1\), the states \(s\in\{((x_\text{home},y_\text{home}),1,0,1,0),((x_\text{home},y_\text{home}),1,1,1,0)\}\) receive \(1\) to compensate for delivering \(R_1\) to the agent’s home position \((x_\text{home},y_\text{home})\).

2. Analogously, \({\boldsymbol{r}}_2\) accounts for the cases where \(R_2\) gets transported to \((x_\text{home},y_\text{home})\), thus assigning the states \(s\in\{((x_\text{home},y_\text{home}),0,1,1,0),((x_\text{home},y_\text{home}),1,1,1,0)\}\) with reward \(1\).

3. To account for when the agent loses as the consequence of an enemy clash, \({\boldsymbol{r}}_3\) sets the reward to \(-1\) for any state \(s\) having \(h_\text{enemy}=1\).

Subsequently, we combine all rewards by applying the following non-linear function, \(f({\boldsymbol{d}})=\left({\boldsymbol{r}}_1^\top{\boldsymbol{d}}\right)^{\frac{1}{2}}+{\boldsymbol{r}}_2^\top{\boldsymbol{d}}+{\boldsymbol{r}}_3^\top{\boldsymbol{d}}\). With \(f({\boldsymbol{d}})\)¹, we assign higher priority when retrieving \(R_1\), e.g., simulating it being more valuable than \(R_2\).

The environment hyper-parameters are defined in 5.

11.6 Compute↩︎

The illustrative environments were simulated on a laptop CPU (Intel 11th Gen i5-1135G7) with 16   of RAM. Additionally, the experiments for the grid environments were deployed on a server with a dual CPU (AMD EPYC 9224 24-Core) and 770   of RAM. In the latter, due to the large number of CPUs, we effectively parallelize every experiment, allocating one CPU per independent run. 6 7 display the runtime obtained when running ERM-MCTS on the illustrative and grid environments, respectively.

References↩︎

---

1. Transforming this objective to handle costs, we first set \({\boldsymbol{c}}_i=-{\boldsymbol{r}}_i,\forall i\in\{1,\ldots,3\}\) and then use \(f({\boldsymbol{d}})=\text{sign}({\boldsymbol{c}}_1^\top{\boldsymbol{d}}){|{\boldsymbol{c}}_1^\top{\boldsymbol{d}}|}^{\frac{1}{2}}+{\boldsymbol{c}}_2^\top{\boldsymbol{d}}+{\boldsymbol{c}}_3^\top{\boldsymbol{d}}\).↩︎