June 30, 2026
We study policy optimization for online episodic tabular Markov decision processes with unknown transition kernels, aiming for best-of-both-worlds guarantees together with data-dependent regret bounds. Recent work [1], [2] has shown that policy optimization can adapt to both adversarial and stochastic losses with first-order, second-order, and path-length bounds, but only under known transitions, leaving open whether such data-dependent guarantees are achievable by policy optimization when the transition kernel is unknown. We resolve this by developing a new algorithm based on optimistic follow-the-regularized-leader that attains these guarantees under unknown transitions. The key ingredient is a new design of optimistic \(Q\)-function estimators together with a data-dependent transition bonus that controls estimator bias through the loss-prediction error. Our analysis further identifies an unavoidable transition-dependent complexity term that captures the intrinsic cost of estimating the transition kernel. As a result, we obtain first-order, second-order, and path-length bounds with the transition-dependent complexity term while simultaneously achieving gap-dependent \(\mathrm{polylog}(T)\) regret in the stochastic regime.
We study policy optimization for online finite-horizon episodic tabular Markov decision processes (MDPs) with unknown transitions. In this setting, the learner interacts with an MDP over \(T\) episodes. In each episode, the environment selects a loss function while the learner selects a policy. The learner then executes it and receives feedback on the loss function. The goal is to minimize the regret, the gap between the cumulative loss of the learner and that of the best fixed policy in hindsight. Policy optimization directly updates the policy and is a central paradigm in reinforcement learning (e.g., [3], [4]). For tabular MDPs, this approach updates a separate action distribution for each state [5], [6], which is often more computationally efficient and practical than global optimization over the occupancy-measure space.
The difficulty of tabular MDPs depends on how the loss sequence is generated. In the adversarial regime, the worst-case regret scales as \(\widetilde{\mathcal{O}}(\sqrt{T})\) [6], [7]. In contrast, in the stochastic regime, one can often achieve much faster gap-dependent \(\mathrm{polylog}(T)\) regret [8], [9]. This contrast has motivated best-of-both-worlds algorithms, which aim to achieve near-optimal guarantees in both the adversarial and stochastic regimes with a single algorithm [1], [10], [11].
Beyond adapting to the underlying regime, another natural goal is to obtain regret bounds that scale with the difficulty of the realized loss sequence in the adversarial regime. Algorithms only with worst-case regret guarantees ignore the structure of losses and are often overly pessimistic in practical environments. Motivated by this, data-dependent bounds, such as first-order, second-order, and path-length bounds, have been actively studied in online learning, including prediction with expert advice and multi-armed bandits [12]–[16]. These depend, respectively, on the comparator’s cumulative loss, the magnitude of the loss fluctuations, and the temporal variation of the losses.
Such data-dependent bounds have also been studied in episodic MDPs, but existing data-dependent guarantees in episodic MDPs are limited either in the algorithmic approach or in the transition model. Under unknown transitions, [17] obtained a first-order bound, but only through occupancy-measure-based global optimization rather than policy optimization. For policy optimization, such data-dependent bounds have so far been obtained only under known transitions: [1] established best-of-both-worlds guarantees with a first-order bound, and [2] provided algorithms achieving first-order, second-order, and path-length bounds simultaneously. Consequently, whether policy optimization can attain any data-dependent bound under unknown transitions, let alone combine first-order, second-order, and path-length adaptivity with best-of-both-worlds guarantees, was left open by [1], [2]. This raises the following question.
Can policy optimization achieve data-dependent bounds such as first-order, second-order, and path-length bounds in episodic MDPs with unknown transitions?
We answer this question affirmatively by developing a policy optimization algorithm for episodic tabular MDPs with unknown transitions under bandit feedback, the harder setting in which the learner must estimate both the losses and the transition kernel from the realized trajectories. In the adversarial regime, its regret adapts to \(\min\{L(\pi),HT-L(\pi),Q_\infty,V_1\}\), covering first-order, second-order, and path-length complexities, together with a transition-dependent complexity term \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) in 2 that captures the intrinsic cost of estimating the transition kernel and can be connected to the first-order complexity \(L(\pi)\) (see 8.2 for details). The same algorithm also enjoys gap-dependent \(\mathrm{polylog}(T)\) regret in the stochastic regime, achieving the best-of-both-worlds guarantee (see 3). To our knowledge, this is the first data-dependent guarantee for policy optimization under unknown transitions. It is worth noting that this is also the first algorithm for tabular MDPs achieving best-of-both-worlds and data-dependent bounds simultaneously under unknown transitions. A detailed comparison with prior policy-optimization results is given in ¿tbl:tab:comparison?.
As a by-product, our algorithm carries over to the easier full-information setting, where it retains these data-dependent and best-of-both-worlds guarantees and, in the worst case, recovers the \(\widetilde{\mathcal{O}}(\sqrt{H^2S^2AT})\) rate of occupancy-measure-based methods [11], [18] (see 2). This implies that the extra \(H\) factor often attributed to \(Q\)-function estimation is not inherent for policy optimization in the full-information setting under unknown transitions.
Technically, our algorithms build on the optimistic follow-the-regularized-leader framework, with new prediction terms and optimistic \(Q\)-function estimators designed for unknown transitions tailored for each feedback setting (see 3). For the full-information setting, the \(Q\)-function estimator uses the most optimistic transition kernel in the confidence set. For the bandit setting, we incorporate a new data-dependent transition bonus defined in 11 12 into the \(Q\)-function estimator. This bonus keeps the \(Q\)-function estimator optimistic while reducing the estimator bias to the loss-prediction error, with the remaining worst-case terms contributing only lower-order terms.
Our analysis identifies the new transition-dependent complexity term \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\), which captures the cost of estimating the unknown transition kernel. We show that this term is unavoidable under unknown transitions: even for a time-invariant loss sequence with \(Q_\infty=V_1=0\), any algorithm still incurs regret from estimating the unknown transition kernel (see 4). This explains why data-dependent regret bounds under unknown transitions cannot be characterized solely by the loss-complexity measures used in the known-transition setting of [2]. Due to space constraints, we defer a more detailed discussion of related work to 8.1.
We study an episodic tabular Markov decision process (MDP) \(\mathcal{M}=(\mathcal{S},\mathcal{A}, P, H, s_0)\), where \(\mathcal{S}\) is a finite state space with \(S=\relax{\mathcal{S}}\), \(\mathcal{A}\) is a finite action space with \(A=\relax{\mathcal{A}}\), \(H\) is the horizon length, and \(s_0\) is the initial state. The transition kernel \(P\) is fixed but unknown to the learner. For each state-action pair \((s,a)\), \(P(s'\mid s,a)\) denotes the probability of moving to state \(s'\) after taking action \(a\) in state \(s\). We adopt the standard layered MDP assumption [6], [7], [19], where the state space is partitioned into \(H+1\) disjoint layers \(\mathcal{S}_0,\mathcal{S}_1,\ldots,\mathcal{S}_H\). Here, \(\mathcal{S}_0=\{s_0\}\) is the initial layer and \(\mathcal{S}_H=\{s_H\}\) is a terminal absorbing layer. For simplicity, we exclude \(s_H\) from \(\mathcal{S}\), so that \(H\leq S\). Transitions occur only between consecutive layers. Specifically, for any \((s,a)\in\mathcal{S}_h\times\mathcal{A}\) with \(h\in\{0,\ldots,H-1\}\), the distribution \(P(\cdot\mid s,a)\) is supported on \(\mathcal{S}_{h+1}\). We write \(h(s)\) for the layer index of state \(s\).
The learner interacts with the MDP for \(T\) episodes, indexed by \(t=1,\ldots,T\). At the beginning of episode \(t\), the environment selects a loss function \(\ell_t:\mathcal{S}\times\mathcal{A}\to[0,1]\). The learner then selects a policy \(\pi_t\) based on past observations, executes it from \(s_0\) under \(P\), and incurs the losses along the realized trajectory. Here, a stochastic policy maps each state \(s\) to a distribution \(\pi(\cdot\mid s)\) over actions, and for a deterministic policy \(\pi\), we write \(\pi(s)\in\mathcal{A}\) for its selected action at state \(s\). At the end of the episode, the learner receives feedback on \(\ell_t\). In the full-information setting, the feedback is the entire loss function \(\ell_t\), whereas in the bandit setting, it is limited to the realized losses \(\{(s_{t,h},a_{t,h},\ell_t(s_{t,h},a_{t,h}))\}_{h=0}^{H-1}\). We assume \(T\geq\max\{2,S,A\}\) for notational convenience.1
For a policy \(\pi\) and a loss function \(\ell\), the value functions are defined recursively, with terminal condition \(V^\pi(s_H;\ell)=0\). Specifically, the value function \(V^\pi(s;\ell)\) and the \(Q\)-function \(Q^\pi(s,a;\ell)\) satisfy \(V^\pi(s;\ell) = \mathbb{E}_{a\sim \pi(\cdot\mid s)}\brk{Q^\pi(s,a;\ell)}\) and \(Q^\pi(s,a;\ell) = \ell(s,a) + \mathbb{E}_{s'\sim P(\cdot\mid s,a)}\brk{V^\pi(s';\ell)}\). We may overload notation by allowing any function \(g:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\) to play the role of the loss function, in which case \(V^\pi(s;g)\) and \(Q^\pi(s,a;g)\) are defined accordingly. The goal is to minimize the regret \(\text{\rm Reg}_T(\pi)\mathrel{\vcenter{:}}=\mathbb{E}\brk[\big]{\sum_{t=1}^T V^{\pi_t}(s_0;\ell_t)-\sum_{t=1}^T V^{\pi}(s_0;\ell_t)}\) against any fixed comparator policy \(\pi\), and we denote by \(\mathring{\pi}\in \mathop{\mathrm{arg\,min}}_{\pi}\mathbb{E}\brk[\big]{\sum_{t=1}^T V^{\pi}(s_0;\ell_t)}\) an optimal policy in hindsight.
For \(N\in\mathbb{N}\), let \([N]\mathrel{\vcenter{:}}=\{1,\dots,N\}\), given a vector \(x\), let \(\norm{x}_p\) be its \(\ell_p\)-norm for \(p\in[1,\infty]\), and for a scalar \(z\), let \([z]_+\mathrel{\vcenter{:}}= \max\{z,0\}\). For a set \(\mathcal{K}\), let \(\Delta(\mathcal{K})\) denote the set of probability distributions over \(\mathcal{K}\). If \(f\) and \(g\) are functions with \(g(x)>0\), we write \(f\lesssim g\) or \(f=\mathcal{O}(g)\) when \(f(x)\leq c g(x)\) holds on the relevant domain for some universal constant \(c>0\), and use \(\widetilde{\mathcal{O}}(\cdot)\) to hide logarithmic factors. Let \(\{\mathcal{F}_t\}_{t\geq0}\) be the natural filtration generated by all observations up to the end of episode \(t - 1\), and define \(\mathbb{E}_t[\cdot] = \mathbb{E}[\cdot\mid \mathcal{F}_{t}]\). We use \(\mathbb{1}[\cdot]\) for indicators, which equal \(1\) on the event and \(0\) otherwise. Let \(\mathbb{I}_t(s,a)=\mathbb{1}\brk*{(s_{t,h},a_{t,h})=(s,a),\exists h\in\{0,\ldots,H-1\}}\) denote whether a state-action pair \((s,a)\) is visited in episode \(t\) under policy \(\pi_t\) and transition kernel \(P\), and let \(\mathbb{I}_t(s)=\sum_{a} \mathbb{I}_t(s,a)\). For each layer \(h\), write \(\ell_t(h)\in[0,1]^{\mathcal{S}_h\times\mathcal{A}}\) for the restriction of \(\ell_t\) to \(\mathcal{S}_h\times\mathcal{A}\), and use the same convention for other functions on \(\mathcal{S}\times\mathcal{A}\).
Following [7], we construct confidence sets for the unknown transition kernel. Let \(n_t(s,a,s')\) be the number of observed transitions from \((s,a)\) to \(s'\) up to episode \(t - 1\), and let \(n_t(s,a)\) be the corresponding visitation count. Whenever \(n_t(s,a)=0\), we set it to \(1\) in any denominator. The empirical transition estimate is then defined by \(\widehat{P}_t(s'\mid s,a)=\frac{n_t(s,a,s')}{n_t(s,a)}\) for all \(h\) and \((s,a,s')\in\mathcal{S}_h\times\mathcal{A}\times\mathcal{S}_{h+1}\). The confidence set of the true transition at episode \(t\) is then defined as \[\begin{align} \mathcal{P}_t = \set[\Bigg]{\widetilde{P}:&\relax*{\widetilde{P}(s'\mid s,a) - \widehat{P}_t(s'\mid s, a)} \leq \min\set[\bigg]{2\sqrt{\frac{\widehat{P}_t(s'\mid s, a)\iota}{n_t(s,a)}} + \frac{14\iota}{3n_t(s,a)}, 1},\\[-1mm] & \forall (s,a,s') \in \mathcal{S}_h\times\mathcal{A}\times\mathcal{S}_{h+1},\;\forall h = 0, \dots, H - 1,\; \widetilde{P}(\cdot\mid s, a) \in \Delta(\mathcal{S}_{h+1}).}, \label{def:calP95t} \end{align}\tag{1}\] where \(\iota = \log(SAT/\delta)\). Throughout the paper, we set \(\delta = 1/T^2\).
By [7], the true transition kernel \(P\) belongs to \(\mathcal{P}_t\) for all episodes \(t=1,\ldots,T\) with probability at least \(1-4\delta\). We note that the set \(\mathcal{P}_t\) is rectangular, since its constraints are imposed independently for each state-action pair \((s,a)\).
For a transition kernel \(\widetilde{P}\) and a policy \(\pi\), let \(q^{\widetilde{P},\pi}(s,a)\) be the probability of visiting \((s,a)\) within an episode. We also use \(q^{\widetilde{P},\pi}(s'\mid s,a)\) and \(q^{\widetilde{P},\pi}(s',a'\mid s,a)\) for the corresponding conditional occupancy measures given that \((s,a)\) has been visited and set them to zero whenever \(h(s')<h(s)\). For each state \(s\), define \(q^{\widetilde{P},\pi}(s)\mathrel{\vcenter{:}}= \sum_a q^{\widetilde{P},\pi}(s,a)\), so that \(q^{\widetilde{P},\pi}(s,a)=q^{\widetilde{P},\pi}(s)\pi(a\mid s)\). When \(\widetilde{P}=P\), we simply write \(q^\pi\). We also define the upper and lower occupancies \(\overline{q}_t^\pi(s)\) and \(\underline{q}_t^\pi(s)\) with respect to the confidence set \(\mathcal{P}_t\), and for the learner’s policy \(\pi_t\), the smoothed occupancy \(q_t(s)\) and relative occupancy width \(\rho_t(s)\) using an exploration rate \(\gamma_t>0\): \[\begin{align} \overline{q}_t^\pi(s)\!\mathrel{\vcenter{:}}=\!\max_{\widetilde{P}\in\mathcal{P}_t} q^{\widetilde{P},\pi}(s), \underline{q}_t^\pi(s)\!\mathrel{\vcenter{:}}=\!\min_{\widetilde{P}\in\mathcal{P}_t} q^{\widetilde{P},\pi}(s), q_t(s)\!\mathrel{\vcenter{:}}=\! \overline{q}^{\pi_t}_t(s) + \gamma_t, \rho_t(s) \!\mathrel{\vcenter{:}}=\! \frac{q_t(s) - \underline{q}_t^{\pi_t}(s)}{q_t(s)}. \end{align}\] These quantities can be computed efficiently by the dynamic-programming procedure of [7]. Similarly, for any transition kernel \(\widetilde{P}\), policy \(\pi\), and function \(g:\mathcal{S}\times\mathcal{A}\to\mathbb{R}\), we write \(V^{\widetilde{P},\pi}(s;g)\) and \(Q^{\widetilde{P},\pi}(s,a;g)\) for the value and \(Q\)-functions computed under \(\widetilde{P}\) and \(\pi\). When \(\widetilde{P}=P\), we again omit the transition kernel and write \(V^\pi(s;g)\) and \(Q^\pi(s,a;g)\).
We consider three regimes for the loss sequence \(\ell_1,\ldots,\ell_T\). In the adversarial regime, we impose no assumption on the losses. At the beginning of each episode \(t\), the environment selects an arbitrary loss function \(\ell_t\in[0,1]^{S\times A}\), which may depend on the learner’s algorithm, as well as on past trajectories and observed losses, but not on the learner’s internal randomness.
In the stochastic regime, the loss functions \(\ell_1,\ldots,\ell_T\) are sampled i.i.d. from a fixed but unknown distribution \(\mathcal{D}\). The stochastic regime with adversarial corruption interpolates between these two settings. In this regime, there is an underlying (non-corrupted) loss sequence \(\ell'_1,\ldots,\ell'_T\) sampled i.i.d. from an unknown distribution \(\mathcal{D}\), while the learner observes a possibly corrupted loss sequence \(\ell_1,\ldots,\ell_T\). The corruptions may depend arbitrarily on the past trajectories and observed losses, and their total magnitude is measured by \(\mathcal{C}\mathrel{\vcenter{:}}= \mathbb{E}\brk[\big]{\sum_{t=1}^T\sum_{h=0}^{H-1}\|\ell'_t(h)-\ell_t(h)\|_\infty} \in [0, HT]\). When \(\mathcal{C}=0\), this regime reduces to the stochastic regime; when \(\mathcal{C}\) is arbitrary, it coincides with the adversarial regime. For the stochastic and corrupted stochastic regimes, let \(\mu(s,a)\mathrel{\vcenter{:}}= \mathbb{E}_{\ell'\sim\mathcal{D}}\brk*{\ell'(s,a)}\) be the mean uncorrupted loss function. Assume that the optimal policy for \(\mu\) is unique, and let it be denoted by \({\pi^\star}\). We define the suboptimality gap of a state-action pair \((s,a)\) by \(\Delta\colon\mathcal{S}\times\mathcal{A}\to[0,H]\) as \(\Delta(s,a)\mathrel{\vcenter{:}}= Q^{\pi^\star}(s,a;\mu)-\min_{a'\in\mathcal{A}}Q^{\pi^\star}(s,a';\mu)\) and let \(\Delta_{\min}\mathrel{\vcenter{:}}= \min_{s, a\neq {\pi^\star}(s)}\Delta(s,a) > 0\).
We use the following data-dependent complexity measures to state refined regret bounds. The first-order quantity introduced by [17] is \(L(\pi)\mathrel{\vcenter{:}}=\mathbb{E}\brk[\big]{\sum_{t=1}^T V^\pi(s_0;\ell_t)}\), which is the cumulative loss of a fixed comparator policy \(\pi\). The second-order quantity \(Q_\infty\mathrel{\vcenter{:}}=\min_{\ell^\star\in[0,1]^{S\times A}}\mathbb{E}\brk[\big]{\sum_{t=1}^T\sum_{h=0}^{H-1}\norm{\ell_t(h)-\ell^\star(h)}_\infty^2}\), which is small when the losses stay close to a fixed baseline loss, and the path-length quantity \(V_1\mathrel{\vcenter{:}}=\mathbb{E}\brk[\big]{\sum_{t=1}^{T-1}\nrm{\ell_{t+1}-\ell_t}_1}\), which is small when the losses vary slowly over time, are defined as in [2].
Under unknown transitions, our regret bounds contain a transition-dependent complexity term that reflects the difficulty of estimating the transition kernel. For a loss sequence \(\ell=\{\ell_t\}_{t=1}^T\) and a policy sequence \(\pi_{1:T}=\{\pi_t\}_{t=1}^T\), define \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell) \mathrel{\vcenter{:}}= \sup_{\forall t,s,a:\;|\phi_t(s,a)|=\ell_t(s,a)} \sum_{t=1}^T\sum_{s,a} q^{\pi_t}(s,a) \mathsf{Var}_{s'\sim P(\cdot\mid s,a)} \brk*{V^{\pi_t}(s';\phi_t)} \in [0,H^2T]. \label{def:qtrans95loss} \end{align}\tag{2}\] This quantity measures the variance of future cumulative losses across next states. Thus, transition uncertainty affects regret only when possible next states have different values \(V^{\pi_t}(s';\phi_t)\). This mirrors transition-variance terms in stochastic MDPs, such as \(\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi^\star}(s'; \mu)}\) [8], [20]. Our definition replaces the optimal stochastic value \(V^{\pi^\star}(\,\cdot\,;\mu)\) with learner-induced value functions so that the same idea applies to arbitrary loss sequences. As discussed later, this transition-dependent term is unavoidable under unknown transitions, since learning the transition kernel can itself cause regret even when \(Q_\infty = V_1 = 0\). Furthermore, this term recovers a first-order regret bound through a self-bounding argument. In particular, the relation \(\mathbb{E}[Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)]\leq H(L(\pi)+\text{\rm Reg}_T(\pi))\), together with a regret bound of the form \(\text{\rm Reg}_T(\pi)\lesssim \sqrt{\mathbb{E}[Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)]}+J\), implies \(\text{\rm Reg}_T(\pi)\lesssim \sqrt{HL(\pi)}+J\). Thus, the transition-dependent complexity is at least as adaptive as the first-order complexity, while it can be sharper because it only counts transition uncertainty that affects future cumulative losses. We defer a detailed explanation and a simple illustrative example to 8.2.
We are now ready to present our policy optimization algorithm for both the full-information and bandit settings. The common template is given in 1, with the feedback-dependent definitions specified in 2. Both algorithms follow the same policy optimization template and use new \(Q\)-function estimators and prediction terms to handle unknown transitions and adversarial losses. We describe the key components below.
Our algorithms rely on the optimistic follow-the-regularized-leader (OFTRL) framework [21]–[23], which incorporates a prediction of the next loss vector into the standard FTRL. Our policy update is based on the OFTRL policy-optimization scheme of [2], which was developed for known transitions, and we modify it to handle unknown transitions.
Policy optimization reduces the MDP problem to a separate multi-armed bandit problem at each state. By the performance-difference lemma [24], the regret can be written as \(\text{\rm Reg}_T(\pi)=\mathbb{E}\brk[\big]{\sum_{s} q^{\pi}(s)\sum_{t}\inpr{\pi_t(\cdot\mid s) - \pi(\cdot\mid s), Q^{\pi_t}(s,\,\cdot\,;\ell_t)}}\). Thus, the \(Q\)-function plays the role of the loss vector for updating the action distribution at each state. Under unknown transitions, we first form the loss prediction \(m_t\in[0,1]^{S\times A}\) and then compute its \(Q\)-function using an optimistic transition kernel from the confidence set. The OFTRL update with a convex regularizer \(\psi_t\) is defined, for each state \(s \in \mathcal{S}\), by \[\begin{align} \pi_t(\cdot\mid s)\! =\! \mathop{\mathrm{arg\,min}}_{\pi(\cdot\mid s)\in\Delta(\mathcal{A})} \set[\bigg]{\!\inpr[\bigg]{\pi(\cdot\mid s), \sum_{\tau=1}^{t-1}\prn*{\widehat{Q}_\tau(s, \cdot) - B_\tau(s, \cdot)} + Q^{\underline{P}^m_t, \pi_t}(s, \cdot;m_t)} + \psi_t(\pi(\cdot\mid s))\!}. \label{eq:opt95pi95OFTRL} \end{align}\tag{3}\] Here, \(\widehat Q_\tau\) is a \(Q\)-function estimator, \(B_\tau\) is a dilated exploration bonus, and \(Q^{\underline{P}^m_t,\pi_t}(s,\cdot;m_t)\) is the optimistic prediction of \(Q^{\pi_t}(s,\cdot;\ell_t)\). The transition kernel \(\underline{P}^m_t\in\mathcal{P}_t\) is chosen to minimize \(Q^{\widetilde{P},\pi_t}(s,a;m_t)\) over \(\widetilde{P}\in\mathcal{P}_t\) for each state-action pair \((s,a)\). Both \(\underline{P}^m_t\) and \(Q^{\underline{P}^m_t,\pi_t}(s,\cdot;m_t)\) are computable by backward dynamic programming over the layers, since for \(s\in\mathcal{S}_h\), computing this \(Q\)-function only requires later-layer values and policies. In our algorithms, we use the log-barrier regularizer \(\psi_t(\pi(\cdot\mid s))=\sum_{a\in\mathcal{A}}\frac{1}{\eta_t(s,a)}\log\prn[\big]{\frac{1}{\pi(a\mid s)}}\) , where \(\eta_t(s,a)>0\) are state-action-wise time-varying, data-dependent learning rates. In both the full-information and bandit settings, the algorithm uses the same update rule 3 , and differs only in the construction of \(\widehat Q_t\), \(B_t\), and \(\eta_t(s,a)\).
The optimistic prediction \(Q^{\underline{P}^m_t,\pi_t}(s,\cdot;m_t)\) is obtained from a loss prediction sequence \(m_t\). We construct \(m_t\) from the realized trajectory, starting with \(m_1(s,a)=1/2\) for all \((s,a)\) and setting \[\begin{align} m_{t+1}(s,a)=\begin{cases} (1-\xi)m_t(s,a)+\xi\ell_t(s,a), & \text{if } \mathbb{I}_t(s,a)=1,\\ m_t(s,a), & \text{if } \mathbb{I}_t(s,a)=0, \end{cases} \label{def:predictor95sequence} \end{align}\tag{4}\] where \(\xi\in(0,1/2)\) is a step size. Since the transition kernel is unknown, we use this predictor even in the full-information setting, where it is useful for deriving path-length bounds.
In policy optimization, a naive state-wise update of the action distributions can lead to insufficient exploration. To address this, [6] introduced a dilated exploration bonus \(B_t(s,a)\), which is constructed in the same form as a \(Q\)-function, \[\begin{align} &B_t(s,a) = b_t(s) + \prn*{1 + \frac{1}{H}}\max_{\widetilde{P}\in\mathcal{P}_t}\mathbb{E}_{s'\sim \widetilde{P}(\cdot\mid s,a), a'\sim \pi_t(\cdot\mid s')} \brk*{B_t(s',a')}. \label{def:dilated95bonus} \end{align}\tag{5}\] Since \(B_t\) is subtracted from the \(Q\)-function estimator in 3 , a state-action pair \((s,a)\) with a larger dilated exploration bonus is evaluated more optimistically and is therefore more likely to be visited. This leads to the following lemma.
Lemma 1 ([6], Lemma B.2). Suppose that \(b_t(s)\) is a nonnegative function. Suppose also that, for a comparator policy \(\pi\), there exists \(J^\pi\geq 0\) such that \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\sum_{t=1}^T\sum_a \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}}\\ &\leq J^{\pi} + \mathbb{E}\brk*{\sum_{t = 1}^T\sum_s q^{\pi}(s) b_t(s)} + \mathbb{E}\brk*{\frac{1}{H} \sum_{t = 1}^T\sum_s\sum_a q^{\pi}(s) \pi_t(a\mid s)B_t(s,a)}. \label{eq:key95lemma95ineq95main} \end{align}\qquad{(1)}\] Then, it holds that \(\text{\rm Reg}_T(\pi) \leq J^{\pi} + 3 \, \mathbb{E}\brk[\Big]{\sum_{t = 1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)} + \mathbb{E}\brk[\big]{HT\mathbb{1}[P\notin\mathcal{P}_t, \exists t\in[T]]}\), where \(\overline{P}^B_t \in \mathcal{P}_t\) simultaneously attains the maxima in 5 for all \((s,a)\).
From this lemma, once the condition ?? is verified, it remains only to evaluate the value function \(V^{\overline{P}^B_t, \pi_t}(s_0;b_t)\) of the bonus under the learner’s own policies \(\pi_t\), rather than a comparator policy \(\pi\). For further intuition on the dilated bonus, we refer the reader to [1], [6].
Input: \(\gamma_t = \frac{1}{t}\), \(\eta_1 = \frac{1}{12H^3}\), \(m_1 = \frac{1}{2}\), \(\xi =
\frac{1}{4}\), \(\psi_t(\pi(\cdot\mid s))=\sum_{a\in\mathcal{A}}\frac{1}{\eta_t(s,a)}\log\prn[\big]{\frac{1}{\pi(a\mid s)}}\)
Update \(\mathcal{P}_t\) by 1 .
For each \(s\in \mathcal{S}\), compute \(\pi_t\) by the OFTRL update 3 .
Set \(Y_t\gets 1\) in the full-information setting, and \(Y_t\gets \mathbb{1}\brk[\big]{\max_{s,a}\frac{\eta_t(s,a)}{q_t(s)}\le \frac{1}{50\sqrt{H^3S^3A}}}\) in the bandit setting.
If \(Y_t=0\), insert a virtual episode and shift the indices of real episodes.
If \(Y_t=1\), execute \(\pi_t\), observe \(\{(s_{t,h},a_{t,h})\}_{h=0}^{H-1}\), and receive the corresponding loss feedback.
Compute \(\widehat Q_t\), \(\eta_{t+1}\), and \(b_t\) as in 2, and compute \(B_t\)
by 5 .
Compute loss prediction \(m_{t+1}(s,a)\) by 4 .
Figure 1: Data-Dependent Policy Optimization with Unknown Transitions.
Full-information.
\(Q\)-function estimator. \[\begin{align} \widehat{Q}_t(s,a) = Q^{\underline{P}^\ell_t, \pi_t}(s,a;\ell_t) \qquad \text{with}\quad
\underline{P}^\ell_t \in \mathop{\mathrm{arg\,min}}_{\widetilde{P}\in \mathcal{P}_t}Q^{\widetilde{P}, \pi_t}(s,a;\ell_t) \text{ for all } (s,a). \label{eq:Q-estimate95full}
\end{align}\tag{6}\] Learning rate and exploration bonus. \[\begin{align} \frac{1}{\eta_{t+1}(s,a)} &= \frac{1}{\eta_{t}(s,a)} + \frac{\eta_t(s,a)\zeta_t(s,a)}{\log(T)}, \tag{7}\\
b_t(s) &= 7\sum_{a}\prn*{\frac{1}{\eta_{t+1}(s,a)} - \frac{1}{\eta_t(s,a)}}\log(T), \tag{8}
\end{align}\] \[\begin{align}
\!\!\!\!\text{where}\quad \zeta_t(s, a) = \pi_t(a\mid s)(1 - \pi_t(a \mid s))\sum_{b}\pi_t(b \mid s)\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)}^2. \label{eq:zeta95full}
\end{align}\tag{9}\]
Bandit feedback.
\(Q\)-function estimator. Let \(L_{t,h}=\sum_{h'=h}^{H-1}\ell_t(s_{t,h'},a_{t,h'})\), \(M_{t,h}=\sum_{h'=h}^{H-1}m_t(s_{t,h'},a_{t,h'})\), and \(D_{t,h}=\sum_{h'=h}^{H-1}\relax{\ell_t(s_{t,h'},a_{t,h'})-m_t(s_{t,h'},a_{t,h'})}\), and define
\[\begin{align} \widehat Q_t(s,a) = Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)+\frac{\mathbb{I}_t(s,a)\prn*{L_{t,h(s)}-M_{t,h(s)}}}{q_t(s)\pi_t(a\mid s)}Y_t-C_t(s,a)Y_t,
\label{eq:Q-estimate95bandit}
\end{align}\tag{10}\] \[\begin{align} \text{where}\quad c_t(s,a) &= \rho_t(s)\frac{\mathbb{I}_t(s,a)D_{t,h(s)}}{q_t(s)\pi_t(a\mid s)}+\rho_t(s)^2H, \tag{11}\\ C_t(s,a) &=
c_t(s,a)+\max_{\widetilde{P}\in\mathcal{P}_t}\mathbb{E}_{s'\sim\widetilde{P}(\cdot\mid s,a),\,a'\sim\pi_t(\cdot\mid s')}\brk*{c_t(s',a')+C_t(s',a')}. \tag{12}
\end{align}\] Learning rate and exploration bonus. Let \((s_t^\dagger,a_t^\dagger)\) be any
maximizer of \(\eta_t(s,a)/q_t(s)\), and define \[\begin{align} \frac{1}{\eta_{t+1}(s,a)} &= \begin{cases}
\frac{1}{\eta_t(s,a)}+\frac{\eta_t(s,a)\zeta_t(s,a)}{q_t(s)^2\log(T)}, & \text{if } t \text{ is a real episode}, \\ \frac{1}{\eta_t(s,a)}\prn[\Big]{1+\frac{\mathbb{1}\{(s_t^\dagger,a_t^\dagger)=(s,a)\}}{324H\log(T)}}, & \text{if } t \text{ is a
virtual episode}, \end{cases} \tag{13}\\ b_t(s) &= 6\sum_a\prn*{\frac{1}{\eta_{t+1}(s,a)}-\frac{1}{\eta_t(s,a)}}\log(T)+3\sum_a\eta_t(s,a)\pi_t(a\mid s)^2C_t(s,a)^2Y_t, \tag{14}
\end{align}\] \[\begin{align} \text{where}\quad
\zeta_t(s,a) &= \prn*{\mathbb{I}_t(s,a)-\pi_t(a\mid s)\mathbb{I}_t(s)}^2\prn*{L_{t,h(s)}-M_{t,h(s)}}^2. \label{eq:zeta95bandit}
\end{align}\tag{15}\]
Figure 2: Definitions of the \(Q\)-function estimator \(\widehat Q_t\), the learning-rate update \(\eta_{t+1}\), and the local exploration bonus \(b_t\) used in 1..
The main new component is the \(Q\)-function estimator. The estimator differs between the full-information and bandit settings, and this choice also affects the learning-rate update and the local exploration bonus \(b_t\). We collect the definitions in 2 and explain the idea below. Note that the quantities \(\underline{P}_t^\ell\), \(\underline{P}_t^m\), \(B_t\), and \(C_t\) in 2 are efficiently computable by backward dynamic programming with a state-action-wise greedy procedure over the confidence set, as in [6], [7].
In the full-information setting, the learner observes the entire loss function \(\ell_t\) after each episode, so the \(Q\)-function estimator can be formed directly from the entire loss \(\ell_t\). We choose \(\underline{P}_t^\ell\in\mathcal{P}_t\) so that it minimizes \(Q^{\widetilde{P},\pi_t}(s,a;\ell_t)\) over \(\widetilde{P}\in\mathcal{P}_t\) for all state-action pairs \((s,a)\), and design the new optimistic \(Q\)-function estimator by 6 . On the event that \(P\in\mathcal{P}_t\) for all \(t\), this estimator is optimistic, and the resulting bias is one-sided. This is the only additional bias caused by unknown transitions, and it is controlled by the transition-dependent complexity \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) in the regret analysis.
The corresponding learning-rate update and local exploration bonus are given in 7 8 . The update of the learning rate is designed so that the penalty term \((1/\eta_{t+1}(s,a) - 1/\eta_t(s,a))\log(T)\) matches the stability term \(\eta_t(s,a)\zeta_t(s,a)\), where \(\zeta_t\) is defined in 9 . Thus, both terms are controlled through the same data-dependent quantity \(\zeta_t\). The local exploration bonus \(b_t\) is directly induced by this learning-rate update, with transition-estimation error included through \(\zeta_t\), so no separate transition bonus is needed in the full-information setting. Since the estimator uses the observed loss directly, the stability condition typically needed when analyzing OFTRL is easy to satisfy without introducing virtual episodes. Accordingly, Algorithm 1 sets \(Y_t=1\) throughout the full-information setting.
In the bandit setting, the learner observes losses only along the realized trajectory. To deal with this, our new \(Q\)-function estimator 10 therefore combines three ingredients: the optimistic prediction \(Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)\), estimator of the prediction-error \(Q^{\pi_t}(s,a;\ell_t-m_t)\), and the transition bonus \(C_t(s,a)\). The main challenge is designing the transition bonus. It has to account for both loss-estimation error from bandit feedback and transition-estimation error from the unknown transition kernel. More precisely, the local transition bonus \(c_t(s,a)\) must upper bound the estimator bias, while \(c_t(s,a)\) itself remains controllable by data-dependent quantities. In particular, it should satisfy a relation of the form \(\mathbb{E}_t[c_t(s,a)] \geq \rho_t(s)\relax{Q^{\pi_t}(s,a;\ell_t - m_t)}\) for all \(t,s,a\). The simpler local transition bonus \(c_t(s)=\rho_t(s)H\) used in [1] controls the estimator bias only through a worst-case \(\widetilde{\mathcal{O}}(\sqrt T)\) bound and thus does not yield the desired data-dependent bound.
To tackle this challenge, we instead define the data-dependent local transition bonus \(c_t(s,a)\) in 11 and its recursive version \(C_t(s,a)\) in 12 , both constructed from the bandit feedback. The key point in our estimator is that, in the local transition bonus \(c_t(s,a)\), the part that does not depend on the prediction error scales as \(\rho_t(s)^2\) rather than \(\rho_t(s)\). This makes the worst-case contribution of the estimator bias only \(\mathrm{polylog}(T)\), and the other terms can be bounded by data-dependent quantities. In addition, we use the prediction term \(Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)\), which is not automatically a lower estimate of \(Q^{\pi_t}(s,a;\ell_t)\), as also noted by [2]. We therefore define \(C_t\) with an extra margin, so that the estimator remains optimistic without losing data-dependent control.
The learning-rate update and local exploration bonus are given in 13 14 . As in the full-information setting, the penalty term of regret \((1/\eta_{t+1}(s,a)-1/\eta_t(s,a))\log(T)\) is chosen to match the log-barrier stability term \(\eta_t(s,a)\zeta_t(s,a)/q_t(s)^2\), where \(\zeta_t\) is defined in 15 . The local exploration bonus \(b_t\) contains the term induced by this learning-rate update, together with an additional term involving \(C_t(s,a)^2\), which accounts for transition-estimation error.
In the bandit setting, \(B_t(s,a)\) can scale as \(1/q_t(s)^2\) through 14 , so the resulting bonus-induced stability term cannot be controlled by the adaptive learning-rate update alone when \(q_t(s)\) is small. Therefore, following [1], we insert virtual episodes to control the ratio \(\eta_t(s,a)/q_t(s)\). If \(\max_{s,a}\eta_t(s,a)/q_t(s)>1/(50\sqrt{H^3S^3A})\) at the beginning of episode \(t\), we set \(Y_t=0\), make episode \(t\) virtual, and shift the indices of all subsequent real episodes. The algorithm then uses the zero loss function and only updates the pair \((s_t^\dagger,a_t^\dagger)\in\mathop{\mathrm{arg\,max}}_{s,a}\eta_t(s,a)/q_t(s)\), multiplying \(1/\eta_t(s_t^\dagger,a_t^\dagger)\) by \(1+\frac{1}{324H\log(T)}\). It still computes \(\widehat Q_t\), \(b_t\), and \(B_t\), but keeps the prediction \(m_t\) and the confidence set \(\mathcal{P}_t\) unchanged. Since the number of virtual episodes is \(\mathcal{O}(HSA\log^2(T))\), their effect is lower-order, and we still use \(T\) to denote the total number of episodes.2
We now state the guarantees of 1. The first theorem is for the full-information setting, and the second is for the bandit setting. In both cases, the algorithm achieves first-order, second-order, and path-length bounds in the adversarial regime, and at the same time achieves a gap-dependent \(\mathrm{polylog}(T)\) bound in the stochastic regime. The transition-dependent complexity \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) captures transition uncertainty that affects future cumulative losses and can imply first-order dependence on \(L(\pi)\) through self-bounding.
Theorem 2. In the full-information setting, for any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^2SA \log^2(T) \min\set*{L(\pi),HT-L(\pi),Q_\infty,V_1}}\\ &\qquad + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + HS^3A^{\frac{3}{2}}\log^2(T). \end{align}\] Under the stochastic regime with adversarial corruption, it simultaneously ensures \(\text{\rm Reg}_T(\pi) \lesssim U + \sqrt{U\mathcal{C}} + HS^3A^{\frac{3}{2}}\log^2(T)\), where \(U = \frac{H^2S^2A\log^2(T)}{\Delta_{\min}}\).
The above bound recovers the \(\widetilde{\mathcal{O}}(\sqrt{H^2S^2AT})\) worst-case rate of occupancy-measure-based algorithms [11], [18]. To our knowledge, this is the first policy-optimization result to remove the extra \(H\) factor [1], [6]. In the stochastic regime, the leading gap-dependent term also has sharper dependence on \(H\), \(S\), and \(A\) than that of [11], up to logarithmic factors.
Theorem 3. In the bandit setting, for any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^3S^2A\log^2(T) \min\set*{L(\pi), HT - L(\pi), Q_{\infty}, V_1}}\\ &\qquad+\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \end{align}\] Under the stochastic regime with adversarial corruption, it simultaneously ensures \(\text{\rm Reg}_T(\pi) \lesssim U + \sqrt{U\mathcal{C}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)\), where \(U = \frac{H^2S^4A\log^2(T)}{\Delta_{\min}}\).
In the worst case, the bound for the adversarial regime recovers the worst-case rate of [1]. For the first-order bound, the \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) term is absorbed by the first-order term through the self-bounding argument. Thus, the final bound does not contain an explicit transition-dependent term. In the stochastic regime, the gap-dependent term incurs an additional factor \(S^2/H^2\) compared with [1] due to the transition bonus, and whether this dependence can be improved remains open. However, the main contribution of 2 3 is their data-dependent regret bounds under unknown transitions. Together, these theorems give the first policy-optimization guarantees under unknown transitions with first-order, second-order, and path-length adaptivity.
We show that the data-dependent terms in our adversarial guarantees cannot be avoided in general. First, by running a standard hard instance for adversarial MDPs with unknown transitions [25], [26] for the first \(T'\le T\) episodes and setting all later losses to zero, the complexities satisfy \(L(\pi)=\mathcal{O}(HT')\), \(Q_\infty=\mathcal{O}(HT')\), \(V_1=\mathcal{O}(SAT')\), and \(\mathbb{E}[Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)]=\mathcal{O}(H^2T')\), while \(\max_{\pi}\text{\rm Reg}_T(\pi)\ge\Omega(\sqrt{H^2SAT'})\). Thus, one can obtain \(\Omega(\sqrt{HSA L(\pi)})\), \(\Omega(\sqrt{HSAQ_\infty})\), \(\Omega(\sqrt{H^2 V_1})\) and \(\Omega(\sqrt{SA \mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}})\) for the adversarial regime. Closing the remaining gaps is not specific to our data-dependent setting, and remains open even for worst-case policy optimization [6].
The following proposition further shows that a transition-dependent term is necessary under unknown transitions. This follows from the lower-bound constructions of [25] and [26] use time-invariant loss sequences, so the regret arises solely from transition uncertainty.
Proposition 4. There exists an episodic MDP with unknown transitions and a time-invariant loss sequence such that \(Q_\infty=V_1=0\), while any algorithm suffers \(\max_{\pi}\text{\rm Reg}_T(\pi)=\Omega(\sqrt{H^2SAT})\). This lower bound holds even under full-information feedback.
We sketch the proof for the bandit setting, which contains the main ideas also used in the full-information setting. The complete proofs are deferred to 12 for the full-information setting and 13 for the bandit setting. We use 1 as the main reduction for policy optimization. Recall that once the inequality in ?? is verified, the lemma reduces the regret analysis to bounding \(\sum_{t=1}^T V^{\overline{P}^B_t,\pi_t}(s_0;b_t)\). We prove ?? by decomposing its left-hand side into \(\mathbb{E}\brk[\big]{\sum_s q^\pi(s)(\textrm{\boldsymbol{reg}}^{\pi}(s)+\textrm{\boldsymbol{bias}}^{\pi}(s)+\textrm{\boldsymbol{error}}^{\pi}(s))}\), where \[\begin{align} \\[-10pt] \textrm{\boldsymbol{reg}}^{\pi}(s) &\!=\! \sum_{t,a} \prn{\pi_t(a\mid s)\!-\!\pi(a\mid s)} \prn*{\!\widehat{Q}_t(s,a) \!-\! B_t(s,a)\!},\\[-2pt] \textrm{\boldsymbol{bias}}^{\pi}(s) &\!=\! \sum_{t,a} \prn{\pi_t(a\mid s)\!-\!\pi(a\mid s)} \prn*{\!Q^{\pi_t}(s,a;\ell_t) \!-\! \widehat{Q}_t(s,a) \!+\! Q^{\underline{P}^m_t, \pi_t}(s, a; m_t) \!-\! Q^{\pi_t}(s, a; m_t)\!},\\[-2pt] \textrm{\boldsymbol{error}}^{\pi}(s) &\!=\! \sum_{t,a} \prn{\pi_t(a\mid s)\!-\!\pi(a\mid s)} \prn*{\!Q^{\pi_t}(s, a; m_t) \!-\! Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)\!}. \end{align}\] Here, \(\textrm{\boldsymbol{bias}}^{\pi}(s)\) is the estimator bias, while \(\textrm{\boldsymbol{error}}^{\pi}(s)\) comes from evaluating the prediction using \(\underline{P}_t^m\) instead of the true transition kernel. The standard analysis in 45 bounds \(\textrm{\boldsymbol{reg}}^{\pi}(s)\) as \[\begin{align} \mathbb{E}[\textrm{\boldsymbol{reg}}^{\pi}(s)] &\leq \mathcal{O}\prn*{HS^2A\log(T)} + \mathbb{E}\brk*{\sum_{t=1}^Tb_t(s)} + \mathbb{E}\brk*{\frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a)}. \end{align}\] The remaining terms are controlled in 46 47 by \[\begin{align} \\[-10pt] \mathbb{E}\brk*{\!\sum_s q^\pi(s)\textrm{\boldsymbol{bias}}^{\pi}(s)\!} &\lesssim \widetilde{\mathcal{O}}\prn*{\!\sqrt{\!H^2S^2A \iota^2 \mathbb{E}\brk*{\sum_{t:Y_t=1} \sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\prn{\ell_t - m_t}^2)}\!}},\\[-2pt] \mathbb{E}\brk*{\!\sum_s q^\pi(s)\textrm{\boldsymbol{error}}^{\pi}(s)\!} &\lesssim \widetilde{\mathcal{O}}\prn*{\sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}}}. \end{align}\] The key point is that \(\textrm{\boldsymbol{bias}}^{\pi}(s)\) is reduced to the prediction error \(\ell_t-m_t\). This follows from the design of \(c_t(s,a)\), where the part independent of the prediction error scales with \(\rho_t(s)^2\) rather than \(\rho_t(s)\), and \(\sum_{t:Y_t=1}\sum_s q^{\pi_t}(s)\rho_t(s)^2\) is only polylogarithmic in \(T\) (see 43). The error term is captured by \(Q_{\mathrm{trans}}^{\pi_{1:T}}(m)\), which can be reduced to \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) through \(Q_{\mathrm{trans}}^{\pi_{1:T}}(m) \leq 2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell) + 2H\sum_{t:Y_t=1} V^{\pi_t}(s_0;(\ell_t-m_t)^2)\) in 20. Thus, all remaining terms reduce to the prediction error \(\ell_t-m_t\) and \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\), yielding the desired first-order, second-order, and path-length bounds.
For the stochastic regime with adversarial corruption, we use the same decomposition as above. We show in 46 47 that the individual terms can be bounded by factors of the form \(\sum_s\sum_{a\neq\pi^\star(s)}\sqrt{\mathbb{E}\brk[\big]{\sum_{t:Y_t=1} q^{\pi_t}(s,a)}}\) or \(\sqrt{\mathbb{E}\brk[\big]{\sum_{t:Y_t=1}\sum_{s,a}[q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)]_+}}\). These quantities are then controlled by a self-bounding argument following [1], yielding the desired gap-dependent bound under adversarial corruption.
In this work, we resolved the open problem raised by [1] on data-dependent regret bounds for policy optimization in MDPs with unknown transitions. Our algorithms achieve refined best-of-both-worlds guarantees, adapting to \(L(\pi)\), \(Q_\infty\), and \(V_1\) in the adversarial regime while attaining a gap-dependent \(\mathrm{polylog}(T)\) bound in the stochastic regime with adversarial corruption. A natural future direction is to obtain variance-aware gap-dependent bounds as in [9], and to investigate whether the refined data-dependent guarantees can be achieved under aggregate feedback.
TT is supported by JSPS KAKENHI Grant Number JP26K21297 and KY is partially supported by JSPS KAKENHI Grant Number JP24H00703.
For the reader’s convenience, 1 collects the main notation used throughout the paper.
| Symbol | Meaning |
|---|---|
| Tabular MDPs | |
| \(\mathcal{M}=(\mathcal{S},\mathcal{A},P,H,s_0)\) | Episodic tabular MDP |
| \(\mathcal{S}, S\) | State space and its size \(S=|\mathcal{S}|\) |
| \(\mathcal{A}, A\) | Action space and its size \(A=|\mathcal{A}|\) |
| \(P\) | Transition kernel |
| \(H\) | Horizon length |
| \(T\) | Number of episodes |
| \(h(s)\) | Layer index of state \(s\) |
| \(s_{t,h},\,a_{t,h}\) | State / action at step \(h\) in episode \(t\) |
| \(\ell_t(s,a)\) | Loss assigned to \((s,a)\) in episode \(t\) |
| \(\pi_t\) | Policy in episode \(t\) |
| \(\Reg_T(\pi)\) | Regret over \(T\) episodes with comparator \(\pi\) |
| \(\I_t(s,a)\) | Visitation indicator of \((s,a)\) in episode \(t\) |
| \(n_t(s,a)\) | Number of visits to \((s,a)\) up to episode \(t-1\) |
| \(V^{\pi}(s;\ell)\) | Value function under policy \(\pi\) from state \(s\) with loss \(\ell\) |
| \(Q^{\pi}(s,a;\ell)\) | \(Q\)-function under policy \(\pi\) from \((s,a)\) with loss \(\ell\) |
| \(q^{\pi}(s),\;q^{\pi}(s,a)\) | Occupancy measure under policy \(\pi\) |
| \(q^{\pi}(s',a'\mid s,a)\) | Conditional occupancy from \((s,a)\) to \((s',a')\) under \(\pi\) |
| \(\pio\) | Optimal policy for the regret |
| \(\ell'_t(s,a)\) | Uncorrupted i.i.d. loss |
| \(\calC\) | Corruption budget |
| \(\mu(s,a)\) | Mean of \(\ell'_t(s,a)\) |
| \(\pist\) | Optimal deterministic policy under \(\mu\) |
| \(\Delta(s,a), \Delta_{\min}\) | Suboptimality gap at \((s,a)\), and minimum positive gap |
| Data-dependent complexity measures | |
| \(L(\pi)\) | First-order complexity |
| \(Q_\infty\) | Second-order complexity |
| \(V_1\) | Path-length complexity |
| \(\Qtrans^{\pi_{1:T}}(\ell)\) | Transition-dependent complexity in [def:qtrans95loss] |
| Notation for [alg:OFTRL95unknown] | |
| \(\psi_t\) | Regularizer in episode \(t\) |
| \(\eta_t(s,a)\) | Learning rate for \((s,a)\) in episode \(t\) |
| \(m_t(s,a)\) | Loss prediction for \((s,a)\) |
| \(\zeta_t(s,a)\) | Data-dependent term for updating \(\eta_t(s,a)\) |
| \(\hat Q_t(s,a)\) | \(Q\)-function estimator |
| \(b_t(s), B_t(s,a)\) | Exploaration bonus and its recursive \(Q\)-function version |
| \(c_t(s,a), C_t(s,a)\) | Transition bonus and its recursive \(Q\)-function version |
| \(Y_t\in\{0,1\}\) | Episode indicator (\(Y_t=1\) real, \(Y_t=0\) virtual) |
| \(\mathcal{T}_r,\;\mathcal{T}_v\) | Sets of real and virtual episodes |
| \(\overq^{\pi}_t(s), \underq^{\pi}_t(s)\) | Upper and lower occupancy measures |
| \(\gamma_t, q_t(s)\coloneqq\overq^{\pi_t}_t(s)+\gamma_t\) | Exploration rate and smoothed state occupancy |
| \(\rho_t(s)\coloneqq (q_t(s) - \underq_t^{\pi_t}(s))/q_t(s)\) | relative occupancy width |
| \(L_{t,h(s)}, M_{t,h(s)}\) | Realized / predicted suffix loss from layer \(h(s)\) |
| \(D_{t,h(s)}\) | Suffix prediction error from layer \(h(s)\) |
| \(\calP_t\) | Confidence set at episode \(t\) |
| \(\underP^\ell_t\) | Transition in \(\calP_t\) minimizing \(Q(s,a;\ell_t)\) for all \((s,a)\) |
| \(\underP^m_t\) | Transition in \(\calP_t\) minimizing \(Q(s,a;m_t)\) for all \((s,a)\) |
| \(\overP^B_t\) | Transition in \(\calP_t\) maximizing \(B_t(s,a)\) for all \((s,a)\) |
| \(\overP^C_t\) | Transition in \(\calP_t\) maximizing \(C_t(s,a)\) for all \((s,a)\) |
We formalize the conditional occupancy measure \(q^{\pi_t}(s',a'\mid s,a)\) as follows: \[\begin{align} \begin{cases} 0 & \text{if } h(s') < h(s),\\ 0 & \text{if } h(s') = h(s) \text{ and } (s',a') \neq (s,a),\\ 1 & \text{if } (s',a') = (s,a),\\ \Pr\set*{(s_{t,h(s')},a_{t,h(s')})=(s',a') \mid (s_{t,h(s)},a_{t,h(s)})=(s,a)}, & \text{if } h(s') > h(s). \end{cases} \end{align}\]
For \(\underline{P}^\ell_t\), \(\underline{P}^m_t\), \(\overline{P}^B_t\), and \(\overline{P}^C_t\), we choose kernels in \(\mathcal{P}_t\) that attain the required state-action-wise minimum or maximum for all \((s,a)\) simultaneously. This is possible since \(\mathcal{P}_t\) is rectangular, and these kernels are efficiently computable by backward dynamic programming with a state-action-wise greedy procedure over the confidence set, as in [7] and [6].
Adversarial MDPs have been studied since the works of [27], [28], and the episodic formulation was later introduced by [29]. Algorithms for online tabular MDPs can be broadly divided into two approaches: occupancy-measure based global optimization and policy optimization. Global optimization solves an optimization problem over the set of occupancy measures, and has led to strong regret guarantees in several settings [7], [29]. However, global optimization can be computationally demanding because it optimizes over the occupancy measure space of the MDP. Policy optimization takes a more local approach by optimizing an action distribution at each state, which is often practical and computationally efficient [5], [6].
Under known transitions and bandit feedback, global optimization achieves the minimax-optimal regret rate \(\widetilde{\mathcal{O}}(\sqrt{HSAT})\) [29]. In contrast, the best known bound for policy optimization is \(\widetilde{\mathcal{O}}(\sqrt{H^3SAT})\), achieved via dilated exploration [6]. Compared with the global optimization rate, this bound incurs an extra factor of \(H\), often attributed to \(Q\)-function estimation.
Unknown transitions introduce an additional difficulty, since the learner must also estimate the transition kernel. Among global optimization, the best known regret bounds were obtained by [18] for full-information feedback and [7] for bandit feedback, both of order \(\widetilde{\mathcal{O}}(\sqrt{H^2S^2AT})\). For policy optimization under bandit feedback, the best known guarantee was established by [6], with regret \(\widetilde{\mathcal{O}}(\sqrt{H^4S^2AT})\). On the other hand, minimax lower bounds of order \(\Omega(\sqrt{H^2SAT})\) were established by [25], [26], leaving gaps in the dependence on the horizon and the number of states. A line of work has sought to close these gaps, and [30] proposed a policy-optimization algorithm under full-information feedback with regret \(\widetilde{\mathcal{O}}(\sqrt{H^6SAT})\). More recent work has studied policy optimization under more challenging feedback models, including aggregate feedback [31], [32].
In the stochastic setting, near-optimal regret guarantees have been established by both model-based and value-based methods [20], [25], [33], [34]. Beyond such gap-independent guarantees, a line of work has studied gap-dependent \(\mathrm{polylog}(T)\) regret bounds for episodic tabular MDPs [8], [35], [36].
The best-of-both-worlds guarantee aims to obtain near-optimal regret in both adversarial and stochastic regimes using a single algorithm. This idea was first studied for multi-armed bandits by [37], and was later developed in several directions [38]–[40]. A common approach is to use follow-the-regularized-leader with carefully chosen regularizers, which allows the algorithm to remain robust in the adversarial regime while improving automatically in the stochastic regime [15], [41], [42]. The stochastic-regime analysis is often based on a self-bounding technique [41], [43], and the same idea is useful for handling stochastic losses with adversarial corruption.
Best-of-both-worlds guarantees have also been studied in tabular MDPs. For global optimization, [10], [11] first developed best-of-both-worlds algorithms, with subsequent extensions to adversarial transitions and aggregate feedback [32], [44]. In policy optimization, [1] established best-of-both-worlds guarantees under bandit feedback for several regularizers, including Tsallis entropy, Shannon entropy, and log-barrier regularization, while [2] later obtained refined data-dependent guarantees using the OFTRL framework under known transitions.
Data-dependent regret bounds adapt to the realized loss sequence instead of depending only on worst-case parameters. They have been widely studied in online learning, including learning with expert advice [45], multi-armed bandits [46], and online convex optimization [47]. Typical examples in adversarial regimes include first-order bounds based on the comparator’s cumulative loss [12]–[14], second-order bounds based on loss fluctuations [15], [42], [48], and path-length bounds based on temporal variation [15], [16], [42].
In tabular MDPs, it is also important to obtain regret bounds that adapt to data-dependent complexity measures. The first such result was obtained by [17], who derived a first-order bound for MDPs with unknown transitions via global optimization. For policy optimization, [1] established best-of-both-worlds guarantees under unknown transitions and first-order data-dependent guarantees under known transitions, leaving open whether data-dependent guarantees are possible under unknown transitions. More recently, [2] showed that, under known transitions, OFTRL-based policy optimization can adapt to multiple complexity measures, including first-order, second-order, and path-length measures.
A related line of work has studied gap-dependent bounds in stochastic regimes. For multi-armed bandits, [49] introduced variance-aware gap-dependent bounds, and related ideas have been incorporated into best-of-both-worlds analyses (e.g., [50]). For tabular MDPs, recent works have also studied variance-aware gap-dependent \(\mathrm{polylog}(T)\) regret bounds [2], [8], [9], [35].
These studies left open the case of data-dependent policy optimization under unknown transitions, especially when combined with best-of-both-worlds guarantees. Our work addresses these gaps by showing that policy optimization can achieve refined data-dependent and best-of-both-worlds guarantees under unknown transitions, up to an additional transition-dependent complexity term.
In this part, we discuss the role of the transition-dependent complexity \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\), which captures the difficulty caused by unknown transitions. By 14, we have \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell) \leq H\sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\ell_t(s,a)^2. \label{eq:qtrans95example} \end{align}\tag{16}\] Since \(\ell_t(s,a)\in[0,1]\), taking expectations gives \[\begin{align} \mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)} &\leq H\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\ell_t(s,a)} = H\prn*{L_T(\pi)+\text{\rm Reg}_T(\pi)} \label{eq:qtrans95first95order95relation} \end{align}\tag{17}\] for each policy \(\pi\). From this inequality, we can show that the transition-dependent bound recovers a regret bound depending on the first-order complexity. Suppose that for the any comparator policy \(\pi\), there exists an absolute constant \(c>0\) such that \[\begin{align} \text{\rm Reg}_T(\pi) \leq c\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}+J \end{align}\] for some \(J > 0\), as the regret bounds in 2 3. Then, using the inequality 17 , the regret is upper bounded as \[\begin{align} \text{\rm Reg}_T(\pi) &\leq c\sqrt{HS^2A\log^2(T)\prn*{L(\pi)+\text{\rm Reg}_T(\pi)}}+J\\ &\leq c\sqrt{HS^2A\log^2(T)L(\pi)}+c\sqrt{HS^2A\log^2(T)\text{\rm Reg}_T(\pi)}+J\\ &\leq c\sqrt{HS^2A\log^2(T)L(\pi)}+\frac{1}{2}\text{\rm Reg}_T(\pi)+\frac{1}{2}c^2HS^2A\log^2(T) +J \label{eq:adv95bandit95first95order95with95qtrans} \end{align}\tag{18}\] where the last inequality follows from the AM–GM inequality. Thus, we obtain \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{HS^2A\log^2(T)L(\pi)}+ HS^2A\log^2(T)+J. \end{align}\] This implies that \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) is at least as adaptive as the first-order complexity. However, the comparison in 16 can be loose. Transition uncertainty contributes to \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) only when it creates variation in the future cumulative loss.
The following example shows that \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) can be small even when the cumulative loss is large. Consider an \(H=3\) layered MDP with layers \(\mathcal{S}_0=\{s_0\}\), \(\mathcal{S}_1=\{x_1,x_2\}\), and \(\mathcal{S}_2=\{y\}\), followed by the terminal state. For each action \(a\), the transition \(P(\cdot\mid s_0,a)\) may be stochastic and unknown over \(\{x_1,x_2\}\). For every action, the transitions from \(x_1\) and \(x_2\) go deterministically to \(y\), and the transition from \(y\) goes deterministically to the terminal state.
Let the losses at \(s_0\), \(x_1\) and \(x_2\) be zero, and let the loss at \(y\) be one: \[\begin{align} \ell_t(s_0,a)=0, \qquad \ell_t(x_i,a)=0 \quad (i\in\{1,2\}), \qquad \ell_t(y,a)=1, \qquad \forall a\in\mathcal{A},\;\forall t \in [T]. \end{align}\] Since every trajectory passes through \(y\) and the loss at \(y\) is one, the cumulative loss is linear in \(T\), and in particular \[\begin{align} \sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\ell_t(s,a)^2 = T. \end{align}\] However, for any sequence \(\{\phi_t\}_{t=1}^T\) satisfying \(|\phi_t(s,a)|=\ell_t(s,a)\), we have \[\begin{align} V^{\pi_t}(x_1;\phi_t)=V^{\pi_t}(x_2;\phi_t). \end{align}\] Hence, for every action \(a\), \[\begin{align} \mathsf{Var}_{s'\sim P(\cdot\mid s_0,a)} \left[ V^{\pi_t}(s';\phi_t) \right] =0. \end{align}\] The other transitions are deterministic, so no transition contributes to \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\), and thus \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)=0\).
In this example, the right-hand side of 16 is of order \(HT\), whereas \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)=0\). By adding deterministic layers after \(y\), the same construction gives an \(\mathcal{O}(H^2T)\) term on the right-hand side while keeping \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)=0\). This shows that \(Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)\) can be much smaller than the upper bound in 16 17 .
We define the following good event \(\mathcal{E}=\mathcal{E}_1 \cap \mathcal{E}_2\), which holds with high probability as shown in 8.
\[\begin{align} \mathcal{E}_1 &= \set*{P\in\mathcal{P}_t,\;\forall t=1,\dots,T}, \\ \mathcal{E}_2 &= \set*{\sum_{t=1}^{T}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}} \frac{q^{\pi_t}(s,a)}{n_t(s,a)} \lesssim \relax{\mathcal{S}_h}A\log T+\log(H/\delta),\;\forall h = 0, \dots, H-1}. \\ \end{align}\]
Lemma 5 (Event \(\mathcal{E}_1\), [7], Lemma 2). With probability at least \(1 - 4\delta\), we have \(P \in \mathcal{P}_t\) for all \(t = 1, \dots, T\).
Lemma 6 ([11], Corollary D.3.4). If \(P\in \mathcal{P}_t\), then for all \(\widetilde{P}\in \mathcal{P}_t\) \[\begin{align} \relax*{\widetilde{P}(s'\mid s, a) - P(s'\mid s,a)} \leq \min\set*{4\sqrt{\frac{P(s'\mid s, a)\iota}{n_t(s,a)}} + \frac{40\iota}{3n_t(s,a)}, 1} \end{align}\]
Lemma 7 (Event \(\mathcal{E}_2\), [7], Lemma 10). We have with probability at least \(1 - \delta\) \[\begin{align} \sum_{t=1}^{T}\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \frac{q^{\pi_t}(s,a)}{n_t(s,a)} &\lesssim \relax{\mathcal{S}_h}A\log T + \log(H/\delta) \end{align}\] for all layers \(h\).
Definition 8. Define \(\mathcal{E}\) to be the event that \(P \in \mathcal{P}_t\) for all \(t\) and the bound in 7 holds. In this case, \(\Pr(\mathcal{E}) \geq 1 - 5\delta\).
Lemma 9 (Performance difference lemma). For any policies \(\pi_1\) and \(\pi_2\), and any loss function \(\ell:S\times A \to \mathbb{R}\), \[\begin{align} V^{\pi_1}(s_0; \ell) - V^{\pi_2}(s_0; \ell) = \sum_s q^{\pi_2}(s)\sum_a \prn*{\pi_1(a \mid s) - \pi_2(a \mid s)}Q^{\pi_1}(s,a;\ell). \end{align}\]
Lemma 10 ([1], Lemma C.2). For any policies \(\pi_1\) and \(\pi_2\), and any function \(L:S\times A \to \mathbb{R}\), define \[\begin{align} \ell(s,a) = L(s,a) - \mathbb{E}_{s'\sim P(\cdot \mid s, a),\, a'\sim \pi_1(\cdot \mid s')}\brk*{L(s',a')}. \end{align}\] Then, for all \((s,a)\), \(Q^{\pi_1}(s,a;\ell) = L(s,a)\). Consequently, \[\begin{align} \sum_s q^{\pi_2}(s)\sum_a \prn*{\pi_1(a \mid s) - \pi_2(a \mid s)}L(s,a) = V^{\pi_1}(s_0; \ell) - V^{\pi_2}(s_0; \ell). \end{align}\]
Lemma 11 (Occupancy measure difference, [11] Lemma D.3.1). For any transition functions \(P_1,P_2\) and any policy \(\pi\), \[\begin{align} q^{P_1,\pi}(s) - q^{P_2,\pi}(s) &= \sum_{h=0}^{h(s)-1}\sum_{u\in \mathcal{S}_h}\sum_{v\in\mathcal{A}}\sum_{w\in \mathcal{S}_{h+1}}q^{P_1,\pi}(u,v)\prn*{P_1(w\mid u,v)-P_2(w\mid u,v) } q^{P_2,\pi}(s\mid w) \\ & = \sum_{h=0}^{h(s)-1}\sum_{u\in \mathcal{S}_h}\sum_{v\in\mathcal{A}}\sum_{w\in \mathcal{S}_{h+1}}q^{P_2,\pi}(u,v)\prn*{P_1(w\mid u,v)-P_2(w\mid u,v) } q^{P_1,\pi}(s\mid w). \end{align}\]
Lemma 12. Suppose that the event \(\mathcal{E}\) holds. Let \(F : \mathcal{S}_{h+1} \to [-\overline{F},\overline{F}]\). Then, \[\begin{align} \relax*{\sum_{s' \in S_{h + 1}}\prn*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} F(s')} &\lesssim \sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{F(s')}\iota}{n_t(s,a)}} +\frac{\relax{\mathcal{S}_{h+1}}\overline{F}\iota}{n_t(s,a)} \end{align}\] for all state-action pairs \((s,a) \in \mathcal{S}_h \times \mathcal{A}\).
Proof. Since \(\sum_{s' \in S_{h + 1}}\prn*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} = 0\), we may subtract any constant from \(F\). Let \[\begin{align} c = \mathbb{E}_{s' \sim P(\cdot \mid s, a)}\brk*{F(s')}. \end{align}\] Then, \[\begin{align} \sum_{s' \in S_{h + 1}}\prn*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} F(s') = \sum_{s' \in S_{h + 1}}\prn*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} \prn*{F(s') - c} \end{align}\] Therefore, \[\begin{align} &\relax*{\sum_{s' \in S_{h + 1}}\prn*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} F(s')} \\ &\leq \sum_{s' \in S_{h + 1}}\relax*{\widetilde{P}_t(s'\mid s, a) - P(s' \mid s, a)} \relax*{F(s') - c}\\ &\lesssim \sum_{s' \in S_{h + 1}}\sqrt{\frac{P(s'\mid s, a)\iota}{n_t(s,a)}} \relax*{F(s') - c} + \sum_{s' \in S_{h + 1}}\frac{\iota}{n_t(s,a)} \relax*{F(s') - c} and the assumption that \mathcal{E} holds}\\ &\leq \sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\sum_{s' \in S_{h + 1}} P(s'\mid s, a)\prn*{F(s') - c}^2 \iota}{n_t(s,a)}} +\frac{2\relax{\mathcal{S}_{h+1}}\overline{F}\iota}{n_t(s,a)}\le 2\overline{F}}\\ &\leq \sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{F(s')}\iota}{n_t(s,a)}} +\frac{2\relax{\mathcal{S}_{h+1}}\overline{F}\iota}{n_t(s,a)}. \end{align}\] ◻
Lemma 13. Suppose that the event \(\mathcal{E}\) holds. Let \(\widetilde{P}_t\) be a transition kernel in \(\mathcal{P}_t\), and let \(g_t(s,a)\in[0,G]\). Then, it holds that \[\begin{align} \sum_{t=1}^T \sum_{s,a} \relax*{ q^{\pi_t}(s) - q^{\widetilde{P}_t, \pi_t}(s)} \pi_t(a\mid s) g_t(s,a) \lesssim \sqrt{S^2A\iota^2Q_{\mathrm{trans}}^{\pi_{1:T}}(g)} + HS^3AG\iota^2, \end{align}\] where \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(g) \coloneq \sup_{\forall t,\forall s,a\; |\phi_t(s,a)|=g_t(s,a)} \sum_{t=1}^T\sum_{s,a} q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t)}. \end{align}\]
Proof. For each layer \(h \in \{0, \dots, H-1\}\), let \(W_h \mathrel{\vcenter{:}}= \mathcal{S}_h \times \mathcal{A} \times \mathcal{S}_{h+1}\). Throughout the proof, whenever summation indices are omitted, expressions such as \(\sum_{u,v,w}\) or \(\sum_{x,y,z}\) stand for \(\sum_{h=0}^{H-1} \sum_{(s,a,s') \in W_h}\).
Let \(\phi_t(s,a) = \mathrm{sgn}\prn*{q^{\pi_t}(s) - q^{\widetilde{P}_t,\pi_t}(s)}g_t(s,a)\). According to 11, we have \[\begin{align} &\sum_{s,a}\relax*{ q^{\pi_t}(s) - q^{\widetilde{P}_t,\pi_t}(s)}\pi_t(a\mid s)g_t(s,a) \\ &= \sum_{s,a}\prn*{q^{\pi_t}(s) - q^{\widetilde{P}_t,\pi_t}(s)}\pi_t(a\mid s)\phi_t(s,a)\\ &= \sum_{s,a}\sum_{u,v,w} q^{\pi_t}(u,v)\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } q^{\widetilde{P}_t,\pi_t}(s\mid w)\pi_t(a\mid s)\phi_t(s,a) \\ &= \underbrace{\sum_{s,a}\sum_{u,v,w} q^{\pi_t}(u,v)\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } q^{\pi_t}(s\mid w)\pi_t(a\mid s)\phi_t(s,a)}_{\mathrm{\boldsymbol{term}}_1} \\ &\qquad + \underbrace{\sum_{s}\sum_a\sum_{u,v,w} q^{\pi_t}(u,v)\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } \prn*{q^{\widetilde{P}_t, \pi_t}(s \mid w) - q^{\pi_t}(s\mid w)}\pi_t(a\mid s)\phi_t(s,a)}_{\mathrm{\boldsymbol{term}}_2}. \end{align}\] We first bound \(\mathrm{\boldsymbol{term}}_1\). \[\begin{align} \mathrm{\boldsymbol{term}}_1 &\leq \sum_{u,v,w} q^{\pi_t}(u,v)\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } \sum_{s,a} q^{\pi_t}(s,a\mid w)\phi_t(s,a)\\ &= \sum_{u,v,w} q^{\pi_t}(u,v)\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } V^{\pi_t}(w; \phi_t)\\ &\leq \sum_{u,v} q^{\pi_t}(u,v)\relax*{\sum_w\prn*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } V^{\pi_t}(w; \phi_t)}\\ &\leq \sum_{h=0}^{H-1}\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}} q^{\pi_t}(u,v)\prn*{\sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\mathsf{Var}_{w \sim P(\cdot \mid u, v)}\brk*{V^{\pi_t}(w; \phi_t)}\iota}{n_t(u, v)}} +\frac{\relax{\mathcal{S}_{h+1}}HG\iota}{n_t(u, v)} } }\\ &= \underbrace{\sum_{h=0}^{H-1}\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}} q^{\pi_t}(u,v)\prn*{\sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\mathsf{Var}_{w \sim P(\cdot \mid u, v)}\brk*{V^{\pi_t}(w; \phi_t)}\iota}{n_t(u, v)}}}}_{\mathrm{\boldsymbol{term}}_{1a}}\\ &\qquad +\underbrace{\sum_{h=0}^{H-1}\relax{\mathcal{S}_{h+1}}HG\iota\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}}\frac{q^{\pi_t}(u,v)}{n_t(u, v)}}_{\mathrm{\boldsymbol{term}}_{1b}}\\ \end{align}\] For \(\mathrm{\boldsymbol{term}}_{1a}\), \[\begin{align} \sum_{t = 1}^T\mathrm{\boldsymbol{term}}_{1a} &\leq \sum_{t = 1}^T\sum_{h=0}^{H-1}\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}} q^{\pi_t}(u,v)\prn*{\sqrt{\frac{\relax*{\mathcal{S}_{h+1}}\mathsf{Var}_{w \sim P(\cdot \mid u, v)}\brk*{V^{\pi_t}(w; \phi_t)}\iota}{n_t(u, v)}}}\\ &\leq \sqrt{\sum_{t = 1}^T\sum_{h=0}^{H-1}\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}}\relax*{\mathcal{S}_{h+1}}\frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}}\\ &\qquad \times \sqrt{\sum_{t = 1}^T\sum_{h=0}^{H-1}\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}}q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}} \\ &\lesssim \sqrt{\prn*{\sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\relax{\mathcal{S}_h}A\iota^2}\prn*{\sum_{t = 1}^T\sum_{u,v} q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}}}and the assumption that \mathcal{E} holds}\\ &\leq \sqrt{S^2A\iota^2\prn*{\sum_{t = 1}^T\sum_{u,v} q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}}}, \label{eq:var95term1a} \end{align}\tag{19}\] where the last inequality uses \(\sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\relax{\mathcal{S}_h} \leq S^2\).
For \(\mathrm{\boldsymbol{term}}_{1b}\), \[\begin{align} \sum_{t = 1}^T\mathrm{\boldsymbol{term}}_{1b} &\leq \sum_{t = 1}^T\sum_{h=0}^{H-1}\relax{\mathcal{S}_{h+1}}HG\iota\sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}}\frac{q^{\pi_t}(u,v)}{n_t(u, v)}\\ &\lesssim \sum_{t = 1}^T\sum_{h=0}^{H-1}\relax{\mathcal{S}_h}\relax{\mathcal{S}_{h+1}}HAG\iota^2and the assumption that \mathcal{E} holds}\\ &\leq HS^2AG\iota^2. \label{eq:var95term1b} \end{align}\tag{20}\] Thus, combining 19 20 , we obtain \[\begin{align} \sum_{t = 1}^T\mathrm{\boldsymbol{term}}_{1} \lesssim \sqrt{S^2A\iota^2\prn*{\sum_{t = 1}^T\sum_{u,v} q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}}} + HS^2AG\iota^2. \end{align}\]
We next bound \(\mathrm{\boldsymbol{term}}_2\). \[\begin{align} &\mathrm{\boldsymbol{term}}_2\\ &\leq \sum_s\sum_{u,v,w} q^{\pi_t}(u,v)\relax*{P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } \relax*{q^{\widetilde{P}_t, \pi_t}(s \mid w) - q^{\pi_t}(s\mid w)}\sum_a \pi_t(a\mid s)\relax{\phi_t(s,a)} \\ &\leq G\sum_s\sum_{u,v,w} q^{\pi_t}(u,v)\relax*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } \sum_{x,y,z} q^{\pi_t}(x,y\mid w)\relax*{ P(z\mid x,y) - \widetilde{P}_t(z\mid x,y) }q^{\widetilde{P}_t,\pi_t} (s\mid z) }\\ &\leq HG\sum_s\sum_{u,v,w} q^{\pi_t}(u,v)\relax*{ P(w\mid u,v) - \widetilde{P}_t(w\mid u,v) } \sum_{x,y,z} q^{\pi_t}(x,y\mid w)\relax*{ P(z\mid x,y) - \widetilde{P}_t(z\mid x,y)}\\ &\lesssim HG\sum_{u,v,w} \sum_{x,y,z} q^{\pi_t}(u,v)\prn*{\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}} + \frac{\iota}{n_t(u,v)}}q^{\pi_t}(x,y\mid w)\min\set*{\sqrt{\frac{P(z\mid x,y)\iota}{n_t(x,y)}} + \frac{\iota}{n_t(x,y)}, 1}and the assumption that \mathcal{E} holds}\\ &\leq HG\underbrace{\sum_{u,v,w}\sum_{x,y,z} q^{\pi_t}(u,v)\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi_t}(x,y\mid w)\sqrt{\frac{P(z\mid x,y)\iota}{n_t(x,y)}} }_{\mathrm{\boldsymbol{term}}_{2a}}\\ &\qquad + HG\underbrace{\sum_{u,v,w}\sum_{x,y,z}q^{\pi_t}(u,v) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}} q^{\pi_t}(x,y\mid w)\min\set*{\frac{\iota}{n_t(x,y)}, \;1}}_{\mathrm{\boldsymbol{term}}_{2b}}\\ &\qquad + HG\underbrace{\sum_{u,v,w}\sum_{x,y,z} q^{\pi_t}(u,v)\frac{\iota}{n_t(u,v)}q^{\pi_t}(x,y\mid w)}_{\mathrm{\boldsymbol{term}}_{2c}}. \end{align}\] For \(\mathrm{\boldsymbol{term}}_{2a}\), \[\begin{align} &\mathrm{\boldsymbol{term}}_{2a}\\ & =\sum_{u,v,w}\sum_{x,y,z} \sqrt{\frac{q^{\pi_t}(u,v) P(z\mid x,y) q^{\pi_t}(x,y\mid w) \iota}{n_t(u,v)}}\sqrt{\frac{q^{\pi_t}(u,v) P(w\mid u,v) q^{\pi_t}(x,y\mid w) \iota}{n_t(x,y)}} \\ &\leq \sqrt{\sum_{u,v,w}\sum_{x,y,z}\frac{q^{\pi_t}(u,v) P(z\mid x,y) q^{\pi_t}(x,y\mid w) \iota}{n_t(u,v)}}\sqrt{\sum_{u,v,w}\sum_{x,y,z}\frac{q^{\pi_t}(u,v) P(w\mid u,v) q^{\pi_t}(x,y\mid w) \iota}{n_t(x,y)}}\\ &\leq \sqrt{H\sum_{u,v,w}\frac{q^{\pi_t}(u,v) \iota}{n_t(u,v)}}\sqrt{H\sum_{x,y,z}\frac{q^{\pi_t}(x,y)\iota}{n_t(x,y)}} \\ &\leq HS \sum_{u,v} \frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)},\label{eq:var95term2a} \end{align}\tag{21}\] where the third line follows from the facts that \(\sum_{(x,y,z) \in W_h} q^{\pi_t}(x,y \mid w) P(z \mid x,y) \le 1\) and \(\sum_{(u,v,w) \in W_h} q^{\pi_t}(u,v) P(w \mid u,v) q^{\pi_t}(x,y \mid w) \le q^{P,\pi_t}(x,y)\) for any layer \(h \in \{0, \dots, H-1\}\).
For \(\mathrm{\boldsymbol{term}}_{2b}\), \[\begin{align} &\mathrm{\boldsymbol{term}}_{2b} \\ &\leq \sum_{u,v,w}\sum_{x,y,z} q^{\pi_t}(u,v) \prn*{P(w\mid u,v) + \frac{\iota}{n_t(u,v)}}q^{P,\pi_t}(x,y\mid w) \min\left\{\frac{\iota}{n_t(x,y)}, \;1\right\} \\ &\leq \sum_{u,v,w}\sum_{x,y,z} q^{\pi_t}(u,v)P(w\mid u,v)q^{\pi_t}(x,y\mid w) \frac{\iota}{n_t(x,y)} + \sum_{u,v,w}\sum_{x,y,z} q^{\pi_t}(u,v)\frac{\iota}{n_t(u,v)}q^{\pi_t}(x,y\mid w) \\ &\leq H\sum_{x,y,z}q^{\pi_t}(x,y)\frac{\iota}{n_t(x,y)} + S\sum_{u,v,w}q^{\pi_t}(u,v)\frac{\iota}{n_t(u,v)} \\ &\lesssim S^2\sum_{u,v}\frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}, \label{eq:var95term2b} \end{align}\tag{22}\] where the third line follows from \(\sum_{(u,v,w) \in W_h} q^{\pi_t}(u,v) P(w \mid u,v) q^{\pi_t}(x,y \mid w) \le q^{\pi_t}(x,y)\) and \(\sum_{(x,y,z) \in W_h} q^{\pi_t}(x,y \mid w) \leq |\mathcal{S}_{h + 1}|\).
Similarly, \[\begin{align} \mathrm{\boldsymbol{term}}_{2c} \leq S\sum_{u,v,w}q^{\pi_t}(u,v)\frac{\iota}{n_t(u,v)} \leq S^2\sum_{u,v}\frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}. \label{eq:var95term2c} \end{align}\tag{23}\] Combining 21 22 23 , we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_2 &\lesssim HS^2G\sum_{u,v}\frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}. \end{align}\] Therefore, by 7, \[\begin{align} \sum_{t=1}^T \mathrm{\boldsymbol{term}}_2 \lesssim \sum_{h = 0}^{H - 1} HS^2G\relax{\mathcal{S}_h}A\iota^2 \leq HS^3AG\iota^2. \end{align}\] Combining the bounds on \(\mathrm{\boldsymbol{term}}_1\) and \(\mathrm{\boldsymbol{term}}_2\), we conclude that \[\begin{align} &\sum_{t=1}^T \sum_{s,a} \relax*{ q^{\pi_t}(s) - q^{\widetilde{P}_t, \pi_t}(s)} \pi_t(a\mid s) g_t(s,a)\\ &\lesssim \sqrt{S^2A\iota^2\prn*{\sum_{t = 1}^T\sum_{u,v} q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}}} + HS^3AG\iota^2. \end{align}\] Here, we define \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(g) \coloneq \sup_{\forall t,\forall s,a\; |\phi_t(s,a)|=g_t(s,a)} \sum_{t = 1}^T\sum_{u,v} q^{\pi_t}(u,v)\mathsf{Var}_{w \sim P(\cdot \mid u,v)}\brk*{V^{\pi_t}(w; \phi_t)}, \end{align}\] which completes the proof. ◻
Lemma 14. For any nonnegative functions \(g_t : \mathcal{S}\times\mathcal{A}\to \mathbb{R}_{\geq 0}\), \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(g) \coloneq \sup_{\forall t,\forall s,a\; |\phi_t(s,a)|=g_t(s,a)} \sum_{t=1}^T\sum_{s,a} q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t)} \end{align}\] satisfies \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(g) \leq H\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s,a)g_t(s,a)^2. \end{align}\]
Proof. Since \(|\phi_t(s,a)| = g_t(s,a)\) for all \(s,a\), it suffices to prove that for each \(t\) and each such function \(\phi_t\), \[\sum_{s,a} q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t)} \leq H\sum_{s,a} q^{\pi_t}(s,a)\phi_t(s,a)^2 .\]
Let \((X_{t,0},A_{t,0},X_{t,1},A_{t,1},\dots,X_{t,H})\) be the random trajectory generated by \((P,\pi_t)\). For \(h=0,1,\dots,H\) and \(x\in \mathcal{S}_h\), define \(\mathcal{V}_{t,H}= 0\) and \(\mathcal{M}_{t,H} = 0\) and \[\mathcal{G}_{t,h} \mathrel{\vcenter{:}}= \sum_{h'=h}^{H-1}\phi_t(X_{t,h'}, A_{t,h'}),\quad \mathcal{V}_{t,h}(x) \mathrel{\vcenter{:}}= \mathbb{E}_{\pi_t}[\mathcal{G}_{t,h}\mid X_{t,h}=x],\quad \mathcal{M}_{t,h}(x) \mathrel{\vcenter{:}}= \mathbb{E}_{\pi_t}[\mathcal{G}_{t,h}^2\mid X_{t,h}=x].\]
Since \(\mathcal{G}_{t,h} = \phi_t(X_{t,h},A_{t,h}) + \mathcal{G}_{t,h+1}\), we have for any state \(x\) \[\begin{align} \mathcal{V}_{t,h}(x) = \sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t, h+1}(x')}, \end{align}\] and \[\begin{align} \mathcal{M}_{t,h}(x) &= \sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a)^2+2\phi_t(x,a)\sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}\\ &\qquad +\sum_a \pi_t(a\mid x)\sum_{x'} P(x'\mid x,a)\mathcal{M}_{t,h+1}(x'). \end{align}\] Then, we have \[\begin{align} &\mathcal{M}_{t,h}(x) - \mathcal{V}_{t,h}(x)^2\\ &= \sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a)^2 + 2\phi_t(x,a)\sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x') + \sum_{x'} P(x'\mid x,a)\mathcal{M}_{t,h+1}(x')}\\ &\qquad - \prn*{\sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}}^2 \\ &= \sum_a \pi_t(a\mid x)\sum_{x'} P(x'\mid x,a)\prn*{\mathcal{M}_{t,h+1}(x') - \mathcal{V}_{t,h+1}(x')^2} \\ &\qquad + \sum_a \pi_t(a\mid x)\prn*{\sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')^2 - \prn*{\sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}^2} \\ &\qquad + \sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}^2\\ &\qquad - \prn*{\sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}}^2\\ &\geq \sum_a \pi_t(a\mid x)\sum_{x'} P(x'\mid x,a)\prn*{\mathcal{M}_{t,h+1}(x') - \mathcal{V}_{t,h+1}(x')^2} \\ &\qquad + \sum_a \pi_t(a\mid x)\mathsf{Var}_{x' \sim P(\cdot \mid x,a)}\brk*{\mathcal{V}_{t,h+1}(x')}, \end{align}\] where the last inequality follows from Jensen’s inequality \[\begin{align} &\prn*{\sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}}^2 \\ &\leq \sum_a \pi_t(a\mid x)\prn*{\phi_t(x,a) + \sum_{x'} P(x'\mid x,a)\mathcal{V}_{t,h+1}(x')}^2. \end{align}\] Thus, we have \[\begin{align} \sum_a \pi_t(a\mid x)\mathsf{Var}_{x' \sim P(\cdot \mid x,a)}\brk*{\mathcal{V}_{t,h+1}(x')} &\leq \prn*{\mathcal{M}_{t,h}(x) - \mathcal{V}_{t,h}(x)^2}\\ &\qquad - \sum_a \pi_t(a\mid x)\sum_{x'} P(x'\mid x,a)\prn*{\mathcal{M}_{t,h+1}(x') - \mathcal{V}_{t,h+1}(x')^2}. \\ \end{align}\]
By multiplying \(q^{\pi_t}(x)\) and sum over \(x\in \mathcal{S}_h\), we obtain \[\begin{align} \sum_{(x,a)\in \mathcal{S}_h\times\mathcal{A}} q^{\pi_t}(x,a)\mathsf{Var}_{x' \sim P(\cdot \mid x,a)}\brk*{\mathcal{V}_{t,h+1}(x')} &\leq \sum_{x\in \mathcal{S}_h} q^{\pi_t}(x)\prn*{\mathcal{M}_{t,h}(x) - \mathcal{V}_{t,h}(x)^2}\\ &\qquad -\sum_{x'\in \mathcal{S}_{h+1}} q^{\pi_t}(x')\prn*{\mathcal{M}_{t,h+1}(x') - \mathcal{V}_{t,h+1}(x')^2}. \end{align}\] Summing over \(h=0,1,\dots,H-1\) and \(\mathcal{V}_{t,h+1}(x') = V^{\pi_t}(x'; \phi_t)\), we obtain \[\begin{align} \sum_{s,a} q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t)} \leq \mathcal{M}_{t,0}(s_0) - \mathcal{V}_{t,0}(s_0)^2 \leq \mathcal{M}_{t,0}(s_0). \label{eq:Var95leq95M} \end{align}\tag{24}\]
Finally, \[\begin{align} \mathcal{M}_{t,0}(s_0) &= \mathbb{E}_{\pi_t}\brk*{\prn*{\sum_{h=0}^{H-1}\phi_t(X_{t,h}, A_{t,h})}^2}\\ &\leq H\mathbb{E}_{\pi_t}\brk*{\sum_{h=0}^{H-1}\phi_t(X_{t,h}, A_{t,h})^2} = H\sum_{s,a} q^{\pi_t}(s,a)\phi_t(s,a)^2, \label{eq:M95leq95phi} \end{align}\tag{25}\] where the inequality follows from the Cauchy–Schwarz inequality. Combining 24 25 completes the proof. ◻
By following the same argument as for \(\mathrm{\boldsymbol{term}}_2\) in the proof of 13, we obtain the following bound, whose lower-order term improves that of [1] by a factor of \(S\).
Lemma 15. Suppose that the event \(\mathcal{E}\) holds. Let \(\widetilde{P}_t^s\) be a transition kernel in \(\mathcal{P}_t\) which may depend on \(s\), and let \(g_t(s)\in[0,G]\). Then \[\begin{align} \sum_{t=1}^T \sum_s \relax*{ q^{\pi_t}(s) - q^{\widetilde{P}_t^s, \pi_t}(s)} g_t(s) \lesssim \sqrt{HS^2A \iota^2 \sum_{t=1}^T \sum_s q^{\pi_t}(s)g_t(s)^2} + HS^3AG\iota^2. \end{align}\]
Lemma 16. Suppose that the event \(\mathcal{E}\) holds. Let \(\widetilde{P}^{s,\widetilde{s}}_t\) be a transition kernel in \(\mathcal{P}_t\) which may depend on \(s,\widetilde{s}\), and let \(g_t(s)\in[0,G]\). Then \[\begin{align} &\sum_{t=1}^T \sum_{\widetilde{s}}q^{\pi_t}(\widetilde{s})\sum_{s} \relax*{ q^{\pi_t}(s\mid \widetilde{s}) - q^{\widetilde{P}^{s,\widetilde{s}}_t, \pi_t}(s\mid \widetilde{s})} g_t(s) \\ &\lesssim \sqrt{H^3 S^2 A \iota^2 \sum_{t=1}^T \sum_{s} q^{\pi_t}(s)g_t(s)^2} + HS^3AG\iota^2. \end{align}\]
Proof. For each layer \(h \in \{0, \dots, H-1\}\), let \(W_h \mathrel{\vcenter{:}}= \mathcal{S}_h \times \mathcal{A} \times \mathcal{S}_{h+1}\). Throughout the proof, whenever summation indices are omitted, expressions such as \(\sum_{u,v,w}\) or \(\sum_{x,y,z}\) stand for \(\sum_{h=0}^{H-1} \sum_{(s,a,s') \in W_h}\).
First, we prove that for any \(\widetilde{s}\), and policy \(\pi\), \[\begin{align} \sum_s\relax*{ q^{\pi}(s\mid \widetilde{s}) - q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid \widetilde{s})} &\lesssim \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s}) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi}(s\mid w) + HS^2\sum_{u,v}\frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)}. \end{align}\]
By 11, we have \[\begin{align} &\sum_s\relax*{ q^{\pi}(s\mid \widetilde{s}) - q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid \widetilde{s})} \\ & \leq \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid w) }\\ & \leq \sum_s \sum_{u,v,w} q^{P,\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } q^{\pi}(s\mid w) \\ & \quad + \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } \prn*{q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid w) - q^{\pi}(s\mid w)} \\ & \leq \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } q^{\pi}(s\mid w) \\ & \quad + \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } \sum_{x,y,z} q^{\pi}(x,y\mid w)\relax*{ P(z\mid x,y) - \widetilde{P}^{s,\widetilde{s}}_t(z\mid x,y)}q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid z)}\\ & \leq \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } q^{\pi}(s\mid w) \\ & \quad + H\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\relax*{ P(w\mid u,v) - \widetilde{P}^{s,\widetilde{s}}_t(w\mid u,v) } \sum_{x,y,z} q^{\pi}(x,y\mid w)\relax*{ P(z\mid x,y) - \widetilde{P}^{s,\widetilde{s}}_t(z\mid x,y)} \\ &\lesssim \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\prn*{\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}} + \frac{\iota}{n_t(u,v)}}q^{\pi}(s\mid w) \\ &\qquad + H\sum_{u,v,w} \sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s})\prn*{\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}} + \frac{\iota}{n_t(u,v)}}q^{\pi}(x,y\mid w)\min\set*{\sqrt{\frac{P(z\mid x,y)\iota}{n_t(x,y)}} + \frac{\iota}{n_t(x,y)}, 1}and the assumption that \mathcal{E} holds}\\ &\leq \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s}) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi}(s\mid w)\\ &\qquad + \underbrace{ \sum_s\sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s})\frac{\iota}{n_t(u,v)}q^{\pi}(s\mid w)}_{\mathrm{\boldsymbol{term}}_1} \\ &\qquad + H\underbrace{\sum_{u,v,w}\sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s})\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi}(x,y\mid w)\sqrt{\frac{P(z\mid x,y)\iota}{n_t(x,y)}} }_{\mathrm{\boldsymbol{term}}_2}\\ &\qquad + H\underbrace{\sum_{u,v,w}\sum_{x,y,z}q^{\pi}(u,v\mid \widetilde{s}) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}} q^{\pi}(x,y\mid w)\min\set*{\frac{\iota}{n_t(x,y)}, \;1}}_{\mathrm{\boldsymbol{term}}_3}\\ &\qquad + H\underbrace{\sum_{u,v,w}\sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s})\frac{\iota}{n_t(u,v)}q^{\pi}(x,y\mid w)}_{\mathrm{\boldsymbol{term}}_4}. \end{align}\] We bound \(\mathrm{\boldsymbol{term}}_1\) to \(\mathrm{\boldsymbol{term}}_4\) separately. \[\begin{align} \mathrm{\boldsymbol{term}}_1 \leq \sum_s\sum_{u,v,w} \frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)}\leq S^2\sum_{u,v} \frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)}. \label{eq:conditional95term1} \end{align}\tag{26}\] For \(\mathrm{\boldsymbol{term}}_2\), we have \[\begin{align} &\mathrm{\boldsymbol{term}}_2\\ & =\sum_{u,v,w}\sum_{x,y,z} \sqrt{\frac{q^{\pi}(u,v\mid \widetilde{s}) P(z\mid x,y) q^{\pi}(x,y\mid w) \iota}{n_t(u,v)}}\sqrt{\frac{q^{\pi}(u,v\mid \widetilde{s}) P(w\mid u,v) q^{\pi}(x,y\mid w) \iota}{n_t(x,y)}} \\ &\leq \sqrt{\sum_{u,v,w}\sum_{x,y,z}\frac{q^{\pi}(u,v\mid \widetilde{s}) P(z\mid x,y) q^{\pi}(x,y\mid w) \iota}{n_t(u,v)}}\sqrt{\sum_{u,v,w}\sum_{x,y,z}\frac{q^{\pi}(u,v\mid \widetilde{s}) P(w\mid u,v) q^{\pi}(x,y\mid w) \iota}{n_t(x,y)}}\\ &\leq \sqrt{H\sum_{u,v,w}\frac{q^{\pi}(u,v\mid \widetilde{s}) \iota}{n_t(u,v)}}\sqrt{H\sum_{x,y,z}\frac{q^{\pi}(x,y\mid \widetilde{s})\iota}{n_t(x,y)}} \\ &\leq HS \sum_{u,v} \frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)},\label{eq:conditional95term2} \end{align}\tag{27}\] where the third line follows from the facts that \(\sum_{(x,y,z) \in W_h} q^{\pi}(x,y \mid w) P(z \mid x,y) \leq 1\) and \(\sum_{(u,v,w) \in W_h} q^{\pi}(u,v \mid \widetilde{s}) P(w \mid u,v) q^{\pi}(x,y \mid w) \le q^{\pi}(x,y \mid \widetilde{s})\) for any layer \(h \in \{0, \dots, H-1\}\).
For \(\mathrm{\boldsymbol{term}}_3\), we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\leq \sum_{u,v,w}\sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s}) \prn*{P(w\mid u,v) + \frac{\iota}{n_t(u,v)}}q^{\pi}(x,y\mid w) \min\set*{\frac{\iota}{n_t(x,y)}, 1} \\ &\leq \sum_{u,v,w}\sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s})P(w\mid u,v)q^{\pi}(x,y\mid w) \frac{\iota}{n_t(x,y)}\\ &\qquad + \sum_{u,v,w}\sum_{x,y,z} q^{\pi}(u,v\mid \widetilde{s})\frac{\iota}{n_t(u,v)}q^{\pi}(x,y\mid w) \\ &\leq H\sum_{x,y,z}q^{\pi}(x,y \mid \widetilde{s})\frac{\iota}{n_t(x,y)} + S\sum_{u,v,w}q^{\pi}(u,v \mid \widetilde{s})\frac{\iota}{n_t(u,v)} \\ &\lesssim S^2\sum_{u,v}\frac{q^{\pi}(u,v \mid \widetilde{s})\iota}{n_t(u,v)}, \label{eq:conditional95term3} \end{align}\tag{28}\] where the third line follows from \(\sum_{(u,v,w) \in W_h} q^{\pi}(u,v \mid \widetilde{s}) P(w \mid u,v) q^{\pi}(x,y \mid w) \leq q^{\pi}(x,y \mid \widetilde{s})\) and \(\sum_{(x,y,z) \in W_h} q^{\pi}(x,y \mid w) \leq \relax{\mathcal{S}_{h+1}}\), and the last line uses \(\sum_{(u,v,w)\in W_h} q^{P,\pi}(u,v \mid \widetilde{s}) \le |\mathcal{S}_{h+1}| \sum_{(u,v)\in \mathcal{S}_h \times \mathcal{A}} q^{\pi}(u,v \mid \widetilde{s})\).
Similarly, we have \[\begin{align} \mathrm{\boldsymbol{term}}_4 \leq S\sum_{u,v,w}q^{\pi}(u,v\mid \widetilde{s})\frac{\iota}{n_t(u,v)} \leq S^2\sum_{u,v}\frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)}. \label{eq:conditional95term4} \end{align}\tag{29}\] Combining with 26 27 28 29 , we obtain \[\begin{align} \sum_s \relax*{ q^{\pi}(s\mid \widetilde{s}) - q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi}(s\mid \widetilde{s})} &\lesssim \sum_s \sum_{u,v,w} q^{\pi}(u,v\mid \widetilde{s}) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi}(s\mid w) + S^2\sum_{u,v}\frac{q^{\pi}(u,v\mid \widetilde{s})\iota}{n_t(u,v)}. \label{eq:cond95occup95diff} \end{align}\tag{30}\]
We now apply 30 . Then \[\begin{align} &\sum_{t=1}^T \sum_{\widetilde{s}}q^{\pi_t}(\widetilde{s})\sum_s\relax*{ q^{\pi_t}(s\mid \widetilde{s}) - q^{\widetilde{P}^{s,\widetilde{s}}_t,\pi_t}(s\mid \widetilde{s})}g_t(s) \\ &\lesssim \sum_{t=1}^T \sum_{\widetilde{s}}q^{\pi_t}(\widetilde{s}) \prn*{ \sum_s\sum_{u,v,w} q^{\pi_t}(u,v\mid \widetilde{s}) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi_t}(s\mid w) + S^2\sum_{u,v}\frac{q^{\pi_t}(u,v\mid \widetilde{s})\iota}{n_t(u,v)} } g_t(s) \\ &\leq H\sum_{t=1}^T\prn*{ \sum_s\sum_{u,v,w} q^{\pi_t} (u,v)\sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi_t}(s\mid w) + S^2\sum_{u,v}\frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)} } g_t(s) \\ &\leq H\sum_{t=1}^T \prn*{\sum_s\sum_{u,v,w} q^{\pi_t}(u,v) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi_t}(s\mid w)}g_t(s) + HS^2G \sum_{t=1}^T \sum_{u,v} \frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}. \label{eq:conditional95decomp} \end{align}\tag{31}\] We first bound the first term in 31 . Fix \(h\). Then \[\begin{align} &H\sum_{t=1}^T \sum_s \prn*{\sum_{(u,v,w) \in W_h} q^{\pi_t}(u,v) \sqrt{\frac{P(w\mid u,v)\iota}{n_t(u,v)}}q^{\pi_t}(s\mid w)}g_t(s) \\ &\leq H\sum_{t=1}^T \sum_s \prn*{ \sum_{(u,v,w) \in \mathcal{S}_h\times\mathcal{A}\times\mathcal{S}_{h+1}} q^{\pi_t}(u,v)\prn*{\alpha P(w\mid u,v) g_t(s)^2 + \frac{\iota}{\alpha n_t(u,v)}}q^{\pi_t}(s\mid w)} \\ &= \alpha H \sum_{t=1}^T\sum_s\sum_{(u,v,w) \in \mathcal{S}_h\times\mathcal{A}\times\mathcal{S}_{h+1}} q^{\pi_t}(u,v) P(w\mid u,v) q^{\pi_t}(s\mid w)g_t(s)^2\\ &\qquad + \frac{H}{\alpha}\sum_{t=1}^T \sum_s \sum_{(u,v,w) \in \mathcal{S}_h\times\mathcal{A}\times\mathcal{S}_{h+1}} \frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)}q^{\pi_t}(s\mid w) \\ &\leq \alpha H \sum_{t=1}^T\sum_s q^{\pi_t}(s)g_t(s)^2 + \frac{H^2\relax{\mathcal{S}_{h+1}}}{\alpha}\sum_{t=1}^T \sum_{(u,v) \in \mathcal{S}_h \times \mathcal{A}} \frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)} \\ &\lesssim \alpha H \sum_{t=1}^T\sum_s q^{\pi_t}(s)g_t(s)^2 + \frac{H^2\relax{\mathcal{S}_{h+1}}\relax{\mathcal{S}_h}A\iota^2}{\alpha}and the assumption that \mathcal{E} holds}\\ &\lesssim \sqrt{H^3 \relax{\mathcal{S}_h}\relax{\mathcal{S}_{h+1}} A \iota^2 \sum_{t=1}^T \sum_s q^{P, \pi_t}(s)g_t(s)^2}\\ &\leq (\relax{\mathcal{S}_h} + \relax{\mathcal{S}_{h+1}})\sqrt{H^3 A \iota^2 \sum_{t=1}^T \sum_s q^{P, \pi_t}(s)g_t(s)^2}. \\ \end{align}\] Summing over \(h\) gives \[\begin{align} \sum_{h=0}^{H-1}(\relax{\mathcal{S}_h} + \relax{\mathcal{S}_{h+1}})\sqrt{H^3 A \iota^2 \sum_{t=1}^T \sum_s q^{\pi_t}(s)g_t(s)^2} \lesssim \sqrt{H^3 S^2A \iota^2 \sum_{t=1}^T \sum_s q^{\pi_t}(s)g_t(s)^2}. \label{eq:conditional95decomp1} \end{align}\tag{32}\] It remains to bound the second term in 31 . By 7 and the event \(\mathcal{E}\), \[\begin{align} HS^2G \sum_{t=1}^T \sum_{u,v} \frac{q^{\pi_t}(u,v)\iota}{n_t(u,v)} &\lesssim HS^3AG\iota^2. \label{eq:conditional95decomp2} \end{align}\tag{33}\] Combining 32 33 , we complete the proof. ◻
Lemma 17. It holds that \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a} q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s, a;\prn*{\ell_t - m_t}^2)} \lesssim H\mathbb{E}\brk*{\sum_{t=1}^TV^{\pi_t}(s_0;\prn*{\ell_t - m_t}^2)} + HS^3A\iota^2. \end{align}\]
Proof. On the event\(\bar{\mathcal{E}}\), \(\sum_{t=1}^T\sum_{s,a} q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P},\pi_t}(s,a;(\ell_t-m_t)^2)=\mathcal{O}(H^2T)\) gives a contribution \(\mathcal{O}(H^2T\delta)\), which is absorbed by taking \(\delta=\mathcal{O}(1/T^2)\).
Hence, it suffices to prove the bound on \(\mathcal{E}\). For each \(t\), let \(\overline{P}^Q_t\in\mathcal{P}_t\) be such that \[\begin{align} Q^{\overline{P}^Q_t,\pi_t}(s,a;\prn*{\ell_t-m_t}^2) = \max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn*{\ell_t-m_t}^2) \end{align}\] for all \((s,a)\). Then, \[\begin{align} &\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a) Q^{\overline{P}^Q_t, \pi_t}(s, a;\prn*{\ell_t - m_t}^2)\\ &\leq \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a) Q^{\pi_t}(s, a;\prn*{\ell_t - m_t}^2\\ &\qquad + \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a)\prn*{Q^{\overline{P}^Q_t, \pi_t}(s, a;\prn*{\ell_t - m_t}^2) -Q^{\pi_t}(s, a;\prn*{\ell_t - m_t}^2)}. \label{eq:qmax95decomp} \end{align}\tag{34}\] For the first term, we have \[\begin{align} &\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s,a)Q^{\pi_t}(s, a;\prn*{\ell_t - m_t}^2)\\ &= \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s,a) \sum_{s',a'}q^{\pi_t}(s',a'\mid s, a)\prn*{\ell_t(s',a') - m_t(s',a')}^2 \\ &\leq H\sum_{t=1}^T \sum_{s',a'}q^{\pi_t}(s',a')\prn*{\ell_t(s',a') - m_t(s',a')}^2 \\ &= H \sum_{t=1}^TV^{\pi_t}(s_0; \prn*{\ell_t - m_t}^2) \label{eq:qmax95first} \end{align}\tag{35}\] For the second term, we obtain \[\begin{align} &\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a)\prn*{Q^{\overline{P}^Q_t, \pi_t}(s, a;\prn*{\ell_t - m_t}^2) -Q^{\pi_t}(s, a;\prn*{\ell_t - m_t}^2)} \\ &= \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a)\sum_{s'}\prn*{q^{\overline{P}^Q_t, \pi_t}(s'\mid s,a) - q^{\pi_t}(s'\mid s,a)}\sum_{a'}\pi_t(a'\mid s')\prn*{\ell_t(s',a') - m_t(s',a')}^2 \\ &= \sum_{t=1}^T \sum_{s} q^{\pi_t}(s)\sum_{s'}\prn*{q^{\overline{P}^Q_t, \pi_t}(s'\mid s) - q^{\pi_t}(s'\mid s)}\sum_{a'}\pi_t(a'\mid s')\prn*{\ell_t(s',a') - m_t(s',a')}^2 \\ &\leq \sqrt{H^3S^2A \iota^2 \sum_{t=1}^T \sum_{s',a'} q^{\pi_t}(s',a')\prn*{\ell_t(s',a') - m_t(s',a')}^2} + HS^3A\iota^2 }\\ &= \sqrt{H^3S^2A \iota^2 \sum_{t=1}^T V^{\pi_t}(s_0; \prn*{\ell_t - m_t}^2)} + HS^3A\iota^2.\label{eq:qmax95second} \end{align}\tag{36}\]
Combining 34 35 36 , we obtain \[\begin{align} &\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a) Q^{\overline{P}^Q_t, \pi_t}(s, a;\prn*{\ell_t - m_t}^2)\\ &\leq H\sum_{t=1}^TV^{\pi_t}(s_0;\prn*{\ell_t - m_t}^2) + \sqrt{H^3S^2A \iota^2 \sum_{t=1}^T V^{\pi_t}(s_0; \prn*{\ell_t - m_t}^2)} + HS^3A\iota^2\\ &\lesssim H\sum_{t=1}^TV^{\pi_t}(s_0;\prn*{\ell_t - m_t}^2) + HS^3A\iota^2, \end{align}\] where the last line follows from the AM–GM inequality since \(ax + \sqrt{bx} \leq ax + ax + \frac{b}{2a}\) for \(a,b,x \geq 0\). Taking expectations completes the proof. ◻
Lemma 18 ([2], Lemma E.9). It holds that \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\mathbb{I}_t(s,a)(L_{t,h(s)} - M_{t,h(s)})^2} \leq H^2\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\mathbb{I}_t(s,a)(\ell_t(s,a) - m_t(s,a))^2}. \end{align}\]
Lemma 19 ([2], Lemma F.12). Suppose \(m_t\) is defined in 4 with some constant \(\xi \in (0, \frac{1}{2})\). Then, for any comparator policy \(\pi\), it holds that \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^TV^{\pi_t}(s_0;\prn*{\ell_t-m_t}^2)} &= \mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\mathbb{I}_t(s,a)(\ell_t(s,a) - m_t(s,a))^2}\\ &\lesssim \min\set*{L(\pi) + \text{\rm Reg}_T(\pi), HT - L(\pi) - \text{\rm Reg}_T(\pi), Q_{\infty}, V_1} + SA. \end{align}\]
Lemma 20. For any sequence of policies \(\{\pi_t\}_{t=1}^T\) and any prediction sequence \(\{m_t\}_{t=1}^T\), it holds that \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(m) &\leq 2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell) + 2H\sum_{t=1}^T V^{\pi_t}(s_0;(\ell_t-m_t)^2). \label{eq:qtrans95m95to95ell} \end{align}\qquad{(2)}\]
Proof. Fix any sequence \(\phi_t\) such that \(\relax*{\phi_t(s,a)}=m_t(s,a)\) for all \(t,s,a\). Define \[\begin{align} \phi_t^\ell(s,a)\mathrel{\vcenter{:}}= \mathrm{sgn}(\phi_t(s,a))\ell_t(s,a),\qquad \phi_t^\Delta(s,a)\mathrel{\vcenter{:}}= \phi_t(s,a)-\phi_t^\ell(s,a). \end{align}\] Then \[\begin{align} \relax*{\phi_t^\ell(s,a)}=\ell_t(s,a),\qquad \relax*{\phi_t^\Delta(s,a)}=\relax*{\ell_t(s,a)-m_t(s,a)}. \end{align}\] Moreover, for any state \(s\), \[\begin{align} V^{\pi_t}(s;\phi_t)=V^{\pi_t}(s;\phi_t^\ell)+V^{\pi_t}(s;\phi_t^\Delta). \end{align}\] By using \(\mathsf{Var}(X+Y)\leq 2\mathsf{Var}(X)+2\mathsf{Var}(Y)\), we have \[\begin{align} &\sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t)}\\ &\leq 2\sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t^\ell)}+2\sum_{t=1}^T\sum_{s,a}q^{\pi_t}(s,a)\mathsf{Var}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\pi_t}(s';\phi_t^\Delta)}\\ &\leq 2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)+2Q_{\mathrm{trans}}^{\pi_{1:T}}(\relax{\ell-m}). \end{align}\] Since this holds for any sequence \(\phi_t\) with \(\relax*{\phi_t(s,a)}=m_t(s,a)\), taking the supremum yields \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(m)\leq 2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)+2Q_{\mathrm{trans}}^{\pi_{1:T}}(\relax{\ell-m}). \end{align}\] Finally, 14 with \(g_t=\relax{\ell_t-m_t}\) gives \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(\relax{\ell-m})\leq H\sum_{t=1}^T V^{\pi_t}(s_0;(\ell_t-m_t)^2). \end{align}\] Therefore, \[\begin{align} Q_{\mathrm{trans}}^{\pi_{1:T}}(m)\leq 2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)+2H\sum_{t=1}^T V^{\pi_t}(s_0;(\ell_t-m_t)^2). \end{align}\] ◻
Lemma 21 ([1]). For any two policies \(\pi_1\) and \(\pi_2\), it holds that \[\begin{align} \sum_{s,a}\relax*{q^{\pi_1}(s,a) - q^{\pi_2}(s,a)} \leq H \sum_{s,a}q^{\pi_1}(s)\relax*{\pi_1(a \mid s)- \pi_2(a\mid s)}. \end{align}\]
Lemma 22. For any policy \(\pi\) and layer \(h\), it holds that \[\begin{align} \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi}(s,a) - q^{{\pi^\star}}(s,a)} \leq 2\sum_{h'=0}^{h } \sum_{s \in \mathcal{S}_{h'}}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s,a). \end{align}\]
Proof. Since \({\pi^\star}\) is deterministic, we have \[\begin{align} &\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi}(s,a) - q^{{\pi^\star}}(s,a)} \\ &= \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi}(s)\pi(a\mid s) - q^{{\pi^\star}}(s)\mathbb{1}\brk*{{a = {\pi^\star}(s)}}} \\ &\leq \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi}(s)\pi(a\mid s) - q^{\pi}(s)\mathbb{1}\brk*{{a = {\pi^\star}(s)}}} + \sum_{s\in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)} \\ &= \sum_{s \in \mathcal{S}_h}\sum_{a \neq {\pi^\star}(s)}q^{\pi}(s,a) + \sum_{s\in \mathcal{S}_h}q^{\pi}(s)(1 - \pi({\pi^\star}(s)\mid s)) + \sum_{s\in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)} \\ &= 2\sum_{s \in \mathcal{S}_h}\sum_{a \neq {\pi^\star}(s)}q^{\pi}(s,a) + \sum_{s\in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)}. \label{eq:occupancy95action95l1} \end{align}\tag{37}\] Next, by the triangle inequality, \[\begin{align} &\sum_{s'\in \mathcal{S}_{h+1}}\relax*{q^{\pi}(s') - q^{{\pi^\star}}(s')} \\ &= \sum_{s'\in \mathcal{S}_{h+1}}\relax*{\sum_{s \in \mathcal{S}_h}\sum_{a}\prn*{q^{\pi}(s)\pi(a\mid s) P(s'\mid s, a) - q^{{\pi^\star}}(s)\mathbb{1}\brk*{a = {\pi^\star}(s)}P(s'\mid s, {\pi^\star}(s))}} \\ &\leq\sum_{s'\in \mathcal{S}_{h+1}}\relax*{\sum_{s \in \mathcal{S}_h}\prn*{q^{\pi}(s)P(s'\mid s, {\pi^\star}(s)) - q^{{\pi^\star}}(s)P(s'\mid s, {\pi^\star}(s))}} \\ &\qquad + \sum_{s'\in \mathcal{S}_{h+1}}\relax*{\sum_{s \in \mathcal{S}_h}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s)\pi(a\mid s) P(s'\mid s, a) + \sum_{s \in \mathcal{S}_h}q^{\pi}(s)(\pi({\pi^\star}(s)\mid s) - 1)P(s'\mid s, {\pi^\star}(s))} \\ &\leq \sum_{s'\in \mathcal{S}_{h+1}}\sum_{s \in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)}P(s'\mid s, {\pi^\star}(s)) \\ &\qquad + \sum_{s'\in \mathcal{S}_{h+1}}\sum_{s \in \mathcal{S}_h}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s,a) \relax*{P(s'\mid s, a) -P(s'\mid s, {\pi^\star}(s))} \\ &\leq \sum_{s \in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)} + 2\sum_{s \in \mathcal{S}_h}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s,a),\label{eq:state95l195recursion} \end{align}\tag{38}\] where the last inequality follows from \(\sum_{s'\in \mathcal{S}_{h+1}}\relax*{P(s'\mid s, a) -P(s'\mid s, {\pi^\star}(s))} \leq 2\) for all state-action pairs \((s,a)\).
Applying 38 recursively and using \(q^\pi(s_0)=q^{{\pi^\star}}(s_0)\), we obtain \[\begin{align} \sum_{s \in \mathcal{S}_h}\relax*{q^{\pi}(s) - q^{{\pi^\star}}(s)} \leq 2\sum_{h'=0}^{h - 1} \sum_{s \in \mathcal{S}_{h'}}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s,a). \label{eq:state95l195bound} \end{align}\tag{39}\] Combining 37 39 yields \[\begin{align} \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi}(s,a) - q^{{\pi^\star}}(s,a)} \leq 2\sum_{h'=0}^{h } \sum_{s \in \mathcal{S}_{h'}}\sum_{a\neq {\pi^\star}(s)}q^{\pi}(s,a). \end{align}\] This completes the proof. ◻
Lemma 23 ([7, Sec. 2.1]). Under the stochastic regime with adversarial corruption, for any sequence of policies \(\{\pi_t\}_{t=1}^T\), the regret satisfies the following \((\Delta,2\mathcal{C},T)\) self-bounding constraint: \[\text{\rm Reg}_T(\mathring{\pi})\geq \mathbb{E}\brk*{\sum_{t=1}^T \sum_{s}\sum_{a\neq {\pi^\star}(s)} q^{\pi_t}(s,a)\Delta(s,a)} - 2 \mathcal{C}. \label{eq:adversarial95regime95with95self-bounding95constraints}\qquad{(3)}\]
Lemma 24. Let \(G:S\times A\to\mathbb{R}_{\geq 0}\) be any nonnegative function. Under the stochastic regime with adversarial corruption, for any \(\alpha>0\), it holds that \[\begin{align} \sum_{s}\sum_{a\neq\pi^\star(s)}G(s,a)\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s,a)}} &\leq \alpha(\text{\rm Reg}_T(\mathring{\pi}) + 2\mathcal{C}) + \sum_{s}\sum_{a\neq\pi^\star(s)} \frac{G(s,a)^2}{4\alpha\Delta(s,a)}. \label{eq:selfbound95pit} \end{align}\qquad{(4)}\]
Proof. For each \((s,a)\) with \(a\neq \pi^\star(s)\), the AM–GM inequality and 23 imply \[\begin{align} &\sum_{s}\sum_{a\neq\pi^\star(s)}G(s,a)\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s,a)}} \\ &\leq \alpha\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s}\sum_{a\neq\pi^\star(s)} q^{\pi_t}(s,a)\Delta(s,a)} + \sum_{s}\sum_{a\neq\pi^\star(s)} \frac{G(s,a)^2}{4\alpha\Delta(s,a)}\\ &\leq \alpha(\text{\rm Reg}_T(\mathring{\pi}) + 2\mathcal{C}) + \sum_{s}\sum_{a\neq\pi^\star(s)} \frac{G(s,a)^2}{4\alpha\Delta(s,a)}. } \end{align}\] ◻
Lemma 25. Let \(G(h)\) be any nonnegative function defined on layer \(h\). Under the stochastic regime with adversarial corruption, for any \(\alpha>0\), it holds that \[\begin{align} &\sum_{h = 0}^{H-1}\sqrt{G(h)\mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\mathring{\pi}}(s,a)}_{+}}}\\ &\qquad\leq \alpha(\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}) + \frac{\prn*{\sum_{h = 0}^{H-1}\sqrt{G(h)}}^2}{4\alpha\Delta_{\min}}. \end{align}\]
Proof. Applying 22, we obtain \[\begin{align} &\mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\mathring{\pi}}(s,a)}_{+}}\\ &= \mathbb{E}\brk*{\frac{1}{2}\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\relax*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}}\\ &\leq \mathbb{E}\brk*{\frac{1}{2}\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\prn*{\relax*{q^{\pi_t}(s,a)-q^{{\pi^\star}}(s,a)} + \relax*{q^{\mathring{\pi}}(s,a)-q^{{\pi^\star}}(s,a)}}}\\ &\leq \mathbb{E}\brk*{\sum_{t=1}^T\sum_{h'=0}^{h } \sum_{s \in \mathcal{S}_{h'}}\sum_{a\neq {\pi^\star}(s)}\prn*{q^{\pi_t}(s,a) + q^{\mathring{\pi}}(s,a)}}. \end{align}\] We next bound the comparator contribution. Using the uncorrupted mean loss function \(\mu\), we have \[\begin{align} &\mathbb{E}\brk*{\sum_{t=1}^T\sum_s\sum_{a\neq{\pi^\star}(s)}q^{\mathring{\pi}}(s,a)\Delta(s,a)}\\ &=\mathbb{E}\brk*{\sum_{t=1}^T\prn*{V^{\mathring{\pi}}(s_0;\mu)-V^{{\pi^\star}}(s_0;\mu)}}\\ &=\mathbb{E}\brk*{\sum_{t=1}^T\prn*{V^{\mathring{\pi}}(s_0;\mu)-V^{\mathring{\pi}}(s_0;\ell_t)}}+\mathbb{E}\brk*{\sum_{t=1}^T\prn*{V^{\mathring{\pi}}(s_0;\ell_t)-V^{{\pi^\star}}(s_0;\ell_t)}}\\ &\qquad+\mathbb{E}\brk*{\sum_{t=1}^T\prn*{V^{{\pi^\star}}(s_0;\ell_t)-V^{{\pi^\star}}(s_0;\mu)}}. \end{align}\] The middle term is nonpositive by the definition of \(\mathring{\pi}\). Moreover, since \(\mathbb{E}[\ell'_t]=\mu\), for any policy \(\pi\), \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T\relax*{V^\pi(s_0;\ell_t)-V^\pi(s_0;\mu)}} &\leq\mathbb{E}\brk*{\sum_{t=1}^T\relax*{V^\pi(s_0;\ell_t)-V^\pi(s_0;\ell'_t)}} \\ &\leq\mathbb{E}\brk*{\sum_{t=1}^T\sum_{h=0}^{H-1}\nrm*{\ell_t(h)-\ell'_t(h)}_\infty} =\mathcal{C}. \end{align}\] Therefore, \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T\sum_s\sum_{a\neq{\pi^\star}(s)}q^{\mathring{\pi}}(s,a)\Delta(s,a)} &\leq 2\mathcal{C}. \label{eq:pio95delta95self} \end{align}\tag{40}\] Combining 40 with 23, we get \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T\sum_s\sum_{a\neq{\pi^\star}(s)}\prn*{q^{\pi_t}(s,a)+q^{\mathring{\pi}}(s,a)}\Delta(s,a)} &\leq \text{\rm Reg}_T(\mathring{\pi})+4\mathcal{C}. \label{eq:self95bound95pi95t95pio} \end{align}\tag{41}\] Then, for any \(\alpha > 0\), we have \[\begin{align} &\sum_{h = 0}^{H-1}\sqrt{G(h)\mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\mathring{\pi}}(s,a)}_{+}}}\\ &\leq \sum_{h = 0}^{H-1}\sqrt{G(h)\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s}\sum_{a\neq {\pi^\star}(s)}\prn*{q^{\pi_t}(s,a) + q^{\mathring{\pi}}(s,a)}}}\\ &\leq \sum_{h = 0}^{H-1}\sqrt{G(h)}\sqrt{\frac{1}{\Delta_{\min}}\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s}\sum_{a\neq {\pi^\star}(s)}\prn*{q^{\pi_t}(s,a) + q^{\mathring{\pi}}(s,a)}\Delta(s,a)}}\\ &\leq \sum_{h = 0}^{H-1}\sqrt{G(h)}\sqrt{\frac{\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}}{\Delta_{\min}}}\\ &\leq \alpha(\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}) + \frac{\prn*{\sum_{h = 0}^{H-1}\sqrt{G(h)}}^2}{4\alpha\Delta_{\min}}, \end{align}\] where we use the AM–GM inequality. ◻
Lemma 26. Let \(G>0\). Under the stochastic regime with adversarial corruption, for any \(\alpha>0\), it holds that \[\begin{align} \sqrt{G \mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}_+}}&\leq \alpha\prn*{\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}} + \frac{HG}{2\alpha\Delta_{\min}}. \end{align}\]
Proof. Applying 21 to compare any policy \(\pi\) with \({\pi^\star}\), we get \[\begin{align} \sum_{t=1}^T\sum_{s,a}\relax*{q^{\pi}(s,a)-q^{{\pi^\star}}(s,a)} &\leq H \sum_{t=1}^T\sum_{s,a}q^{\pi}(s)\relax*{\pi(a \mid s)- {\pi^\star}(a\mid s)}\\ &\leq H \sum_{t=1}^T\sum_{s}\sum_{a \neq {\pi^\star}(s)}q^{\pi}(s)\pi(a \mid s)\\ &\qquad + H \sum_{t=1}^T\sum_{s}q^{\pi}(s)\prn*{1 - \pi({\pi^\star}(s) \mid s)}\\ &\leq 2H \sum_{t=1}^T\sum_{s}\sum_{a \neq {\pi^\star}(s)}q^{\pi}(s)\pi(a \mid s). \end{align}\] Using this bound with \(\pi=\pi_t\) and \(\pi=\mathring{\pi}\), for any \(\alpha>0\) we obtain \[\begin{align} &\sqrt{G\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}_+}}\\ &\leq \sqrt{G\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\relax*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}}}\\ &\leq \sqrt{G\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}\prn*{\relax*{q^{\pi_t}(s,a)-q^{{\pi^\star}}(s,a)} + \relax*{q^{\mathring{\pi}}(s,a)-q^{{\pi^\star}}(s,a)}}}}\\ &\leq \sqrt{2HG\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s}\sum_{a \neq {\pi^\star}(s)}\prn*{q^{\pi_t}(s,a) + q^{\mathring{\pi}}(s,a}}}\\ &\leq \sqrt{2HG\frac{1}{\Delta_{\min}}\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s}\sum_{a \neq {\pi^\star}(s)}\prn*{q^{\pi_t}(s,a) + q^{\mathring{\pi}}(s,a)}\Delta(s,a)}}\\ &\leq \sqrt{2HG\frac{\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}}{\Delta_{\min}}}and \ref{eq:pio95delta95self}}\\ &\leq \alpha(\text{\rm Reg}_T(\mathring{\pi}) + 4\mathcal{C}) + \frac{HG}{2\alpha\Delta_{\min}}, \label{eq:self95bounding95min} \end{align}\tag{42}\] where the last inequality follows from the AM–GM inequality. ◻
In this section, we provide a regret analysis of optimistic follow-the-regularized-leader (OFTRL) for the MDP setting.
Lemma 27 (OFTRL with policy optimization, [2], Lemma C.4). Suppose that a sequence of probability vectors \(p_1, \dots, p_T \in \triangle(\mathcal{A})\) is given by OFTRL in \[\begin{align} p_t = \mathop{\mathrm{arg\,min}}_{\Delta(\mathcal{A})}\set*{\inpr*{p, \sum_{\tau = 1}^{t - 1}\ell_\tau + m_t} + \psi_t(p)}, \qquad \psi_t(p) = \sum_a \frac{1}{\eta_t(a)}\log\prn*{\frac{1}{p(a)}} \end{align}\] with \(\eta_1(a)=\eta_1\) for all \(a\), and let losses \(\set{\ell_t}_{t=1}^T\), loss predictions \(\set{m_t}_{t=1}^{T+1}\) and \(\set{x_t}_{t=1}^T\) be such that \[\begin{align} \eta_t(a) p_t(a)(\ell_t(a) - m_t(a) + x_t) &\geq -\frac{1}{2} \label{eq:OFTRL95constraint95policy} \end{align}\qquad{(5)}\] for all \(t,a\). Then for any \(u\in\triangle(\mathcal{A})\), the OFTRL algorithm achieves \[\begin{align} \sum_{t=1}^T \inpr*{ p_t-u, \ell_t} &\leq \frac{A\log(AT^2)}{\eta_1} + \sum_{t=1}^T \sum_a \prn*{\frac{1}{\eta_{t+1}(a)} - \frac{1}{\eta_t(a)}}\log(AT^2)\\ &\qquad + \sum_{t=1}^T \sum_a \eta_t(a)p_t(a)^2(\ell_t(a)-m_t(a) + x_t)^2\\ &\qquad + \frac{1}{T^2}\sum_{t=1}^T \inpr*{ -u + \frac{1}{A}\mathbf{1}, \ell_t } + 2\nrm*{m_{T+1}}_{\infty}. \end{align}\]
In this section, we prove the best-of-both-worlds results for the full-information setting. We present the bound for the adversarial regime in 37 and the bound for the stochastic regime with adversarial corruption in 38. Together, these results establish 2.
Building on the policy optimization framework of [1], [6], we use the following key lemma to derive our regret bounds.
Lemma 28 (Restatement of 1). Suppose that \(b_t(s)\) is a nonnegative loss function. Suppose also that, for a comparator policy \(\pi\), there exists \(J^\pi\geq 0\) such that \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\sum_{t=1}^T\sum_a \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}}\\ &\leq J^{\pi} + \mathbb{E}\brk*{\sum_{t = 1}^T\sum_s q^{\pi}(s) b_t(s)} + \mathbb{E}\brk*{\frac{1}{H} \sum_{t = 1}^T\sum_s\sum_a q^{\pi}(s) \pi_t(a\mid s)B_t(s,a)} .\label{eq:key95lemma95ineq} \end{align}\qquad{(6)}\] Then, it holds that \[\begin{align} \text{\rm Reg}_T(\pi) \leq J^{\pi} + 3 \, \mathbb{E}\brk*{\sum_{t = 1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)} + \mathbb{E}\brk*{HT\mathbb{1}[P\notin\mathcal{P}_t, \exists t\in[T]]}, \end{align}\] where \(\overline{P}^B_t \in \mathcal{P}_t\) simultaneously attains the maxima in 5 for all \((s,a)\).
To show ?? , we choose \(b_t\) in 8 and decompose the LHS of ?? as \[\begin{align} &\sum_s q^{\pi}(s) \sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}\\ &= \sum_s q^{\pi}(s) \underbrace{\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{\widehat{Q}_t(s,a) - B_t(s,a)}}_{\textrm{\boldsymbol{reg}}^{\pi}(s)}\\ &\qquad + \sum_s q^{\pi}(s) \underbrace{\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s,a;\ell_t) - \widehat{Q}_t(s,a)}}_{\textrm{\boldsymbol{bias}}^{\pi}(s)}. \label{eq:decompose95full} \end{align}\tag{43}\]
Lemma 29 (full-information setting). The variables \(b_t(s)\) in 8 and \(B_t(s,a)\) in 5 satisfy \[\begin{align} B_t(s,a) \leq 2 \label{eq:bound95eta95pi95B95full} \end{align}\qquad{(7)}\] for all episodes \(t\) and state-action pairs \((s,a)\).
Proof. We first upper bound the dilated bonus-to-go \(B_t(s,a)\) by unrolling the dilated recursion. Since \((1+1/H)^H\le 3\), we obtain \[\begin{align} B_t(s,a) \leq 3\sum_{s'} q^{\overline{P}^B_t, \pi_t}(s'\mid s,a)\,b_t(s'). \label{eq:dilated95bonus95upper} \end{align}\tag{44}\]
By the definition of \(b_t\) in 8 and the learning-rate update 7 , \[\begin{align} b_t(s) &= 7\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T)\\ &\leq 7\sum_{a} \eta_t(s,a)\zeta_t(s,a)\\ &\leq 7\eta_1H^2, \end{align}\] where we used \(\sum_a \zeta_t(s,a)\le H^2\) and \(\eta_t(s,a)\le \eta_1\).
Using \(\eta_1 = \frac{1}{12H^3}\) and 44 , we get \[\begin{align} B_t(s,a) &\leq 3\sum_{s'} q^{\overline{P}^B_t, \pi_t}(s'\mid s,a)b_t(s') \leq 21\eta_1H^3 \leq 2.\label{eq:bound95B95full} \end{align}\tag{45}\] ◻
Lemma 30 ([1], Lemma G.3). Let \(\eta_1>0, \eta_2, \eta_3, \ldots\) be updated by \[\begin{align} \frac{1}{\eta_{t+1}} = \frac{1}{\eta_t} + \eta_t \phi_t\qquad \forall t\geq 1 \end{align}\] with \(0\leq \phi_t\leq \eta_t^{-2}\). Then, \[\begin{align} \frac{1}{\eta_{t+1}} \geq \frac{1}{2}\sqrt{\sum_{\tau=1}^{t+1} \phi_\tau}. \end{align}\]
Lemma 31 (full-information setting). Suppose that the learning rates are updated according to 7 . Then, it holds \[\begin{align} \eta_t(s,a) \leq \frac{2\sqrt{\log(T)}}{\sqrt{\sum_{\tau = 1}^t\zeta_\tau(s,a)} } \end{align}\] for all episodes \(t\) and state-action pairs \((s,a)\).
Proof. Fix \((s,a)\) and define \(\phi_t(s,a)=\frac{\zeta_t(s,a)}{\log(T)}\). Then the update rule of learning rates can be written as \[\begin{align} \frac{1}{\eta_{t+1}(s,a)} = \frac{1}{\eta_t(s,a)} + \eta_t(s,a) \phi_t(s,a). \end{align}\] To apply 30, we verify that \(\phi_t(s,a)\le \frac{1}{\eta_t(s,a)^2}\). Indeed, \[\begin{align} \phi_t(s,a)\eta_t(s,a)^2 = \frac{\eta_t(s,a)^2\zeta_t(s,a)}{\log(T)} \leq \frac{\eta_1^2H^2}{\log(T)} \leq \frac{H^2}{\log(T)}\cdot \frac{1}{144H^6} \leq 1, \end{align}\] which follows from \(\eta_1 = \frac{1}{12H^3}\).
Then, by 30, we have \[\begin{align} \eta_t(s,a) \leq \frac{2}{\sqrt{\sum_{\tau = 1}^t \phi_\tau }} \leq \frac{2\sqrt{\log(T)}}{\sqrt{\sum_{\tau = 1}^t\zeta_\tau(s,a)} }. \end{align}\] ◻
In this part, we upper bound the right-hand side of ?? .
Lemma 32 (full-information setting). For each state \(s\in\mathcal{S}\) and any comparator policy \(\pi\), it holds that \[\begin{align} \textrm{\boldsymbol{reg}}^{\pi}(s) &\leq O\prn*{H^3A\log(T)} + \sum_{t=1}^Tb_t(s) + \frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a). \end{align}\]
Proof. We will apply 27 with \(p_t = \pi_t(\cdot \mid s)\) and \(\ell_t = \widehat{Q}_t(s,a) - B_t(s,a)\) for each \(s \in \mathcal{S}\). To do so, in what follows, we will check the conditions of 27. Let \[\begin{align} x_t &= \inpr*{-\pi_t(\cdot \mid s) , \widehat{Q}_t(s,\cdot) - Q^{\underline{P}^m_t, \pi_t}(s, \cdot;m_t)}\\ &= -\sum_{a}\pi_t(a\mid s) \prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t)} \end{align}\] and verify that for all \((s,a)\), \(\eta_t(s,a)\pi_t(a\mid s)\prn*{\widehat{Q}_t(s,a)-B_t(s,a) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) + x_t}\geq -{1}/{2}\). Indeed, \[\begin{align} &\eta_t(s,a)\pi_t(a\mid s)\prn*{\widehat{Q}_t(s,a)-B_t(s,a)-Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) + x_t} \\ &\geq \eta_t(s,a)\pi_t(a\mid s) \prn*{-H-B_t(s,a)-H} \\ &\geq -4\eta_1H \geq -\frac{1}{2}, \end{align}\] where the bounds in the second lines use \(-H\leq \widehat{Q}_t(s,a)-Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) \leq H\), \(B_t(s,a)\leq 2\leq 2H\) from 29.
Thus, 27 implies that, for any comparator policy \(\pi\), \[\begin{align} & \textrm{\boldsymbol{reg}}^{\pi}(s)\\ &\leq \frac{A\log(AT^2)}{\eta_1} + \sum_{t=1}^T \sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(AT^2)\\ &\qquad + \sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a \mid s)^{2}\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) - B_t(s, a)+ x_t}^2 \\ &\qquad + \frac{1}{T^2}\sum_{t=1}^T \inpr*{ -\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}, \widehat{Q}_t(s, \cdot) - B_t(s, \cdot) } + 2\nrm*{Q^{\underline{P}^m_t, \pi_t}(s, \cdot;m_{T+1})}_{\infty} \\ &\leq \frac{3A\log(T)}{\eta_1} + \frac{6H}{T} + 2H + 3\sum_{t=1}^T\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T) ^m_t, \pi_t}(s, \cdot;m_{T+1})}_{\infty} \leq H} \\ &\qquad + 2\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) +x_t}^2\\ &\qquad + 2\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2} B_t(s, a)^2 \\ &\leq \mathcal{O}(H^3A\log(T)) + 3\sum_{t=1}^T\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T) \\ &\qquad + 2\underbrace{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t)+x_t}^2}_{\text{stability-term}}\\ &\qquad + \frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a). \label{eq:regterm95full95mid} \end{align}\tag{46}\] Here, the second inequality follows from \[\begin{align} &\frac{1}{T^2}\sum_{t=1}^T \inpr*{ -\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}, \widehat{Q}_t(s, \cdot) - B_t(s, \cdot) }\\ &\leq \frac{1}{T^2}\nrm*{-\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}}_1\nrm*{\sum_{t=1}^T\prn*{\widehat{Q}_t(s, \cdot) - B_t(s, \cdot)} }_\infty \\ &\leq \frac{2T\cdot 3H}{T^2} = \frac{6H}{T}, \end{align}\] where we used \(\nrm*{-\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}}_1\leq 2\), \(\relax*{\widehat{Q}_t(s,a)} \leq H\) and \(B_t(s,a) \leq 2\) from 29.
It remains to bound the . Define \[\begin{align} \mathcal{Q}(s,a) \coloneq Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t). \end{align}\] Since \[\begin{align} x_t = -\sum_b \pi_t(b \mid s)\mathcal{Q}(s,b), \end{align}\] we have \[\begin{align} \prn*{\mathcal{Q}(s,a) - \sum_{b}\pi_t(b \mid s) \mathcal{Q}(s,b)}^2 &= \prn*{(1 - \pi_t(a \mid s))\mathcal{Q}(s,a) - \sum_{b\neq a}\pi_t(b \mid s) \mathcal{Q}(s,b)}^2\\ &\leq 2(1 - \pi_t(a \mid s))^2\mathcal{Q}(s,a)^2 + 2\prn*{\sum_{b\neq a}\pi_t(b \mid s) \mathcal{Q}(s,b)}^2\\ &\leq 2(1 - \pi_t(a \mid s))^2\mathcal{Q}(s,a)^2\\ &\qquad + 2(1 - \pi_t(a \mid s))\sum_{b\neq a}\pi_t(b \mid s) \mathcal{Q}(s,b)^2, \end{align}\] where the last inequality uses the Cauchy–Schwarz inequality. Thus, we obtain \[\begin{align} \pi_t(a \mid s)^2\prn*{\mathcal{Q}(s,a) - \sum_{b}\pi(b \mid s) \mathcal{Q}(s,b)}^2 &\leq 2\pi_t(a \mid s)(1 - \pi_t(a \mid s))\sum_{b}\pi_t(b \mid s) \mathcal{Q}(s, b)^2\\ \end{align}\] Therefore, \[\begin{align} &\eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) + x_t}^2 \\ &\leq 2\eta_t(s,a)\pi_t(a\mid s)(1 - \pi_t(a \mid s))\sum_{b}\pi_t(b \mid s)\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)}^2\\ &\leq 2\eta_t(s,a)\zeta_t(s, a), \end{align}\] where the last equality follows from the definition of \(\zeta_t(s, a) = \pi_t(a\mid s)(1 - \pi_t(a \mid s))\sum_{b}\pi_t(b \mid s)\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)}^2\).
Then, is evaluated as \[\begin{align} \text{stability-term} &= \sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) + x_t}^2 \\ &\leq 2\sum_{t=1}^T \sum_a \eta_t(s,a)\zeta_t(s,a) \\ &\leq 2\sum_{t=1}^T \sum_a\prn*{\frac{1}{\eta_{t+1}(s,a)} - \frac{1}{\eta_t(s,a)}}\log(T), \end{align}\] where the last inequality follows from 7 .
Combining the above bound on the stability-term with 46 , we obtain, for any policy \(\pi\), \[\begin{align} \textrm{\boldsymbol{reg}}^{\pi}(s) &\leq \mathcal{O}\prn*{H^3A\log(T)} + 7\sum_{t=1}^T\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T) \\ &\qquad + \frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a)\\ &= \mathcal{O}\prn*{H^3A\log(T)} + \sum_{t=1}^Tb_t(s) + \frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a). \end{align}\] ◻
Lemma 33 (full-information setting). Suppose that the event \(\mathcal{E}\) holds. Then, for any comparator policy \(\pi\), it holds that \[\begin{align} \sum_s q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s) &\lesssim \sqrt{S^2A\iota^2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)} + HS^3A\iota^2. \label{eq:bound95bias951} \end{align}\qquad{(8)}\] Also, \[\begin{align} \sum_s q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s) &\lesssim \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}\\ &\qquad + HS^2A\iota^2. \label{eq:bound95bias952} \end{align}\qquad{(9)}\]
Proof. \[\begin{align} \sum_s q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s) &=\sum_s q^{\pi}(s) \sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s, a; \ell_t) - Q^{\underline{P}^\ell_t, \pi_t}(s, a; \ell_t)}\\ &= \sum_{t=1}^T \sum_{s,a} \prn*{q^{\pi_t}(s,a) - q^{\pi}(s,a)} z_t(s,a), \end{align}\] where, for each \(t\), 10 is applied with \[\begin{align} L(s,a)=Q^{\pi_t}(s, a; \ell_t) - Q^{\underline{P}^\ell_t, \pi_t}(s, a; \ell_t). \end{align}\] Thus, \[\begin{align} z_t(s,a) &= Q^{\pi_t}(s, a; \ell_t) - Q^{\underline{P}^\ell_t, \pi_t}(s, a; \ell_t) \\ &\quad - \mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\pi_t}(s', a'; \ell_t) - Q^{\underline{P}^\ell_t, \pi_t}(s', a'; \ell_t)}. \end{align}\] Expanding the definition of \(z_t(s,a)\), we obtain \[\begin{align} z_t(s,a) &= \mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\pi_t}(s', a'; \ell_t)} - \mathbb{E}_{s'\sim \underline{P}^\ell_t(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\underline{P}^\ell_t, \pi_t}(s', a'; \ell_t)}\\ &\qquad - \mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\pi_t}(s', a'; \ell_t) - Q^{\underline{P}^\ell_t, \pi_t}(s', a'; \ell_t)} \\ &= \mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\underline{P}^\ell_t, \pi_t}(s', a'; \ell_t)} - \mathbb{E}_{s'\sim \underline{P}^\ell_t(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{Q^{\underline{P}^\ell_t, \pi_t}(s', a'; \ell_t)} \\ &= \mathbb{E}_{s'\sim P(\cdot\mid s,a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)} - \mathbb{E}_{s'\sim \underline{P}^\ell_t(\cdot\mid s,a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}. \end{align}\] By the definition of \(\underline{P}^\ell_t\), which minimizes \(Q(s,a;\ell_t)\) for all \((s,a)\), we have \(z_t(s,a) \geq 0\).
First, we show ?? . By the fact that \(z_t(s,a) \geq 0\), \[\begin{align} \sum_{t=1}^T \sum_{s,a} \prn*{q^{\pi_t}(s,a) - q^{\pi}(s,a)} z_t(s,a) &\leq \sum_{t=1}^T V^{\pi_t}(s_0; z_t)\\ &\leq \sum_{t=1}^T \prn*{V^{\pi_t}(s_0; \ell_t) - V^{\underline{P}^\ell_t, \pi_t}(s_0; \ell_t)} }\\ &= \sum_{t=1}^T \sum_{s} \prn*{q^{\pi_t}(s) - q^{\underline{P}^\ell_t, \pi_t}(s)}\sum_a\pi_t(a \mid s) \ell_t(s,a). \label{eq:bound95bias95195mid} \end{align}\tag{47}\]
Applying 13 to 47 , we obtain \[\begin{align} \sum_{t=1}^T \sum_{s,a} \prn*{q^{\pi_t}(s,a) - q^{\pi}(s,a)} z_t(s,a) &\lesssim \sqrt{S^2A\iota^2Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)} + HS^3A\iota^2. \end{align}\] This proves ?? .
Next, we show ?? . By 12, \[\begin{align} z_t(s,a) &= \sum_{s'}\prn*{P(s' \mid s, a) - \underline{P}^\ell_t(s' \mid s,a)}V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)\\ &\lesssim \sqrt{\frac{\relax*{S_{h(s) + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}} +\frac{\relax{S_{h(s)+1}}H\iota}{n_t(s,a)}. \end{align}\] Using \(z_t(s,a)\geq 0\), we further have \[\begin{align} &\sum_{t=1}^T \sum_{s,a} \prn*{q^{\pi_t}(s,a) - q^{\pi}(s,a)} z_t(s,a)\\ &\leq \sum_{t=1}^T \sum_{s,a} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} z_t(s,a)\\ &\lesssim \sum_{t=1}^T \sum_{s,a} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} \prn*{\sqrt{\frac{\relax*{S_{h(s) + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}} +\frac{\relax{S_{h(s)+1}}H\iota}{n_t(s,a)}}\\ &\leq \sum_{h = 0}^{H-1}\sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} \sqrt{\frac{\relax*{S_{h + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}}\\ &\qquad + \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s,a) \frac{\relax{S_{h(s)+1}}H\iota}{n_t(s,a)}. \label{eq:bound95bias95295mid} \end{align}\tag{48}\]
Here, by the Cauchy–Schwarz inequality, the first term in 48 is bounded as \[\begin{align} &\sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} \sqrt{\frac{\relax*{S_{h + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}}\\ &\leq \sqrt{\sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}\\ &\qquad \times \sqrt{\sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} \frac{\relax*{S_{h + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}}. \end{align}\] The term inside the second square root is further bounded as \[\begin{align} &\sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+} \frac{\relax*{S_{h + 1}}\mathsf{Var}_{s' \sim P(\cdot \mid s, a)}\brk*{V^{\underline{P}^\ell_t, \pi_t}(s'; \ell_t)}\iota}{n_t(s,a)}\\ &\leq H^2\relax*{S_{h + 1}} \sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} \frac{q^{\pi_t}(s,a) \iota}{n_t(s,a)} \\ &\lesssim H^2 \relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2. } \end{align}\] The second term in 48 is bounded as \[\begin{align} \sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s,a) \frac{\relax{S_{h(s)+1}}H\iota}{n_t(s,a)} &\leq \sum_{h = 0}^{H - 1}\relax*{S_{h + 1}} \sum_{t=1}^T \sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}} q^{\pi_t}(s,a) \frac{H\iota}{n_t(s,a)} \\ &\lesssim H\sum_{h = 0}^{H - 1} \relax*{S_{h + 1}} \relax*{S_{h}}A\iota^2 \\ &\leq HS^2A\iota^2. \end{align}\] Therefore, \[\begin{align} &\sum_{t=1}^T \sum_{s,a} \prn*{q^{\pi_t}(s,a) - q^{\pi}(s,a)} z_t(s,a) \\ &\lesssim \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}} + HS^2A\iota^2\\ \end{align}\] This proves ?? , and hence completes the proof. ◻
Lemma 34 (full-information setting). Suppose that the event \(\mathcal{E}\) holds. Then, it holds that \[\begin{align} &\sum_{t=1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)\lesssim \log(T)\sum_{s,a}\sqrt{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)} + H^{\frac{3}{2}}S^{\frac{5}{2}}A^{\frac{3}{2}}\log(T)\iota. \end{align}\]
Proof. By the definition of \(b_t\) and the \(\eta_t\), we have \[\begin{align} &\sum_{t=1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)\\ &= \sum_{t=1}^T\sum_s q^{\overline{P}^B_t, \pi_t}(s)b_t(s)\\ &\lesssim \sum_{t=1}^T\sum_{s,a} q^{\overline{P}^B_t,\pi_t}(s) \prn*{\frac{1}{\eta_{t+1}(s,a)}-\frac{1}{\eta_t(s,a)}}\log(T) \\ &\lesssim \sum_{t=1}^T\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \eta_t(s,a)\zeta_t(s,a)\\ &\lesssim \sqrt{\log(T)}\sum_{t=1}^T\sum_{s,a} \frac{q^{\overline{P}^B_t, \pi_t}(s)\sqrt{\zeta_t(s,a)}\times \sqrt{\zeta_t(s,a)}}{\sqrt{\sum_{\tau=1}^t\zeta_\tau(s,a) }} }\\ &\leq \sqrt{\log(T)}\sum_{s,a}\sqrt{\sum_{t=1}^T\frac{\zeta_t(s,a)}{ \sum_{\tau=1}^t \zeta_\tau(s,a)}}\sqrt{\sum_{t = 1}^T q^{\overline{P}^B_t, \pi_t}(s)^2\zeta_t(s,a)}\\ &\lesssim \log(T)\sum_{s,a}\sqrt{\sum_{t=1}^T q^{\overline{P}^B_t, \pi_t}(s)\zeta_t(s,a)}, \label{eq:res95qBzeta} \end{align}\tag{49}\] where the last inequality uses \[\begin{align} \sqrt{\sum_{t=1}^T\frac{\zeta_t(s,a)}{ \sum_{\tau=1}^t \zeta_\tau(s,a)}} \leq \sqrt{1 + \log\prn*{\sum_{t=1}^T\zeta_t(s,a)}} \lesssim \sqrt{\log(T)} \end{align}\] together with \(q^{\overline{P}^B_t, \pi_t}(s)^2 \leq q^{\overline{P}^B_t, \pi_t}(s)\).
We further bound \(\sum_{t=1}^T q^{\overline{P}^B_t, \pi_t}(s)\zeta_t(s,a)\) for each fixed \((s,a)\). We decompose it as \[\begin{align} \sum_{t=1}^T q^{\overline{P}^B_t, \pi_t}(s) \zeta_t(s,a) &= \sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a) + \sum_{t=1}^T \prn*{q^{\overline{P}^B_t, \pi_t}(s) - q^{\pi_t}(s)} \zeta_t(s,a). \label{eq:regret95overB} \end{align}\tag{50}\] To bound the second term, define \[\begin{align} g^{s,a}_t(s') \coloneq \mathbb{1}\brk*{s' = s}\zeta_t(s,a) \leq H^2. \end{align}\] Applying 15 with this choice of \(g_t^{s,a}\) yields \[\begin{align} \sum_{t=1}^T \prn*{q^{\overline{P}^B_t, \pi_t}(s) - q^{\pi_t}(s)} \zeta_t(s,a) &= \sum_{t=1}^T \sum_{s'} \prn*{q^{\overline{P}^B_t, \pi_t}(s') - q^{\pi_t}(s')} g^{s,a}_t(s')\\ &\lesssim \sqrt{HS^2A \iota^2\sum_{t=1}^T \sum_{s'} q^{\pi_t}(s')\prn*{g^{s,a}_t(s')}^2} + H^3S^3A\iota^2\\ &= \sqrt{HS^2A \iota^2 \sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)^2} + H^3S^3A\iota^2\\ &\leq \sqrt{H^3S^2A \iota^2 \sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)} + H^3S^3A\iota^2. \end{align}\] Combining this with 50 , we obtain \[\begin{align} \sum_{t=1}^T q^{\overline{P}^B_t, \pi_t}(s) \zeta_t(s,a) &\lesssim \sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a) + \sqrt{H^3S^2A \iota^2 \sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)} + H^3S^3A\iota^2 \\ &\lesssim \sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a) + H^3S^3A\iota^2, \label{eq:qB95to95q} \end{align}\tag{51}\] where the second line follows from the AM–GM inequality since \(ax + \sqrt{bx} \leq ax + ax + \frac{b}{2a}\) for \(a,b,x \geq 0\).
By 49 51 , we obtain \[\begin{align} \sum_{t=1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t) &\lesssim \log(T)\sum_{s,a}\sqrt{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a) + H^3S^3A\iota^2}\\ &\leq \log(T)\sum_{s,a}\sqrt{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)} + H^{\frac{3}{2}}S^{\frac{5}{2}}A^{\frac{3}{2}}\log(T)\iota. \end{align}\] This completes the proof. ◻
Lemma 35 (full-information setting). For any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim HS^3A^{\frac{3}{2}}\iota^2 + \sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)}}\\ &\qquad + \min\set[\Bigg]{\sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}},\\ &\qquad\qquad\qquad \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}}}. \end{align}\]
Proof. By the definition of the regret decomposition in 43 , \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}}\\ &= \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{reg}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)}\\ &= \mathcal{O}\prn*{H^4 A\log(T)} + \mathbb{E}\brk*{\sum_{t = 1}^T\sum_s q^{\pi}(s) b_t(s)} + \mathbb{E}\brk*{\frac{1}{H} \sum_{t = 1}^T\sum_s q^{\pi}(s)\sum_a \pi_t(a\mid s)B_t(s,a)}\\ &\qquad + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)}, \end{align}\] where we used 32. Combining this with 28 and \(\delta=\mathcal{O}(1/T^2)\), for any comparator policy \(\pi\), we obtain \[\begin{align} \text{\rm Reg}_T (\pi) &\leq \mathcal{O}\prn*{H^4 A\log(T)} + 3\mathbb{E}\brk*{\sum_{t = 1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)}. \end{align}\] Here, by 34 and Jensen’s inequality, with \(\delta=1/T^2\), we obtain \[\begin{align} \mathbb{E}\brk*{\sum_{t=1}^T V^{\overline{P}^B_t,\pi_t}(s_0;b_t)} &\lesssim \log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)}} + H^{\frac{3}{2}}S^{\frac{5}{2}}A^{\frac{3}{2}}\log(T)\iota\\ &\qquad + \mathcal{O}(\delta)\times \mathcal{O}(HT), \end{align}\] where we used the trivial bound \(\sum_{t=1}^T V^{\overline{P}^B_t,\pi_t}(s_0;b_t)\leq \mathcal{O}(HT)\) on \(\bar{\mathcal{E}}\) from 29.
Then, using \(\delta = \frac{1}{T^2}\), we have \[\begin{align} \text{\rm Reg}_T (\pi)&\lesssim H^{\frac{3}{2}}S^{\frac{5}{2}}A^{\frac{3}{2}}\log(T)\iota + \log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)}} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)}, \label{eq:regret95with95qb} \end{align}\tag{52}\]
By 52 33 and Jensen’s inequality, together with the trivial bound \(\mathcal{O}(H^2T)\) on \(\bar{\mathcal{E}}\) and the choice \(\delta=\mathcal{O}(1/T^2)\), we obtain, for any policy \(\pi\), \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim H^{\frac{3}{2}}S^{\frac{5}{2}}A^{\frac{3}{2}}\log(T)\iota + \log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s)\zeta_t(s,a)}}\\ &\qquad + \min\set[\Bigg]{\sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}},\\ &\qquad\qquad\qquad \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}}}\\ &\qquad + HS^3A\iota^2 + \mathcal{O}(\delta)\times\mathcal{O}(H^2T)\\ &\lesssim HS^3A^{\frac{3}{2}}\iota^2 + \sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)}}\\ &\qquad + \min\set[\Bigg]{\sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}},\\ &\qquad\qquad\qquad \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}}}. \end{align}\] This completes the proof. ◻
Lemma 36 (full-information setting). It holds that \[\begin{align} &\sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)}} \\ &\lesssim \sqrt{HSA \log^2(T) \mathbb{E}\brk*{\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a) \max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s, a;\prn*{\ell_t - m_t}^2)}}. \end{align}\]
Proof. By the definition of \(\zeta_t(s,a)\), we have \[\begin{align} &\sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s) \zeta_t(s,a)} } \\ &= \sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s,a)(1 - \pi_t(a \mid s))\sum_{b}\pi_t(b \mid s)\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)}^2}}\\ &\leq \sqrt{SA\log^2(T)\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s} q^{\pi_t}(s)\sum_{b}\pi_t(b \mid s)\prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)}^2}}, \label{eq:common95bound95adv95full} \end{align}\tag{53}\] where the inequality follows from the Cauchy–Schwarz inequality.
We next bound \(Q^{\underline{P}^\ell_t, \pi_t}(s, b;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, b;m_t)\). \[\begin{align} &Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t)\\ &= \prn*{ Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;\ell_t)} + \prn*{Q^{\underline{P}^m_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t)}\\ &\leq 0 + Q^{\underline{P}^m_t, \pi_t}(s, a;\ell_t - m_t) \\ &\leq \max_{\widetilde{P}\in \mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\relax*{\ell_t - m_t}). \end{align}\] Similarly, \[\begin{align} Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t) \geq -\max_{\widetilde{P}\in \mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\relax*{\ell_t - m_t}) \end{align}\] Here, for any transition kernel \(\widetilde{P}\), we have \[\begin{align} Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t}) &= \sum_{s',a'} q^{\widetilde{P},\pi_t}(s',a'\mid s,a)\relax{\ell_t(s',a')-m_t(s',a')} \\ &\leq \sqrt{\sum_{s',a'} q^{\widetilde{P},\pi_t}(s',a'\mid s,a)} \sqrt{\sum_{s',a'} q^{\widetilde{P},\pi_t}(s',a'\mid s,a)(\ell_t(s',a')-m_t(s',a'))^2} \\ &\leq \sqrt{H Q^{\widetilde{P},\pi_t}(s,a;(\ell_t-m_t)^2)}. \end{align}\]
Therefore, \[\begin{align} \prn*{Q^{\underline{P}^\ell_t, \pi_t}(s, a;\ell_t) - Q^{\underline{P}^m_t, \pi_t}(s, a;m_t)}^2 &\leq \prn*{\max_{\widetilde{P}\in \mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\relax*{\ell_t - m_t})}^2 \\ &\leq H\max_{\widetilde{P}\in \mathcal{P}_t}Q^{\widetilde{P}, \pi_t}(s,a;\prn*{\ell_t - m_t}^2) \label{eq:max95square95to95H} \end{align}\tag{54}\] Combining this with 53 completes the proof. ◻
Theorem 37 (full-information setting). For any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^2SA \log^2(T) \min\set*{L(\pi),HT-L(\pi),Q_\infty,V_1}}\\ &\qquad + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + HS^3A^{\frac{3}{2}}\log^2(T). \end{align}\]
Proof. By 35 36 17 and the choice \(\delta=1/T^2\), which implies \(\iota\lesssim \log(T)\), we have, for any policy \(\pi\), \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim HS^3A^{\frac{3}{2}}\log^2(T) + \sqrt{HSA \log^2(T) \mathbb{E}\brk*{\sum_{t=1}^T \sum_{s,a} q^{\pi_t}(s, a) \max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s, a;\prn*{\ell_t - m_t}^2)}}\\ &\qquad + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}\\ &\lesssim HS^3A^{\frac{3}{2}}\log^2(T) + \sqrt{H^2SA \log^2(T) \mathbb{E}\brk*{\sum_{t=1}^TV^{\pi_t}(s_0;\prn*{\ell_t - m_t}^2)}}\\ &\qquad + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}. \end{align}\] Combining this with 19, we obtain \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^2SA \log^2(T) \min\set*{L(\pi) + \text{\rm Reg}_T(\pi),HT-L(\pi)-\text{\rm Reg}_T(\pi),Q_\infty,V_1}}\\ &\qquad + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + HS^3A^{\frac{3}{2}}\log^2(T). \label{eq:adv95bound95mid} \end{align}\tag{55}\] Now, we derive the first-order bound. By 55 , there exists an absolute constant \(c>0\) such that \[\begin{align} \text{\rm Reg}_T(\pi) &\leq cHS^3A^{\frac{3}{2}}\log^2(T) + c\sqrt{H^2SA \log^2(T) L(\pi)}+c\sqrt{H^2SA \log^2(T) \text{\rm Reg}_T(\pi)}\\ &\qquad + c\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}\\ &\leq cHS^3A^{\frac{3}{2}}\log^2(T) + c\sqrt{H^2SA \log^2(T) L(\pi)}+\frac{c^2}{2}H^2SA \log^2(T) + \frac{1}{2}\text{\rm Reg}_T(\pi)\\ &\qquad + c\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}, \end{align}\] where the last inequality follows from the AM–GM inequality. Thus, we obtain \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim HS^3A^{\frac{3}{2}}\log^2(T) + \sqrt{H^2SA \log^2(T) L(\pi)} + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}. \label{eq:first95order95bound1} \end{align}\tag{56}\] Furthermore, we derive the bound in terms of \(HT-L(\pi)\). If \(\text{\rm Reg}_T(\pi)<0\), the desired upper bound is trivial. Otherwise, \(\text{\rm Reg}_T(\pi)\ge 0\) and 55 gives \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim HS^3A^{\frac{3}{2}}\log^2(T) + \sqrt{H^2SA\log^2(T)\prn*{HT-L(\pi)-\text{\rm Reg}_T(\pi)}} + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} \\ &\le HS^3A^{\frac{3}{2}}\log^2(T) + \sqrt{H^2SA\log^2(T)\prn*{HT-L(\pi)}} + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} . \label{eq:first95order95bound2} \end{align}\tag{57}\]
Theorem 38 (full-information setting). Under the stochastic regime with adversarial corruption, 1 guarantees that, for any policy \(\pi\), \[\begin{align} \text{\rm Reg}_T(\pi) \lesssim U + \sqrt{U\mathcal{C}} + HS^3A^{\frac{3}{2}}\log^2(T), \end{align}\] where \(U = \frac{H^2S^2A\log^2(T)}{\Delta_{\min}}\).
Proof. Since \(\text{\rm Reg}_T(\pi) \leq \text{\rm Reg}_T(\mathring{\pi})\) for all \(\pi\), it suffices to bound \(\text{\rm Reg}_T(\mathring{\pi})\). From 35 and the choice \(\delta=1/T^2\), which implies \(\iota\lesssim \log(T)\), we have \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) &\lesssim HS^3A^{\frac{3}{2}}\log^2(T) + 2H\log(T)\sum_{s}\sum_{a\neq {\pi^\star}(s)}\sqrt{\mathbb{E}\brk*{\sum_{t=1}^T q^{\pi_t}(s,a)}}\\ &\qquad + \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\log^2(T) \mathbb{E}\brk*{\sum_{t=1}^T\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\mathring{\pi}}(s,a)}_{+}}}. \end{align}\] Applying 24 25, for any \(\alpha > 0\), we get \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) &\leq \alpha(2\text{\rm Reg}_T(\mathring{\pi}) + 6\mathcal{C}) + \mathcal{O}\prn*{\sum_{s}\sum_{a\neq {\pi^\star}(s)}\frac{H^2\log^2(T)}{\alpha\Delta(s,a)} + HS^3A^{\frac{3}{2}}\log^2(T)}\\ &\qquad + \mathcal{O}\prn*{\frac{\prn*{\sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\log^2(T)}}^2}{4\alpha\Delta_{\min}}}. \end{align}\] Here, \[\begin{align} \prn*{\sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\log^2(T)}}^2 &\leq \prn*{\sum_{h = 0}^{H-1}\frac{1}{2}\prn*{\relax*{S_{h}} + \relax*{S_{h + 1}}}\sqrt{H^2A\log^2(T)}}^2 \\ &= \prn*{S\sqrt{H^2A\log^2(T)}}^2 = H^2S^2A\log^2(T). \label{eq:coeff95gap2} \end{align}\tag{58}\] Then, using 58 and choosing \(\alpha = \min\set*{\frac{1}{4}, \sqrt{\frac{U}{\mathcal{C}}}}\) with \(U = \frac{H^2S^2A\log^2(T)}{\Delta_{\min}}\), we get \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) \lesssim U + \sqrt{U\mathcal{C}} + HS^3A^{\frac{3}{2}}\log^2(T). \end{align}\] Therefore, since \(\text{\rm Reg}_T(\pi)\leq \text{\rm Reg}_T(\mathring{\pi})\) for all \(\pi\) by the definition of \(\mathring{\pi}\), the desired bound follows. ◻
In this section, we prove the best-of-both-worlds results for the bandit setting. We present the bound for the adversarial regime in 50 and the bound for the stochastic regime with adversarial corruption in 51. Together, these results establish 3.
Throughout this section, virtual episodes are inserted into the episode sequence, shifting the indices of subsequent real episodes. All algorithmic sums over \(t=1,\ldots,T\) include virtual episodes, while the data-dependent complexity measures in the main statements are defined only over real episodes. We denote by \(\mathcal{T}_r=\{t\in[T]:Y_t=1\}\) and \(\mathcal{T}_v=\{t\in[T]:Y_t=0\}\) the sets of real and virtual episodes, with sizes \(\relax{\mathcal{T}_r}\) and \(\relax{\mathcal{T}_v}\), respectively.
Following the full-information setting, 28 reduces the regret analysis to proving ?? . With the choice of \(b_t\) in 14 , we decompose the left-hand side of ?? as \[\begin{align} &\sum_s q^{\pi}(s) \sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}\\ &= \sum_s q^{\pi}(s) \underbrace{\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{\widehat{Q}_t(s,a) - B_t(s,a)}}_{\textrm{\boldsymbol{reg}}^{\pi}(s)}\\ &\qquad + \sum_s q^{\pi}(s) \underbrace{\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s,a;\ell_t) - \widehat{Q}_t(s,a) + Q^{\underline{P}^m_t, \pi_t}(s, a; m_t) - Q^{\pi_t}(s, a; m_t)}}_{\textrm{\boldsymbol{bias}}^{\pi}(s)}\\ &\qquad + \sum_s q^{\pi}(s) \underbrace{\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s, a; m_t) - Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)}}_{\textrm{\boldsymbol{error}}^{\pi}(s)}.\label{eq:decompose95bandit} \end{align}\tag{59}\]
Lemma 39 (bandit setting). It holds that \[\begin{align} q_t(s)\pi_t(a\mid s)C_t(s,a) &\leq 6HS, \qquad \mathbb{E}_t\brk*{C_t(s,a)} \leq 4HS \label{eq:qC95size95bound} \end{align}\qquad{(10)}\] for any episode \(t\) and state-action pair \((s,a)\).
Proof. We first prove a conditional occupancy property. For any \(\widetilde{P}\in\mathcal{P}_t\), any states \(s,s'\) with \(h(s')>h(s)\), and any actions \(a,a'\), we have \[\begin{align} q_t(s)\pi_t(a\mid s)q^{\widetilde{P},\pi_t}(s',a'\mid s,a) &\leq q_t(s')\pi_t(a'\mid s'). \label{eq:conditional95occupancy95domination} \end{align}\tag{60}\] Hence, \[\begin{align} \overline{q}_t^{\pi_t}(s') &\geq \overline{q}_t^{\pi_t}(s)\pi_t(a\mid s)q^{\widetilde{P},\pi_t}(s'\mid s,a). \end{align}\] Using \(q_t(x)=\overline{q}_t^{\pi_t}(x)+\gamma_t\) and \(\pi_t(a\mid s)q^{\widetilde{P},\pi_t}(s'\mid s,a)\leq 1\), we obtain \[\begin{align} q_t(s')\pi_t(a'\mid s') &=\prn*{\overline{q}_t^{\pi_t}(s')+\gamma_t}\pi_t(a'\mid s')\\ &\geq \prn*{\overline{q}_t^{\pi_t}(s)+\gamma_t}\pi_t(a\mid s)q^{\widetilde{P},\pi_t}(s'\mid s,a)\pi_t(a'\mid s')\\ &=q_t(s)\pi_t(a\mid s)q^{\widetilde{P},\pi_t}(s',a'\mid s,a), \end{align}\] which proves 60 .
Now, by the definition of \(C_t\) and 60 with \(\widetilde{P}=\overline{P}^C_t\), \[\begin{align} &q_t(s)\pi_t(a\mid s)C_t(s,a)\\ &\leq 2q_t(s)\pi_t(a\mid s)\sum_{s',a'}q^{\overline{P}^C_t,\pi_t}(s',a'\mid s,a)\rho_t(s')\frac{\mathbb{I}_t(s',a')\relax{L_{t,h(s')}-M_{t,h(s')}}}{q_t(s')\pi_t(a'\mid s')}\\ &\qquad+2Hq_t(s)\pi_t(a\mid s)\sum_{s',a'}q^{\overline{P}^C_t,\pi_t}(s',a'\mid s,a)\rho_t(s')^2\\ &\leq 2\sum_{s',a'}\rho_t(s')\mathbb{I}_t(s',a')D_{t,h(s')}+2H\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2. \label{eq:qC95bound95middle} \end{align}\tag{61}\] Since \(\rho_t(s')\leq 1\), \(D_{t,h(s')}\leq H\), \[\begin{align} 2\sum_{s',a'}\rho_t(s')\mathbb{I}_t(s',a')D_{t,h(s')} &\leq 2H^2. \end{align}\] Moreover, using \(\rho_t(s')\leq 1\) and \(\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\leq S + S\gamma_t \leq 2S\), \[\begin{align} 2H\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2 &\leq 4HS. \end{align}\] Combining the last three displays gives \[\begin{align} q_t(s)\pi_t(a\mid s)C_t(s,a) &\leq 6HS. \end{align}\] This proves the first inequality in ?? .
We next prove the second inequality. Let \(\overline{P}^C_t\) be a measurable transition kernel that attains the maximum in 12 . Then, it holds that \[\begin{align} C_t(s,a) &\leq 2\sum_{s',a'}q^{\overline{P}^C_t,\pi_t}(s',a'\mid s,a)c_t(s',a'). \end{align}\] By using \(q^{\overline{P}^C_t,\pi_t}(s',a'\mid s,a)\leq \pi_t(a'\mid s')\), we get \[\begin{align} \mathbb{E}_t\brk*{C_t(s,a)} &\leq 2\sum_{s',a'}\pi_t(a'\mid s')\mathbb{E}_t\brk*{c_t(s',a')}. \end{align}\]
Therefore, using \(\mathbb{E}_t\brk*{c_t(s',a')} \leq 2H\), \[\begin{align} \mathbb{E}_t\brk*{C_t(s,a)} &\leq 2\sum_{s',a'}\pi_t(a'\mid s')\cdot 2H =4HS, \end{align}\] which proves the second inequality. ◻
Lemma 40 (bandit setting). The variables \(b_t(s)\) in 14 and \(B_t(s,a)\) in 5 satisfy \[\begin{align} \eta_t(s,a)\pi_t(a\mid s)B_t(s,a) \leq \frac{1}{6H}, \quad B_t(s,a) \leq \frac{7\sqrt{HS^3A}}{\gamma_t} \label{eq:bound95eta95pi95B} \end{align}\qquad{(11)}\] for any episode \(t\) and state-action pair \((s,a)\).
Proof. Let \(R_t = \max_{s,a}\frac{\eta_t(s,a)}{q_t(s)}\).
We first consider the case when \(t\) is a real episode. In real episodes, by the definition of \(b_t\), \[\begin{align} b_t(s) &= 6\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T) + 3 \sum_a \eta_t(s,a)\pi_t(a\mid s)^2 C_t(s,a)^2 \end{align}\] For the first term, the learning-rate update gives \[\begin{align} &6\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)}- \frac{1}{\eta_t(s, a)}}\log(T)\\ &= \frac{6}{q_t(s)^2}\sum_a \eta_t(s,a)\zeta_t(s,a)\\ &= \frac{6}{q_t(s)^2}\sum_a \eta_t(s,a)\prn*{\mathbb{I}_t(s,a)-\pi_t(a\mid s)\mathbb{I}_t(s)}^2\prn*{L_{t,h(s)}-M_{t,h(s)}}^2\\ &\leq \frac{6H^2}{q_t(s)^2}\max_a\eta_t(s,a) \sum_a\prn*{\mathbb{I}_t(s,a)-\pi_t(a\mid s)\mathbb{I}_t(s)}^2\\ &\leq \frac{12H^2}{q_t(s)^2}\max_a\eta_t(s,a)\\ &\leq \frac{12H^2}{q_t(s)} \max_a\frac{\eta_t(s,a)}{q_t(s)} . \label{eq:bonus95term95upper95real1} \end{align}\tag{62}\] For the second term, by 39, \[\begin{align} 3 \sum_a \eta_t(s,a)\pi_t(a\mid s)^2 C_t(s,a)^2 & = 3\sum_{a} \frac{\eta_t(s,a)q_t(s)^2\pi_t(a\mid s)^2 C_t(s,a)^2}{q_t(s)^2}\\ &\leq \frac{108H^2S^2\sum_a \eta_t(s,a)}{q_t(s)^2} }\\ &\leq \frac{108H^2S^2A}{q_t(s)}\max_{a}\frac{\eta_t(s, a)}{q_t(s)} . \label{eq:bonus95term95upper95real2} \end{align}\tag{63}\] Combining 62 63 , we obtain \[\begin{align} b_t(s) &\leq \frac{12H^2 + 108H^2S^2A}{q_t(s)}\max_{a}\frac{\eta_t(s, a)}{q_t(s)} \leq \frac{120H^2S^2A}{q_t(s)}\max_{a}\frac{\eta_t(s, a)}{q_t(s)} \label{eq:bonus95term95upper95real} \end{align}\tag{64}\] Using 64 44 , we have \[\begin{align} B_t(s,a) &\leq 3\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t, \pi_t}(s'\mid s,a)b_t(s')\\ &\leq 360H^2S^2AR_t \prn*{\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t, \pi_t}(s'\mid s,a)\frac{1}{q_t(s')}} \\ &\leq 360H^2S^2AR_t \prn*{\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t,\pi_t}(s'\mid s,a)\frac{1}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s)q^{\overline{P}^B_t,\pi_t}(s'\mid s,a) + \gamma_t}} \\ &\leq 360H^2S^2AR_t \prn*{\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h}\frac{1}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t}} \\ &\leq 360H^2S^3AR_t\cdot\frac{1}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t} \label{eq:bound95B95real}\\ & \leq \frac{7\sqrt{HS^3A}}{\gamma_t}, \end{align}\tag{65}\] where in the last line we used \(R_t \leq \frac{1}{50\sqrt{H^3S^3A}}\) that holds in real episodes.
By using 65 , we also have \[\begin{align} \eta_t(s,a)\pi_t(a \mid s)B_t(s,a) &\leq 360H^2S^3AR_t\cdot\frac{\eta_t(s,a)\pi_t(a\mid s)}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t} \\ &\leq 360H^2S^3AR_t\cdot\frac{\eta_t(s,a)}{q_t(s)}\\ &\leq 360H^2S^3AR_t^2\\ &\leq \frac{1}{6H}, \end{align}\] where we used \(\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s)+\gamma_t \geq q_t(s)\pi_t(a\mid s)\), \(\frac{\eta_t(s,a)}{q_t(s)}\leq R_t\).
We next consider the case when \(t\) is a virtual episode. In a virtual episode, only the single pair \((s_t^\dagger,a_t^\dagger)\) is updated, and thus \[\begin{align} b_t(s) &= 6\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T) \\ &\leq \sum_{a} \frac{\mathbb{1}\{(s^\dagger_t, a^\dagger_t) = (s,a)\}}{54\eta_t(s,a)H}\\ &= \sum_{a} \frac{\mathbb{1}\{(s^\dagger_t, a^\dagger_t) = (s,a)\}}{54Hq_t(s)} \cdot \frac{1}{\max_{s', a'}\frac{\eta_t(s', a')}{q_t(s')}} }_{s,a} \frac{\eta_t(s,a)}{q_t(s)}} \\ &= \frac{\mathbb{1}\{s^\dagger_t = s\}}{q_t(s)}\cdot \frac{1}{54H\max_{s', a'}\frac{\eta_t(s', a')}{q_t(s')}}. \label{eq:bonus95term95upper95virtual} \end{align}\tag{66}\]
Using 66 44 , we have \[\begin{align} B_t(s,a) &\leq 3\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t,\pi_t}(s'\mid s,a)b_t(s')\\ &\leq \frac{1}{18HR_t} {\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t,\pi_t}(s'\mid s,a)\frac{\mathbb{1}\set{s^\dagger_t = s'}}{q_t(s')}}\\ &\leq \frac{1}{18HR_t} {\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h} q^{\overline{P}^B_t,\pi_t}(s'\mid s,a)\frac{\mathbb{1}\set{s_t^\dagger = s'}}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s)q^{\overline{P}^B_t,\pi_t}(s'\mid s,a) + \gamma_t}}\\ &\leq \frac{1}{18HR_t} {\sum_{h = h(s)}^{H-1}\sum_{s' \in \mathcal{S}_h}\frac{\mathbb{1}\set{s_t^\dagger = s'}}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t}}\\ &\leq \frac{1}{18HR_t} \frac{1}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t} \label{eq:bound95B95virtual}\\ & \leq \frac{3\sqrt{HS^3A}}{\gamma_t}, \end{align}\tag{67}\] where in the last inequality we used \(R_t > \frac{1}{50\sqrt{H^3S^3A}}\) in a virtual episode. By using 67 , we also have \[\begin{align} \eta_t(s,a)\pi_t(a \mid s)B_t(s,a) &\leq \frac{1}{18HR_t} \frac{\eta_t(s,a)\pi_t(a\mid s)}{\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s) + \gamma_t}\\ &\leq \frac{1}{18HR_t} \frac{\eta_t(s,a)}{q_t(s)} \\ &\leq \frac{1}{18H}\leq \frac{1}{6H}, \end{align}\] where we used \(\overline{q}^{\pi_t}_t(s)\pi_t(a\mid s)+\gamma_t \geq q_t(s)\pi_t(a\mid s)\), \(\frac{\eta_t(s,a)}{q_t(s)}\leq R_t\). This completes the proof. ◻
The following two lemmas are based on the [2]. We adapt the proof to our choice of \(\gamma_t\) and our virtual-episode rule, and include the details for completeness.
Lemma 41 (bandit setting). Suppose that the learning rates are updated according to 13 . Then, it holds \[\begin{align} \eta_t(s,a) \leq \frac{2\sqrt{\log(T)}}{\sqrt{\sum_{\tau\leq t: \tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2} }} \end{align}\] for any episode \(t\) and state-action pair \((s,a)\).
Proof. Let \(\phi_t(s,a)=\frac{\zeta_t(s,a)}{q_t(s)^2\log(T)}\) in real episodes and \(\phi_t(s,a)=\frac{\mathbb{I}\{(s_t^\dagger,a_t^\dagger)=(s,a)\}}{324\eta_t(s,a)^2H\log(T)}\) in virtual episodes. Then the update rule of learning rates can be written as \[\begin{align} \frac{1}{\eta_{t+1}(s,a)} = \frac{1}{\eta_t(s,a)} + \eta_t(s,a) \phi_t(s,a). \end{align}\] To apply 30, it suffices to show that \(\phi_t(s,a)\le \frac{1}{\eta_t(s,a)^2}\). This is clear for virtual episodes. For real episodes, \[\begin{align} \phi_t(s,a)\eta_t(s,a)^2 = \frac{\eta_t(s,a)^2\zeta_t(s,a)}{q_t(s)^2\log(T)} \leq \frac{H^2}{\log(T)}\prn*{\frac{\eta_t(s,a)}{q_t(s)}}^2 \leq \frac{H^2}{\log(T)}\cdot \frac{1}{50^2H^3S^3A} \leq 1, \end{align}\] which follows from \(\zeta_t(s,a) \leq H^2\) and \(\frac{\eta_t(s,a)}{q_t(s)}\leq \frac{1}{50\sqrt{H^3S^3A}}\) in real episodes.
Then, by 30, we have \[\begin{align} \eta_t(s,a) \leq \frac{2}{\sqrt{\sum_{\tau\leq t} \phi_\tau }} \leq \frac{2\sqrt{\log(T)}}{\sqrt{\sum_{\tau\leq t: \tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2} }}. \end{align}\] ◻
In the following lemma only, we use \(T\) to denote the number of real episodes rather than the total number of episodes.
Lemma 42 (bandit setting). When the number of real episodes is \(T\), the number of virtual episodes \(\relax{\mathcal{T}_v}\) is upper bounded by \[\begin{align} \relax{\mathcal{T}_v} \lesssim HSA \log^2(T). \end{align}\]
Proof. By the definition of virtual episodes, whenever \(t\in\mathcal{T}_v\), there exists a pair \((s,a)\) such that \(\frac{\eta_t(s,a)}{q_t(s)} > \frac{1}{50\sqrt{H^3S^3A}}\). Moreover, in virtual episodes, the corresponding learning rate will shrink by a factor of \(\prn*{1+\frac{1}{324H\log(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v})}}\) for a state-action pair \((s_t^\dagger,a_t^\dagger)\).
Therefore, for each fixed \((s,a)\), a virtual update on this pair can occur only while \[\begin{align} \eta_t(s,a)>\frac{1}{50\sqrt{H^3S^3A}(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v})} \end{align}\] Thus, the number of virtual updates on this pair is at most the number of multiplicative shrink steps needed to reduce \(\eta_t(s,a)\) from \(\eta_1\) to \(1/(50\sqrt{H^3S^3A}(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v}))\). Hence, \[\begin{align} \relax{\mathcal{T}_v} &\lesssim SA \frac{\log\prn*{\eta_1\sqrt{H^3S^3A}(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v})}}{\log\prn*{1+\frac{1}{324H\log(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v})}}} \\ &\lesssim HSA\log^2(\relax{\mathcal{T}_r}+\relax{\mathcal{T}_v}), \end{align}\] where we used \(\log(1+x)\ge x/2\) for \(x\in(0,1]\), \(\eta_1=1/(12H^3)\), and \(H,S,A\leq \relax{\mathcal{T}_r}\). This implies \[\begin{align} \relax{\mathcal{T}_v}\lesssim HSA\log^2(\relax{\mathcal{T}_r}). \end{align}\] Since \(\relax{\mathcal{T}_r}=T\), the desired bound follows. ◻
Lemma 43 (bandit setting). Suppose that the event \(\mathcal{E}\) holds. Then, it holds that \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s)\rho_t(s)^2 &\lesssim HS^3A\iota^2. \end{align}\]
Proof. Recall that \(\rho_t(s)=\dfrac{q_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}\) and \(q_t(s)=\overline{q}_t^{\pi_t}(s)+\gamma_t\). Then, we have \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s,a} q^{\pi_t}(s,a)\rho_t(s,a)^2 &\leq \sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s) \prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s) + \gamma_t}{q_t(s)}}^2 \\ &\leq \sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s) \prn*{2\prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}}^2 + 2\prn*{\frac{\gamma_t}{q_t(s)}}^2} \\ &\leq 2\sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s) \prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}}^2 + 2\sum_{t \in \mathcal{T}_r}\sum_{s} \gamma_t \\ \end{align}\] We bound the first term. By 15, we obtain \[\begin{align} &\sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s) \prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}}^2 \\ &\leq \sum_{t \in \mathcal{T}_r}\sum_{s} \prn*{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)} \frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}\\ &\leq c\sqrt{HS^2A \iota^2 \sum_{t \in \mathcal{T}_r} \sum_s q^{\pi_t}(s)\prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}}^2} + cHS^3A\iota^2. \end{align}\] Now use the fact that if \(x \leq a\sqrt{x}+ b\) with \(x,a,b \geq 0\), then \(x\leq 2a^2 + 2b\), we obtain \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s) \prn*{\frac{\overline{q}^{\pi_t}_t(s) - \underline{q}^{\pi_t}_t(s)}{q_t(s)}}^2 &\lesssim HS^2A \iota^2 + HS^3A\iota^2 \lesssim HS^3A\iota^2. \end{align}\] We also have \[\begin{align} 2\sum_{t \in \mathcal{T}_r}\sum_{s} \gamma_t \leq 2S\log(T) \end{align}\] Combining the above bounds gives \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s} q^{\pi_t}(s)\rho_t(s)^2 &\lesssim HS^3A\iota^2. \end{align}\] ◻
Lemma 44 (bandit setting). Suppose that the event \(\mathcal{E}\) holds. Then, it holds that \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s} q_t(s)\rho_t(s)^2 &\lesssim HS^3A\iota^2. \end{align}\]
Proof. We decompose \[\begin{align} &\sum_{t \in \mathcal{T}_r}\sum_{s'}q_t(s')\rho_t(s')^2\\ &=\underbrace{\sum_{t \in \mathcal{T}_r}\sum_{s'}q^{\pi_t}(s')\rho_t(s')^2}_{\mathrm{\boldsymbol{term}}_a} +\underbrace{\sum_{t \in \mathcal{T}_r}\sum_{s'}\prn*{\overline{q}_t^{\pi_t}(s')-q^{\pi_t}(s')}\rho_t(s')^2}_{\mathrm{\boldsymbol{term}}_b}+\underbrace{\sum_{t \in \mathcal{T}_r}\sum_{s'}\gamma_t\rho_t(s')^2}_{\mathrm{\boldsymbol{term}}_c}. \label{eq:bound95squared95Ctorho} \end{align}\tag{68}\] For \(\mathrm{\boldsymbol{term}}_a\), by 43, \[\begin{align} \mathrm{\boldsymbol{term}}_a \lesssim HS^3A\iota^2. \label{eq:term95a95rho95bound} \end{align}\tag{69}\] For \(\mathrm{\boldsymbol{term}}_b\), applying 15 with \(g_t(s)=\rho_t(s)^2\) and 43, we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_b &\lesssim \sqrt{HS^2A\iota^2\sum_{t \in \mathcal{T}_r}\sum_s q^{\pi_t}(s)\rho_t(s)^4}+HS^3A\iota^2\\ &\lesssim \sqrt{HS^2A\iota^2\cdot S^4A\log(T)\iota}+HS^3A\iota^2\\ &\lesssim HS^3A\iota^2, \label{eq:term95b95rho95bound} \end{align}\tag{70}\] For \(\mathrm{\boldsymbol{term}}_c\), since \(\rho_t(s')\leq 1\), \[\begin{align} \mathrm{\boldsymbol{term}}_c &\leq \sum_{t \in \mathcal{T}_r}\sum_{s'}\gamma_t \lesssim S\log(T). \label{eq:term95c95rho95bound} \end{align}\tag{71}\] Thus, we have \[\begin{align} \sum_{t \in \mathcal{T}_r}\sum_{s} q_t(s)\rho_t(s)^2 &\lesssim HS^3A\iota^2. \end{align}\] ◻
Now we upper bound the right-hand side of 59 .
Lemma 45 (bandit setting). For each state \(s \in \mathcal{S}\) and any comparator \(\pi\), it holds that \[\begin{align} \mathbb{E} \brk*{\textrm{\boldsymbol{reg}}^{\pi}(s) } &\leq \mathcal{O}\prn*{HS^2A\log(T)} + \mathbb{E}\brk*{\sum_{t=1}^Tb_t(s)} + \mathbb{E}\brk*{\frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a)}. \end{align}\]
Proof. We will apply 27 with \(p_t = \pi_t(\cdot \mid s)\) and \(\ell_t = \widehat{Q}_t(s,a) - B_t(s,a)\) for each \(s \in \mathcal{S}\). To do so, in what follows, we will check the conditions of 27. Let \[\begin{align} x_t &= \inpr*{-\pi_t(\cdot \mid s) , \widehat{Q}_t(s,\cdot) - Q^{\underline{P}^m_t, \pi_t}(s,\cdot; m_t) + C_t(s,\cdot)Y_t}= -\dfrac{\mathbb{I}_t(s)(L_{t,h(s)} - M_{t,h(s)})}{q_t(s)}Y_t \end{align}\] and verify that for all \((s,a)\), \(\eta_t(s,a)\pi_t(a\mid s)\prn*{\widehat{Q}_t(s,a)-B_t(s,a) - Q^{\underline{P}^m_t, \pi_t}(s,a; m_t) + x_t}\geq -{1}/{2}\). Recall that in a virtual episode we have \(Y_t=0\) and \(\ell_t(s,a)=0\) for all state-action pairs \((s,a)\).
Then, we have \[\begin{align} &\eta_t(s,a)\pi_t(a\mid s)\prn*{\widehat{Q}_t(s,a)-B_t(s,a)-Q^{\underline{P}^m_t, \pi_t}(s,a; m_t) + x_t} \\ &= \eta_t(s,a)\pi_t(a\mid s) \prn[\bigg]{\frac{\mathbb{I}_t(s,a)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)\pi_t(a\mid s)}Y_t- \frac{\mathbb{I}_t(s)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)}Y_t\\ & -B_t(s,a) - C_t(s,a)Y_t} \\ &\geq -\frac{2\eta_t(s,a)H}{q_t(s)}Y_t -\eta_t(s, a)\pi_t(a\mid s) B_t(s,a) -\eta_t(s, a)\pi_t(a\mid s) C_t(s,a)Y_t\\ &\geq -\frac{1}{3} - \frac{1}{6H} \geq -\frac{1}{2}. \end{align}\] where we used \(0\le L_{t,h(s)},M_{t,h(s)}\le H\) and \(C_t\ge 0\). The last inequality uses \(\eta_t(s,a)/q_t(s)\leq 1/(50\sqrt{H^3S^3A})\) in real episodes, 39 40. Indeed, \[\begin{align} \frac{2\eta_t(s,a)H}{q_t(s)} +\eta_t(s,a)\pi_t(a\mid s)C_t(s,a) &=\frac{\eta_t(s,a)}{q_t(s)}\prn*{2H+q_t(s)\pi_t(a\mid s)C_t(s,a)}\\ &\leq \frac{2H+6HS}{50\sqrt{H^3S^3A}} \leq \frac{1}{3}, \end{align}\] and \[\begin{align} \eta_t(s,a)\pi_t(a\mid s)B_t(s,a) &\leq \frac{1}{6H}. \end{align}\]
Hence, by 27, we obtain \[\begin{align} & \mathbb{E}[\textrm{\boldsymbol{reg}}^{\pi}(s)]\\ &\leq \frac{A\log(AT^2)}{\eta_1} + \mathbb{E}\brk*{\sum_{t=1}^T \sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(AT^2)} \\ &\qquad + \mathbb{E}\brk*{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a \mid s)^{2}\prn*{\prn*{\widehat{Q}_t(s, a) - Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)} - B_t(s, a) + x_t}^2} \\ &\qquad + \mathbb{E}\brk*{\frac{1}{T^2}\sum_{t=1}^T \inpr*{ -\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}, \widehat{Q}_t(s, \cdot) - B_t(s, \cdot) }} + 2\mathbb{E}\brk*{\nrm*{Q^{\underline{P}^m_t, \pi_t}(s, \cdot; m_{T+1})}_{\infty}} \\ &\leq \mathcal{O}\prn*{HS^2A\log(T)} + 3\mathbb{E}\brk*{\sum_{t=1}^T\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T)} ^m_t, \pi_t}(s, \cdot; m_{T+1})}_{\infty}\leq H} \\ &\qquad + 3\mathbb{E}\brk[\Bigg]{\underbrace{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{\frac{\mathbb{I}_t(s,a)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)\pi_t(a\mid s)}Y_t-\frac{\mathbb{I}_t(s)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)}Y_t}^2}_{\text{stability-term}}}\\ &\qquad + 3\mathbb{E}\brk*{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{C_t(s,a)Y_t}^2}+ 3\mathbb{E}\brk*{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2} B_t(s, a)^2}. \label{eq:regterm95policy95mid} \end{align}\tag{72}\] Here, the second inequality follows from \[\begin{align} &\mathbb{E}\brk*{\frac{1}{T^2}\sum_{t=1}^T \inpr*{ -\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}, \widehat{Q}_t(s, \cdot) - B_t(s, \cdot) }} \\ &= \frac{1}{T^2}\inpr*{ -\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}, \mathbb{E}\brk*{\sum_{t=1}^T\mathbb{E}_t\brk*{\widehat{Q}_t(s, \cdot) - B_t(s, \cdot)}} } \\ &\leq \frac{1}{T^2}\nrm*{-\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}}_1\nrm*{\mathbb{E}\brk*{\sum_{t=1}^T\mathbb{E}_t\brk*{\widehat{Q}_t(s, \cdot) - B_t(s, \cdot)}} }_\infty \\ &\leq \frac{2T(6HS +7\sqrt{HS^3A}T)}{T^2} = \frac{12HS}{T} + 14\sqrt{HS^3A} \leq \mathcal{O}\prn*{HS^2A\log(T)}, \end{align}\] where we used \(\nrm*{-\pi(\cdot \mid s) + \frac{1}{A}\mathbf{1}}_1\leq 2\), \(\relax*{\mathbb{E}_t[\widehat{Q}_t(s,a)]} \leq 6HS\) and \(B_t(s,a) \leq \frac{7\sqrt{HS^3A}}{\gamma_t}\) from 39 40, which together imply \(\nrm*{\mathbb{E}\brk*{\sum_{t=1}^T\mathbb{E}_t\brk*{\widehat{Q}_t(s, \cdot) - B_t(s, \cdot)}} }_\infty \leq T(6HS + 7\sqrt{HS^3A}T)\).
We further evaluate the in the last inequality as \[\begin{align} &\eta_t(s, a)\pi_t(a\mid s)^{2}\prn*{\frac{\mathbb{I}_t(s,a)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)\pi_t(a\mid s)}Y_t-\frac{\mathbb{I}_t(s)(L_{t,h(s)}-M_{t,h(s)})}{q_t(s)}Y_t}^2 \\ &= \eta_t(s,a)\prn*{\frac{\mathbb{I}_t(s,a) (L_{t,h(s)} - M_{t,h(s)})}{q_t(s)} - \frac{\pi_t(a\mid s) \mathbb{I}_t(s) (L_{t,h(s)} - M_{t,h(s)})}{q_t(s)}}^2Y_t \\ &= \frac{\eta_t(s,a)}{q_t(s)^2}(\mathbb{I}_t(s,a) - \pi_t(a\mid s) \mathbb{I}_t(s))^2 (L_{t,h(s)} - M_{t,h(s)})^2Y_t\\ &= \frac{\eta_t(s,a)\zeta_t(s,a)}{q_t(s)^2} Y_t, \label{eq:stab195bound} \end{align}\tag{73}\] where the last equality follows from the definition of \(\zeta_t(s,a) = (\mathbb{I}_t(s,a) - \pi_t(a\mid s) \mathbb{I}_t(s))^2 (L_{t,h(s)} - M_{t,h(s)})^2\). Thus, we have \[\begin{align} \text{stability-term} &= \sum_{t=1}^T \sum_a \frac{\eta_t(s,a)\prn*{\zeta_t(s,a)}}{q_t(s)^2}Y_t \\ &\leq \sum_{t=1}^T \sum_a\prn*{\frac{1}{\eta_{t+1}(s,a)} - \frac{1}{\eta_t(s,a)}}\log(T), \end{align}\] where the last inequality follows from 13 .
Then, together with 72 and 40, we obtain \[\begin{align} \mathbb{E}[\textrm{\boldsymbol{reg}}^{\pi}(s)] &\leq \mathcal{O}\prn*{HS^2A\log(T)} + 6\mathbb{E}\brk*{\sum_{t=1}^T\sum_a \prn*{\frac{1}{\eta_{t+1}(s, a)} - \frac{1}{\eta_t(s, a)}}\log(T)} \\ &\qquad + 3\mathbb{E}\brk*{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2}C_t(s,a)^2Y_t}\\ &\qquad + 3\mathbb{E}\brk*{\sum_{t=1}^T \sum_a \eta_t(s, a)\pi_t(a\mid s)^{2} B_t(s, a)^2}\\ &= \mathcal{O}\prn*{HS^2A\log(T)} + \mathbb{E}\brk*{\sum_{t=1}^Tb_t(s)} + \mathbb{E}\brk*{\frac{1}{H}\sum_{t=1}^T\sum_a \pi_t(a \mid s) B_t(s,a)}. \end{align}\] ◻
Lemma 46 (bandit setting). For any comparator policy \(\pi\), the following two bounds hold: \[\begin{align} &\mathbb{E}\brk*{\sum_{s}q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s) }\\ &\lesssim \sqrt{H^2S^2A \iota^2 \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r} \sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\prn{\ell_t - m_t}^2)}} + H^2S^3A\iota^2 \end{align}\] and \[\begin{align} &\mathbb{E}\brk*{\sum_{s}q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s) }\\ &\lesssim \sqrt{H^3S^2A \iota^2 \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r} \sum_{s,a}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_+}}\\ &\qquad + \sum_{h=0}^{H-1}\sqrt{H^2S^2\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\iota^2 \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}}+ H^2S^3A\iota^2. \end{align}\]
Proof. By the definition of \(c_t(s,a)\) 11 , \[\begin{align} \mathbb{E}_t\brk*{c_t(s,a)} &=\rho_t(s)\frac{q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\relax*{\ell_t - m_t}) + \rho_t(s)^2H. \label{eq:ct95cond95mean} \end{align}\tag{74}\] Then, we obtain \[\begin{align} \mathbb{E}_t\brk*{c_t(s,a)} &\geq \rho_t(s) \frac{q^{\pi_t}(s)}{q_t(s)}\relax*{Q^{\pi_t}(s,a;\ell_t-m_t)} +\rho_t(s)\prn*{1-\frac{q^{\pi_t}(s)}{q_t(s)}}\relax*{Q^{\pi_t}(s,a;\ell_t-m_t)} \\ &=\rho_t(s)\relax*{Q^{\pi_t}(s,a;\ell_t-m_t)} \tag{75}\\ &\geq\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}\relax*{Q^{\pi_t}(s,a;\ell_t-m_t)} . \tag{76} \end{align}\]
Thus, by the definition of \(\widehat{Q}_t\), we have \[\begin{align} &\mathbb{E}_t\brk*{Q^{\pi_t}(s,a;\ell_t)-\widehat{Q}_t(s,a)+Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)-Q^{\pi_t}(s,a;m_t)}\\ &=Q^{\pi_t}(s,a;\ell_t)-\prn*{Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)+\frac{q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)Y_t-\mathbb{E}_t\brk*{C_t(s,a)}Y_t}\\ &\qquad +Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)-Q^{\pi_t}(s,a;m_t)\\ &=\frac{q_t(s)-q^{\pi_t}(s)Y_t}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)}Y_t\\ &= \begin{cases} -Q^{\pi_t}(s,a;m_t) & \text{if } Y_t=0,\\[2ex] \dfrac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)} & \text{if } Y_t=1. \end{cases} \label{eq:cond95bias95new95c} \end{align}\tag{77}\]
When \(Y_t=1\), by 76 and \(C_t(s,a)\geq c_t(s,a)\) pathwise, \[\begin{align} &\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)}\\ &\geq -\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}\relax*{Q^{\pi_t}(s,a;\ell_t-m_t)}+\mathbb{E}_t\brk*{c_t(s,a)}\geq 0. \label{eq:real95bias95nonnegative95new95c} \end{align}\tag{78}\] When \(Y_t=0\), using \(Q^{\pi_t}(s,a;m_t)\leq H\) and the fact that the virtual episode has zero loss, we obtain \[\begin{align} -H &\leq \mathbb{E}_t\brk*{Q^{\pi_t}(s,a;\ell_t)-\widehat{Q}_t(s,a)+Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)-Q^{\pi_t}(s,a;m_t)}\leq 0. \label{eq:virtual95bias95upper95new95c} \end{align}\tag{79}\]
Therefore, \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\textrm{\boldsymbol{bias}}^{\pi}(s)}\\ &=\mathbb{E}\brk*{\sum_{t=1}^T\sum_{s,a}q^{\pi}(s)(\pi_t(a\mid s)-\pi(a\mid s))\mathbb{E}_t\brk*{Q^{\pi_t}(s,a;\ell_t)-\widehat{Q}_t(s,a)+Q^{\underline{P}^m_t,\pi_t}(s,a;m_t)-Q^{\pi_t}(s,a;m_t)}}\\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi}(s)(\pi_t(a\mid s)-\pi(a\mid s))\prn*{\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)}}}\\ &\qquad+\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_v}\sum_{s,a}q^{\pi}(s)\pi(a\mid s)H}\\ &=\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)}+H\relax{\mathcal{T}_v}. \label{eq:bias95to95zt95new95c} \end{align}\tag{80}\] The additive term \(H\relax{\mathcal{T}_v} \lesssim H^3SA\log^2(T)\) by 42 is lower order and will be absorbed into the final lower-order term. The last equality follows by applying 10, for each real episode \(t\in\mathcal{T}_r\), with \[\begin{align} L(s,a)=\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)}. \end{align}\] Thus, \[\begin{align} z_t(s,a)&=\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s,a)}\\ &\qquad-\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\frac{q_t(s')-q^{\pi_t}(s')}{q_t(s')}Q^{\pi_t}(s',a';\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s',a')}}. \label{eq:def95zt95new95c} \end{align}\tag{81}\]
We first reduce the bound to the high-probability event \(\mathcal{E}\). By \(\mathbb{E}_t\brk*{C_t(s,a)} \leq \mathcal{O}(HS)\) from 39, \(\mathbb{E}_t\brk*{z_t(s,a)}\leq \mathcal{O}(HS)\) holds. Therefore, we have \[\begin{align} &\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)}\\ &\leq\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)\mathrel{}\middle|\mathrel{}\mathcal{E}}+\Pr(\bar{\mathcal{E}})\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)\mathrel{}\middle|\mathrel{}\bar{\mathcal{E}}}\\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)\mathrel{}\middle|\mathrel{}\mathcal{E}}+\mathcal{O}(\delta) \times \mathcal{O}(HT) \times \mathcal{O}(HS)\\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)\mathrel{}\middle|\mathrel{}\mathcal{E}}+\mathcal{O}\prn*{\frac{H^2S}{T}}. \label{eq:high95prob95bias} \end{align}\tag{82}\] In the following, we work on the event \(\mathcal{E}\) and bound the first term on the right-hand side of 82 .
On \(\mathcal{E}\), since \(P\in\mathcal{P}_t\), we can upper bound \(\mathbb{E}_t\brk*{c_t(s,a)}\) as \[\begin{align} \mathbb{E}_t\brk*{c_t(s,a)} &\leq \rho_t(s)Q^{\pi_t}(s,a;\relax*{\ell_t-m_t})+\rho_t(s)^2H\\ &\leq \rho_t(s)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})+\rho_t(s)^2H\\ &\leq \frac{\overline{q}_t^{\pi_t}(s)-\underline{q}_t^{\pi_t}(s)}{q_t(s)}\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})+\frac{\gamma_t H}{q_t(s)}+\rho_t(s)^2H. \label{eq:ct95upper95for95term1} \end{align}\tag{83}\]
We next derive lower and upper bounds on \(z_t(s,a)\). Since \(P\in\mathcal{P}_t\) on \(\mathcal{E}\), the definition of \(C_t(s,a)\) 12 implies \[\begin{align} \mathbb{E}_t\brk*{C_t(s,a)} &\geq\mathbb{E}_t\brk*{c_t(s,a)}+\mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a') + C_t(s',a')}}. \label{eq:Et95C95lower} \end{align}\tag{84}\]
Combining 81 84 76 , we have for any real episode \(t \in \mathcal{T}_r\) \[\begin{align} z_t(s,a)&\geq \frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{c_t(s,a)}\\ &\qquad +\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a')} - \frac{q_t(s')-q^{\pi_t}(s')}{q_t(s')}Q^{\pi_t}(s',a';\ell_t-m_t)}\\ &\geq 0. \label{eq:zt95geq095new95c} \end{align}\tag{85}\]
On the other hand, let \(\overline{P}^C_t\) be the measurable transition kernel that attains the maximum in 12 for the realized table \(c_t\). Then, for any real episode \(t \in \mathcal{T}_r\) \[\begin{align} z_t(s,a)&=\frac{q_t(s)-q^{\pi_t}(s)}{q_t(s)}Q^{\pi_t}(s,a;\ell_t-m_t)+\mathbb{E}_t\brk*{c_t(s,a)}\\ &\qquad+\mathbb{E}_t\brk*{\mathbb{E}_{s'\sim \overline{P}^C_t(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{c_t(s',a')+C_t(s',a')}}\\ &\qquad-\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\frac{q_t(s')-q^{\pi_t}(s')}{q_t(s')}Q^{\pi_t}(s',a';\ell_t-m_t)+\mathbb{E}_t\brk*{C_t(s',a')}}\\ &\leq 2\mathbb{E}_t\brk*{c_t(s,a)} +\mathbb{E}_t\brk*{\mathbb{E}_{s'\sim \overline{P}^C_t(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{c_t(s',a')+C_t(s',a')}}\\ &\qquad - \mathbb{E}_{s'\sim P(\cdot\mid s,a), a'\sim\pi_t(\cdot\mid s')}\brk*{ - \mathbb{E}_t\brk*{c_t(s',a')} + C_t(s',a') } }\\ &\leq 2\mathbb{E}_t\brk*{c_t(s,a)}+2\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a')}}\\ &\qquad+\mathbb{E}_t\brk*{\sum_{s',a'}\relax*{\overline{P}^C_t(s'\mid s,a)-P(s'\mid s,a)}\pi_t(a'\mid s')\prn*{c_t(s',a')+C_t(s',a')}}. \label{eq:zt95upper95before95unroll95new95c} \end{align}\tag{86}\]
By the definition of \(C_t(s,a)\) 12 , we have \[\begin{align} \sum_a\pi_t(a\mid s)\prn*{c_t(s,a)+C_t(s,a)}&=2\sum_{s',a'}q^{\overline{P}^C_t,\pi_t}(s',a'\mid s)c_t(s',a'). \label{eq:C95unroll95new95c} \end{align}\tag{87}\] Combining 86 and 87 , we obtain for any real episode \(t \in \mathcal{T}_r\) \[\begin{align} z_t(s,a)&\leq 2\mathbb{E}_t\brk*{c_t(s,a)}+2\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a')}}\\ &\qquad+2\mathbb{E}_t\brk*{\sum_{s'}\relax*{\overline{P}^C_t(s'\mid s,a)-P(s'\mid s,a)}\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')}. \label{eq:zt95upper95new95c} \end{align}\tag{88}\]
Therefore, by 85 and 88 , \[\begin{align} &\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}z_t(s,a)\\ &\leq \sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+z_t(s,a) }\\ &\leq 2\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_t\brk*{c_t(s,a)}}_{\mathrm{\boldsymbol{term}}_1}\\ &\qquad+2\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a')}}}_{\mathrm{\boldsymbol{term}}_2}\\ &\qquad+2\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_t\brk*{\sum_{s'}\relax*{\overline{P}^C_t(s'\mid s,a)-P(s'\mid s,a)}\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')}}_{\mathrm{\boldsymbol{term}}_3}. \end{align}\]
Let \[\begin{align} G^{\pi_t} \coloneq \min\set*{\sum_{s,a}q^{\pi_t}(s,a)\prn*{\max_{\widetilde{P}\in\mathcal{P}_t} Q^{\widetilde{P}, \pi_t}(s,a;\relax{\ell_t - m_t})}^2,\; H\sum_{s,a}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_+}. \label{def:proof95G} \end{align}\tag{89}\] We first bound \(\mathrm{\boldsymbol{term}}_1\). By 83 , \[\begin{align} \mathrm{\boldsymbol{term}}_1&=\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_t\brk*{c_t(s,a)}\\ &\leq \underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{\overline{q}_t^{\pi_t}(s)-\underline{q}_t^{\pi_t}(s)}\frac{\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}{q_t(s)}\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}_{\mathrm{\boldsymbol{term}}_{1a}}\\ &\qquad+\underbrace{H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\frac{\gamma_t}{q_t(s)}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}_{\mathrm{\boldsymbol{term}}_{1b}}\\ &\qquad+\underbrace{H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\rho_t(s)^2}_{\mathrm{\boldsymbol{term}}_{1c}}. \label{eq:term195split95new95c} \end{align}\tag{90}\]
We first control \(\mathrm{\boldsymbol{term}}_{1a}\). Define \[\begin{align} g_t^{(1)}(s)&\coloneq\sum_a\frac{\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}{q_t(s)}\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t}) \leq H. \label{eq:def95g195term1} \end{align}\tag{91}\] Thus, by 15, \[\begin{align} \mathrm{\boldsymbol{term}}_{1a}&=\sum_{t\in\mathcal{T}_r}\sum_s\prn*{\overline{q}_t^{\pi_t}(s)-\underline{q}_t^{\pi_t}(s)}g_t^{(1)}(s)\\ &\lesssim \sqrt{HS^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_s q^{\pi_t}(s)\prn*{g_t^{(1)}(s)}^2}+H^2S^3A\iota^2. \label{eq:term1a95width95bound} \end{align}\tag{92}\] Here, by the Cauchy–Schwarz inequality, \[\begin{align} &\sum_s q^{\pi_t}(s)\prn*{g_t^{(1)}(s)}^2\\ &\leq \sum_s q^{\pi_t}(s)\prn*{\sum_a\frac{\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}{q_t(s)}}\prn*{\sum_a\frac{\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}{q_t(s)}\prn*{\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}^2}\\ &\leq \sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\prn*{\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}^2\\ &\leq H\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn{\ell_t-m_t}^2) }\\ &\leq HG^{\pi_t}, \label{eq:g195square95to95G} \end{align}\tag{93}\] Therefore, \[\begin{align} \mathrm{\boldsymbol{term}}_{1a}&\lesssim \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}G^{\pi_t}}+H^2S^3A\iota^2. \label{eq:term1a95final95new95c} \end{align}\tag{94}\] Next, \(\mathrm{\boldsymbol{term}}_{1b}\) can be bounded by \[\begin{align} \mathrm{\boldsymbol{term}}_{1b}&\leq H\sum_{t\in\mathcal{T}_r}\sum_s\gamma_t\frac{q^{\pi_t}(s)}{q_t(s)}\leq HS\sum_{t\in\mathcal{T}_r}\gamma_t\lesssim HS\log(T). \label{eq:term1b95final95new95c} \end{align}\tag{95}\] Finally, \(\mathrm{\boldsymbol{term}}_{1c}\) can be bounded by using 43, \[\begin{align} \mathrm{\boldsymbol{term}}_{1c}&\leq H\sum_{t\in\mathcal{T}_r}\sum_s q^{\pi_t}(s)\rho_t(s)^2 \lesssim H^2S^3A\iota^2. \label{eq:term1c95final95new95c} \end{align}\tag{96}\] Combining 94 95 96 , we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_1&\lesssim \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}G^{\pi_t}}+H^2S^3A\iota^2. \label{eq:term195final95new95c} \end{align}\tag{97}\]
We next bound \(\mathrm{\boldsymbol{term}}_2\). By 83 , we have \[\begin{align} \mathrm{\boldsymbol{term}}_2&=\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\mathbb{E}_t\brk*{c_t(s',a')}}\\ &\leq \underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_{s'\sim P(\cdot\mid s,a),a'\sim\pi_t(\cdot\mid s')}\brk*{\frac{\overline{q}_t^{\pi_t}(s')-\underline{q}_t^{\pi_t}(s')}{q_t(s')}\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t})}}_{\mathrm{\boldsymbol{term}}_{2a}}\\ &\qquad+\underbrace{H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\brk*{\frac{\gamma_t}{q_t(s')}}}_{\mathrm{\boldsymbol{term}}_{2b}}\\ &\qquad+\underbrace{H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\brk*{\rho_t(s')^2}}_{\mathrm{\boldsymbol{term}}_{2c}}. \label{eq:term295split95new95c} \end{align}\tag{98}\]
We first bound \(\mathrm{\boldsymbol{term}}_{2a}\). Define \[\begin{align} g_t^{(2)}(s')&\coloneq\frac{\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+P(s'\mid s,a)}{q_t(s')}\sum_{a'}\pi_t(a'\mid s')\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t})\\ &\leq \sum_{a'}\pi_t(a'\mid s')\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t}) \leq H.\tag{99} \tag{100} \end{align}\] Therefore, by 15, \[\begin{align} \mathrm{\boldsymbol{term}}_{2a}&=\sum_{t\in\mathcal{T}_r}\sum_{s'}\prn*{\overline{q}_t^{\pi_t}(s')-\underline{q}_t^{\pi_t}(s')}g_t^{(2)}(s')\\ &\lesssim \sqrt{HS^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s'}q^{\pi_t}(s')\prn*{g_t^{(2)}(s')}^2}+H^2S^3A\iota^2. \label{eq:term2a95width95bound} \end{align}\tag{101}\] Moreover, by using the Cauchy–Schwarz inequality, we obtain \[\begin{align} &\sum_{s'}q^{\pi_t}(s')\prn*{g_t^{(2)}(s')}^2\\ &\leq \sum_{s'}q^{\pi_t}(s')\prn*{\sum_{a'}\pi_t(a'\mid s')\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t})}^2\\ &\leq \sum_{s',a'}q^{\pi_t}(s',a')\prn*{\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t})}^2\\ &\leq H\sum_{s',a'}q^{\pi_t}(s',a')\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\prn{\ell_t-m_t}^2),\label{eq:g295square95bound} \end{align}\tag{102}\] where the last inequality follows from 54 .
Simultaneously, it holds that \[\begin{align} &\sum_{s'}q^{\pi_t}(s')\prn*{g_t^{(2)}(s')}^2\\ &\leq H\sum_{s'}q^{\pi_t}(s')\sum_{s,a,a'}P(s'\mid s,a)\pi_t(a'\mid s')\frac{\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+}{q_t(s')}\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s',a';\relax{\ell_t-m_t})\\ &\leq H^2\sum_{s,a,s',a'}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+P(s'\mid s,a)\pi_t(a'\mid s')\\ &\leq H^2\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+. \label{eq:g295square95bound2} \end{align}\tag{103}\] From 102 103 , we have \[\begin{align} \sum_{s'}q^{\pi_t}(s')\prn*{g_t^{(2)}(s')}^2\leq HG^{\pi_t}. \end{align}\] Hence, \[\begin{align} \mathrm{\boldsymbol{term}}_{2a}&\lesssim \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}G^{\pi_t}}+H^2S^3A\iota^2. \label{eq:term2a95final95new95c} \end{align}\tag{104}\]
Next, \(\mathrm{\boldsymbol{term}}_{2b}\) can be bounded by \[\begin{align} \mathrm{\boldsymbol{term}}_{2b}&\leq H\sum_{t\in\mathcal{T}_r}\sum_{s'}\gamma_t\frac{\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+P(s'\mid s,a)}{q_t(s')}\leq HS\sum_{t\in\mathcal{T}_r}\gamma_t\lesssim HS\log(T). \label{eq:term2b95final95new95c} \end{align}\tag{105}\] Finally, by the 43, \[\begin{align} \mathrm{\boldsymbol{term}}_{2c}&\leq H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+P(s'\mid s,a)\rho_t(s')^2\\ &\leq \sum_{s'}q^{\pi_t}(s')\rho_t(s')^2\lesssim H^2S^3A\iota^2. \label{eq:term2c95final95new95c} \end{align}\tag{106}\] Combining 104 105 106 , we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_2&\lesssim \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}G^{\pi_t}}+H^2S^3A\iota^2. \label{eq:term295final95new95c} \end{align}\tag{107}\]
We finally bound \(\mathrm{\boldsymbol{term}}_3\). Here, for any transition \(\widetilde{P}_t\in\mathcal{P}_t\), we use \[\begin{align} \sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+ \sum_{s'}P(s'\mid s,a)q^{\widetilde{P}_t,\pi_t}(s'',a''\mid s') &\leq H\overline{q}_t^{\pi_t}(s'')\pi_t(a''\mid s''). \label{eq:term395splicing95bound} \end{align}\tag{108}\] By the definition of the confidence set, we have \[\begin{align} &\mathrm{\boldsymbol{term}}_3\\ &\lesssim \sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_t\brk*{\sum_{s'}\prn*{\sqrt{\frac{P(s'\mid s,a)\iota}{n_t(s,a)}}+\frac{\iota}{n_t(s,a)}}\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')} }\\ &\lesssim \sum_{t\in\mathcal{T}_r}\sum_{s,a,s'}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\prn*{\alpha P_t(s'\mid s,a)+\frac{\iota}{\alpha n_t(s,a)}}\sum_{s'',a''}\mathbb{E}_t\brk*{q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')} \\ &\lesssim \alpha\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\sum_{s'}P(s'\mid s,a)\mathbb{E}_t\brk*{\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')}}_{\mathrm{\boldsymbol{term}}_{3a}}\\ &\qquad+\frac{1}{\alpha}\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\frac{\iota}{n_t(s,a)}\mathbb{E}_t\brk*{\sum_{s'}\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')}}_{\mathrm{\boldsymbol{term}}_{3b}}. \end{align}\]
For \(\mathrm{\boldsymbol{term}}_{3a}\), using 108 with \(\widetilde{P}_t=\overline{P}_t^C\), \[\begin{align} \mathrm{\boldsymbol{term}}_{3a} &\leq H\sum_{t\in\mathcal{T}_r}\sum_{s,a}\overline{q}_t^{\pi_t}(s)\pi_t(a\mid s)\mathbb{E}_t[c_t(s,a)]\\ &\leq H\underbrace{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\prn*{\overline{q}_t^{\pi_t}(s)-\underline{q}_t^{\pi_t}(s)}\pi_t(a\mid s)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}_{\mathrm{\boldsymbol{term}}_{3a,1}}\\ &\qquad+H^2\underbrace{\sum_{t\in\mathcal{T}_r}\sum_s\gamma_t}_{\mathrm{\boldsymbol{term}}_{3a,2}}+H^2\underbrace{\sum_{t\in\mathcal{T}_r}\sum_s \overline{q}_t^{\pi_t}(s)\rho_t(s)^2}_{\mathrm{\boldsymbol{term}}_{3a,3}}. } \end{align}\] By 15 and the Cauchy–Schwarz inequality, \[\begin{align} \mathrm{\boldsymbol{term}}_{3a,1} &\lesssim \sqrt{HS^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_s q^{\pi_t}(s)\prn*{\sum_a\pi_t(a\mid s)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}^2}+H^2S^3A\iota^2\\ &\leq \sqrt{HS^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\prn*{\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\relax{\ell_t-m_t})}^2}+H^2S^3A\iota^2\\ &\leq \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn{\ell_t-m_t}^2)}+H^2S^3A\iota^2. } \end{align}\] Furthermore, by the definition of \(\gamma_t\) and 44, \[\begin{align} \mathrm{\boldsymbol{term}}_{3a,2} &\lesssim S\log(T),\\ \mathrm{\boldsymbol{term}}_{3a,3} &\lesssim HS^3A\iota^2. \end{align}\] Therefore, \[\begin{align} \mathrm{\boldsymbol{term}}_{3a} &\lesssim \sqrt{H^4S^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn{\ell_t-m_t}^2)}+H^3S^3A\iota^2. \label{eq:term3a95maxQ95bound} \end{align}\tag{109}\] For \(\mathrm{\boldsymbol{term}}_{3b}\), since \(q^{\overline{P}_t^C,\pi_t}(s'',a''\mid s')\leq \pi_t(a''\mid s'')\), \(\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+\leq q^{\pi_t}(s,a)\), we have \[\begin{align} \mathrm{\boldsymbol{term}}_{3b} &\leq \sum_{h=0}^{H-1}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\frac{q^{\pi_t}(s,a)\iota}{n_t(s,a)}\sum_{s'\in\mathcal{S}_{h+1}}\sum_{s'',a''}\pi_t(a''\mid s'')\mathbb{E}_t[c_t(s'',a'')]\\ &\lesssim HS\sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\frac{q^{\pi_t}(s,a)\iota}{n_t(s,a)} _t[c_t(s'',a'')] \leq 2H}\\ &\lesssim HS\sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\relax*{\mathcal{S}_h}A\iota^2 }\\ &\lesssim HS^3A\iota^2. \label{eq:term3b95count95bound} \end{align}\tag{110}\] Combining 109 and 110 , we obtain, for any \(\alpha\in(0,1]\), \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim \alpha\sqrt{H^4S^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn{\ell_t-m_t}^2)}\\ &\qquad+\alpha H^3S^3A\iota^2+\frac{HS^3A\iota^2}{\alpha}. \end{align}\] Choosing \(\alpha=1/H\) gives \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim \sqrt{H^2S^2A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn{\ell_t-m_t}^2)} + H^2S^3A\iota^2. \label{eq:term395maxQ95bound} \end{align}\tag{111}\]
We also derive a bound in terms of the positive occupancy difference. Recall that \(\mathrm{\boldsymbol{term}}_3\) can be upper-bounded by \[\begin{align} \sum_{t\in\mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\pi}(s,a)}_+\mathbb{E}_t\brk*{\sum_{s'}\prn*{\sqrt{\frac{P(s'\mid s,a)\iota}{n_t(s,a)}}+\frac{\iota}{n_t(s,a)}}\sum_{s'',a''}q^{\overline{P}^C_t,\pi_t}(s'',a''\mid s')c_t(s'',a'')}. \end{align}\] Since \(q^{\overline{P}_t^C,\pi_t}(s'',a''\mid s')\leq \pi_t(a''\mid s'')\), \(\sum_{s'\in\mathcal{S}_{h+1}}\sqrt{P(s'\mid s,a)}\leq \sqrt{|\mathcal{S}_{h+1}|}\), and \(\sum_{s'',a''}\pi_t(a''\mid s'')\mathbb{E}_t[c_t(s'',a'')]\lesssim HS\), we have \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim HS\underbrace{\sum_{h=0}^{H-1}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+\sqrt{\frac{|\mathcal{S}_{h+1}|\iota}{n_t(s,a)}}}_{\mathrm{\boldsymbol{term}}_{3c}}\\ &\qquad+HS\underbrace{\sum_{h=0}^{H-1}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+\frac{|\mathcal{S}_{h+1}|\iota}{n_t(s,a)}}_{\mathrm{\boldsymbol{term}}_{3d}}. \end{align}\] For \(\mathrm{\boldsymbol{term}}_{3c}\), by the Cauchy–Schwarz inequality and 7, \[\begin{align} \mathrm{\boldsymbol{term}}_{3c} &\leq \sum_{h=0}^{H-1}\sqrt{\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}\sqrt{\relax*{\mathcal{S}_{h+1}}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\frac{q^{\pi_t}(s,a)\iota}{n_t(s,a)}}\\ &\lesssim \sum_{h=0}^{H-1}\sqrt{\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}. \end{align}\] For \(\mathrm{\boldsymbol{term}}_{3d}\), by 7, \[\begin{align} \mathrm{\boldsymbol{term}}_{3d} &\leq \sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\frac{q^{\pi_t}(s,a)\iota}{n_t(s,a)}\\ &\lesssim \sum_{h=0}^{H-1}\relax*{\mathcal{S}_{h+1}}\relax*{\mathcal{S}_h}A\iota^2 \lesssim S^2A\iota^2. \end{align}\] Therefore, \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim \sum_{h=0}^{H-1}\sqrt{H^2S^2\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}+HS^3A\iota^2. \label{eq:term395positive95occupancy95layer95bound} \end{align}\tag{112}\] The desired bound follows by combining 82 , the definition of \(G^{\pi_t}\) in 89 , and the bounds in 97 107 111 112 , and then applying Jensen’s inequality. ◻
Lemma 47 (bandit setting). For any comparator policy \(\pi\), it holds that \[\begin{align} \mathbb{E}\brk*{\sum_s q^{\pi}(s)\textrm{\boldsymbol{error}}^{\pi}(s)} &\lesssim \sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}} + HS^3A\iota^2. \label{eq:bound95error951} \end{align}\qquad{(12)}\] Also, \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\textrm{\boldsymbol{error}}^{\pi}(s)}\\ &\lesssim \sum_{h = 0}^{H-1}\sqrt{H^2\relax*{S_{h}}\relax*{S_{h + 1}}A\iota^2 \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{(s,a) \in \mathcal{S}_h \times \mathcal{A}}\brk*{q^{\pi_t}(s,a) - q^{\pi}(s,a)}_{+}}} + H^2S^2A\iota^2. \label{eq:bound95error952} \end{align}\qquad{(13)}\]
Proof. The proof is identical to that of 33, with the loss sequence \(\ell_t\) replaced by the prediction sequence \(m_t\). The only additional point is that we first separate the contribution of virtual episodes.
\[\begin{align} &\sum_s q^{\pi}(s)\textrm{\boldsymbol{error}}^{\pi}(s) \\ &= \sum_s q^{\pi}(s) \sum_{t=1}^T\sum_{a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s, a; m_t) - Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)}\\ &\leq \sum_s q^{\pi}(s) \sum_{t\in\mathcal{T}_r}\sum_{a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s, a; m_t) - Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)} + 2H^2\relax{\mathcal{T}_v}\\ &\lesssim \sum_s q^{\pi}(s) \sum_{t\in\mathcal{T}_r}\sum_{a} \prn*{\pi_t(a\mid s) - \pi(a\mid s)} \prn*{Q^{\pi_t}(s, a; m_t) - Q^{\underline{P}^m_t, \pi_t}(s, a; m_t)} + H^3SA\log^2(T), \end{align}\] where we used 42. The resulting virtual-episode contribution is absorbed into the lower-order term. It remains to bound the first term in expectation, which follows from 33.
Applying the first bound of 33 with \(\ell_t\) replaced by \(m_t\) gives, on \(\mathcal{E}\), \[\begin{align} \sum_s q^\pi(s)\textrm{\boldsymbol{error}}^{\pi}(s) &\lesssim \sqrt{S^2A\iota^2Q_{\mathrm{trans}}^{\pi_{1:T}}(m)} +HS^3A\iota^2. \end{align}\] Taking expectations, using Jensen’s inequality, and combining with the trivial bound \(\mathcal{O}(H^2T)\) on \(\bar{\mathcal{E}}\) with \(\delta=\mathcal{O}(1/T^2)\), we obtain \[\begin{align} \mathbb{E}\brk*{\sum_s q^{\pi}(s)\textrm{\boldsymbol{error}}^{\pi}(s)} &\lesssim \sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}} +HS^3A\iota^2 +\delta H^2T\\ &\lesssim \sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}} +HS^3A\iota^2, \end{align}\] which proves ?? .
Similarly, applying the second bound of 33 with \(\ell_t\) replaced by \(m_t\) gives, on \(\mathcal{E}\), \[\begin{align} \sum_s q^\pi(s)\textrm{\boldsymbol{error}}^{\pi}(s) &\lesssim \sum_{h=0}^{H-1}\sqrt{H^2\relax*{S_h}\relax*{S_{h+1}}A\iota^2\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}\\ &\qquad+H^2S^2A\iota^2. \end{align}\] Taking expectations, applying Jensen’s inequality, and combining with the trivial bound \(\mathcal{O}(H^2T)\) on \(\bar{\mathcal{E}}\) with \(\delta=\mathcal{O}(1/T^2)\), we obtain \[\begin{align} &\mathbb{E}\brk*{\sum_s q^\pi(s)\textrm{\boldsymbol{error}}^{\pi}(s)}\\ &\lesssim \sum_{h=0}^{H-1}\sqrt{H^2\relax*{S_h}\relax*{S_{h+1}}A\iota^2\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}} +H^2S^2A\iota^2+\delta H^2T\\ &\lesssim \sum_{h=0}^{H-1}\sqrt{H^2\relax*{S_h}\relax*{S_{h+1}}A\iota^2\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^\pi(s,a)}_+}} +H^2S^2A\iota^2. \end{align}\] This proves ?? . ◻
Lemma 48 (bandit setting). It holds that \[\begin{align} \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s)\eta_t(s, a)\pi_t(a\mid s)^{2}C_t(s,a)^2} \lesssim H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2. \end{align}\]
Proof. By the decomposition of \(q_t(s)\pi_t(a\mid s)C_t(s,a)\) in 61 , we have \[\begin{align} &q_t(s)\pi_t(a\mid s)C_t(s,a)\\ &\leq 2\sum_{s',a'}\rho_t(s')\mathbb{I}_t(s',a')D_{t,h(s')}+2H\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2. \label{eq:squared95C95decomp} \end{align}\tag{113}\] Thus, we obtain \[\begin{align} \mathbb{E}_t\brk*{\sum_{s,a}q_t(s)^2\pi_t(a\mid s)^2C_t(s,a)^2} &\lesssim SA\mathbb{E}_t\brk*{\prn*{\sum_{s',a'}\rho_t(s')\mathbb{I}_t(s',a')D_{t,h(s')}}^2}\\ &\quad+H^2SA\prn*{\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2}^2. \label{eq:C-square-decomp} \end{align}\tag{114}\] For the first term of 114 , the Cauchy–Schwarz inequality gives \[\begin{align} \mathbb{E}_t\brk*{\prn*{\sum_{s',a'}\rho_t(s')\mathbb{I}_t(s',a')D_{t,h(s')}}^2} &\leq H\mathbb{E}_t\brk*{\sum_{s',a'}\rho_t(s')^2\mathbb{I}_t(s',a')D_{t,h(s')}^2}\\ &\leq H^3\mathbb{E}_t\brk*{\sum_{s',a'}\rho_t(s')^2\mathbb{I}_t(s',a')}\\ &=H^3\sum_{s'}q^{\pi_t}(s')\rho_t(s')^2\\ &\leq H^3\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2. \label{eq:first-C-square} \end{align}\tag{115}\] For the second term of 114 , since \[\begin{align} \sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2\leq S, \end{align}\] we have \[\begin{align} \prn*{\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2}^2 &\leq S\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2. \end{align}\] Hence, \[\begin{align} \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}q_t(s)^2\pi_t(a\mid s)^2C_t(s,a)^2} &\lesssim H^2S^2A\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s',a'}q_t(s')\pi_t(a'\mid s')\rho_t(s')^2}\\ &= H^2S^2A\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s'}q_t(s')\rho_t(s')^2}. \label{eq:squared95C95to95rho} \end{align}\tag{116}\]
Combining 44 with the trivial bound \[\begin{align} \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s'}q_t(s')\rho_t(s')^2} \lesssim HS^3A\iota^2+\delta\cdot \mathcal{O}(HT) \lesssim HS^3A\iota^2, \end{align}\] and 116 , we get \[\begin{align} \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}q_t(s)^2\pi_t(a\mid s)^2C_t(s,a)^2} &\lesssim H^2S^2A\cdot HS^3A\iota^2\\ &\lesssim H^3S^5A^2\iota^2. \label{eq:squared95C95final95bound} \end{align}\tag{117}\] Therefore, \[\begin{align} &\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \eta_t(s, a)\pi_t(a\mid s)^{2}C_t(s,a)^2} \\ &= \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \frac{\eta_t(s, a)q_t(s)^2\pi_t(a\mid s)^{2}C_t(s,a)^2}{q_t(s)^2}} \\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q_t(s)^2\pi_t(a\mid s)^{2}C_t(s,a)^2\max_{s,a}\frac{\eta_t(s, a)}{q_t(s)}} \\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q_t(s)^2\pi_t(a\mid s)^{2}C_t(s,a)^2} \cdot \frac{1}{50\sqrt{H^3S^3A}}\\ &\lesssim H^3S^5A^2\iota^2 \cdot \frac{1}{\sqrt{H^3S^3A}}.}\\ &= H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2. \end{align}\] ◻
Lemma 49 (bandit setting). It holds that \[\begin{align} &\mathbb{E}\brk*{\sum_{t=1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)}\lesssim \log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r} \zeta_t(s,a)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2. \end{align}\]
Proof. We have \[\begin{align} &\mathbb{E}\brk*{\sum_{t=1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)}\\ &= \mathbb{E}\brk*{\sum_{t=1}^T\sum_s q^{\overline{P}^B_t,\pi_t}(s) b_t(s)}\\ &\lesssim \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t,\pi_t}(s) \prn*{\frac{1}{\eta_{t+1}(s,a)}-\frac{1}{\eta_t(s,a)}}\log(T)} \\ &\qquad + \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_v}\sum_{s,a} q^{\overline{P}^B_t,\pi_t}(s) \prn*{\frac{1}{\eta_{t+1}(s,a)}-\frac{1}{\eta_t(s,a)}}\log(T)}\\ &\qquad + \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \eta_t(s, a)\pi_t(a\mid s)^{2}C_t(s,a)^2}\\ &\lesssim \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \frac{\eta_t(s,a)\zeta_t(s,a)}{q_t(s)^2} } + \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_v}\sum_{s,a} q^{\overline{P}^B_t, \pi_t}(s) \frac{\mathbb{1}\{(s^\dagger_t, a^\dagger_t) = (s,a)\}}{\eta_t(s,a)H}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2 }\\ &\leq \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} \frac{\eta_t(s,a)\zeta_t(s,a)}{q_t(s)}} + \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_v} \frac{q^{\overline{P}^B_t, \pi_t} (s_t^\dagger)}{\eta_t(s_t^\dagger,a_t^\dagger)H}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2\\ &\lesssim \mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a}\frac{\eta_t(s,a)\zeta_t(s,a)}{q_t(s)}} + \sum_{t\in\mathcal{T}_v} \sqrt{HS^3A}+ H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2 {q_t(s)} > \frac{1}{50\sqrt{H^3S^3A}} in virtual episodes} \\ &\lesssim \sqrt{\log(T)}\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r}\sum_{s,a} \frac{\frac{\sqrt{\zeta_t(s,a)}}{q_t(s)}\times \sqrt{\zeta_t(s,a)}}{\sqrt{\sum_{\tau\leq t: \tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2} }}} + \sqrt{HS^3A}\relax{\mathcal{T}_v} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2}\\ &\lesssim \sqrt{\log(T)}\mathbb{E}\brk*{\sum_{s,a}\sqrt{\sum_{t\in\mathcal{T}_r}\frac{\frac{\zeta_t(s,a)}{q_t(s)^2}}{ \sum_{\tau\leq t: \tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2}}}\sqrt{\sum_{t\in\mathcal{T}_r} \zeta_t(s,a)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2 }\\ &\lesssim \log(T)\mathbb{E}\brk*{\sum_{s,a}\sqrt{\sum_{t\in\mathcal{T}_{r}} \zeta_t(s,a)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2, \end{align}\] where the last inequality follows from \[\begin{align} \sqrt{\sum_{t\in\mathcal{T}_r}\frac{\frac{\zeta_t(s,a)}{q_t(s)^2}}{ \sum_{\tau\leq t:\tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2}}} \leq \sqrt{1 + \log\prn*{ \sum_{\tau\in\mathcal{T}_r} \frac{\zeta_\tau(s,a)}{q_\tau(s)^2}}} \lesssim \sqrt{\log(T)}. \end{align}\] Finally, the desired bound follows from Jensen’s inequality. ◻
We now summarize the bounds in the bandit setting proved above and plug them into the regret decomposition. By the definition of the regret decomposition in 59 , \[\begin{align} &\mathbb{E}\brk*{\sum_s q^{\pi}(s)\sum_{t,a} \prn*{\pi_t(a\mid s) - \pi(a\mid s) } \prn*{Q^{\pi_t}(s,a;\ell_t) - B_t(s,a)}}\\ &= \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{reg}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{error}}^{\pi}(s)}\\ &\leq \mathcal{O}\prn*{H^2S^2A\log(T)} + \mathbb{E}\brk*{\sum_{t=1}^Tq^{\pi}(s)b_t(s)} + \mathbb{E}\brk*{\frac{1}{H}\sum_{t=1}^T\sum_a q^{\pi}(s)\pi_t(a \mid s) B_t(s,a)}\\ &\qquad + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{error}}^{\pi}(s)} \end{align}\] where we used 45. Combining this with 28 and \(\delta =\frac{1}{T^2}\), for any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\leq \mathcal{O}\prn*{H^2S^2A\log(T)} + 3\mathbb{E}\brk*{\sum_{t = 1}^T V^{\overline{P}^B_t, \pi_t}(s_0;b_t)}\\ &\qquad + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{error}}^{\pi}(s)}\\ &\lesssim H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2 + \log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t\in\mathcal{T}_r} \zeta_t(s,a)}}\\ &\qquad + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{bias}}^{\pi}(s)} + \mathbb{E}\brk*{\sum_s q^{\pi}(s) \cdot \textrm{\boldsymbol{error}}^{\pi}(s)}, \label{eq:regret95sum95bandit} \end{align}\tag{118}\] where the second line follows from 49.
Theorem 50 (bandit setting). For any comparator policy \(\pi\), 1 guarantees \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^3S^2A\log^2(T) \min\set*{L(\pi), HT - L(\pi), Q_{\infty}, V_1}}\\ &\qquad+\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \end{align}\]
Proof. By 118 46 47, for any comparator policy \(\pi\), we have \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\iota^2+\log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\zeta_t(s,a)}}\\ &\qquad+\sqrt{H^2S^2A\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{s,a}q^{\pi_t}(s,a)\max_{\widetilde{P}\in\mathcal{P}_t}Q^{\widetilde{P},\pi_t}(s,a;\prn*{\ell_t-m_t}^2)}}\\ &\qquad+\sqrt{S^2A\iota^2\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}}. \end{align}\] Here, since \(\delta=\mathcal{O}(1/T^2)\), we have \(\iota\lesssim\log(T)\). Hence, \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)+\underbrace{\log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\zeta_t(s,a)}}}_{\mathrm{\boldsymbol{term}}_1}\\ &\qquad+\underbrace{\sqrt{H^3S^2A\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}V^{\pi_t}(s_0;\prn*{\ell_t-m_t}^2)}}}_{\mathrm{\boldsymbol{term}}_2}\\ &\qquad+\underbrace{\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(m)}}}_{\mathrm{\boldsymbol{term}}_3}, \label{eq:bandit95regret95decomp95adv} \end{align}\tag{119}\] where the last inequality uses 17 on \(\mathcal{E}\) together with the trivial bound \(\mathcal{O}(H^2T)\) on \(\bar{\mathcal{E}}\), whose contribution is absorbed since \(\delta=\mathcal{O}(1/T^2)\).
For \(\mathrm{\boldsymbol{term}}_1\), by the definition of \(\zeta_t(s,a)\), we have \[\begin{align} \mathrm{\boldsymbol{term}}_1 &= \sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r} (\mathbb{I}_t(s,a) - \pi_t(a\mid s) \mathbb{I}_t(s))^2(L_{t,h(s)} - M_{t,h(s)})^2}}\\ &\leq \sqrt{SA\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{s,a} (\mathbb{I}_t(s,a) - \pi_t(a\mid s) \mathbb{I}_t(s))^2(L_{t,h(s)} - M_{t,h(s)})^2}}\\ &\leq \sqrt{SA\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{s,a} 2\mathbb{I}_t(s,a)(L_{t,h(s)} - M_{t,h(s)})^2}} \\ &\lesssim \sqrt{H^2SA\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{s,a}\mathbb{I}_t(s,a)(\ell_t(s,a) - m_t(s,a))^2}}, \label{eq:common95bound95adv} \end{align}\tag{120}\] where the third line uses \(\sum_a(\mathbb{I}_t(s,a)-\pi_t(a\mid s)\mathbb{I}_t(s))^2\le 2\mathbb{I}_t(s)\) for each fixed state \(s\), and the last inequality follows from 18.
Applying 19 to 120 , we obtain, for any policy \(\pi\), \[\begin{align} \mathrm{\boldsymbol{term}}_1 &\lesssim \sqrt{H^2SA\log^2(T)\min\set*{L(\pi) + \text{\rm Reg}_T(\pi), HT - L(\pi) - \text{\rm Reg}_T(\pi), Q_{\infty}, V_1}}\\ &\qquad + HSA\log(T).\label{eq:adv95bandit95term1} \end{align}\tag{121}\]
For \(\mathrm{\boldsymbol{term}}_2\), 19 similarly yields, for any policy \(\pi\), \[\begin{align} \mathrm{\boldsymbol{term}}_2 &\lesssim \sqrt{H^3S^2A\log^2(T)\min\set*{L(\pi) + \text{\rm Reg}_T(\pi), HT - L(\pi) - \text{\rm Reg}_T(\pi), Q_{\infty}, V_1}}\\ &\qquad + \sqrt{H^3S^3}A\log(T).\label{eq:adv95bandit95term2} \end{align}\tag{122}\] For \(\mathrm{\boldsymbol{term}}_3\), 20 yields \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}+\sqrt{HS^2A\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}V^{\pi_t}(s_0;\prn*{\ell_t-m_t}^2)}}. \end{align}\] Applying 19 to the second term, we obtain, for any policy \(\pi\), \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\lesssim \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}\\ &\qquad +\sqrt{HS^2A\log^2(T)\min\set*{L(\pi)+\text{\rm Reg}_T(\pi),HT-L(\pi)-\text{\rm Reg}_T(\pi),Q_\infty,V_1}}\\ &\qquad+\sqrt{HS^3}A\log(T). \label{eq:adv95bandit95term3} \end{align}\tag{123}\] Combining 121 122 123 with 119 , we obtain \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^3S^2A\log^2(T)\min\set*{L(\pi)+\text{\rm Reg}_T(\pi),HT-L(\pi)-\text{\rm Reg}_T(\pi),Q_\infty,V_1}}\\ &\qquad+\sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}+H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \label{eq:adv95bandit95mid} \end{align}\tag{124}\] Below, we derive the first-order bound. By 124 , there exists an absolute constant \(c>0\) such that \[\begin{align} \text{\rm Reg}_T(\pi) &\leq c\sqrt{H^3S^2A \log^2(T) L(\pi)}+c\sqrt{H^3S^2A \log^2(T) \text{\rm Reg}_T(\pi)}\\ &\qquad + c\sqrt{S^2A\log^2(T) \mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + cH^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)\\ &\leq c\sqrt{H^3S^2A \log^2(T) L(\pi)}+\frac{c^2}{2}H^3S^2A \log^2(T) + \frac{1}{2} \text{\rm Reg}_T(\pi)\\ &\qquad + c\sqrt{S^2A\log^2(T) \mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + cH^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T), \end{align}\] where the last inequality follows from the AM–GM inequality. Thus, we obtain \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^3S^2A \log^2(T) L(\pi)} + \sqrt{S^2A\log^2(T) \mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \label{eq:first95order95bandit1} \end{align}\tag{125}\] Furthermore, we derive the bound in terms of \(HT-L(\pi)\). If \(\text{\rm Reg}_T(\pi)<0\), the desired upper bound is trivial. Otherwise, \(\text{\rm Reg}_T(\pi)\ge 0\) and 124 gives \[\begin{align} \text{\rm Reg}_T(\pi) &\lesssim \sqrt{H^3S^2A\log^2(T)\prn*{HT-L(\pi)-\text{\rm Reg}_T(\pi)}} + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}}\\ &\qquad + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)\\ &\leq \sqrt{H^3S^2A\log^2(T)\prn*{HT-L(\pi)}} + \sqrt{S^2A\log^2(T)\mathbb{E}\brk*{Q_{\mathrm{trans}}^{\pi_{1:T}}(\ell)}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \label{eq:first95order95bandit2} \end{align}\tag{126}\]
Theorem 51 (bandit setting). Under the stochastic regime with adversarial corruption, 1 with \(\delta=\frac{1}{T^2}\) guarantees that, for any policy \(\pi\), \[\begin{align} \text{\rm Reg}_T(\pi) \lesssim U + \sqrt{U\mathcal{C}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T), \end{align}\] where \(U = \frac{H^2S^4A\log^2(T)}{\Delta_{\min}}\).
Proof. Since \(\text{\rm Reg}_T(\pi) \leq \text{\rm Reg}_T(\mathring{\pi})\) for all \(\pi\), it suffices to bound \(\text{\rm Reg}_T(\mathring{\pi})\). By 118 46 47 and \(\delta=\mathcal{O}(1/T^2)\), we have \(\iota\lesssim\log(T)\). Hence, there exists an absolute constant \(c>0\) such that \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) &\leq cH^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)+\underbrace{c\log(T)\sum_{s,a}\sqrt{\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\zeta_t(s,a)}}}_{\mathrm{\boldsymbol{term}}_1}\\ &\qquad+\underbrace{c\sqrt{H^3S^2A\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{s,a}\brk*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}_+}}}_{\mathrm{\boldsymbol{term}}_2}\\ &\qquad+\underbrace{c\sum_{h=0}^{H-1}\sqrt{H^2S^2\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}\sum_{(s,a)\in\mathcal{S}_h\times\mathcal{A}}\brk*{q^{\pi_t}(s,a)-q^{\mathring{\pi}}(s,a)}_+}}}_{\mathrm{\boldsymbol{term}}_3}. \label{eq:bandit95regret95decomp95sto} \end{align}\tag{127}\] For \(\mathrm{\boldsymbol{term}}_1\), by the definition of \(\zeta_t(s,a)\), we have for any \(\alpha>0\), \[\begin{align} \mathrm{\boldsymbol{term}}_1 &= c\sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}(\mathbb{I}_t(s,a)-\pi_t(a\mid s)\mathbb{I}_t(s))^2(L_{t,h(s)}-M_{t,h(s)})^2}}\\ &\leq cH\sum_{s,a}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}q^{\pi_t}(s,a)(1-\pi_t(a\mid s))}}\\ &\leq 2cH\sum_s\sum_{a\neq{\pi^\star}(s)}\sqrt{\log^2(T)\mathbb{E}\brk*{\sum_{t \in \mathcal{T}_r}q^{\pi_t}(s,a)}}\\ &\leq \alpha\prn*{\text{\rm Reg}_T(\mathring{\pi})+2\mathcal{C}}+\sum_s\sum_{a\neq{\pi^\star}(s)}\frac{c^2H^2\log^2(T)}{4\alpha\Delta(s,a)}, \label{eq:sto95bandit95term1} \end{align}\tag{128}\] where the last line follows from 24.
For \(\mathrm{\boldsymbol{term}}_2\), 26 gives \[\begin{align} \mathrm{\boldsymbol{term}}_2 &\leq \alpha\prn*{\text{\rm Reg}_T(\mathring{\pi})+4\mathcal{C}}+\frac{c^2H^4S^2A\log^2(T)}{4\alpha\Delta_{\min}}. \label{eq:sto95bandit95term2} \end{align}\tag{129}\]
For \(\mathrm{\boldsymbol{term}}_3\), 25 gives \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\leq \alpha\prn*{\text{\rm Reg}_T(\mathring{\pi})+4\mathcal{C}}+\frac{c^2\prn*{\sum_{h=0}^{H-1}\sqrt{H^2S^2\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\log^2(T)}}^2}{4\alpha\Delta_{\min}}. \end{align}\] Moreover, by the AM–GM inequality, \[\begin{align} \prn*{\sum_{h=0}^{H-1}\sqrt{H^2S^2\relax*{\mathcal{S}_h}\relax*{\mathcal{S}_{h+1}}A\log^2(T)}}^2 &\leq \prn*{\sum_{h=0}^{H-1}\frac{1}{2}\prn*{\relax*{\mathcal{S}_h}+\relax*{\mathcal{S}_{h+1}}}\sqrt{H^2S^2A\log^2(T)}}^2\\ &\leq \prn*{\sqrt{H^2S^4A\log^2(T)}}^2 = H^2S^4A\log^2(T). \end{align}\] Thus, we obtain \[\begin{align} \mathrm{\boldsymbol{term}}_3 &\leq \alpha\prn*{\text{\rm Reg}_T(\mathring{\pi})+4\mathcal{C}}+\frac{c^2H^2S^4A\log^2(T)}{4\alpha\Delta_{\min}}. \label{eq:sto95bandit95term3} \end{align}\tag{130}\] Combining 128 129 130 with 127 , we obtain \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) &\leq \alpha(3\text{\rm Reg}_T(\mathring{\pi}) + 10\mathcal{C}) + \mathcal{O}\prn*{\frac{H^2S^4A\log^2(T)}{4\alpha\Delta_{\min}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T)}. \end{align}\] Finally, choosing \(\alpha = \min\set*{\frac{1}{6}, \sqrt{\frac{U}{\mathcal{C}}}}\) with \(U = \frac{H^2S^4A\log^2(T)}{\Delta_{\min}}\), we get \[\begin{align} \text{\rm Reg}_T(\mathring{\pi}) \lesssim U + \sqrt{U\mathcal{C}} + H^{\frac{3}{2}}S^{\frac{7}{2}}A^{\frac{3}{2}}\log^2(T). \end{align}\] Therefore, since \(\text{\rm Reg}_T(\pi)\leq \text{\rm Reg}_T(\mathring{\pi})\) for all \(\pi\) by the definition of \(\mathring{\pi}\), the desired bound follows. ◻