1 Why IRL and DDC?↩︎

In the forward reinforcement-learning problem, the reward function is a fixed, known input. The world handed us \(r\) and asked us to learn \(Q^\star\). In a great many real-world problems, however, \(r\) is exactly what we want to learn, and the closest thing we have to it is a log of decisions made by someone who knew what they were doing.

• A labor economist watches workers choose between job offers and wants to recover the utility function that rationalizes their choices [10].

• A marketing analyst sees customers’ coupon redemptions and purchase histories, and wants to recover preferences in order to predict response to a new offer.

• A health-economics researcher observes physicians’ treatment decisions and wants to quantify the implicit risk-benefit tradeoffs they are making.

• A robotics or autonomous-driving engineer watches expert pilots fly a helicopter or expert drivers navigate traffic, and wants to extract a reward function that, when fed to a downstream RL algorithm, reproduces the expert’s behavior [11].

In each case the data is offline: a fixed log of state-action-(next-state) tuples from one or more experts, and the reward itself is never directly observed [12].

Two literatures have been chipping away at this problem for decades, under almost identical probabilistic structure but completely different vocabularies:

• Dynamic discrete choice (DDC) models from structural econometrics [2], [13], [14]: the agent’s reward is observed plus a private idiosyncratic shock, generally Gumbel-distributed, which the econometrician integrates out.

• Inverse reinforcement learning (IRL) with the entropy-regularized (maximum-entropy) variant of [5], [15]: the agent maximizes expected return plus an entropy bonus on its policy, producing the same softmax choice rule.

Our first order of business (Section 2) is to prove these are the same model. Sections 3–5 then survey the methods each tradition developed, and Section 6 presents the unified ERM treatment of [9] that fixes the issues both traditions inherited.

2 Preliminaries↩︎

This section sets up the model, defines the soft Bellman operator, proves its contraction, and, most importantly, proves the equivalence between the DDC and entropy-regularized IRL formulations. Readers familiar with standard MDP notation can skim the MDP setup, but should not skip Section 2.6 (the Gumbel-max identity) or Section 2.7 (the equivalence theorem).

2.1 Markov Decision Processes↩︎

We consider an infinite-horizon discounted MDP \(\mathcal{M}= (\mathcal{S}, \mathcal{A}, P, \nu_0, r, \beta)\) where \(\mathcal{S}\) is a (possibly continuous) state space, \(\mathcal{A}\) is a finite action space, \(P : \mathcal{S}\times \mathcal{A}\to \Delta(\mathcal{S})\) is the transition kernel, \(\nu_0 \in \Delta(\mathcal{S})\) is the initial distribution, \(r : \mathcal{S}\times \mathcal{A}\to [-R_{\max}, R_{\max}]\) is the (unknown) reward, and \(\beta \in (0, 1)\) is the discount. A stationary Markov policy is a measurable map \(\pi : \mathcal{S}\to \Delta(\mathcal{A})\) with \(\pi(a \mid s)\) the probability of \(a\) at \(s\). We write \(\mathbb{P}_{\nu_0, \pi}\) and \(\mathbb{E}_{\nu_0, \pi}\) for the distribution and expectation over the resulting trajectory \(\tau = (s_0, a_0, s_1, a_1, \ldots)\).

The discounted state-action occupancy of \(\pi\) is \[d^\pi(s, a) \;:=\; (1 - \beta) \sum_{t=0}^\infty \beta^t\, \mathbb{P}_{\nu_0, \pi}(s_t = s,\, a_t = a),\] a probability measure on \(\mathcal{S}\times \mathcal{A}\). The offline dataset is \[\mathcal{D}\;=\; \bigl\{(s_i, a_i, s_i')\bigr\}_{i=1}^N,\] sampled iid from \(d^{\pi^\star}\) for a single expert policy \(\pi^\star\). Note what is missing: we do not observe rewards in \(\mathcal{D}\): only states, actions, and next states. That omission is precisely what makes the problem “inverse,” and it is the canonical setup of both DDC and IRL.

2.2 Function Spaces, Norms, Projections↩︎

Let \(\mathcal{Q}:= \{Q : \mathcal{S}\times \mathcal{A}\to \mathbb{R}\;\text{measurable, bounded}\}\). The sup-norm is \(\|Q\|_\infty := \sup_{s, a} |Q(s,a)|\). For \(\rho \in \Delta(\mathcal{S}\times \mathcal{A})\) we write \(\|Q\|_\rho^2 := \mathbb{E}_{(s,a) \sim \rho}[Q(s,a)^2]\) and \(\langle Q_1, Q_2 \rangle_\rho := \mathbb{E}_\rho[Q_1 Q_2]\), so \(L^2(\rho)\) is a Hilbert space. For a closed linear subspace \(\mathcal{F}\subseteq L^2(\rho)\), the \(L^2(\rho)\)-projection is the unique element \(\Pi_{\mathcal{F}, \rho} Q \in \mathcal{F}\) satisfying \(\|Q-\Pi_{\mathcal{F}, \rho}Q\|_\rho = \inf_{f \in \mathcal{F}}\|Q-f\|_\rho\). This restriction to closed linear subspaces matters: projections onto arbitrary closed nonconvex sets may fail to be unique. For closed linear subspaces, \(\Pi_{\mathcal{F}, \rho}\) is a non-expansion: \(\|\Pi Q_1 - \Pi Q_2\|_\rho \le \|Q_1 - Q_2\|_\rho\) for all \(Q_1, Q_2\). We will need this fact, both the projection itself and its non-expansion property, when analyzing temporal-difference methods in Section 4.4.

2.3 The Dynamic Discrete Choice Formulation↩︎

The DDC setup follows the Type 1 Extreme Value (T1EV) derivation in [9]. At each period the agent observes the deterministic flow payoff \(r(s_t,a_t)\) plus an idiosyncratic alternative-specific shock \(\epsilon_{t,a_t}\). We use the unit-scale Gumbel distribution with location parameter \(\delta\): \[\begin{align} F_\delta(x)&=\exp\!\bigl(-\exp(-(x-\delta))\bigr),\tag{1}\\ f_\delta(x)&=\exp\!\bigl(-((x-\delta)+\exp(-(x-\delta)))\bigr),\tag{2}\\ \mathbb{E}[\epsilon]&=\delta+\gamma_E=:\mu_\epsilon, \tag{3} \end{align}\] where \(\gamma_E\) is the Euler–Mascheroni constant. The mean-zero T1EV normalization used for the baseline DDC–MaxEnt equivalence is \[\delta=-\gamma_E, \qquad \mu_\epsilon=0.\]

The DDC agent maximizes \[\pi^\star \in \operatorname*{arg\,max}_{\pi}\; \mathbb{E}_{\nu_0,\pi,G}\!\left[ \sum_{t=0}^\infty \beta^t\bigl(r(s_t,a_t)+\epsilon_{t,a_t}\bigr) \right], \label{eq:ddc-objective}\tag{4}\] where the expectation averages over the Markov chain and the idiosyncratic shocks. The policy is induced by choosing the alternative with the largest shock-augmented choice-specific value. The derivation in Section 2.6 shows that, under the mean-zero normalization \(\delta=-\gamma_E\), this DDC problem implies \[\begin{align} \pi^\star(a\mid s) &=\frac{\exp(Q^\star(s,a))}{\sum_{a'\in\mathcal{A}}\exp(Q^\star(s,a'))}, \tag{5}\\ Q^\star(s,a) &=r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\!\left[ \log\sum_{a'\in\mathcal{A}}\exp(Q^\star(s',a')) \right]. \tag{6} \end{align}\] If \(\delta\neq-\gamma_E\), the same derivation gives the same softmax rule but adds the constant \(\beta(\delta+\gamma_E)\) to the right side of the \(Q\) Bellman equation.

2.4 The Entropy-Regularized IRL Formulation↩︎

The MaxEnt-IRL setup in [9] instead treats the agent as choosing a stochastic stationary policy and explicitly rewards entropy. With regularization coefficient \(\lambda\), the forward problem is \[\pi^\star \in \operatorname*{arg\,max}_{\pi}\; \mathbb{E}_{\nu_0,\pi}\!\left[ \sum_{t=0}^\infty \beta^t\bigl(r(s_t,a_t)+\lambda\operatorname{H}(\pi(\cdot\mid s_t))\bigr) \right], \label{eq:irl-objective}\tag{7}\] where \(\operatorname{H}(\pi(\cdot\mid s)):=-\sum_a\pi(a\mid s)\log\pi(a\mid s)\). Throughout the main text, and in the equivalence theorem below, we set \[\lambda=1,\] which is the entropy scale corresponding to unit-scale T1EV shocks. Section 2.6 derives the same softmax rule 5 and soft Bellman equation 6 from this entropy-regularized problem.

2.5 The Soft Bellman Operator↩︎

The log-sum-exp is a smooth surrogate for \(\max\), sandwiched between it and a \(\log|\mathcal{A}|\) slack: \(\max_{a'} Q(s', a') \le \operatorname{LSE}(Q(s', \cdot)) \le \max_{a'} Q(s', a') + \log|\mathcal{A}|\). The two facts below are what we need to treat \(\mathcal{T}_{\mathrm{soft}}\) as well-behaved as the ordinary hard Bellman operator.

By Banach’s fixed-point theorem on \((\mathcal{Q}, \|\cdot\|_\infty)\), \(\mathcal{T}_{\mathrm{soft}}\) has a unique fixed point \(Q^\star\) and soft value iteration \(Q_{k+1} := \mathcal{T}_{\mathrm{soft}} Q_k\) converges geometrically: \(\|Q_k - Q^\star\|_\infty \le \beta^k \|Q_0 - Q^\star\|_\infty\). So far this is identical to the hard-Bellman story; the action is in what comes next.

2.6 DDC–MaxEnt-IRL Equivalence↩︎

2.6.0.1 MaxEnt-IRL derivation.

Define the value and choice-specific value functions by \[\begin{align} V^\star(s) &:=\mathbb{E}_{\pi^\star}\!\left[\sum_{h=0}^\infty\beta^h \bigl(r(s_h,a_h)+\operatorname{H}(\pi^\star(\cdot\mid s_h))\bigr)\;\middle|\;s_0=s\right],\\ Q^\star(s,a)&:=r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[V^\star(s')]. \end{align}\] At a fixed state \(s\), the Bellman recursion is the finite-dimensional entropy-regularized choice problem \[\begin{align} V^\star(s) &=\max_{q\in\Delta(\mathcal{A})} \left\{ \sum_{a\in\mathcal{A}}q_a\left(r(s,a)+\beta\mathbb{E}[V^\star(s')\mid s,a]\right)+\operatorname{H}(q) \right\} \notag\\ &=\max_{q\in\Delta(\mathcal{A})} \left\{\sum_{a\in\mathcal{A}}q_aQ^\star(s,a)+\operatorname{H}(q)\right\}. \label{eq:irl-bellman-v-choice} \end{align}\tag{8}\] By Lemma 5, \[\begin{align} V^\star(s)&=\log\sum_{a\in\mathcal{A}}\exp(Q^\star(s,a)), \tag{9}\\ \pi^\star(a\mid s)&=\frac{\exp(Q^\star(s,a))}{\sum_{a'\in\mathcal{A}}\exp(Q^\star(s,a'))}. \tag{10} \end{align}\] Substituting 9 into the definition of \(Q^\star\) gives the choice-specific Bellman equation \[Q^\star(s,a) =r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\!\left[ \log\sum_{a'\in\mathcal{A}}\exp(Q^\star(s',a')) \right]. \label{eq:irl-q-lse-bellman}\tag{11}\] The paper also records the equivalent continuation representation. Since 10 implies \[Q^\star(s',a')-\log\pi^\star(a'\mid s') =\log\sum_{b\in\mathcal{A}}\exp(Q^\star(s',b)) \qquad \text{for every }a',\] Equation 11 can be written as \[Q^\star(s,a) =r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi^\star(\cdot\mid s')}\!\left[ Q^\star(s',a')-\log\pi^\star(a'\mid s') \right]. \label{eq:irl-q-policy-continuation}\tag{12}\] This is the analogue of Equation (A5) in the paper.

2.6.0.2 DDC derivation.

Let \(\epsilon_{t,a}\overset{\mathrm{i.i.d.}}{\sim}\operatorname{Gumbel}(\delta,1)\) across actions and dates. The shock-realized value and its shock average are \[\begin{align} V^\star(s,\epsilon) &:=\max_{a\in\mathcal{A}}\left\{r(s,a)+\epsilon_a+\beta\,\mathbb{E}[\bar V^\star(s')\mid s,a]\right\},\\ \bar V^\star(s)&:=\mathbb{E}_\epsilon[V^\star(s,\epsilon)], \end{align}\] and the DDC choice-specific value is \[Q^\star(s,a):=r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a)}[\bar V^\star(s')]. \label{eq:ddc-q-definition-paper-style}\tag{13}\] Thus \[V^\star(s,\epsilon)=\max_{a\in\mathcal{A}}\{Q^\star(s,a)+\epsilon_a\}. \label{eq:ddc-shock-realized-value}\tag{14}\] For each action \(a\), \(Q^\star(s,a)+\epsilon_a\) is \(\operatorname{Gumbel}(Q^\star(s,a)+\delta,1)\). Lemma 4 gives \[\begin{align} \bar V^\star(s) &=\delta+\gamma_E+\log\sum_{a\in\mathcal{A}}\exp(Q^\star(s,a)), \tag{15}\\ \pi^\star(a\mid s) &=\mathbb{P}\left\{a=\operatorname*{arg\,max}_{b\in\mathcal{A}}(Q^\star(s,b)+\epsilon_b)\right\} =\frac{\exp(Q^\star(s,a))}{\sum_{a'\in\mathcal{A}}\exp(Q^\star(s,a'))}. \tag{16} \end{align}\] Substituting 15 into 13 yields \[Q^\star(s,a) =r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a)}\!\left[ \log\sum_{a'\in\mathcal{A}}\exp(Q^\star(s',a'))+\delta+\gamma_E \right]. \label{eq:ddc-q-bellman-delta}\tag{17}\] Under the mean-zero T1EV normalization \(\delta=-\gamma_E\), this becomes exactly 6 . This is Equation (3) in the paper.

The same DDC Bellman equation can also be written in the policy-continuation form used in the paper. Conditional on the selected next action \(a'\), Lemma 4 gives \[\mathbb{E}\left[Q^\star(s',a')+\epsilon_{a'}\;\middle|\;a'=\operatorname*{arg\,max}_b(Q^\star(s',b)+\epsilon_b)\right] =Q^\star(s',a')+\delta+\gamma_E-\log\pi^\star(a'\mid s').\] Therefore \[Q^\star(s,a) =r(s,a)+\beta\,\mathbb{E}_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi^\star(\cdot\mid s')}\!\left[ Q^\star(s',a')+\delta+\gamma_E-\log\pi^\star(a'\mid s') \right]. \label{eq:ddc-q-policy-continuation}\tag{18}\] When \(\delta=-\gamma_E\), Equation 18 is identical to the MaxEnt-IRL continuation representation 12 .

2.7 The DDC–IRL Equivalence Theorem↩︎

2.8 The Sample Soft Bellman Operator and Double Sampling↩︎

Given \(\mathcal{D}= \{(s_i, a_i, s_i')\}_{i=1}^N\), the sample soft Bellman operator is \[\hat{\mathcal{T}}_{\mathrm{soft}} Q(s, a, s') \;:=\; r(s, a) + \beta\,\operatorname{LSE}(Q(s', \cdot)).\] For each fixed \((s, a)\), \(\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[\hat{\mathcal{T}}_{\mathrm{soft}} Q(s, a, s')] = \mathcal{T}_{\mathrm{soft}} Q(s, a)\): one sample is unbiased for the operator’s value. Squaring breaks this. Define \[\begin{align} \mathcal{L}_{\mathrm{BE}}(Q)(s, a) &\;:=\; \bigl(Q(s, a) - \mathcal{T}_{\mathrm{soft}} Q(s, a)\bigr)^{\!2},\tag{19}\\ \mathcal{L}_{\mathrm{TD}}(Q)(s, a, s') &\;:=\; \bigl(Q(s, a) - \hat{\mathcal{T}}_{\mathrm{soft}} Q(s, a, s')\bigr)^{\!2},\tag{20} \end{align}\] and write \(V_Q(s') := \operatorname{LSE}(Q(s', \cdot))\).

The discrepancy \(\beta^2 \operatorname{Var}_{s' \sim P(\cdot \mid s, a)}(V_Q(s'))\) is the double-sampling bias. It is a \(Q\)-dependent population quantity that does not vanish with more data, so its gradient is nonzero at \(Q^\star\). Minimizing the empirical squared TD residual is not the same as minimizing the population Bellman error. Two independent next-state samples from the same \((s, a)\) would suffice to break the bias, but offline RL never gives us two next-states at the same \((s, a)\). Section 6.4 fixes this by deriving the bi-conjugate correction explicitly.

With the equivalence and the sampling subtlety in hand, we proceed to identification (Section 3), the classical DDC methods (Section 4), the IRL methods (Section 5), and the unified ERM treatment (Section 6).

3 Identification: What Can Be Recovered from Behavior Alone?↩︎

Before discussing how to estimate the reward, we have to confront a more basic question: is the reward identifiable from offline expert data at all?

Without further structure, the answer is no. This is not a quirk of any particular estimator; it is a fact about the model. We make it precise, then describe the two normalizations, one from econometrics and one from machine learning, that resolve it, and assess what each delivers [1], [13], [16].

3.1 The Identification Failure↩︎

Suppose two reward functions \(r\) and \(r'\) produce, via the soft Bellman equation, the same optimal policy \(\pi^\star\) and the same trajectory distribution under \(P\). Then the offline expert data \(\mathcal{D}\) has identical distribution under \((r, \pi^\star)\) and \((r', \pi^\star)\), and no estimator based on \(\mathcal{D}\) alone can possibly distinguish them. Worse, the indeterminacy is large: the soft-Bellman fixed point is invariant under state-only potential-based shaping in the sense of [17], and we can write down explicitly an entire family of rewards that all produce the same expert behavior. This non-identification was established in the DDC framework by [1] (their Proposition 1); we restate it in the entropy-regularized language for self-containedness.

[1] proved that, in the DDC framework, this is essentially the only non-identification: any two per-period utilities producing the same choice probabilities under the same \((P, \beta)\) differ by a state-only potential. [16] gave the corresponding result explicitly in the entropy-regularized MDP language. To make the result fully concrete:

Lemma 7 establishes the “if” direction; the “only if” is the content of [1] and [16]. The structural takeaway is the following:

From expert behavior alone, the reward function is identifiable only up to a state-dependent potential. To identify it any more sharply, we need additional structure: either restrictions on the form of \(r\), or normalizations that fix the potential.

The DDC and modern-IRL literatures take very different roads at exactly this fork. We examine both.

3.2 The Anchor-Action Assumption (Magnac–Thesmar)↩︎

The classical DDC route is to declare the value of \(r\) at one designated action per state [1].

In the Rust bus-engine application, the anchor action is “replace the engine”; its payoff (the cost of a new engine) is part of the model specification, not something to be estimated from data. The anchor action is allowed to depend on \(s\).

A small but important point: the anchor-action assumption is a normalization, not a restriction. It fixes the location of the reward without restricting which preferences over \((s, a)\) pairs the model can represent. It is the standard identification device in structural econometrics for DDC [1], [13], [14], [18].

3.3 The Adversarial-IRL Route and Its Limits↩︎

The machine-learning tradition takes a different fork at the identification problem of Lemma 7. The DDC route (Section 3.2) fixes the shaping potential by fiat: declare one action’s reward known at every state. Adversarial IRL [5] instead tries to learn the split between reward and potential, the state-dependent term whose Bellman difference can change the numerical reward without changing behavior, by building the potential-based shaping invariance of [17] directly into the structure of a discriminator. This subsection asks how much that buys.

The guiding distinction is simple. From transitions \((s,a,s')\), expert behavior directly reveals which actions look better relative to other actions at the same state; it does not directly reveal how much of that attractiveness comes from immediate reward rather than continuation value. AIRL is an attempt to learn this split: one part should be the transferable reward, and the other part should be the shaping/value correction. The question is whether the split is actually identified.

The AIRL language and objective.↩︎

[5] build on the MaxEnt-IRL framework of [15] and the generative-adversarial framework of [6], [19]. AIRL introduces a binary classifier \(D\), called the discriminator. Given a transition \(x=(s,a,s')\), \(D(x)\) is trained to be large on expert transitions and small on transitions generated by the current policy \(\pi\). Before writing \(D\) itself, AIRL specifies the reward-like score that will enter the classifier. That score is constrained to the potential-shaped form \[f_{g,h}(s, a, s') \;:=\; g(s) \;+\; \beta\, h(s') \;-\; h(s), \label{eq:airl-parametrization}\tag{26}\] where \(g\) is intended as the transferable reward and \(h\) as the shaping potential, which is equivalently the soft value term that should telescope away. In words, AIRL writes the expert-like score as \[\text{reward part} + \text{future potential} - \text{current potential}.\] The hope is that the reward part \(g\) survives transfer, while the potential terms only correct for where the transition starts and ends. We write the state-only reward-head version here because it is the version under which AIRL’s disentanglement claim can be true; if \(g\) is allowed to depend on \((s,a)\), the same ambiguity returns immediately. In the terminology of [16], “state-only” is an action-independent reward restriction. It is a real identifying restriction: recovering such a reward also needs decomposability conditions on how actions move between states, not only the adversarial objective.

This potential-shaped quantity is only a score. To turn such a score into a classifier, the adversarial-IRL derivation starts from the standard GAN/Bayes-classifier density-ratio identity. For a binary classifier trained to classify expert samples against current-policy samples, the Bayes classifier with equal class priors has \[D^\star(x)=\frac{p_E(x)}{p_E(x)+p_\pi(x)}, \qquad \log\frac{D^\star(x)}{1-D^\star(x)} = \log p_E(x)-\log p_\pi(x). \label{eq:bayes-log-density-ratio}\tag{27}\] Thus the classifier logit is a log density ratio. The next step is to specify the two trajectory densities whose ratio will be used. Write a finite trajectory as \(\tau=(s_0,a_0,s_1,a_1,\ldots,s_{H-1},a_{H-1},s_H)\). The density-ratio motivation used in GAN-guided cost learning and AIRL starts from an energy model over trajectories: keep the environment part of the trajectory probability, \(\nu_0(s_0)\prod_t P(s_{t+1}\mid s_t,a_t)\), and tilt it toward trajectories whose accumulated learned score \(\sum_t f(s_t,a_t,s_{t+1})\) is large. Exponential tilting gives the expert-side density \[p_f(\tau) \;\propto\; \nu_0(s_0)\prod_{t=0}^{H-1}P(s_{t+1}\mid s_t,a_t) \exp\!\Bigl(\sum_{t=0}^{H-1} f(s_t,a_t,s_{t+1})\Bigr),\] whereas the current policy generates trajectories according to \[p_\pi(\tau) \;=\; \nu_0(s_0)\prod_{t=0}^{H-1}\pi(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t).\] Taking the log-ratio cancels the common initial-state and transition factors: \[\log p_f(\tau)-\log p_\pi(\tau) \;=\; \sum_{t=0}^{H-1}\bigl[f(s_t,a_t,s_{t+1})-\log\pi(a_t\mid s_t)\bigr] \;+\; \text{constant}.\] By the Bayes log-density-ratio identity 27 , the corresponding trajectory classifier has log-odds \[\log \frac{D_f^{\pi}(\tau)}{1-D_f^{\pi}(\tau)} \;=\; \sum_{t=0}^{H-1}\bigl[f(s_t,a_t,s_{t+1})-\log\pi(a_t\mid s_t)\bigr] \;+\; C_f, \label{eq:airl-trajectory-log-odds}\tag{28}\] where \(C_f\) absorbs the trajectory-energy normalizer and any class-prior constant. This trajectory classifier is the GAN-GCL starting point.

3.3.0.1 From trajectory odds to the local AIRL discriminator.

AIRL [5] now makes a one-step localization of the trajectory-level odds relation. Since the trajectory logit in 28 is a sum of per-transition terms, the natural local contribution of a transition \((s,a,s')\) is \[f(s,a,s')-\log\pi(a\mid s).\] AIRL therefore defines the transition-level discriminator by the odds equation \[\log \frac{D_f^{\pi}(s,a,s')}{1-D_f^{\pi}(s,a,s')} \;=\; f(s,a,s')-\log\pi(a\mid s). \label{eq:airl-log-odds}\tag{29}\] Equivalently, \[D_f^{\pi}(s,a,s') \;=\; \frac{\exp f(s,a,s')}{\exp f(s,a,s')+\pi(a\mid s)}. \label{eq:airl-generic-discriminator}\tag{30}\] How can we justify this localization? The clean way is not to view it as literally marginalizing the trajectory Bayes classifier down to one transition. A true marginal classifier for a single transition would have to account for the full distribution of times and histories that can produce the same \((s,a,s')\). The justification is more local: because the trajectory logit in 28 is additive, the term \(f(s,a,s')-\log\pi(a\mid s)\) is the one-transition contribution suggested by that logit. The lemma below records why this is the right surrogate for AIRL: it gives the local discriminator gradient used in the adversarial update and the entropy-regularized reward used in the policy update.

Thus the disappearance of the trajectory-level \(\sum_t\) is the answer to a localization question, not a new cancellation in the likelihood ratio: AIRL keeps the one-transition contribution because it has the right discriminator-gradient and policy-objective identities, not because the full trajectory Bayes classifier literally collapses to one step.

This is also the right place to keep one interpretive caveat in view. The trajectory-energy model above and its corresponding localization is not the same object as the dynamic soft-control model in Section 2.4. It is myopic in the specific sense emphasized by [16]: the trajectory likelihood is tilted by accumulated one-step scores \(f(s_t,a_t,s_{t+1})\), whereas a dynamically optimal policy evaluates an action through immediate reward plus the continuation value induced by the transition kernel. In the dynamic model, \[\log \pi^\star(a\mid s) = Q^\star(s,a)-V^\star(s) = r(s,a)+\beta\,\mathbb{E}[V^\star(s')\mid s,a]-V^\star(s).\] So the local AIRL score should first be read as a policy-rationalizing advantage-like object. It becomes a reward only after a separate decomposition argument explains which part is immediate payoff and which part is value shaping.

Specializing the generic local discriminator 30 to the structured AIRL score \(f=f_{g,h}\) from 26 gives \[D_{g,h}^{\pi}(s, a, s') \;=\; \frac{\exp f_{g,h}(s, a, s')}{\exp f_{g,h}(s, a, s') + \pi(a \mid s)}. \label{eq:airl-discriminator}\tag{31}\] The \(\pi(a\mid s)\) term has an intuitive role: a generated transition may contain action \(a\) simply because the current policy already takes \(a\) often at state \(s\). AIRL therefore asks whether the expert-like score \(\exp f_{g,h}(s,a,s')\) is large relative to the current policy’s own probability of producing that action. This is why the discriminator is policy-dependent, written \(D_{g,h}^{\pi}\).

In the trained model \(g = g_\theta\) and \(h = h_\phi\) are parametrized, with \(f_{\theta,\phi}:=f_{g_\theta,h_\phi}\) and \(D_{\theta,\phi}^{\pi} := D_{g_\theta,h_\phi}^{\pi}\). We write \(f_{g,h}\) when reasoning about identification at the level of functions, and \(f_{\theta,\phi}\) when reasoning about the optimization. Also, in the adversarial objectives below, the notation \((s,a,s')\sim d^\pi\) means \((s,a)\sim d^\pi\) followed by \(s'\sim P(\cdot\mid s,a)\); equivalently it is the transition-augmented discounted occupancy.

The two training updates now follow from the two roles of this classifier.

• First, fix the current policy \(\pi\). The policy-generated transitions are the negative samples for the discriminator. If \(y=1\) denotes an expert transition and \(y=0\) a generated transition, then the discriminator log-likelihood is \[y\log D_{\theta,\phi}^{\pi}(s,a,s')+(1-y)\log\!\bigl(1-D_{\theta,\phi}^{\pi}(s,a,s')\bigr).\] Averaging this log-likelihood over expert samples from \(d^{\pi^\star}\) and generated samples from \(d^\pi\), and dropping the irrelevant class-prior factor, gives the discriminator objective: increase \(\log D_{\theta,\phi}^{\pi}\) on expert transitions and increase \(\log(1-D_{\theta,\phi}^{\pi})\) on policy transitions.

• Second, fix \((\theta,\phi)\) and update the policy. AIRL gives the policy the same discriminator’s log-odds as its reward signal. Substituting \(g=g_\theta\) and \(h=h_\phi\) into the log-odds identity 29 gives \[\log D_{\theta,\phi}^{\pi}(s,a,s') - \log\!\bigl(1-D_{\theta,\phi}^{\pi}(s,a,s')\bigr) = f_{\theta,\phi}(s,a,s')-\log \pi(a\mid s).\] Thus averaging the log-odds over transitions from \(d^\pi\) gives an entropy-regularized policy objective: the policy optimizer sees the learned score \(f_{\theta,\phi}\) together with the entropy term induced by \(-\log\pi(a\mid s)\), rather than a raw logistic score. Operationally, the discriminator says which transitions look expert-like relative to the current policy, and the policy update tries to make those high-log-odds transitions more common.

Putting these two derived objectives together, AIRL alternates: \[\begin{align} \text{Discriminator } (\max_{\theta, \phi}):\;\; &\mathbb{E}_{(s, a, s') \sim d^{\pi^\star}}[\log D_{\theta,\phi}^{\pi}(s, a, s')] + \mathbb{E}_{(s, a, s') \sim d^\pi}[\log(1 - D_{\theta,\phi}^{\pi}(s, a, s'))], \label{eq:airl-disc-update}\\ \text{Policy } (\max_\pi):\;\; &\mathbb{E}_{(s, a, s') \sim d^\pi}\!\bigl[\log D_{\theta,\phi}^{\pi} - \log(1 - D_{\theta,\phi}^{\pi})\bigr] \;=\; \mathbb{E}_{d^\pi}\!\bigl[f_{\theta,\phi} - \log \pi(a \mid s)\bigr], \nonumber \end{align}\tag{32}\] so the policy maximizes entropy-regularized return under the implicit reward \(f_{\theta,\phi}\). The modeling bet is that 26 splits the soft advantage cleanly into a reward part and a shaping part. The bet can pay off only if that split is unique; the rest of this subsection is about when it is.

Identification in AIRL↩︎

We are finally ready to discuss the identification logic of AIRL. We discuss that:

• When the adversarial game actually reaches its saddle, under deterministic transitions, the AIRL discriminator identifies the soft advantage \(A^\star(s, a) = \log \pi^\star(a \mid s)\). This is exactly the expert’s conditional choice probabilities (the same information Hotz–Miller inversion provides) and nothing more.

• Converting that advantage into a reward requires a separate decomposition argument from [5]. We show that it works, but under four restrictions: a state-only reward, deterministic dynamics, a connectivity-and-coverage condition on the transition graph, and an attained exact saddle.

• Each of the four is load-bearing. Drop any one and the shaping ambiguity of Lemma 7 returns. Stochastic transitions, in particular, reintroduce it through a transition-kernel completeness condition that AIRL cannot enforce.

3.3.0.2 The saddle identifies the advantage, not the reward.

The next statement is conditional and population-level. It assumes (a) the policy player has reached the expert soft-optimal policy, so \(d^\pi = d^{\pi^\star}\) on the support under discussion, and (b) the discriminator class can realize its Bayes optimum there. These assumptions say that expert and generated transitions are truly indistinguishable to the classifier, and that the classifier class is rich enough to express the corresponding optimum. If either fails, the conclusions below do not apply: the logit would describe the current policy rather than the expert, or the AIRL class would not contain the Bayes-optimal discriminator.

Under (a)–(b), the best discriminator cannot tell the two sources apart, so \(D^\star \equiv \tfrac12\) on the matched support. Since the policy has matched the expert, the \(\pi\) appearing in 31 is now \(\pi^\star\). Hence \(D_{\theta,\phi}^{\pi^\star} = \tfrac12\) exactly when \(\exp f_{\theta,\phi} = \pi^\star(a \mid s)\), which implies \[f_{\theta,\phi}(s, a, s') \;=\; \log \pi^\star(a \mid s) \;=\; Q^\star(s, a) - V^\star(s) \;=:\; A^\star(s, a), \qquad V^\star(s) := \operatorname{LSE}(Q^\star(s, \cdot)). \label{eq:airl-saddle}\tag{33}\] So an attained, realizable saddle pins down the soft advantage \(A^\star\). Two consequences are worth stating plainly.

This is not new information. In population, \(A^\star(s, a) = \log \pi^\star(a \mid s)\) is just the expert’s log conditional choice probability. Hotz–Miller inversion 21 already extracts the same object through the log-probability differences \(\log \pi^\star(a \mid s) - \log \pi^\star(a_s \mid s)\). This leads to the following Remark 6.

This is the key interpretive point. The advantage says how attractive action \(a\) is relative to the baseline value of state \(s\). It does not say whether that attractiveness came from high immediate reward, a favorable next-state distribution, or a potential-shaping term. Behavior may tell us that action \(a\) is preferred to action \(b\) at state \(s\), but it does not by itself say which part is immediate payoff and which part is future continuation value. So the equality \(f_{\theta,\phi}=A^\star\) is not yet reward recovery; it is only the object that reward recovery must decompose.

Exact equality may be unreachable in stochastic environments. The target \(A^\star(s, a)\) does not depend on the realized next state \(s'\), whereas the AIRL class \(g(s) + \beta h(s') - h(s)\) generally does. So in a stochastic environment the pointwise equality 33 can fail within the parametrized class, before any reward question is even posed. We return to this in the discussion of restriction (iii) below; until then, transitions are deterministic.

3.3.0.3 From advantage to reward: decomposable deterministic dynamics.

Assume now \(s' = T(s, a)\) deterministic. Reward recovery still does not follow for free. If the AIRL reward head is allowed to be a state-action function, then for any bounded \(h : \mathcal{S}\to \mathbb{R}\) the choice \[g_h(s, a) \;:=\; \log \pi^\star(a \mid s) - \beta\, h(T(s, a)) + h(s)\] satisfies \(g_h(s, a) + \beta h(T(s, a)) - h(s) = \log \pi^\star(a \mid s)\) exactly, so infinitely many decompositions of the same saddle logit coexist. Thus writing \(f=g+\beta h'-h\) does not, by itself, tell us which piece is the reward. AIRL needs extra structure that makes the split unique.

The relevant property is not that next states identify current states; it is almost the opposite; enough overlap in one-step reachability that a next-state-only term cannot hide inside a sum of a current-state term and a next-state term. Define the one-step successor and predecessor sets \[\mathcal{S}_{+} := \{x \in \mathcal{S}: \exists\,(s, a) \text{ with } P(x \mid s, a) > 0\}, \qquad \mathcal{S}_{-} := \{s \in \mathcal{S}: \exists\,(a, x) \text{ with } P(x \mid s, a) > 0\}.\] In a standard MDP \(\mathcal{S}_{-} = \mathcal{S}\), but \(\mathcal{S}_{+} = \mathcal{S}\) only if every state has a one-step in-neighbor. Build an undirected graph on \(\mathcal{S}_{+}\), placing an edge between successors \(x\) and \(y\) whenever they share a predecessor (some \(s\) and actions \(a, b\) with \(P(x \mid s, a) > 0\) and \(P(y \mid s, b) > 0\)) and close the relation transitively. The transition kernel is decomposable on the relevant state space (the Fu–Luo–Levine condition) if this successor graph is connected and, for any claim of equality on all of \(\mathcal{S}\), additionally \(\mathcal{S}_{+} = \mathcal{S}_{-} = \mathcal{S}\).

The useful consequence is the following separation lemma. The domain qualification matters: the equation only sees \(v\) at states that appear as successors, and only sees \(u\) at states that appear as predecessors.

3.3.0.4 All four restrictions are load-bearing.

Proposition 7 is sharp: each substantive restriction removes a different obstruction. Here the state-only restriction combines assumptions (i)–(ii), so the four restrictions are state-only structure, deterministic dynamics, decomposable coverage, and an attained exact saddle.

(i)–(ii) State-only rewards. If the reward head may depend on \((s, a)\), the shaping family of Lemma 7 comes straight back. For any bounded potential \(\Phi : \mathcal{S}\to \mathbb{R}\), set \(h = V^\star + \Phi\) and \[g_\Phi(s, a) := r(s, a) + \Phi(s) - \beta\, \mathbb{E}_{s' \sim P(\cdot \mid s, a)}[\Phi(s')]. \label{eq:airl-state-action-shaping}\tag{35}\] Then \(g_\Phi(s, a) + \beta\, \mathbb{E}[h(s') \mid s, a] - h(s) = A^\star(s, a)\) identically. In deterministic dynamics this expectation is just evaluation at \(T(s,a)\); in stochastic dynamics it is the population version of the same ambiguity. Either way, every potential-shaped reward produces the same soft advantage, so AIRL leaves state-action rewards unidentified beyond the baseline shaping equivalence.

(iii) Deterministic dynamics. Determinism matters because then the next state \(T(s,a)\) is fixed once \((s,a)\) is fixed, so the AIRL term \(h(T(s,a))\) can line up pointwise with the Bellman equation. The proof used exactly this identity: \(A^\star(s, a) = r(s) + \beta V^\star(T(s, a)) - V^\star(s)\). Under stochastic transitions, the Bellman identity carries a conditional expectation, while AIRL’s realized term \(h(s')\) depends on one sampled next state. The most one can ask is the population moment \[\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[f_{g,h}(s, a, s')] = A^\star(s, a). \label{eq:airl-expectation-condition}\tag{36}\] Even granting 36 , reward recovery is not automatic. With a state-only reward and state-only heads, the soft Bellman equation turns 36 into \(g(s) + \beta \mathbb{E}[h(s') \mid s, a] - h(s) = r(s) + \beta \mathbb{E}[V^\star(s') \mid s, a] - V^\star(s)\); with \(\Delta := h - V^\star\) and \(\delta := g - r\) this rearranges to \[\delta(s) = \Delta(s) - \beta\, \mathbb{E}[\Delta(s') \mid s, a] \qquad \forall a. \label{eq:airl-stochastic-delta}\tag{37}\] The left-hand side is independent of \(a\), so \(\mathbb{E}[\Delta(s') \mid s, a]\) must be independent of \(a\) as well. This forces \(\Delta\) to be locally, and then globally, constant only if the transition family is rich enough that the condition “all action-conditioned expectations of \(\Delta\) agree” implies \(\Delta\) is constant: a completeness condition on \(P\). Mere dependence of \(P(\cdot \mid s, a)\) on \(a\) is not enough, since two next-state distributions can share the expectation of a non-constant \(\Delta\). Under the completeness condition one recovers \(h = V^\star + c\) and \(g = r + (1 - \beta)c\) as before; without it, 37 leaves residual shaping ambiguity.

This distinction clarifies the role of transition-aware AIRL variants. [20] tries to address the stochastic-realizability obstruction: replacing AIRL’s realized-next-state term \(h(s')\) by a model-based approximation to \(\mathbb{E}[h(s')\mid s,a]\), equivalently a transition-aware shaping term such as \(R(s,a)+\beta\mathbb{E}_{\hat{T}}[\varphi(s')\mid s,a]-\varphi(s)\), makes the logit compatible with the stochastic Bellman identity up to learned-transition error. That repair does not by itself supply the completeness or normalization condition needed for identification. After the expectation is inserted, the remaining equation is still 37 , so reward recovery still requires that the only potential differences \(\Delta\) whose action-conditioned transition expectations agree are constants.

(iv) Connectivity and coverage. If the successor graph splits into several components, the separation argument forces \(\Delta\) constant only within each component, and different per-component constants give genuinely different decompositions consistent with the same saddle. States that never appear as one-step successors leave \(h\) unconstrained there; states that never appear as predecessors leave \(g\) unconstrained. Deterministic systems with little branching, or with disconnected recurrent classes, routinely fail the condition.

(v) Exact saddle. If the policy player has not matched the expert occupancy, or if the discriminator class cannot realize its Bayes optimum, the equality \(f_{g,h}=A^\star\) need not hold on the expert support. The learned logit then reflects the current policy/discriminator game rather than the expert’s soft advantage, so there is no exact advantage identity to decompose into reward plus shaping terms.

3.3.0.5 Relation to anchor actions.

The bottom line is that AIRL is elegant because it builds potential shaping into the discriminator, but that removes the baseline identification problem only under assumptions such as state-dependent rewards and deterministic transitions. At the perfect saddle, it recovers the expert’s soft advantage; reward recovery requires additional structure that makes the advantage decomposition unique.

To summarize:

• AIRL recovers Hotz-Miller conversion under deterministic transitions. In other words, under deterministic transitions, making the anchor action assumption additionally yields exact reward recovery [1]. However, under stochastic transitions, the Hotz-Miller conversation is not recovered because it stems from the trajectory energy model (which is fundamentally myopic under stochastic transitions [16]).

• AIRL can avoid the anchor-action assumption, following [5], by making extra assumptions such as deterministic transitions and state-only rewards.

For the remainder of these notes, we go back to the anchor action assumption Assumption 1: it identifies state-action rewards directly, on the expert support, without requiring state-only rewards, deterministic dynamics, or a decomposable transition graph.

4 Classical DDC Methods↩︎

We now survey the classical estimation routes that grew out of the Magnac–Thesmar identification: Rust’s nested fixed-point algorithm (Section 4.2), the conditional-choice-probability approach of Hotz–Miller (Section 4.3), and the Adusumilli–Eckardt temporal-difference route (Section 4.4), which itself contains two methods: linear semi-gradient TD and approximate value iteration. Each route gets one of the following ingredients right, but pays a price elsewhere:

• Scalability in the dimension of \(\mathcal{S}\),

• Transition-kernel freedom (no separate estimate of \(P\) required),

• Stability (no deadly-triad-style divergence).

Section 6 is then about how we can address this issue.

The starting point is common to all. By Theorem 5, \(\pi^\star\) pins down \(Q^\star\) once an anchor action is fixed. Operationally, this requires solving two coupled conditions (the softmax MLE condition and the soft Bellman equation at the anchor action) that we record now.

4.1 The Two Pieces: NLL and Bellman↩︎

Notice what the NLL gives us and what it does not. The softmax \(\hat{p}_Q\) is invariant under state-only shifts \(Q(s, a) \mapsto Q(s, a) + c(s)\) (Lemma 2). So fitting \(\pi^\star\) alone pins down \(Q\) only up to such shifts, exactly the shaping ambiguity of Lemma 7. The Bellman condition is what locks down those shifts. The classical methods differ in how they enforce it.

4.2 Rust’s Nested Fixed-Point Algorithm (NFXP)↩︎

[2] proposed to parametrize the reward \(r_\theta\) in a low-dimensional class and treat the Bellman equation as an inner-loop constraint: \[\begin{align} \text{Outer (over \theta):}\quad &\theta_{k+1} = \operatorname*{arg\,min}_\theta\;\widehat\mathbb{E}_{\mathcal{D}}\bigl[-\log \hat{p}_{Q_\theta}(a \mid s)\bigr],\\ \text{Inner (at each \theta):}\quad &Q_\theta := \text{(unique fixed point of } \mathcal{T}^{r_\theta}_{\mathrm{soft}})\text{ computed by soft value iteration.} \end{align}\] Convergence of the inner loop is guaranteed by Lemma 3; consistency of the outer MLE is the classical result of [2]. This is the right thing to do, statistically; the problem is doing it.

4.2.0.1 The computational obstruction.

The inner soft value iteration is over \(\mathcal{S}\times \mathcal{A}\). For each iteration, the LSE \[\operatorname{LSE}(Q(s', \cdot)) \;=\; \log \sum_{a' \in \mathcal{A}} \exp(Q(s', a'))\] must be evaluated at every \(s'\) the outer expectation might query. Discretizing \(\mathcal{S}\) to \(|\mathcal{S}|\) points and running value iteration costs \(\Theta(|\mathcal{S}| \cdot |\mathcal{A}|)\) per iteration to convergence in \(\Theta(\log(1/\varepsilon)/(1-\beta))\) iterations, and the inner loop sits inside the outer loop over \(\theta\). When the underlying state has intrinsic dimension \(d\), fixed-grid discretization to resolution \(h\) produces \(|\mathcal{S}| = \Theta(h^{-d})\), so the inner cost scales as \(h^{-d}\): the curse of dimensionality of dynamic programming. Workarounds (sieve approximations [21], [22], recursive partitioning [23], state aggregation [24], neural-network approximations [25]) push the boundary but do not change the scaling.

4.2.0.2 The transition kernel is also needed.

There is a second problem hidden inside the first. The inner soft value iteration requires the expectation \(\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[\operatorname{LSE}(Q(s', \cdot))]\), which presumes either (i) closed-form access to \(P\) (rare outside textbook problems), or (ii) a separately estimated \(\hat{P}\). In option (ii), error in \(\hat{P}\) propagates through the fixed-point computation; [13] requires \(P\) to be known a priori or consistently estimable in a first stage, a non-trivial demand in high dimensions. A constrained-optimization reformulation (MPEC) that avoids the nested loop in low dimensions is given by [26]: treat both \(\theta\) and the value function as optimization variables and minimize the likelihood subject to the Bellman equation as a constraint. MPEC still requires \(P\).

4.3 Hotz–Miller / Conditional Choice Probabilities (CCP)↩︎

[3] attack the inner-loop bottleneck differently. Their starting point is the observation underpinning 21 : the differences of \(Q^\star\) across actions at the same state are already pinned down by the observed log choice-probability ratios. Identity 22 rearranges to \[V^\star(s) \;=\; Q^\star(s, a_s) - \log \pi^\star(a_s \mid s), \label{eq:HM-V}\tag{38}\] and the linear fixed-point equation 24 that follows is a Bellman-like equation for \(V^\star\) alone, with reward \(-\log \pi^\star(a_s \mid \cdot)\). Once \(V^\star\) is in hand, \(Q^\star(s, a)\) follows from 21 and 38 , and \(r\) follows from the soft Bellman equation 25 .

4.3.0.1 Forward simulation.

The expectation in 24 can be approximated by forward Monte Carlo under the anchor-conditioned kernel: simulate trajectories of length \(T\) starting at \(s\), set the action at each simulated state \(s_t\) to the anchor action \(a_{s_t}\), draw \(s_{t+1}\) from \(\hat{P}(\cdot \mid s_t,a_{s_t})\), and accumulate the per-step pseudo-reward \(-\log \hat{\pi}^\star(a_{s_t} \mid s_t)\). Drawing actions from \(\hat{\pi}^\star\) would solve a different Bellman equation; the anchor equation 24 specifically uses \(P(\cdot\mid s,a_s)\). This sidesteps the high-dimensional fixed-point computation of NFXP entirely. Recursive variants of CCP [27], [28] interleave \(\pi^\star\) updates and value iteration.

4.3.0.2 The statistical obstruction.

We bought the computational savings of CCP at a price, and it is a steep one. Forward simulation requires estimating \(\hat{P}\) in high dimensions, then propagating its error over \(T = \Theta(1/(1-\beta))\) steps. With a continuous or high-dimensional state space, \(\hat{P}\) is itself subject to a curse of dimensionality: a nonparametric kernel density estimator of \(P(s' \mid s, a)\) has rate \(N^{-c/d}\) for some smoothness-dependent \(c\). The statistical complexity blows up exactly where NFXP’s computational complexity did.

4.3.0.3 Recovery of \(r\).

Once \(V^\star\) and \(Q^\star\) are recovered, the reward is read off 25 . This expression requires another \(P\)-expectation; in continuous states it is again subject to transition-estimation error.

4.3.0.4 The summary.

NFXP and CCP are duals. NFXP fits \(r\) in an outer loop and solves the Bellman fixed point in the inner. CCP fits \(\pi^\star\) first and exploits Hotz–Miller inversion to avoid the inner fixed point but needs \(P\) everywhere. Both require either knowing or estimating \(P\), and both impose either computational (\(|\mathcal{S}| \cdot |\mathcal{A}|\) DP table) or statistical (\(\hat{P}\) in \(d\) dimensions) burdens scaling exponentially in \(d\). Neither produces a single differentiable objective whose minimizer is \(Q^\star\).

4.4 Temporal-Difference DDC: Linear Semi-Gradient and Approximate Value Iteration↩︎

The natural next thought is: can we avoid the transition kernel by using temporal-difference updates based directly on observed transitions? [4] propose exactly this. More precisely, they propose two transition-density-free TD-style approaches for the recursive terms that enter CCP estimation:

1. Linear semi-gradient TD, which approximates the recursive terms with basis functions and updates the coefficients by stochastic approximation.

2. Approximate Value Iteration (AVI), which builds a sequence of approximations by freezing the previous continuation-value estimate and solving a fresh nonparametric regression problem at each iteration.

In the full DDC pseudo-likelihood these recursive terms are the continuation-value objects induced by Hotz–Miller inversion. In our anchor-action notation they collapse to the auxiliary value function in 24 . Thus the same analysis covers both Adusumilli–Eckardt methods: one is the semi-gradient form of TD, and the other is fitted/approximate value iteration. The only twist is that we are now bootstrapping the auxiliary value function 24 , not the original value function of an MDP.

4.4.1 Linear Function Approximation and the Projected Bellman Equation↩︎

Fix a feature map \(\phi : \mathcal{S}\to \mathbb{R}^d\) and let \(\Phi\) denote the \(|\mathcal{S}| \times d\) feature matrix (in the finite-state case; the general case is analogous in \(L^2\)). Parametrize \(V_\theta(s) := \phi(s)^\top \theta\), \(\theta \in \mathbb{R}^d\). The hypothesis class is the linear span \(\mathcal{F}:= \{V : V(s) = \phi(s)^\top \theta,\;\theta \in \mathbb{R}^d\}\), a closed subspace of bounded functions on \(\mathcal{S}\).

Fix a distribution \(\rho \in \Delta(\mathcal{S})\) (the data distribution) with associated \(L^2(\rho)\)-inner product \(\langle f, g \rangle_\rho := \mathbb{E}_{s \sim \rho}[f(s) g(s)]\) and norm \(\|f\|_\rho^2 := \langle f, f\rangle_\rho\). The \(L^2(\rho)\)-projection onto \(\mathcal{F}\) is \[\Pi_{\mathcal{F}, \rho} V \;:=\; \operatorname*{arg\,min}_{V_\theta \in \mathcal{F}} \|V - V_\theta\|_\rho^2,\] which is linear in \(V\) and a non-expansion in \(\|\cdot\|_\rho\).

Let \(\mathcal{T}^{\pi_b}\) denote the (linear) Bellman expectation operator under a behavior policy \(\pi_b\) with corresponding kernel \(P^{\pi_b}(s' \mid s) = \sum_a \pi_b(a \mid s) P(s' \mid s, a)\): \[\mathcal{T}^{\pi_b} V(s) \;:=\; r^{\pi_b}(s) + \beta\,\mathbb{E}_{s' \sim P^{\pi_b}(\cdot \mid s)}[V(s')], \qquad r^{\pi_b}(s) := \sum_a \pi_b(a \mid s) r(s, a). \label{eq:bellman-expectation}\tag{39}\] For the TD-DDC application below, \(\pi_b\) will be the anchor-conditioned chain \(\pi_b(a \mid s) = \mathbf{1}[a = a_s]\) and the per-state “reward” will be \(-\log \pi^\star(a_s \mid s)\).

The unique fixed point \(V_{\mathcal{F}, \rho}^\star\) of \(\Pi_{\mathcal{F}, \rho}\mathcal{T}^{\pi_b}\) is the projected Bellman fixed point. It solves the projected Bellman equation \(V_{\mathcal{F}, \rho} = \Pi_{\mathcal{F}, \rho}\mathcal{T}^{\pi_b} V_{\mathcal{F}, \rho}\); in matrix form, \(\theta^\star\) solves \(A\theta = b\) with \[A \;=\; \Phi^\top D_\rho (\Phi - \beta P^{\pi_b}\Phi), \qquad b \;=\; \Phi^\top D_\rho r^{\pi_b}. \label{eq:A-b}\tag{40}\] When \(\rho\) is the stationary distribution of \(P^{\pi_b}\), [29] show that \(A\) is positive definite. Linear TD\((0)\), whose update direction has expectation \(b - A\theta\), then converges to \(A^{-1} b\) with rate determined by the smallest eigenvalue of \(A\).

4.4.2 The Deadly Triad: Off-Policy Divergence of Linear TD↩︎

The on-policy stability of Lemma 11 relies critically on \(\rho\) being the stationary distribution of \(P^{\pi_b}\). The combined presence of (i) linear (or more general) function approximation, (ii) bootstrapping, and (iii) off-policy data is known as the deadly triad [30], [31]. Here is its TD-DDC face.

This is not a pathological construction. [32] document the same phenomenon with neural-network function approximation, where it persists even with mild representation pre-training. And critically for us: the deadly triad applies in particular to the DDC application below. The offline DDC dataset is generated by the expert acting according to \(\pi^\star\) under all actions, so its state marginal is \(d^{\pi^\star}_S\). The TD-DDC update, however, bootstraps along the anchor-conditioned chain whose kernel is \(P(\cdot \mid \cdot, a_s)\), a deterministic “policy” that always picks the state-specific anchor. After restricting to anchor transitions, the effective sampling distribution is \(\rho_A(s) \propto \pi^\star(a_s \mid s)\, d^{\pi^\star}_S(s)\), which is in general not the stationary distribution of \(P(\cdot \mid \cdot, a_s)\). The on-policy contraction of Lemma 11 therefore does not apply to this update, and Example 1’s pathology is on the table.

4.4.3 Approximate Value Iteration as Fitted Value Iteration↩︎

The second Adusumilli–Eckardt method is not the same as the linear semi-gradient recursion above. It is Approximate Value Iteration: freeze the current approximation inside the continuation term, then solve a new regression problem for the next approximation. In the linear case this takes the form \[\theta^+ \;:=\; \operatorname*{arg\,min}_\theta\;\mathbb{E}_{s \sim \rho,\; s' \sim P^{\pi_b}(\cdot \mid s)}\!\bigl[(r^{\pi_b}(s) + \beta\,\phi(s')^\top \theta^- - \phi(s)^\top\theta)^2\bigr], \qquad \theta^- \leftarrow \theta^+, \label{eq:fvi}\tag{41}\] and iterate. The inner problem is a well-conditioned least-squares problem whenever \(\Phi^\top D_\rho \Phi \succ 0\); its closed form is \[\theta^+ \;=\; (\Phi^\top D_\rho \Phi)^{-1} \Phi^\top D_\rho (r^{\pi_b} + \beta P^{\pi_b}\Phi\theta^-) \;=\; (\Phi^\top D_\rho \Phi)^{-1} \Phi^\top D_\rho \mathcal{T}^{\pi_b}(\Phi\theta^-).\] Viewed as a map \(\theta^- \mapsto \theta^+\), this is exactly fitted value iteration (FVI): \(V_{\theta^+} = \Pi_{\mathcal{F}, \rho} \mathcal{T}^{\pi_b} V_{\theta^-}\). In a nonparametric or sieve implementation, the same map is computed by replacing the linear least-squares problem with the corresponding regression learner over the chosen function class. Thus, in these notes, Adusumilli–Eckardt’s AVI method is the DDC specialization of the standard fitted-value-iteration operator.

4.4.4 Application: The Two TD-DDC Methods of Adusumilli–Eckardt↩︎

[4] apply this machinery in the DDC setting with the substitutions of Section 4.3: \(r^{\pi_b}(s) := -\log \pi^\star(a_s \mid s)\) and \(\pi_b(a \mid s) := \mathbf{1}[a = a_s]\). Let \[I_A \;:=\; \{i \in \{1,\ldots,N\}: a_i = a_{s_i}\}\] be the set of observed transitions in which the expert took the anchor action, and write \(\mathcal{D}_A := \{(s_i, s_i') : i \in I_A\}\) for the corresponding sub-dataset with state marginal \(\rho_A\).

4.4.4.1 Method 1: linear semi-gradient TD.

The first method updates the coefficient vector using the current parameter on both sides of the TD error: \[\theta_{t+1} \;=\; \theta_t + \alpha_t\,\bigl(-\log \pi^\star(a_{s_t} \mid s_t) + \beta\,\phi(s_{t+1})^\top \theta_t - \phi(s_t)^\top \theta_t\bigr)\,\phi(s_t), \label{eq:ae-semigrad}\tag{42}\] sampled from transitions \((s_t, a_t = a_{s_t}, s_{t+1})\) in \(\mathcal{D}\).

4.4.4.2 Method 2: approximate value iteration.

The second method freezes the previous value approximation. Starting from \(V_0 \in \mathcal{F}\), define pseudo-outcomes for \(i \in I_A\) by \[Y_i^{(k)} \;:=\; -\log \pi^\star(a_{s_i} \mid s_i) + \beta V_k(s_i'), \label{eq:ae-avi-target}\tag{43}\] and compute \[V_{k+1} \;\in\; \operatorname*{arg\,min}_{f \in \mathcal{F}}\;\frac{1}{|I_A|}\sum_{i \in I_A}\bigl(Y_i^{(k)} - f(s_i)\bigr)^2. \label{eq:ae-avi-regression}\tag{44}\] When \(\mathcal{F}\) is linear, \(V_k(s)=\phi(s)^\top\theta_k\) and 44 is exactly 41 ; when \(\mathcal{F}\) is a sieve or nonparametric class, 44 is the nonparametric AVI regression. Either way, the population operator is \(V_{k+1}=\Pi_{\mathcal{F},\rho_A}\mathcal{T}^{\pi_b}V_k\).

4.4.4.3 Replaying the analysis in the DDC setting.

The two methods have different failure modes.

• The linear semi-gradient update 42 has all three deadly-triad ingredients: function approximation, bootstrapping, and off-policy state weighting relative to the anchor-conditioned chain. By Example 1, it has no general guarantee of convergence to \(V^\star\), the unique solution of 24 .

• The AVI/FVI update 44 avoids the within-iteration moving-target gradient problem by freezing \(V_k\) and solving a supervised regression. But it is still an iterated projected Bellman method. If \(\Pi_{\mathcal{F},\rho_A}\mathcal{T}^{\pi_b}\) is a contraction, Proposition 8 gives convergence; the limit is the projected fixed point \(V_{\mathcal{F},\rho_A}^\star\), which equals \(V^\star\) when \(V^\star\) is realizable in \(\mathcal{F}\) and the projected fixed point is unique, as in Corollary 1. Without such a contraction, even the projected iteration can fail to converge.

4.4.4.4 The honest scorecard.

Both Adusumilli–Eckardt methods avoid the explicit transition kernel that CCP/Hotz–Miller needed: they use observed \((s_t, s_{t+1})\) pairs in place of an explicit \(\mathbb{E}_{P(\cdot \mid s, a)}\). That is a genuine improvement over forward simulation. But the price is not the same for the two methods. Tallying:

• NFXP: statistically efficient, transition-kernel-needing, computationally cursed in \(d\).

• CCP/Hotz–Miller: computationally cheap, transition-estimation-needing, statistically cursed in \(d\).

• Adusumilli–Eckardt linear semi-gradient TD: transition-kernel-free, but deadly-triad-vulnerable.

• Adusumilli–Eckardt AVI/FVI: transition-kernel-free and stable as a sequence of regression problems, but still a bootstrapped projected-fixed-point method; convergence to the true \(V^\star\) requires both stability of the projected operator and realizability of \(V^\star\) in the chosen function class.

None produces a single unbiased objective whose global minimum is the true \(Q^\star\), computable with mild conditioning assumptions on the parametrization and without specifying \(P\).

4.4.5 Why “Neural Orthogonalization” (Target Networks) Does Not Resolve It↩︎

A reader familiar with deep RL might wonder whether the linear semi-gradient instability of Method 1 can be cured by introducing a target network, exactly as in the forward-RL setting. The answer is the same here as there, for the same reason: target networks change which method you are running, not where it converges. We make this precise, then explain why it leaves the underlying issue intact.

4.4.5.1 What the proposition says, and what it doesn’t.

Target stabilization moves Method 1 onto Method 2’s orbit; it does not change Method 2’s destination. The projected-fixed-point obstruction of Corollary 1 is unaffected: even with perfect target stabilization and idealized inner optimization, the limit (when it exists) is the projected fixed point \(V_{\mathcal{F}, \rho_A}^\star\). That limit equals the true auxiliary \(V^\star\) only under additional conditions such as realizability of \(V^\star\) in \(\mathcal{F}\) and uniqueness/stability of the projected fixed point. These conditions are not innocuous for neural function classes in continuous-state, off-policy settings; they are exactly the kind of structural assumptions the deadly-triad analysis (Section 4.4.2) warns us we cannot expect for free.

In particular, three caveats are worth recording:

• Linear \(\mathcal{F}\), off-policy data (\(\rho_A\) not invariant under \(P^{\pi_b}\)): the projected operator can fail to be a contraction; Example 1 applies, and target stabilization does not rescue it.

• Linear \(\mathcal{F}\), on-policy stationary data: the projected operator is a \(\beta\)-contraction by Lemma 11, and target-stabilized Method 1 converges to \(V_{\mathcal{F}, \rho_A}^\star\), the projected fixed point. This limit equals \(V^\star\) if \(V^\star\) lies in \(\mathcal{F}\); otherwise it is a projected approximation.

• Nonlinear \(\mathcal{F}\) (the relevant regime for offline IRL/DDC at scale): \(\Pi_{\mathcal{F}, \rho_A}\) is set-valued, the inner regression may have multiple local minima, and ?? does not uniquely define a single-valued map. Even pointwise selecting an inner minimizer, \(\Pi_{\mathcal{F}, \rho_A}\,\mathcal{T}^{\pi_b}\) need not be Lipschitz, and convergence requires assumptions far beyond what is verifiable in practice.

The methodological takeaway is the following:

Target networks stabilize the trajectory of bootstrapped TD-DDC updates but do not change their destination. The destination is the projected Bellman fixed point of the auxiliary value function 24 , not the auxiliary value function itself.

This rules out the most familiar deep-RL fix and points to the same conclusion: the right escape route in offline IRL/DDC is to abandon bootstrapping entirely in favor of direct (bias-corrected) Bellman-residual minimization. That is the gap Section 6 closes.

5 Modern IRL Methods↩︎

We now turn to the ML/IRL literature. Four families dominate it: adversarial IRL, occupancy matching, IQ-Learn, and offline maximum-likelihood IRL. For each we (a) write out the actual training objective, (b) state precisely what it identifies and what it does not, and (c) compare to the anchor-action route. A general survey of imitation learning is given by [33].

A preview of where the section lands: each modern method makes a different bet against the same identification ambiguity (Lemma 7), and each pays for the bet somewhere: in the form of reward identified, in stability, in compatibility with offline data, or in what the method actually optimizes.

5.1 Adversarial IRL (AIRL)↩︎

Section 3.3 introduced AIRL’s language and formulation: the potential-shaped discriminator 31 and the alternating saddle update 32 . We return to AIRL here as a modern IRL method, focusing on what that formulation delivers computationally and what it still cannot identify.

5.1.0.1 What the saddle delivers.

The textbook saddle calculation needs two idealizations made explicit. First, the policy player must have converged to the expert, so \(d^\pi = d^{\pi^\star}\) on the relevant support. Second, the AIRL discriminator class must be able to realize its Bayes optimum. Under both, \(D^\star \equiv \tfrac12\) on the matched support, and since 31 gives \(D = \tfrac12\) exactly when \(\exp f_{\theta, \phi} = \pi(a \mid s)\), the saddle pins down \[f_{\theta, \phi}(s, a, s') = \log \pi^\star(a \mid s) = A^\star(s, a).\] As emphasized in Section 3.3, this is an advantage identity, not yet a reward identity, and it carries no information beyond the expert’s conditional choice probabilities, the same object Hotz–Miller inversion 21 consumes through log-probability differences. The discriminator’s real burden is therefore not to discover the advantage, but to split it into a reward part and a shaping part. In stochastic environments even the equality above is delicate: the right-hand side \(A^\star(s, a)\) is independent of \(s'\) while the AIRL class \(g(s) + \beta h(s') - h(s)\) is not, so exact \(D \equiv \tfrac12\) within the parametrized class may be unattainable. Section 3.3 treats this carefully through the population moment 36 .

5.1.0.2 Identification limitations.

The self-contained analysis of Section 3.3 (Proposition 7) shows precisely what extra structure makes reward recovery valid, and what goes wrong otherwise. The limitations are structural rather than merely computational:

• With a state-only ground-truth reward, a state-only reward head, deterministic dynamics, an attained exact saddle, and the decomposable-coverage condition, AIRL recovers the reward up to an additive constant: \(g_\theta(s) = r(s) + C\) and \(h_\phi(s) = V^\star(s) + C/(1 - \beta)\).

• If \(g_\theta\) is allowed to depend on \((s, a)\), the potential-shaped family 35 gives the same advantage, so state-action rewards remain unidentified beyond the baseline shaping equivalence of Lemma 7.

• In stochastic environments there is both a realizability gap, because the parametrized logit cannot match the \(s'\)-independent target \(A^\star(s, a)\) pointwise, and an identification gap: even the population moment 36 needs a transition-kernel completeness condition to pin down the shaping potential.

The anchor-action route gives strictly more for these notes: it identifies state-action \(r\) on the expert support without requiring the reward to be state-only and without relying on the AIRL decomposition (cf.Theorem 5 and Theorem 4).

5.1.0.3 Optimization limitations.

On top of this identification ceiling, AIRL inherits the well-documented instabilities of GAN training: the alternating minimax updates on \((\theta, \phi)\) and \(\pi\) can fail to converge, exhibit mode collapse, or settle at a non-optimal saddle. And AIRL is fundamentally an online method: the discriminator update 32 requires sampling from the current policy \(\pi\), which is exactly what offline IRL forbids.

5.2 Occupancy Matching: GAIL↩︎

[6] introduced Generative Adversarial Imitation Learning (GAIL). The starting point is the occupancy-matching objective \[\min_\pi \;D_{\mathrm{JS}}\bigl(d^\pi \,\|\, d^{\pi^\star}\bigr),\] where \(D_{\mathrm{JS}}\) is the Jensen–Shannon divergence. The variational identity behind GAIL is the following. For any two distributions \(p, q\) on a common space \(\mathcal{X}\), the GAN value at the inner \(D\)-optimum is \[\max_{D : \mathcal{X}\to (0, 1)} \;\mathbb{E}_p[\log D] + \mathbb{E}_q[\log(1 - D)] \;=\; -\log 4 + 2\,D_{\mathrm{JS}}(p \,\|\, q), \label{eq:gan-dual}\tag{45}\] with optimum attained at \(D^\star = p / (p + q)\) [19]. The derivation is direct: writing the integrand at a point as \(V(D) := p \log D + q \log(1 - D)\) and setting \(\partial V/\partial D = 0\) gives \(D^\star = p/(p+q)\), and substituting back recovers \(-\log 4 + 2D_{\mathrm{JS}}(p \| q)\) via \(D_{\mathrm{JS}}(p\|q) = \tfrac{1}{2}\operatorname{KL}(p \| (p+q)/2) + \tfrac{1}{2}\operatorname{KL}(q \| (p+q)/2)\). So minimizing the JS over \(\pi\) is equivalent (up to additive constants) to the saddle \[\min_\pi\,\max_{D}\;\mathbb{E}_{(s, a) \sim d^{\pi^\star}}[\log D(s, a)] + \mathbb{E}_{(s, a) \sim d^\pi}[\log(1 - D(s, a))]. \label{eq:gail-saddle}\tag{46}\] GAIL solves 46 with a GAN-style optimizer over a parametrized \(D_\phi\) and \(\pi_\theta\), using on-policy policy-gradient methods such as PPO [34], [35] for the \(\pi_\theta\) update. Related state-marginal-matching variants restrict \(D\) to depend only on \(s\) [36].

5.2.0.1 What GAIL is and isn’t.

GAIL is fundamentally an imitation algorithm: its goal is to make \(d^{\pi_\theta}\) close to \(d^{\pi^\star}\) in JS divergence, and it succeeds at this on many tasks. But the crucial point is that it is not a reward-recovery algorithm. At the saddle, the discriminator satisfies \[\log\frac{D^\star(s, a)}{1 - D^\star(s, a)} \;=\; \log \frac{d^{\pi^\star}(s, a)}{d^{\pi_\theta}(s, a)},\] which is a log Radon–Nikodym derivative of expert vs.current-policy occupancies, not a reward. There is no way to extract \(r\) from \(D^\star\) without further identification assumptions of exactly the kind we already discussed (Theorem 4 and the anchor-action assumption).

5.2.0.2 Bellman consistency is not built in.

The limitation can be seen directly from a one-line Bellman-flow identity. Fix any policy \(\pi\), reward \(r\), and candidate state-action value function \(Q\). Define the policy Bellman operator \[(\mathcal{T}_r^\pi Q)(s,a) :=r(s,a)+\beta\mathbb{E}_{s'\sim P(\cdot\mid s,a),\,a'\sim\pi(\cdot\mid s')}[Q(s',a')].\] Using the normalized discounted occupancy \[d^\pi(s,a)=(1-\beta)\sum_{t\ge 0}\beta^t\mathbb{P}_\pi(s_t=s,a_t=a),\] we have \[\begin{align} \mathbb{E}_{d^\pi}\!\left[Q(s,a)-\beta\mathbb{E}[Q(s',a')\mid s,a]\right] &=(1-\beta)\mathbb{E}_{s_0\sim\nu_0,\,a_0\sim\pi(\cdot\mid s_0)}[Q(s_0,a_0)], \tag{47}\\ \mathbb{E}_{d^\pi}\!\left[Q(s,a)-(\mathcal{T}_r^\pi Q)(s,a)\right] &=(1-\beta)\mathbb{E}[Q(s_0,a_0)]-\mathbb{E}_{d^\pi}[r(s,a)]. \tag{48} \end{align}\] The first line is just telescoping: the second term equals \((1-\beta)\sum_{t\ge 0}\beta^{t+1}\mathbb{E}[Q(s_{t+1},a_{t+1})]\), so subtracting it from \((1-\beta)\sum_{t\ge 0}\beta^t\mathbb{E}[Q(s_t,a_t)]\) leaves only the \(t=0\) term. The second line subtracts the reward.

Equation 48 is a single weighted average of Bellman residuals under the occupancy \(d^\pi\). Matching \(d^\pi\) to \(d^{\pi^\star}\) can at most match such occupancy-weighted averages. It does not imply the pointwise Bellman equation \[Q(s,a)=(\mathcal{T}_r^\pi Q)(s,a)\qquad\text{for every }(s,a).\] Residuals can be positive on one part of the support and negative on another while their \(d^\pi\)-average remains zero. Thus occupancy matching places a distributional constraint on where the policy goes; it does not place an equation-of-motion constraint on a candidate \(Q\).

This also explains why a reward cannot be recovered from GAIL by first extracting some \(Q\) and then applying the Bellman rearrangement \(r=Q-\beta\mathbb{E}[V_Q(s')\mid s,a]\). At a perfect GAIL match the discriminator itself is uninformative (\(D^\star=1/2\)), and any separately fitted \(Q\) is unconstrained by the pointwise Bellman residual unless that residual is explicitly included in the objective. The anchor-action and ERM routes of Section 3.2 and Section 6 do exactly that: they enforce the Bellman equation at the anchor action rather than hoping it follows from occupancy matching.

5.2.0.3 Offline limitations.

GAIL’s policy update is on-policy: it requires sampling from \(\pi_\theta\) to compute \(\mathbb{E}_{d^{\pi_\theta}}[\log(1 - D)]\). An offline variant, ValueDice [37], re-derives the objective using off-policy distribution-ratio estimation via the Fenchel dual of the \(f\)-divergence: instead of sampling from \(\pi_\theta\), it estimates the ratio \(d^{\pi_\theta}/d^{\pi^\star}\) from offline data and substitutes it into the GAIL objective. The empirical performance of these off-policy corrections is mixed; [38] show that improvements over a strong behavior-cloning baseline can be entirely attributed to architectural and regularization differences, not the off-policy correction itself. The deeper issue is the curse of horizon all over again: trajectory importance sampling for \(T\)-step corrections has variance \(\Theta((\rho_{\max})^T)\) where \(\rho_{\max}\) is the per-step importance ratio bound and \(T \sim 1/(1-\beta)\) is the effective horizon, exponentially bad in the horizon for any non-trivial policy shift, as standard importance-sampling calculations show.

5.3 IQ-Learn↩︎

[7] propose IQ-Learn (Inverse soft-Q Learning), which sidesteps both forward simulation and adversarial training by changing variables from \((r, \pi)\) to \((Q, \pi)\). The key identity is the soft Bellman equation 6 , rearranged: \[r_Q(s, a) \;:=\; Q(s, a) - \beta\,\mathbb{E}_{s' \sim P(\cdot \mid s, a)}[V_Q(s')], \qquad V_Q(s) := \operatorname{LSE}(Q(s, \cdot)). \label{eq:iql-reward-recovery}\tag{49}\] Given any \(Q\), equation 49 defines a reward \(r_Q\) for which \(Q\) satisfies the soft Bellman equation by construction. IQ-Learn parametrizes \(Q_\theta\) directly and maximizes \[\mathcal{J}(Q_\theta) \;:=\; \mathbb{E}_{(s, a) \sim d^{\pi^\star}}[r_Q(s, a)] \;-\; (1 - \beta)\,\mathbb{E}_{s_0 \sim \nu_0}[V_Q(s_0)] \;-\; \lambda\,\mathbb{E}_{(s, a) \sim d^{\pi^\star}}[\psi(r_Q(s, a))], \label{eq:iql-objective}\tag{50}\] where \(\psi\) is a strongly convex regularizer (the chi-squared \(\psi(x) = x^2/4\) in the simplest version) and \(\lambda > 0\). To see why the first two terms behave as an entropy-regularized return gap, use the Bellman-flow identity: for any function \(V\) and any policy \(\pi\), \[\mathbb{E}_{(s, a) \sim d^\pi}\!\bigl[V(s) - \beta\,\mathbb{E}_{s' \sim P(\cdot|s, a)}[V(s')]\bigr] \;=\; (1 - \beta)\,\mathbb{E}_{s_0 \sim \nu_0}[V(s_0)], \label{eq:bellman-flow}\tag{51}\] which is the discounted-occupancy form of the telescoping sum \(\sum_t \beta^t(V(s_t) - \beta V(s_{t+1}))\). Applying 51 with \(V = V_Q\) and \(\pi = \pi^\star\), and using the definition of \(r_Q\), \[\mathbb{E}_{(s, a) \sim d^{\pi^\star}}[r_Q(s, a)] \,-\, (1-\beta)\mathbb{E}_{s_0 \sim \nu_0}[V_Q(s_0)] \;=\; \mathbb{E}_{(s, a) \sim d^{\pi^\star}}\!\bigl[Q(s, a) - V_Q(s)\bigr].\] The RHS is exactly \(\mathbb{E}_{(s, a) \sim d^{\pi^\star}}[\log \hat{p}_Q(a \mid s)] = -\mathbb{E}_{s \sim d^{\pi^\star}_S}\!\bigl[\operatorname{KL}(\pi^\star(\cdot|s)\,\|\,\hat{p}_Q(\cdot|s))\bigr] - \mathbb{E}_{s \sim d^{\pi^\star}_S}[\operatorname{H}(\pi^\star(\cdot|s))]\) by the same decomposition as in Lemma 10. So maximizing the first two terms of \(\mathcal{J}\) in \(Q\) is exactly minimizing the KL between the softmax of \(Q\) and the expert policy, i.e., it fits \(Q\) to a value function whose softmax matches \(\pi^\star\), hence to the soft Bellman equation of \(r_Q\) by construction. The chi-squared term keeps \(r_Q\) from blowing up.

5.3.0.1 What IQ-Learn does and does not pin down.

The first two terms of 50 do not fix the state-only shaping ambiguity. Let \(c:\mathcal{S}\to\mathbb{R}\) be bounded and define \(Q_c(s,a):=Q(s,a)+c(s)\). Then \[V_{Q_c}(s)=V_Q(s)+c(s),\qquad r_{Q_c}(s,a)=r_Q(s,a)+c(s)-\beta \mathbb{E}[c(s')\mid s,a].\] The change in the reward term is, by the Bellman-flow identity 51 , \[\mathbb{E}_{d^{\pi^\star}}[r_{Q_c}-r_Q] =\mathbb{E}_{d^{\pi^\star}}\bigl[c(s)-\beta\mathbb{E}[c(s')\mid s,a]\bigr] =(1-\beta)\mathbb{E}_{\nu_0}[c(s_0)].\] The initial-value term changes by exactly the same amount with the opposite sign: \[-(1-\beta)\mathbb{E}_{\nu_0}[V_{Q_c}(s_0)-V_Q(s_0)] =-(1-\beta)\mathbb{E}_{\nu_0}[c(s_0)].\] Therefore the first two terms of IQ-Learn are invariant under every bounded state-only shift \(c\), not merely under a scalar normalization. The regularizer \(-\lambda\mathbb{E}[\psi(r_Q)]\) is the only part of 50 that selects a representative within the potential-shaped equivalence class, and that selection is a penalty preference rather than statistical identification of the ground-truth reward. Thus IQ-Learn can fit a softmax policy and define a Bellman-consistent reward by construction, but without an additional normalization such as the anchor action it does not identify a unique state-action reward. This is materially weaker than the anchor-action identification of Theorem 5, which fixes the shaping function rather than choosing a regularizer-preferred representative.

5.4 Offline Maximum-Likelihood IRL (Zeng et al.)↩︎

[8] propose offline ML-IRL. Stripped of vocabulary, the method is essentially a parametric, neural re-skinning of Hotz–Miller / CCP (Section 4.3) with two modifications: (a) the closed-form auxiliary inversion of Section 4.3 is replaced by an iterated forward-soft-RL inner loop with an explicit reward parametrization \(r_\theta\), driven by an outer softmax-MLE objective; and (b) a conservatism penalty, analogous to Conservative Q-Learning (CQL) of [39], is added inside the soft-RL inner loop to handle distribution shift. Related conservative offline IRL methods appear in [40].

It is worth making the structural parallel to Hotz–Miller explicit, because it is the right way to understand what the method does and does not buy. Both methods (i) estimate \(\hat{P}\) from \(\mathcal{D}\), and (ii) alternate between a forward step that produces a value function from \(\hat{P}\) together with the current reward (or current value) and a fitting step that updates the reward (or the choice probabilities). CCP performs the forward step in closed form by solving the auxiliary linear fixed-point equation 24 and reading off \(r\) from 25 ; Zeng et al.instead solve a parametric soft-RL problem at each outer step. The substrate is the same; only the parametrization and the objective differ.

The procedure is a doubly-nested alternation. At outer iteration \(k\) with current reward parameter \(\theta_k\):

1. Fit a transition model \(\hat{P}_\phi\) from \(\mathcal{D}\) by maximum likelihood (this is done once or refreshed periodically).

2. Inner loop: solve the soft-RL problem under \((r_{\theta_k}, \hat{P}_\phi)\) for the soft optimal \(\pi_k\) via soft value iteration or actor–critic with a CQL-style penalty added to the value function to discourage extrapolation beyond the support of \(\mathcal{D}\).

3. Outer update: \(\theta_{k+1} \leftarrow \theta_k + \eta\, \nabla_\theta \widehat\mathbb{E}_\mathcal{D}[\log \hat{p}_{Q_{\theta_k}}(a \mid s)]\), where \(Q_{\theta_k}\) is the soft \(Q\)-function corresponding to \(\pi_k\).

4. Repeat until convergence.

The conservatism penalty is the one substantive new ingredient, and it is genuinely useful: it handles the unvisited states that vanilla CCP could not, by discouraging the inner-loop value from extrapolating wildly outside the support of \(\mathcal{D}\). Everything else inherits CCP’s structure, as well as CCP’s limitations:

• It needs to estimate \(\hat{P}_\phi\) in high dimensions, with the same statistical curse-of-dimensionality as CCP/Hotz–Miller.

• The inner-loop policy fit must contend with the bias of \(\hat{P}_\phi\) and the distribution-shift concerns that motivate the conservatism penalty.

• The outer loop is nested optimization, which is the very thing we have been trying to avoid since Rust.

[8] prove finite-time guarantees of the form \(\|\widehat r - r\|_2^2 \le \mathcal{O}(N^{-1/2}) + \text{(model error)}\) under linear-MDP assumptions and bounded transition-model error, but the analysis does not extend directly to neural function approximation without further structure.

The honest takeaway is that offline ML-IRL inherits Hotz–Miller’s strengths and weaknesses essentially unchanged; conservatism is orthogonal to the underlying estimator and could in principle be bolted onto any method in this section, including the ERM framework of Section 6.

5.4.0.1 Where modern IRL leaves us.

None of the methods reviewed in this section gives all of the following at once:

• A single (non-nested) gradient-based objective.

• Provable global convergence in nonlinear function classes.

• State-action reward recovery in continuous-state, stochastic-transition environments.

• No transition-kernel estimation.

The next section gives one route through this gap under explicit anchor, empirical Jacobian-conditioning/PL, realizability, and stability assumptions.

6 The Empirical Risk Minimization Approach↩︎

We finally reach the ERM framework of [9]. The easiest way to understand the method is not to start with the minimax objective. Instead, start from the two restrictions that have appeared throughout the notes:

1. the expert’s observed choices identify the within-state differences of \(Q^\star\) through the softmax likelihood;

2. the anchor-action Bellman equation fixes the remaining state-wise level of \(Q^\star\).

The ERM estimator is simply the one-shot version of these two restrictions. It parametrizes \(Q\) directly, rather than parametrizing \(r\) and repeatedly solving the forward Bellman equation. This is the step that removes the nested \(r\mapsto Q\) optimization that classical DDC methods require.

Throughout this section, \(a_s\) denotes the anchor action at state \(s\), and \[r_A(s):=r(s,a_s)\] denotes its known reward. The normalized presentation often sets \(r_A(s)=0\) for all \(s\); the formulation below allows any known anchor payoff, which is the notation used in the paper.

The build-up is as follows. Section 6.1 rewrites the classical identification equations as losses. Section 6.2 combines them into the population expected risk. Section 6.3 explains why the empirical problem cannot be obtained by naively replacing the Bellman expectation by one next-state sample. Section 6.4 introduces the conditional-mean correction that removes that bias. Sections 6.5 and 6.6 state the resulting GLADIUS algorithm and its conditional convergence guarantees.

6.1 From classical DDC equations to two losses↩︎

Recall the two soft optimality equations. If \[V_Q(s):=\operatorname{LSE}(Q(s,\cdot))=\log\sum_{b\in\mathcal{A}}\exp(Q(s,b)), \qquad \hat{p}_Q(a\mid s):={\exp(Q(s,a))\over\sum_{b\in\mathcal{A}}\exp(Q(s,b))},\] then the expert policy and the optimal \(Q\) satisfy \[\begin{align} \pi^\star(a\mid s)&=\hat{p}_{Q^\star}(a\mid s), \tag{52}\\ Q^\star(s,a)&=r(s,a)+\beta\mathbb{E}[V_{Q^\star}(s')\mid s,a]. \tag{53} \end{align}\] The first equation is what likelihood-based estimation uses. The second equation is what dynamic programming, CCP, and Bellman-residual methods try to enforce.

Under the anchor-action assumption, the key Magnac–Thesmar observation is that we do not need the Bellman equation for every action in order to identify \(Q^\star\). The likelihood condition 52 identifies all within-state differences \(Q^\star(s,a)-Q^\star(s,b)\). It is only missing one scalar level per state. The anchor-action Bellman equation supplies exactly that missing scalar: \[Q^\star(s,a_s)=r_A(s)+\beta\mathbb{E}[V_{Q^\star}(s')\mid s,a_s]. \label{eq:erm-anchor-bellman}\tag{54}\] This is the bridge from classical DDC to ERM. Rust’s nested fixed-point method enforces the full Bellman equation after parametrizing \(r\); Magnac–Thesmar identification says that, once the anchor is fixed, the likelihood plus the anchor-action Bellman equation is enough. Therefore, if we parametrize \(Q\) directly, the two identifying restrictions can be expressed as two loss-minimization problems.

The first loss is the negative log-likelihood, \[\mathcal{L}_{\mathrm{NLL}}(Q)(s,a):=-\log \hat{p}_Q(a\mid s). \label{eq:erm-nll-loss}\tag{55}\] By Lemma 10, its population minimizers are precisely the \(Q\) functions whose softmax equals the expert policy on the expert-covered states.

The second loss is the anchor Bellman-error loss. For any candidate \(Q\), define the anchor residual \[\Delta_A(Q)(s) :=r_A(s)+\beta\mathbb{E}[V_Q(s')\mid s,a_s]-Q(s,a_s). \label{eq:erm-anchor-residual}\tag{56}\] Then \(Q\) satisfies the anchor Bellman equation exactly when \(\Delta_A(Q)(s)=0\). Because the expert softmax puts positive probability on every action at each covered state, this is equivalently the minimizer set of \[\mathbb{E}_{(s,a)\sim d^{\pi^\star}} \left[ \mathbf{1}\{a=a_s\}\,\Delta_A(Q)(s)^2 \right]. \label{eq:erm-anchor-loss}\tag{57}\] This is the main conceptual simplification: the Bellman part contains no unknown reward term, because it is evaluated only at the anchor action where \(r_A(s)\) is known.

6.2 Expected risk: the one-shot population problem↩︎

Combining the two losses gives the population’s expected risk \[\mathcal{R}_{\exp}(Q) := \mathbb{E}_{(s,a)\sim d^{\pi^\star}} \left[ -\log\hat{p}_Q(a\mid s) +\mathbf{1}\{a=a_s\}\, \bigl(r_A(s)+\beta\mathbb{E}[V_Q(s')\mid s,a_s]-Q(s,a_s)\bigr)^2 \right]. \label{eq:R-exp}\tag{58}\] This is the population ERM-DDC/IRL objective. It has the flow that the classical methods suggest, but without the classical nesting:

fit the observed choices through NLL, and simultaneously fit the anchor Bellman equation through a squared residual.

The NLL term has no unknown reward in it. The anchor Bellman term also has no unknown reward in it. Thus, at the population level, the inverse problem has become a direct optimization over \(Q\).

6.3 Why squared TD is not the empirical Bellman error↩︎

The expected risk 58 is still a population object. The hard term is \[\mathbb{E}[V_Q(s')\mid s,a_s],\] which is an expectation over the transition kernel. If the transition kernel were known, one could evaluate this term directly. In the offline setting, however, we only observe samples \((s,a,s')\).

The first tempting move is to replace the conditional expectation by the one observed next state. For a general state-action pair, define the sampled Bellman operator \[\widehat{\mathcal{T}}Q(s,a,s'):=r(s,a)+\beta V_Q(s'), \label{eq:erm-sampled-operator}\tag{59}\] and define the sampled TD residual \[\widehat\Delta(Q)(s,a,s'):=\widehat{\mathcal{T}}Q(s,a,s')-Q(s,a). \label{eq:erm-td-residual}\tag{60}\] At anchor actions, \(r(s,a)\) is the known quantity \(r_A(s)\). The sampled operator is unbiased for the Bellman operator: \[\mathbb{E}[\widehat{\mathcal{T}}Q(s,a,s')\mid s,a]=(\mathcal{T}Q)(s,a).\] But the squared TD loss is not unbiased for the squared Bellman error. The variance decomposition gives \[\mathbb{E}\bigl[\widehat\Delta(Q)(s,a,s')^2\mid s,a\bigr] = \bigl((\mathcal{T}Q)(s,a)-Q(s,a)\bigr)^2 + \beta^2\operatorname{Var}(V_Q(s')\mid s,a). \label{eq:erm-td-bias}\tag{61}\] The second term is not a harmless constant: it depends on \(Q\). Therefore, minimizing squared TD error generally does not minimize squared Bellman error. This is the double-sampling problem in this setting. It disappears under deterministic transitions, but not under stochastic transitions.

6.4 Bias correction and the minimax ERM formulation↩︎

The correction is to subtract the conditional variance term in 61 without estimating the transition kernel. The variance has the least-squares representation \[\operatorname{Var}(V_Q(s')\mid s,a) = \min_{z\in\mathbb{R}}\mathbb{E}[(V_Q(s')-z)^2\mid s,a], \qquad z^\star_Q(s,a)=\mathbb{E}[V_Q(s')\mid s,a]. \label{eq:erm-var-regression}\tag{62}\] Promote the scalar minimizer to a function \(\zeta(s,a)\). Combining 61 and 62 yields the corrected Bellman-error identity \[\begin{align} \bigl((\mathcal{T}Q)(s,a)-Q(s,a)\bigr)^2 &= \max_{\zeta} \mathbb{E}\left[ \widehat\Delta(Q)(s,a,s')^2 -\beta^2\bigl(V_Q(s')-\zeta(s,a)\bigr)^2 \mid s,a \right]. \label{eq:erm-corrected-be} \end{align}\tag{63}\] The maximization appears because subtracting the best least-squares error is the same as maximizing the negative squared error.

Plugging the corrected identity into the anchor-weighted expected risk gives the population minimax version of ERM-DDC/IRL: \[\begin{align} \min_Q\max_\zeta\; & \mathbb{E}_{(s,a)\sim d^{\pi^\star},\,s'\sim P(\cdot\mid s,a)} \biggl[ -\log\hat{p}_Q(a\mid s) \notag\\ &\quad +\mathbf{1}\{a=a_s\} \biggl\{ \bigl(r_A(s)+\beta V_Q(s')-Q(s,a)\bigr)^2 -\beta^2\bigl(V_Q(s')-\zeta(s,a)\bigr)^2 \biggr\} \biggr]. \label{eq:erm-pop-minimax} \end{align}\tag{64}\] The anchor indicator is important. The Bellman-identification term is imposed at the anchor action because that is where the reward is known. The auxiliary function \(\zeta\), however, has an additional role: at the solution, \[\zeta^\star_{Q^\star}(s,a)=\mathbb{E}[V_{Q^\star}(s')\mid s,a],\] so it is exactly the continuation-value object needed to recover rewards for all supported state-action pairs: \[r(s,a)=Q^\star(s,a)-\beta\zeta^\star_{Q^\star}(s,a). \label{eq:erm-zeta-reward}\tag{65}\] For this reason, the paper estimates \(\zeta\) over observed state-action pairs, not only at anchor observations. The anchor-weighted corrected Bellman term identifies \(Q^\star\); the all-pair conditional-mean regression makes reward recovery transition-estimation-free.

Replacing the population expectations in 64 and 62 by empirical averages gives the finite-sample objective. For parametrizations \(Q_\theta\) and \(\zeta_\phi\), define the auxiliary regression loss \[\widehat F_\zeta(\phi;\theta) := {1\over N}\sum_{(s,a,s')\in\mathcal{D}} \bigl(V_{Q_\theta}(s')-\zeta_\phi(s,a)\bigr)^2. \label{eq:zeta-regression}\tag{66}\] For fixed \(Q_\theta\), descending this loss estimates the conditional mean \(\mathbb{E}[V_{Q_\theta}(s')\mid s,a]\) on the observed support. Given a current auxiliary function, the empirical \(Q\) objective is \[\begin{align} \widehat f_N(\theta,\phi) := {1\over N}\sum_{(s,a,s')\in\mathcal{D}} \biggl[ &-\log\hat{p}_{Q_\theta}(a\mid s) \notag\\ &+\mathbf{1}\{a=a_s\} \biggl\{ \bigl(r_A(s)+\beta V_{Q_\theta}(s')-Q_\theta(s,a)\bigr)^2 -\beta^2\bigl(V_{Q_\theta}(s')-\zeta_\phi(s,a)\bigr)^2 \biggr\} \biggr]. \label{eq:gladius-objective} \end{align}\tag{67}\] Equivalently, one may view the profiled empirical risk as \[\widehat g_N(\theta):=\widehat f_N(\theta,\widehat\phi(\theta)), \qquad \widehat\phi(\theta)\in\operatorname*{arg\,min}_\phi \widehat F_\zeta(\phi;\theta).\] This is the empirical-risk version of the expected-risk problem above.

6.5 GLADIUS: alternating gradients for the corrected ERM↩︎

GLADIUS solves the empirical problem by alternating the two natural gradient steps. The \(\zeta\) step is a regression step: fit \(\zeta_\phi(s,a)\) to \(V_{Q_\theta}(s')\). The \(Q\) step is a descent step on the NLL plus the anchor-weighted corrected Bellman term.

The language “ascent–descent” comes from the minimax display 64 : maximizing the negative quadratic correction in \(\zeta\) is equivalent to descending the positive regression loss 66 . The practical algorithm uses the regression form because it is clearer and numerically standard.

6.6 PL geometry and convergence↩︎

The objective in 67 is generally nonconvex in \(\theta\). The paper’s convergence argument therefore relies on Polyak–Łojasiewicz geometry, not convexity. A differentiable function \(f\) satisfies a PL inequality with constant \(c>0\) on a set \(U\) if \[{1\over2}\|\nabla f(x)\|_2^2 \ge c\bigl(f(x)-f^\star\bigr), \qquad x\in U. \label{eq:pl-def}\tag{68}\] PL is weaker than convexity but still rules out spurious stationary points on the relevant set.

For the minimax objective, the relevant geometry is two-sided. The profiled \(Q\) objective must satisfy PL in \(\theta\), while the auxiliary regression problem must satisfy the corresponding PL condition in \(\phi\) for the ascent side (equivalently, the regression loss is PL for descent).

Before stating the conditioning assumption, it is useful to say what kind of condition it is. We are not assuming directly that the nonconvex parameter-space objective is PL. Instead, the argument first proves PL-type inequalities in the finite empirical output coordinates, and then transfers them to the parameters through the Jacobians of the output maps \[\theta\mapsto (Q_\theta(z))_{z\in Z_Q}, \qquad \phi\mapsto (\zeta_\phi(\bar z))_{\bar z\in Z_\zeta}.\] This transfer condition is checkable for common parametrizations. For linear models it is exactly a full-row-rank empirical feature condition. For sufficiently wide smooth neural networks, the same type of lower bound is obtained in the usual NTK/lazy-training regime on any fixed finite empirical evaluation set. We state the assumption first and then record these two verification cases as lemmas.

The main takeaway is now straightforward. ERM-DDC/IRL is a transition-estimation-free minimax estimator for \(Q^\star\). It is built by translating the two classical identifying equations into losses, correcting the empirical Bellman term for double-sampling bias, and recovering rewards from the same learned conditional-mean correction \(\widehat\zeta\).

Closing Remarks↩︎

The classical DDC literature gave us identification through the anchor-action argument, plus computational templates such as NFXP and CCP. The modern IRL literature gave us scalable objectives, but often without pointwise Bellman identification of the reward. The ERM framework of [9] combines the useful pieces from both sides: use the likelihood term for within-state action-value differences, use the anchor Bellman equation for the state-wise level, use the bias-corrected TD identity to avoid transition-kernel estimation, and solve the resulting objective by alternating gradients. This is why the section should be read as a build-up from classical DDC to a one-shot gradient-based estimator, rather than as an isolated minimax formula.

7 Deferred Proofs↩︎

7.1 Local AIRL Gradient and Policy-Objective Identities↩︎

For completeness, we give the precise finite-horizon version of Lemma 8. Work on a finite horizon \(H\) and suppress discounting; the discounted version inserts the weight \(\beta^t\) in every time-\(t\) term. Write a transition as \(x_t := (s_t,a_t,s_{t+1})\). For any policy \(\pi\), write \[\rho_t^\pi(s,a,s') \;:=\; \mathbb{P}_{\nu_0,\pi}(s_t=s)\,\pi(a\mid s)\,P(s'\mid s,a)\] for the time-\(t\) transition marginal, and define the policy’s pre-action transition base \[b_t^\pi(s,a,s') \;:=\; \mathbb{P}_{\nu_0,\pi}(s_t=s)\,P(s'\mid s,a), \qquad \rho_t^\pi(s,a,s')=b_t^\pi(s,a,s')\pi(a\mid s).\] For a differentiable transition score \(f_\omega(s,a,s')\), define the finite-horizon maximum-causal-entropy trajectory model \[p_\omega(\tau) \;:=\; \frac{1}{Z_\omega}\,\nu_0(s_0)\prod_{t=0}^{H-1}P(s_{t+1}\mid s_t,a_t) \exp\!\Bigl(\sum_{t=0}^{H-1}f_\omega(s_t,a_t,s_{t+1})\Bigr), \label{eq:airl-mce-trajectory-model}\tag{69}\] and let \(p_{\omega,t}\) denote its transition marginal at time \(t\). The maximum-causal-entropy log-likelihood is \[\mathcal{J}_{\mathrm{MCE}}(\omega) \;:=\; \mathbb{E}_{\nu_0,\pi^\star}[\log p_\omega(\tau)].\] For any sampler \(\mu_t\) whose support contains that of \(p_{\omega,t}\), the same gradient can be written in importance-sampled form. AIRL’s local approximation replaces the intractable marginal \(p_{\omega,t}\) by the policy-based energy measure \[\widehat p_{\omega,t}^{\pi}(s,a,s') \;:=\; \exp(f_\omega(s,a,s'))\,b_t^\pi(s,a,s'), \label{eq:airl-local-energy-marginal}\tag{70}\] and compares it with the policy transition marginal through \[\widehat\mu_{\omega,t}^{\pi}(s,a,s') \;:=\; \tfrac12\widehat p_{\omega,t}^{\pi}(s,a,s')+\tfrac12\rho_t^\pi(s,a,s'). \label{eq:airl-local-mixture}\tag{71}\] The following proposition is the rigorous version of the localization statement used in the main text.

References↩︎