1 Introduction↩︎

Sequential decision-making under uncertainty requires balancing exploitation of current knowledge against exploration to reduce uncertainty. Classical reinforcement learning and optimal control address this through value functions or policy optimization [1], [2], but typically treat reward maximization and uncertainty reduction as separate objectives.

Planning-as-Inference (PAI) offers an alternative by casting control as probabilistic inference [3], [4], connecting control to variational inference and message passing [5]. Standard PAI methods optimize objectives such as expected utility or cross-entropy to preferences, but do not include an explicit epistemic drive to reduce environmental uncertainty.

Active Inference addresses this by minimizing the Expected Free Energy (EFE), unifying instrumental and epistemic objectives [6], [7]. [8] showed that EFE minimization can be reformulated as Variational Free Energy (VFE) minimization on a model augmented with epistemic priors. This brings Active Inference into the variational framework, but leaves open a key distinction: obtaining EFE inside a marginal variational objective is not yet the same as planning over policies. Proper planning additionally requires the planning correction of [9]. This paper makes that separation explicit and derives a message-passing scheme for the combined objective.

This paper combines these two lines of work. Our contributions are:

• We show that proper EFE-based planning requires combining two entropy corrections: the planning correction of [9], which turns the expected-utility variational objective into policy optimization, and the epistemic corrections of [10], which turn marginal VFE minimization into EFE minimization. Together they yield a full variational characterization of EFE-based planning and clarify the difference between EFE as a marginal objective and EFE-based planning.

• We derive a principled message-passing family for these entropy-corrected objectives. Each added entropy term induces a corresponding channel reparameterization that restores Bethe coordinates, resolves the circularity of posterior-dependent epistemic priors, and recovers both variational belief propagation and full active-inference planning within the same derivation.

• We validate the framework on three grid-world environments that span a hierarchy of epistemic demands along two axes, observation scope (global vs.local) and resolution (decisive vs.suggestive). The experiments show where progressively adding corrections matters: the planning correction already helps under decisive observations, while the additional observation-side epistemic corrections matter most under suggestive observations.

2 reviews the generative model and epistemic priors. 3 discusses related work. 4 presents the entropy corrections and their cumulative taxonomy. 5 derives the resulting message-passing family. 6 provides empirical validation, and 7 concludes.

2 Background↩︎

2.1 Generative Model for Sequential Decision-Making↩︎

We consider an agent that maintains a generative model predicting future observations, states, and the consequences of actions. Following standard conventions [5], [9], we write this as a rollout model: \[\begin{align} p(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) = {} & p(\theta) p(x_0) \prod_{t=1}^T p(y_t | x_t, \theta) \notag \\ & \cdot p(x_t | x_{t-1}, u_t, \theta) \, p(u_t)\,, \label{eq:generative95model} \end{align}\tag{1}\]

where \(\boldsymbol{x} = (x_0, \ldots, x_T)\) are latent states, \(\boldsymbol{y} = (y_1, \ldots, y_T)\) are observations, \(\boldsymbol{u} = (u_1, \ldots, u_T)\) are actions, and \(\theta\) are unknown model parameters. Here \(t = 0\) denotes the current time, and the model predicts a rollout into the future over horizon \(T\). The dynamics \(p(x_t | x_{t-1}, u_t, \theta)\) may depend on parameters \(\theta\), capturing model uncertainty. Throughout this paper we work in the discrete regime, so all integrals over \((y_t, x_t, \theta)\) in what follows reduce to finite sums.

To encode goals, we augment the model with preference priors \(\hat{p}(x_t)\) and \(\hat{p}(y_t)\) over desired states and observations [5]: \[\begin{align} \hat{p}(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \propto {} & p(\theta) p(x_0) \prod_{t=1}^T p(y_t | x_t, \theta) \, p(x_t | x_{t-1}, u_t, \theta) \notag \\ & \cdot p(u_t) \, \hat{p}(x_t) \, \hat{p}(y_t)\,. \label{eq:biased95model} \end{align}\tag{2}\] These preference priors can be understood as proportional to exponentiated rewards: \(\hat{p}(x) \propto \exp(R(x))\), connecting planning-as-inference to reward maximization [11]. Together, the rollout model 2 with preferences \(\hat{p}(x_t)\) and \(\hat{p}(y_t)\) defines our planning problem over horizon \(T\): find a policy \(q(u_t | x_{t-1})\) whose induced predicted trajectory agrees with the preferences. The policy is the decision variable; the rollout supplies predictions; the preferences encode the goal.

2.2 Variational Free Energy↩︎

Given a generative model, variational inference approximates the posterior by minimizing the Variational Free Energy (VFE) over a family of tractable distributions \(q\) [12]: \[\label{eq:vfe95def} F_{\hat{p}}[q] = \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \| \hat{p}(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \right]}{\csname l\endcsname[q(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \| \hat{p}(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \csname r\endcsname]}\,.\tag{3}\] Since all variables are unobserved in the planning setting (they represent future quantities), minimizing \(F_{\hat{p}}[q]\) yields beliefs about future trajectories that are consistent with both the dynamics and the preference priors.

2.3 Factor Graphs and the Bethe Approximation↩︎

The generative model 2 factorizes into local terms, which can be represented as a Forney-style factor graph (FFG) [13], [14]. In an FFG, nodes represent factors (probability distributions) and edges represent variables; an edge connects to a node when the variable appears in that factor’s scope. We write \(\mathcal{E}(a)\) for the set of edges (variables) adjacent to factor node \(a\), and \(\mathcal{V}(i)\) for the set of factor nodes adjacent to edge \(i\). The variables in the scope of factor \(a\) are denoted \(\boldsymbol{s}_a\).

The Bethe approximation [15] exploits this structure by constraining the variational distribution to respect the factorization induced by the graph. Each node \(a\) maintains a local belief \(q_a(\boldsymbol{s}_a)\) over its adjacent variables \(\boldsymbol{s}_a\), and each edge \(i\) maintains a singleton belief \(q_i(s_i)\). These beliefs must satisfy local consistency constraints: \[\label{eq:local95consistency} \int q_a(\boldsymbol{s}_a) \, \mathrm{d}\boldsymbol{s}_{a \setminus i} = q_i(s_i) \quad \text{for all } i \in \mathcal{E}(a)\,.\tag{4}\] Under these constraints, with entropy corrections that prevent double-counting of shared variables, the VFE reduces to the Bethe Free Energy: \[\begin{align} F_{\text{Bethe}}[q] = {} & \sum_{a \in \mathcal{V}} \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q_a(\boldsymbol{s}_a) \| f_a(\boldsymbol{s}_a) \right]}{\csname l\endcsname[q_a(\boldsymbol{s}_a) \| f_a(\boldsymbol{s}_a) \csname r\endcsname]} \notag \\ & + \sum_{i \in \mathcal{E}} (d_i - 1) \, \mathbb{H} \ifblank{}{\left[q_i(s_i)\right]}{\csname l\endcsname[q_i(s_i)\csname r\endcsname]}\,, \label{eq:bethe95free95energy} \end{align}\tag{5}\] where \(\mathcal{V}\) is the set of nodes, \(\mathcal{E}\) is the set of edges, \(f_a\) is the factor at node \(a\), and \(d_i\) is the degree (number of connected nodes) of edge \(i\). Minimizing the Bethe Free Energy via message passing yields the belief propagation algorithm; on tree-structured graphs, this recovers exact marginals [16]. Details are provided in Appendix 10.

2.4 Epistemic Priors↩︎

Standard variational inference does not distinguish between variable types: actions, states, observations, and parameters all enter the VFE symmetrically. [10] clarified the epistemic priors \(\tilde{p}(u_t)\), \(\tilde{p}(x_t)\), and \(\tilde{p}(y_t, x_t)\) that encode which variables are controlled, inferred, or observed. These priors augment the generative model: \[\begin{align} \tilde{p}(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \propto {} & \hat{p}(\boldsymbol{y}, \boldsymbol{x}, \boldsymbol{u}, \theta) \notag \\ & \prod_{t=1}^T \tilde{p}(u_t) \, \tilde{p}(x_t) \, \tilde{p}(y_t, x_t)\,. \label{eq:augmented95model} \end{align}\tag{6}\]

Each prior is defined in terms of entropies of conditionals¹ of the variational distribution \(q\): \[\tag{7} \begin{gather} \tilde{p}(u_t) \propto \exp\bigl( \mathrm{h} \ifblank{}{\left[q(x_t, x_{t-1} | u_t)\right]}{\csname l\endcsname[q(x_t, x_{t-1} | u_t)\csname r\endcsname]} - \mathrm{h} \ifblank{}{\left[q(x_{t-1} | u_t)\right]}{\csname l\endcsname[q(x_{t-1} | u_t)\csname r\endcsname]}\bigr)\,, \tag{8} \\ \tilde{p}(x_t) \propto \exp \bigl( \mathbb{E}_{q(\theta | x_t)} \ifblank{}{\left[- \mathrm{h} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]}\right]}{\csname l\endcsname[- \mathrm{h} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]}\csname r\endcsname]}\bigr)\,, \tag{9} \\ \tilde{p}(y_t, x_t) \propto \exp\bigl( \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q(\theta | y_t, x_t) \| q(\theta | x_t) \right]}{\csname l\endcsname[q(\theta | y_t, x_t) \| q(\theta | x_t) \csname r\endcsname]}\bigr)\,. \tag{10} \end{gather}\] [10] showed that the VFE of the augmented model \(F_{\tilde{p}}[q]\) is an upper bound on the expected EFE. A notable feature is that the epistemic priors depend on the variational distribution \(q\) itself, creating a circular dependency that complicates optimization. A central contribution of this paper is to make that circularity explicit as entropy corrections in the objective, rather than leaving it implicit in posterior-dependent priors.

3 Related Work↩︎

3.0.0.1 Planning-as-Inference.

The PAI framework casts optimal control as inference in graphical models [3], [4], connecting control to variational methods and message passing [5]. Closely related formulations include linearly-solvable MDPs [17], path-integral control [18], KL control [19], and stochastic optimal control [20]. A known challenge is optimistic inference: conditioning on goals biases posteriors toward trajectories assuming favorable outcomes [5]. This issue was addressed by [9] with an entropy correction that turns the expected-utility variational objective into a proper control objective by penalizing plans that rely on fortuitous state realizations.

3.0.0.2 Active inference.

Active inference minimizes the Expected Free Energy (EFE), combining instrumental and epistemic value [6], [7], [21]. Existing methods employ specialized procedures: tree search [22], branching [23], or dynamic programming [24]. Several works have sought to unify EFE with variational inference. In a related direction, [25] combined estimation and control via belief propagation. Building on the Generalized Free Energy [26], [27] and [28] modified the VFE to include epistemic terms. Most recently, [10] showed that EFE minimization can be formulated as VFE minimization with epistemic priors, and [29] implemented this via message passing with alternating updates between the posterior and epistemic priors. A separate line of work [30], [31] casts exploration as posterior inference over value functions, targeting uncertainty in the value function itself. This is complementary to the epistemic priors above, which target uncertainty over model parameters \(\theta\). Our contribution is to connect these lines: the de Vries construction provides the EFE correction to a marginal objective, the Lazaro-Gredilla construction provides the planning correction, and their combination yields a principled message-passing formulation of EFE-based planning.

4 Entropy Corrections for EFE-Based Planning↩︎

We now show that the epistemic priors from 2.4 and the planning correction of [9] play different roles. The epistemic priors identify the corrections that transform marginal VFE minimization into EFE minimization. The planning correction turns an expected-utility variational objective into a planning objective over policies. Proper active-inference planning requires both. More broadly, specifying a planning method is a three-way modeling choice: the generative model, the variable-role assignment between controlled, states, parameters, and observed quantities, and the entropy-correction selecting the objective. The AIF-specific commitment lives entirely in the last.

4.1 Entropy Corrections in Variational Inference↩︎

Recall the generative model with preference priors from 2.1. Standard variational inference minimizes the VFE \(F_{\hat{p}}[q]\) of the preference-augmented model without distinguishing variable roles. With no entropy corrections, this is simply marginal inference, or in the control setting, KL control [11]². Different objectives arise by adding entropy corrections to this same baseline. The key question is which corrections are needed for proper EFE-based planning.

4.2 Cross-Entropy Planning↩︎

Marginal variational inference minimizes a cost over the full \(q\), which lets the joint commit to favourable state realizations that the policy alone cannot produce. [9] showed that turning this into planning, where the extracted policy \(q(u_t|x_{t-1})\) actually attains the cost it appears to minimize, requires an entropy correction that penalizes action uncertainty: \[\label{eq:cross95entropy95correction} \sum_{t=1}^T \mathbb{H} \ifblank{}{\left[q(x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_{t-1}, u_t)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(x_{t-1})\right]}{\csname l\endcsname[q(x_{t-1})\csname r\endcsname]} = \sum_{t=1}^T \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\,.\tag{11}\] See Appendix 8.1 for the derivation.

Following the control-as-inference framework [5], rewards can be encoded as preference distributions via \(\hat{p}(x) \propto \exp(R(x))\). As shown in Appendix 8.3, adding the entropy correction 11 to the VFE transforms the objective into minimizing the cross-entropy between the state marginals and the preference distribution: \[\label{eq:cross95entropy95objective} \min_{q} \sum_{t=1}^T \mathbb{H} \ifblank{}{\left[q(x_t), \hat{p}(x_t)\right]}{\csname l\endcsname[q(x_t), \hat{p}(x_t)\csname r\endcsname]} + \text{const}\,,\tag{12}\] where \(\mathbb{H} \ifblank{}{\left[q, \hat{p}\right]}{\csname l\endcsname[q, \hat{p}\csname r\endcsname]} = - \mathbb{E}_{q} \ifblank{}{\left[\log \hat{p}\right]}{\csname l\endcsname[\log \hat{p}\csname r\endcsname]}\) is the cross-entropy. Since \(\mathbb{H} \ifblank{}{\left[q, \hat{p}\right]}{\csname l\endcsname[q, \hat{p}\csname r\endcsname]} = - \mathbb{E}_{q} \ifblank{}{\left[R(x)\right]}{\csname l\endcsname[R(x)\csname r\endcsname]} + \text{const}\), minimizing cross-entropy is equivalent to maximizing expected reward.

We call this cross-entropy planning: the agent maximizes expected reward (equivalently, minimizes cross-entropy to preferences) while committing to a policy.

4.3 EFE as Entropy Corrections↩︎

The epistemic priors introduced in 2 augment the generative model with terms that encode variable roles. The VFE of this augmented model can be expressed as the original VFE plus entropy corrections. This rewriting is an exact algebraic identity.

Each prior contributes a specific correction: \(\tilde{p}(u_t)\) produces \(- \mathbb{H} \ifblank{}{\left[q(x_t | x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_t | x_{t-1}, u_t)\csname r\endcsname]}\), \(\tilde{p}(x_t)\) produces \(+ \mathbb{H} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]}\), and \(\tilde{p}(y_t, x_t)\) contributes a further \(+ \mathbb{H} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(y_t | x_t)\right]}{\csname l\endcsname[q(y_t | x_t)\csname r\endcsname]}\) via the identity \(\mathbb{E}_{q} \ifblank{}{\left[ \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q(\theta|y_t,x_t) \| q(\theta|x_t) \right]}{\csname l\endcsname[q(\theta|y_t,x_t) \| q(\theta|x_t) \csname r\endcsname]}\right]}{\csname l\endcsname[ \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q(\theta|y_t,x_t) \| q(\theta|x_t) \right]}{\csname l\endcsname[q(\theta|y_t,x_t) \| q(\theta|x_t) \csname r\endcsname]}\csname r\endcsname]} = \mathbb{H} \ifblank{}{\left[q(y_t|x_t)\right]}{\csname l\endcsname[q(y_t|x_t)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(y_t|x_t,\theta)\right]}{\csname l\endcsname[q(y_t|x_t,\theta)\csname r\endcsname]}\), producing the factor of two. The interplay yields EFE minimization within a marginal variational objective: minimizing the \(+2\, \mathbb{H} \ifblank{}{\left[q(y_t|x_t,\theta)\right]}{\csname l\endcsname[q(y_t|x_t,\theta)\csname r\endcsname]}\) term concentrates beliefs on state-parameter configurations under which observations are sharply informative, which is the operational meaning of epistemic in AIF. The two channel reparameterizations introduced in 5 play opposing roles consistent with this reading: the dynamics channel divides and spreads belief over reachable states, while the observation channel multiplies and concentrates belief toward informative ones. By itself, however, ?? does not yet yield EFE-based planning, because it lacks the planning correction 11 .

4.4 EFE-Based Planning↩︎

The missing step is to combine the marginal-EFE corrections of 1 with the planning correction of 4.2. Adding only the dynamics-side term \(- \mathbb{H} \ifblank{}{\left[q(x_t | x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_t | x_{t-1}, u_t)\csname r\endcsname]}\) yields an incomplete intermediate objective that is useful as an ablation, but it is not yet full EFE-based planning because it omits the observation-side epistemic corrections.

Appendix 9 proves that the resulting EFE-based planning objective is \[\begin{align} \min_q\; & F_{\hat{p}}[q] + \sum_{t=1}^T \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]} + \sum_{t=1}^T \Bigl( 2 \mathbb{H} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]} \notag \\ & \qquad\quad - \mathbb{H} \ifblank{}{\left[q(x_t | x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_t | x_{t-1}, u_t)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(y_t | x_t)\right]}{\csname l\endcsname[q(y_t | x_t)\csname r\endcsname]} \Bigr). \label{eq:combined95main95text} \end{align}\tag{13}\] The first sum is the planning correction; the second sum is the EFE correction. Only their combination yields proper EFE-based planning.

4.5 Comparison of Objectives↩︎

1 summarizes the progression: the planning correction changes how control is posed (cross-entropy planning), the EFE correction changes what objective is optimized (marginal EFE), and only their combination yields proper EFE-based planning, the objective implemented in our experiments. The channel reparameterizations required for message passing follow directly from these correction terms (5.1); we turn to that next.

5 Message Passing for EFE-Based Planning↩︎

The full EFE-based planning objective 13 contains policy, dynamics, and observation conditional entropy terms: the planning correction \(+ \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\) together with the three EFE corrections \(+2 \mathbb{H} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]}\), \(- \mathbb{H} \ifblank{}{\left[q(x_t | x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_t | x_{t-1}, u_t)\csname r\endcsname]}\), and \(- \mathbb{H} \ifblank{}{\left[q(y_t | x_t)\right]}{\csname l\endcsname[q(y_t | x_t)\csname r\endcsname]}\). In the Bethe framework, these conditionals are ratios of region beliefs, which are not part of the optimization objective. We resolve this by introducing auxiliary conditional distributions (channels) that promote the conditional region beliefs to free variational parameters in the optimization objective, yielding a message-passing family that generalizes standard belief propagation.

5.1 Channel Reparameterization↩︎

We work throughout with Forney factor graphs (FFGs) [13], [32], in which factors are nodes and variables are edges (Figure 1); this makes the locality of channel reparameterization visually explicit, since each correction acts on a single kernel node while the remainder of the graph is unchanged from standard sum-product. The key identity is the variational characterization of conditional entropy (Gibbs’ inequality): \[\label{eq:gibbs95identity} \mathbb{H} \ifblank{}{\left[q(y | x)\right]}{\csname l\endcsname[q(y | x)\csname r\endcsname]} = \min_{r} \mathbb{E}_{q(y, x)} \ifblank{}{\left[-\log r(y | x)\right]}{\csname l\endcsname[-\log r(y | x)\csname r\endcsname]}\,,\tag{14}\] with equality when \(r(y|x) = q(y|x)\). Since the minimum over normalized distributions \(r\) is attained at \(r(y|x) = q(y|x)\), the expression is an equality rather than a bound. We introduce four normalized conditional distributions as channels: \(r_{u|x,t}(u_t | x_{t-1})\), \(r_{x|xu,t}(x_t | x_{t-1}, u_t)\), \(r_{y|x\theta,t}(y_t | x_t, \theta)\), and \(r_{y|x,t}(y_t | x_t)\), as free variational parameters for each time step \(t\) (see Appendix 11 for formal definitions). Substituting 14 into the corrections of 13 yields a well-posed optimization. Since \(+ \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\) carries a positive sign, \(r_{u|x,t}\) enters in the numerator of the dynamics kernel. Since \(+2 \mathbb{H} \ifblank{}{\left[q(y_t | x_t, \theta)\right]}{\csname l\endcsname[q(y_t | x_t, \theta)\csname r\endcsname]}\) carries a positive sign, \(r_{y|x\theta,t}\) enters squared in the numerator of the observation kernel. Since \(- \mathbb{H} \ifblank{}{\left[q(y_t | x_t)\right]}{\csname l\endcsname[q(y_t | x_t)\csname r\endcsname]}\) carries a negative sign, \(r_{y|x,t}\) appears in the denominator. The dynamics correction \(- \mathbb{H} \ifblank{}{\left[q(x_t | x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_t | x_{t-1}, u_t)\csname r\endcsname]}\) is also negative, so \(r_{x|xu,t}\) divides the dynamics factor. This yields the kernels: \[\tag{15} \begin{equation} \tilde{f}_{\mathrm{obs}_t}(y_t, x_t, \theta) = \frac{p(y_t | x_t, \theta)\, r_{y|x\theta,t}^2(y_t | x_t, \theta)}{r_{y|x,t}(y_t | x_t)}, \tag{16} \end{equation} \begin{equation} \tilde{f}_{\mathrm{dyn}_t}(x_t, x_{t-1}, \theta, u_t) = \frac{p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})}{r_{x|xu,t}(x_t | x_{t-1}, u_t)}. \tag{17} \end{equation}\] With these substitutions, the EFE-based planning objective becomes a standard Bethe free energy over the modified factor graph, jointly optimized over beliefs and channels. The kernels 15 replace the original factor functions in the message-passing equations, making the procedure iterative: the channel beliefs \(r\) depend on variational beliefs \(q\) and vice versa. The proof for \(T=1\) is given in Appendix 11. The full scheme comes from the additivity of the Lagrangian and the entropic corrections, see Appendix 11.6 for details.

5.2 Message-Passing Equations↩︎

Since the modified objective has a Bethe form, the stationarity conditions yield sum-product-style message updates. The only difference from standard belief propagation is that each factor uses its kernel 15 in place of the original.

Each factor \(a\) sends to a neighboring factor \(b\) the integral of its kernel over incoming messages on adjacent edges: \[\label{eq:factor95to95var} \mu_{jb}(s_j) \propto \!\int\! \tilde{f}_a(\boldsymbol{s}_a)\!\! \prod_{i \in \mathcal{E}(a) \setminus j}\!\! \mu_{ia}(s_i) \, \mathrm{d}\boldsymbol{s}_{a \setminus j}\,.\tag{18}\] For unmodified factors (priors, data likelihoods), \(\tilde{f}_a = f_a\).

5.2.0.1 Singleton beliefs.

Singleton beliefs are computed by normalizing the product of colliding messages on an edge, \[\label{eq:singleton95beliefs} q^*(s_i) \propto \mu_{ia}(s_i)\mu_{ib}(s_i)\,,\tag{19}\] with \(\{a,b\} = \mathcal{V}(i)\) the nodes adjacent to edge \(i\). The full forward-backward schedule is shown in Fig. 2.

5.2.0.2 Region beliefs

Region beliefs are computed by multiplying the factor function with all inbound messages and normalizing, \[\label{eq:region-belief-update} q^*(\boldsymbol{s}_a) \propto \tilde{f}_a(\boldsymbol{s}_a) \prod_{i \in \mathcal{E}(a)} \mu_{ia}(s_i)\,.\tag{20}\]

5.2.0.3 Channel updates.

At the fixed point, each channel recovers the true conditional under its factor belief: \[\tag{21} \begin{align} r_{u|x,t}^*(u_t | x_{t-1}) & = q_{u|x,t}^*(u_t | x_{t-1})\,, \tag{22} \\ r_{y|x\theta,t}^*(y_t | x_t, \theta) & = q_{y|x\theta,t}^*(y_t | x_t, \theta)\,, \tag{23} \\ r_{y|x,t}^*(y_t | x_t) & = q_{y|x,t}^*(y_t | x_t)\,, \tag{24} \\ r_{x|xu,t}^*(x_t | x_{t-1}, u_t) & = q_{x|xu,t}^*(x_t | x_{t-1}, u_t)\,, \tag{25} \end{align}\] where the beliefs \(q\) are conditionals derived from the respective region beliefs around factors \(f_{\text{obs}_t}\) and \(f_{\text{dyn}_t}\) at time \(t\). The marginal observation channel \(r_{y|x,t}^*\) is obtained by marginalizing \(\theta\) from the observation factor belief (see Appendix 11).

The same construction yields a family of algorithms: VBP uses only the policy reparameterization, a dynamics-only ablation additionally uses the dynamics-side reparameterization, and full EFE-based planning further adds the observation-side reparameterizations. The corresponding VBP derivation is given in Appendix 12.

5.3 Convergence↩︎

The kernels 15 contain opposing channel corrections (4): \(r_{u|x}\) and \(r_{y|x\theta}\) appear in numerators, while \(r_{x|xu}\) and \(r_{y|x}\) appear in denominators, inducing a min-max structure in the joint optimization. Because each channel reparameterization is a local rewrite of a single kernel, the per-update cost matches standard loopy belief propagation up to the channel updates. Standard BP convergence guarantees, however, do not transfer to this min-max setting, and we apply arithmetic damping to channels for each update \(n\): \[\label{eq:damping} r_c^{n} \propto (1 - \lambda)\, r_c^{n-1} + \lambda\, r_c^*\,,\tag{26}\] for each channel \(c \in \{u|x, x|xu, y|x\theta, y|x\}\). Here \(\lambda \in [0, 1]\) is the damping parameter and \(r_c^*\) denotes the newly computed channel from 21 . We select \(\lambda\) per method and environment from a convergence sweep over \(\lambda \in \{0.25, 0.4, 0.5, 0.6, 0.75, 0.9\}\); at the selected \(\lambda\) the channel-based methods reach a stationary VFE plateau within \(15\)–\(150\) iterations (see Appendix 13.5).

6 Experiments↩︎

We design experiments to test the behavioral effect of progressively adding the entropy corrections in 1. We evaluate on three grid-world environments with distinct uncertainty profiles: one dominated by observation noise, one requiring spatial planning to gather distance-dependent observations, and one requiring joint reasoning over dynamics and observations. Full experimental details are deferred to Appendix 13³. All environments use discrete state spaces with exact factor evaluations, isolating the effect of the entropy corrections and channel-augmented schemes from errors introduced by approximate message computation.

6.1 Setup↩︎

6.1.0.1 Environments.

We adapt three classic grid-world environments into epistemic planning benchmarks by treating the environment layout as an unknown parameter \(\theta\) in the generative model 1 . All environments support cardinal movement but differ in observation structure, which determines which entropy corrections are needed. We characterize each environment along two axes: scope, whether a single observation constrains \(\theta\) globally or only locally, and resolution, whether a precise observation decisively identifies \(\theta\) or merely narrows the possibilities (suggestive).

Frozen Lake [33], [34] (global, decisive): The agent observes binary “hole/safe” sensors for every cell on the grid, with noise that increases with distance from the agent. A low-noise reading directly constrains which configurations are consistent, so a single precise observation decisively reveals \(\theta\).

RockSample (\(5,2)\) [35] (local, decisive): Rocks are placed at known positions but have unknown quality (good or bad), defining \(\theta\) (\(4\) configurations). The agent passively observes a binary quality reading for the nearest rock, whose accuracy degrades with distance, and can actively CHECK the closest rock to reveal its quality, wasting a turn. The agent can SAMPLE a rock for a reward or penalty depending on quality, or EXIT for a fixed reward. Observations are local (only the nearest rock is sensed) but decisive: CHECK fully reveals quality, so the epistemic strategy is to approach before sampling.

Wumpus World [36] (local, suggestive): Pit, wumpus, and gold positions define \(\theta\) (\(25\) configurations). The classic dynamics are simplified to isolate the epistemic challenge: the agent has no orientation or inventory and navigates by cardinal movement. The agent observes noisy breeze, stench, and glitter adjacency signals and has uncertain position. Observations are local and suggestive: a breeze indicates a nearby pit but not which neighbor, so even precise readings do not decisively identify \(\theta\), and the agent must triangulate across multiple positions.

Frozen Lake and Wumpus World include a SCAN action that switches observations to near-deterministic at the cost of one time step, with lower prior preference to indicate a higher prior cost. In RockSample, the agent can additionally CHECK the closest rock to reveal its quality at the cost of one time step; passively, observation accuracy degrades with distance. Full details are in Appendix 13.

6.1.0.2 Methods.

We compare five methods. The first four correspond to message-passing implementations of the entropy-corrected objectives in 4.5, with channel configurations as specified in 3:

1. BP: standard belief propagation, no entropy correction.

2. VBP: cross-entropy planning, implemented as the principled channelized scheme from Appendix 12.

3. RM-MP: a dynamics-only ablation, using the planning channel together with the dynamics channel; reduces to VBP under deterministic dynamics.

4. AIF-MP: full EFE-based planning, using the planning, dynamics, and observation channels (3).

5. Nuijten-MP [29]: an alternating approximation to the full active-inference objective that recomputes epistemic priors outside the joint variational optimization over beliefs and channels.

All methods except BP include the planning correction, so the experiments ablate the EFE-side corrections on top of a fixed planning baseline rather than the planning correction itself.

6.2 Results and Discussion↩︎

2 reports performance for all methods across three environments. The results show where the planning correction already matters and where the additional observation-side epistemic corrections become necessary.

4pt

6.2.0.1 Global, decisive observations (Frozen Lake).

Both active inference methods dominate (\({\sim}96\%\) success): AIF-MP achieves \(95.9\%\) and Nuijten-MP \(95.6\%\) (overlapping confidence intervals), substantially outperforming all baselines. Both learn to SCAN, which reveals \(\theta\) directly. RM-MP performs comparably to BP and VBP (\(49.8\%\) vs.\(51.9\%\) and \(54.5\%\), overlapping confidence intervals), indicating that the dynamics-only ablation is neither beneficial nor harmful when observations are already global and decisive. Here a single precise reading resolves uncertainty, so the observation channel offers little additional benefit over the alternating heuristic.

6.2.0.2 Local, decisive observations (RockSample).

The shift from global to local scope means the agent must spatially navigate to gather information. Both active inference methods far outperform baselines: AIF-MP achieves \(99.9\%\) retrieval (reward \(3.06\)) and Nuijten-MP \(99.5\%\) (reward \(3.05\)), with overlapping confidence intervals. BP exits early (\(14.2\%\) retrieval), while VBP and RM-MP produce identical results (\(48.3\%\) retrieval, reward \(1.98\)); this is expected because RockSample has deterministic dynamics, so the dynamics correction reduces to the identity and the dynamics-only ablation collapses to VBP. Both active inference formulations handle this regime comparably because precise local observations are decisive: CHECK fully reveals rock quality, so the additional variational parameters of the observation channel do not improve the epistemic signal.

6.2.0.3 Local, suggestive observations (Wumpus World).

The critical shift is from decisive to suggestive resolution: even precise SCAN readings do not fully disambiguate \(\theta\), since multiple configurations produce the same breeze and stench patterns. AIF-MP is the only method that achieves robust performance (\(47.7\%\)), clearly outperforming Nuijten-MP (\(29.2\%\)) and all baselines; representative trajectories are shown in Appendix 13.3.0.7. VBP (\(35.2\%\)) and RM-MP (\(32.2\%\)) perform comparably (overlapping confidence intervals), both outperforming BP (\(20.7\%\)) and Nuijten-MP but falling short of AIF-MP. This gap is an objective mismatch, not a scheduling artefact: [29] recompute epistemic priors outside the variational objective between belief-propagation sweeps, so the observation channels are never variational parameters and suggestive-observation information cannot enter the prior updates. AIF-MP instead treats all four channels as variational parameters of a single joint objective with closed-form stationary conditions 21 . Under decisive observations the observation channel is near-deterministic and the mismatch is masked (Frozen Lake, RockSample); under suggestive observations the channel carries non-trivial information that must co-adapt with beliefs, which explains the Wumpus gap.

6.2.0.4 Synthesis.

The three environments form a hierarchy of epistemic demands along the scope and resolution axes, with each regime cell illustrated by a single environment; the load-bearing claim is the qualitative pattern across regimes, not the per-environment numbers. Across all environments, the planning correction already explains the jump from BP to VBP, while the dynamics-only ablation is not sufficient for robust epistemic behavior. The full EFE-based planning objective separates most clearly not at the global-to-local transition, but at the decisive-to-suggestive transition, where the observation-side channels matter most.

7 Conclusion↩︎

This paper clarifies the variational structure of active inference planning. 1 shows that the epistemic-prior construction of [10] admits an explicit entropy-corrected reformulation: relative to baseline VFE minimization, it adds a specific set of entropy corrections that yields marginal EFE minimization. This makes explicit which terms contribute the epistemic part of the objective and separates marginal EFE minimization from planning over policies. Proper EFE-based planning additionally requires the planning correction of [9], and the combined objective leads directly to a message-passing construction via channel reparameterization. That construction recovers a family of algorithms, including VBP, a dynamics-only ablation, and full EFE-based planning (3).

Empirically, the distinction between these objectives matters in a structured way. Across our exact discrete benchmarks, the planning correction already explains the improvement from BP to VBP, whereas the full observation-side epistemic corrections matter most when observations are suggestive rather than decisive. This is the regime in which the joint channelized scheme separates most clearly from both simpler planning objectives and the alternating approximation.

7.0.0.1 Limitations and future work.

The opposing signs of the entropy corrections induce a min-max structure in the joint optimization over beliefs and channels, which in practice requires heavy damping for stable convergence (5.3). Standard belief propagation convergence guarantees do not transfer to this setting, and developing convergence theory for the channel-augmented scheme is an open problem. The entropy corrections also introduce an additional tuning parameter \(\lambda\) that controls the strength of the corrections and currently requires manual adjustment. We restrict to discrete state spaces where exact factor evaluations are available, isolating the effect of the channel-augmented scheme from errors introduced by approximate message computation; understanding how the channel reparameterization interacts with further factorization constraints on the variational posterior (e.g., mean-field or structured approximations) is an important direction.

8 Entropy Corrections↩︎

8.1 Entropy Correction for Planning↩︎

We derive the entropy correction that distinguishes planning-as-inference from marginal inference. [9] formulate planning-as-inference using a “planning entropy” that excludes action variables from the trajectory entropy. Here we show that this formulation is equivalent to adding an entropy correction to the standard VFE, and derive the form of this correction.

8.1.1 The Planning Entropy of [9]↩︎

[9] define the planning entropy as: \[\label{eq:lazaro95planning95entropy} \mathbb{H} \ifblank{}{\left[q(x_0)\right]}{\csname l\endcsname[q(x_0)\csname r\endcsname]} + \sum_{t=1}^T \mathbb{H}_{q}[x_t | x_{t-1}, u_t]\,,\tag{27}\] where \(\mathbb{H}_{q}[x_t | x_{t-1}, u_t] = -\int q(x_t, x_{t-1}, u_t) \log q(x_t | x_{t-1}, u_t) \mathrm{d}x_t \mathrm{d}x_{t-1} \mathrm{d}u_t\) denotes the conditional entropy. This differs from the full trajectory entropy \(\mathbb{H} \ifblank{}{\left[q(\boldsymbol{x}, \boldsymbol{u})\right]}{\csname l\endcsname[q(\boldsymbol{x}, \boldsymbol{u})\csname r\endcsname]}\) by excluding the action entropy.

8.1.2 Derivation of the Entropy Correction↩︎

8.1.3 Interpretation↩︎

Since we minimize the VFE (rather than maximize as in [9]), the planning entropy is subtracted from the objective. This means the entropy correction \(\sum_t \mathbb{H} \ifblank{}{\left[q(x_{t-1}, u_t)\right]}{\csname l\endcsname[q(x_{t-1}, u_t)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(x_{t-1})\right]}{\csname l\endcsname[q(x_{t-1})\csname r\endcsname]} = \sum_t \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\) is added to the VFE.

Adding this positive correction penalizes action uncertainty: since we minimize the objective, high \(\mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\) increases the cost, pushing the agent toward a deterministic policy.

8.2 Proof of Theorem 1↩︎

We prove that the VFE of the augmented model 6 decomposes into the original VFE plus entropy correction terms. The proof requires three lemmas, each showing how one epistemic prior contributes to the entropy correction.

8.3 Cross-Entropy Interpretation↩︎

We show that planning-as-inference from [9] is equivalent to minimizing cross-entropy to preference distributions.

8.3.1 Reward as Cross-Entropy↩︎

8.3.2 Connection to Planning-as-Inference↩︎

[9] show that the best exponential utility \[F_\lambda^{\text{planning}} = \frac{1}{\lambda} \log \mathbb{E}_{p(\boldsymbol{x}, \boldsymbol{u})} \ifblank{}{\left[\exp\left(\lambda \sum_{t=1}^T R_t\right)\right]}{\csname l\endcsname[\exp\left(\lambda \sum_{t=1}^T R_t\right)\csname r\endcsname]}\] can be expressed as the result of a variational optimization problem whose objective includes the expected sum of rewards \(\mathbb{E}_{q} \ifblank{}{\left[\sum_t R_t\right]}{\csname l\endcsname[\sum_t R_t\csname r\endcsname]}\) as one of its terms.

By Proposition 4 with \(\hat{p}_\lambda(x_t) \propto \exp(\lambda R_t(x_t))\), this reward term equals (up to a constant) \(-\frac{1}{\lambda}\sum_t \mathbb{H} \ifblank{}{\left[q(x_t), \hat{p}_\lambda(x_t)\right]}{\csname l\endcsname[q(x_t), \hat{p}_\lambda(x_t)\csname r\endcsname]}\). Since the variational bound in [9] has an overall \(\frac{1}{\lambda}\) scaling, this factor cancels, and maximizing \(F_\lambda^{\text{planning}}\) is equivalent to minimizing \(\sum_t \mathbb{H} \ifblank{}{\left[q(x_t), \hat{p}_\lambda(x_t)\right]}{\csname l\endcsname[q(x_t), \hat{p}_\lambda(x_t)\csname r\endcsname]}\) along with the dynamics and entropy terms.

This establishes that with \(\hat{p}_\lambda(x_t) \propto \exp(\lambda R_t(x_t))\), the expected utility becomes cross-entropy to preference distributions. Consequently, the inference procedure from [9] minimizes cross-entropy by optimizing its variational objective. Different values of \(\lambda\) yield different preference distributions (corresponding to different risk attitudes), but the underlying mechanism remains cross-entropy minimization.

9 EFE-based Planning Inference↩︎

10 Bethe Free Energy Details↩︎

This appendix reviews the standard derivation of the Bethe Free Energy (BFE) from the Variational Free Energy (VFE) on factor graphs, following [15] and [37]. The material is collected here for self-containedness and to establish the notation used in 2.3 and subsequent appendices.

10.1 Factorized Generative Model on an FFG↩︎

Consider a generative model that factorizes as \[\label{eq:appx-model-factorization} p(\boldsymbol{s}) = \prod_{a \in \mathcal{V}} f_a(\boldsymbol{s}_a)\,,\tag{29}\] where each factor \(f_a\) depends on a subset of variables \(\boldsymbol{s}_a \subseteq \boldsymbol{s}\). This factorization is represented as a Forney-style factor graph (FFG) \(\mathcal{G} = (\mathcal{V}, \mathcal{E})\) [13], [14] (see 4), where \(\mathcal{V}\) denotes the set of factor nodes and \(\mathcal{E}\) the set of edges (variables). An edge \(i \in \mathcal{E}\) connects to a node \(a \in \mathcal{V}\) whenever variable \(s_i\) appears in the scope of \(f_a\). We write \(\mathcal{E}(a)\) for the edges adjacent to node \(a\) and \(\mathcal{V}(i)\) for the nodes adjacent to edge \(i\). The degree of edge \(i\) is \(d_i = |\mathcal{V}(i)|\), counting the number of factors in which \(s_i\) participates.

10.2 The Bethe Variational Family↩︎

The Bethe approximation [15] parameterizes the variational distribution in terms of local beliefs: a factor belief \(q_a(\boldsymbol{s}_a)\) for each node \(a \in \mathcal{V}\) and a singleton belief \(q_i(s_i)\) for each edge \(i \in \mathcal{E}\). These beliefs define a pseudo-distribution via the Bethe factorization: \[\label{eq:appx-bethe-factorization} q_{\mathrm{Bethe}}(\boldsymbol{s}) = \frac{\prod_{a \in \mathcal{V}} q_a(\boldsymbol{s}_a)}{\prod_{i \in \mathcal{E}} q_i(s_i)^{d_i - 1}}\,.\tag{30}\] The denominator corrects the over-counting: since each variable \(s_i\) appears in \(d_i\) factor beliefs, naively multiplying all \(q_a\) would count \(q_i\) a total of \(d_i\) times. Dividing by \(q_i^{d_i - 1}\) reduces this to a single effective copy. For a variable that appears in only one factor (\(d_i = 1\)), no correction is needed.

This parameterization is valid only when the beliefs satisfy local consistency constraints: \[\tag{31} \begin{align} \int q_a(\boldsymbol{s}_a) \, \mathrm{d}\boldsymbol{s}_{a \setminus i} & = q_i(s_i) \quad \text{for all } a \in \mathcal{V},\; i \in \mathcal{E}(a)\,, \tag{32} \\ \int q_a(\boldsymbol{s}_a) \, \mathrm{d}\boldsymbol{s}_a & = 1 \quad \text{for all } a \in \mathcal{V}\,. \tag{33} \end{align}\] The marginalization constraint 32 requires that each factor belief, when marginalized over all variables except \(s_i\), agrees with the singleton belief \(q_i\). Together with normalization 33 , these constraints define the local polytope \(\mathcal{L}_\mathcal{G}\).

10.3 Derivation: VFE to BFE↩︎

Substituting the model factorization 29 and the Bethe factorization 30 into the VFE yields the BFE.

10.3.0.1 Step 1: Expand the log terms.

Taking the logarithm of the Bethe factorization: \[\label{eq:appx-log-q} \log q_{\mathrm{Bethe}}(\boldsymbol{s}) = \sum_{a \in \mathcal{V}} \log q_a(\boldsymbol{s}_a) - \sum_{i \in \mathcal{E}} (d_i - 1) \log q_i(s_i)\,.\tag{34}\] Similarly, the log model decomposes as: \[\label{eq:appx-log-p} \log p(\boldsymbol{s}) = \sum_{a \in \mathcal{V}} \log f_a(\boldsymbol{s}_a)\,.\tag{35}\]

10.3.0.2 Step 2: Substitute into the VFE.

The VFE is \(F[q] = \int q(\boldsymbol{s}) \log \frac{q(\boldsymbol{s})}{p(\boldsymbol{s})} \, \mathrm{d}\boldsymbol{s}\). Substituting 34 and 35 : \[\label{eq:appx-vfe-expanded} F[q] = \int q(\boldsymbol{s}) \left[ \sum_{a \in \mathcal{V}} \log q_a(\boldsymbol{s}_a) - \sum_{i \in \mathcal{E}} (d_i - 1) \log q_i(s_i) - \sum_{a \in \mathcal{V}} \log f_a(\boldsymbol{s}_a) \right] \mathrm{d}\boldsymbol{s}\,.\tag{36}\]

10.3.0.3 Step 3: Localize expectations using consistency.

The key step exploits the local consistency constraints. For any function \(g(\boldsymbol{s}_a)\) that depends only on the variables in the scope of factor \(a\): \[\label{eq:appx-localization-factor} \int q(\boldsymbol{s}) \, g(\boldsymbol{s}_a) \, \mathrm{d}\boldsymbol{s} = \int q_a(\boldsymbol{s}_a) \, g(\boldsymbol{s}_a) \, \mathrm{d}\boldsymbol{s}_a\,,\tag{37}\] since \(q\) marginalizes to \(q_a\) over \(\boldsymbol{s}_a\) by consistency. Similarly, for any function \(h(s_i)\) of a single variable: \[\label{eq:appx-localization-singleton} \int q(\boldsymbol{s}) \, h(s_i) \, \mathrm{d}\boldsymbol{s} = \int q_i(s_i) \, h(s_i) \, \mathrm{d}s_i\,.\tag{38}\]

Applying these identities to 36 : \[\label{eq:appx-bfe-intermediate} F[q] = \sum_{a \in \mathcal{V}} \int q_a(\boldsymbol{s}_a) \log \frac{q_a(\boldsymbol{s}_a)}{f_a(\boldsymbol{s}_a)} \, \mathrm{d}\boldsymbol{s}_a - \sum_{i \in \mathcal{E}} (d_i - 1) \int q_i(s_i) \log q_i(s_i) \, \mathrm{d}s_i\,.\tag{39}\]

10.3.0.4 Step 4: Identify the BFE.

Recognizing the KL divergence and entropy, one obtains the Bethe Free Energy: \[\label{eq:appx-bfe} F_{\text{Bethe}}[q] = \sum_{a \in \mathcal{V}} \mathbb{D}_{\mathrm{KL}} \ifblank{}{\left[q_a(\boldsymbol{s}_a) \| f_a(\boldsymbol{s}_a) \right]}{\csname l\endcsname[q_a(\boldsymbol{s}_a) \| f_a(\boldsymbol{s}_a) \csname r\endcsname]} + \sum_{i \in \mathcal{E}} (d_i - 1) \, \mathbb{H} \ifblank{}{\left[q_i(s_i)\right]}{\csname l\endcsname[q_i(s_i)\csname r\endcsname]}\,,\tag{40}\] which is 5 in the main text. The first sum penalizes each factor belief for deviating from its corresponding factor, while the second sum adds back the entropy of shared variables to correct for the over-counting inherent in the Bethe factorization.

10.4 Constrained Optimization and Belief Propagation↩︎

The BFE gives rise to the constrained optimization problem \[\label{eq:appx-cbfe} \min_{\{q_a, q_i\} \in \mathcal{L}_\mathcal{G}} F_{\text{Bethe}}[q]\,,\tag{41}\] where \(\mathcal{L}_\mathcal{G}\) is the local polytope defined by the constraints 31 . [15] showed that the stationary points of this constrained problem correspond exactly to the fixed points of the belief propagation (BP) algorithm.

On tree-structured graphs, the BFE equals the exact VFE, and BP converges to the exact posterior marginals [16]. In this case, the Bethe factorization 30 is an exact representation of the global posterior, and the local consistency constraints are sufficient to characterize it.

On graphs with loops, the BFE is an approximation: the Bethe factorization does not generally correspond to a valid probability distribution, and BP may not converge. Nevertheless, when BP does converge, its fixed points remain stationary points of the BFE [15], [38].

In models with shared parameters, such as a temporal chain where \(\theta\) appears in both the dynamics and observation factors at every time step, the resulting loops make the BFE an approximation rather than an exact decomposition. Entropic corrections to the BFE yield modified objectives whose stationarity conditions are derived in Appendix 11.

11 Detailed Message Passing Derivation for the Combined Objective↩︎

This appendix gives a self-contained \(T=1\) derivation of the message-passing scheme for the combined objective. The objective includes both the cross-entropy planning correction and the entropy corrections required for planning-as-inference in Active Inference. Its message-passing structure is organized around four channels, with \(r_{u|x}(u_1|x_0)\) appearing in the numerator of the dynamics kernel.

11.1 Model, Coordinates, and Constraints↩︎

We consider the biased generative model \[\label{eq:comb-detail-generative-model} p(y_1, x_1, x_0, \theta, u_1) = p(\theta)\, p(x_0)\, p(u_1)\, p(x_1 | x_0, \theta, u_1)\, p(y_1 | x_1, \theta)\, \hat{p}_x(x_1)\, \hat{p}_y(y_1).\tag{42}\] The corresponding non-singleton factors are the observation factor \(f_{\mathrm{obs}}(y_1,x_1,\theta)=p(y_1|x_1,\theta)\) and the dynamics factor \(f_{\mathrm{dyn}}(x_1,x_0,\theta,u_1)=p(x_1|x_0,\theta,u_1)\), together with singleton prior and goal factors on \(\theta\), \(x_0\), \(u_1\), \(x_1\), and \(y_1\).

The Bethe coordinates consist of the factor beliefs \[q_{\mathrm{obs}}(y_1, x_1, \theta), \qquad q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1),\] the singleton beliefs are \[q_\theta(\theta), \quad q_{x_0}(x_0), \quad q_{x_1}(x_1), \quad q_{y_1}(y_1), \quad q_{u_1}(u_1),\] and the local-polytope constraints consist of factor normalization, \[\label{eq:comb-detail-factor-normalization} \begin{align} \int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}x_1\, \mathrm{d}\theta & = 1, \\ \int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 & = 1, \end{align}\tag{43}\] and factor-to-singleton consistency, \[\label{eq:comb-detail-consistency} \begin{align} \int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}x_1\, \mathrm{d}\theta & = q_{y_1}(y_1), \\ \int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}\theta & = q_{x_1}(x_1), \\ \int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}x_1 & = q_\theta(\theta), \\ \int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 & = q_{x_1}(x_1), \\ \int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}\theta\, \mathrm{d}u_1 & = q_{x_0}(x_0), \\ \int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}u_1 & = q_\theta(\theta), \\ \int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta & = q_{u_1}(u_1). \end{align}\tag{44}\]

The combined objective uses four normalized channels: \[\label{eq:comb-detail-channels} \begin{align} r_{y|x\theta}(y_1|x_1,\theta), \qquad r_{y|x}(y_1|x_1), \qquad r_{x|xu}(x_1|x_0,u_1), \qquad r_{u|x}(u_1|x_0), \end{align}\tag{45}\] with normalization constraints \[\label{eq:comb-detail-channel-norms} \begin{align} \int r_{y|x\theta}(y_1|x_1,\theta)\, \mathrm{d}y_1 & = 1 \quad \forall (x_1,\theta), \\ \int r_{y|x}(y_1|x_1)\, \mathrm{d}y_1 & = 1 \quad \forall x_1, \\ \int r_{x|xu}(x_1|x_0,u_1)\, \mathrm{d}x_1 & = 1 \quad \forall (x_0,u_1), \\ \int r_{u|x}(u_1|x_0)\, \mathrm{d}u_1 & = 1 \quad \forall x_0. \end{align}\tag{46}\]

We also use the derived marginals \[\label{eq:comb-detail-derived} \begin{align} q_{\mathrm{sep}}(x_1,\theta) & := \int q_{\mathrm{obs}}(y_1,x_1,\theta)\, \mathrm{d}y_1, \\ q_{yx}(y_1,x_1) & := \int q_{\mathrm{obs}}(y_1,x_1,\theta)\, \mathrm{d}\theta, \\ q_{\mathrm{trip}}(x_1,x_0,u_1) & := \int q_{\mathrm{dyn}}(x_1,x_0,\theta,u_1)\, \mathrm{d}\theta, \\ q_{\mathrm{pair}}(x_0,u_1) & := \int q_{\mathrm{trip}}(x_1,x_0,u_1)\, \mathrm{d}x_1, \\ q_{ux}(u_1,x_0) & := \int q_{\mathrm{dyn}}(x_1,x_0,\theta,u_1)\, \mathrm{d}x_1\, \mathrm{d}\theta. \end{align}\tag{47}\] The last quantity is just \(q_{ux}(u_1,x_0) = q_{\mathrm{pair}}(x_0,u_1)\), but it is convenient to name it separately when deriving the policy channel.

11.2 Combined Objective and Modified Kernels↩︎

For \(T=1\), the correction added to the Bethe free energy is \[\label{eq:comb-detail-correction} \Delta F_{\mathrm{comb}} = \underbrace{ 2 \mathbb{H} \ifblank{}{\left[q(y_1|x_1,\theta)\right]}{\csname l\endcsname[q(y_1|x_1,\theta)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(x_1|x_0,u_1)\right]}{\csname l\endcsname[q(x_1|x_0,u_1)\csname r\endcsname]} - \mathbb{H} \ifblank{}{\left[q(y_1|x_1)\right]}{\csname l\endcsname[q(y_1|x_1)\csname r\endcsname]} }_{\Delta^{\mathrm{AIF}}} + \underbrace{ \mathbb{H} \ifblank{}{\left[q(u_1|x_0)\right]}{\csname l\endcsname[q(u_1|x_0)\csname r\endcsname]} }_{\Delta^{\mathrm{planning}}}.\tag{48}\]

Using the variational characterization of conditional entropy, each conditional entropy is reparameterized by a channel. The signs matter:

• \(+ \mathbb{H} \ifblank{}{\left[q(u_1|x_0)\right]}{\csname l\endcsname[q(u_1|x_0)\csname r\endcsname]}\) contributes \(- \mathbb{E}_{q_{\mathrm{dyn}}} \ifblank{}{\left[\log r_{u|x}(u_1|x_0)\right]}{\csname l\endcsname[\log r_{u|x}(u_1|x_0)\csname r\endcsname]}\), so \(r_{u|x}\) enters in the numerator of the dynamics kernel.

• \(- \mathbb{H} \ifblank{}{\left[q(x_1|x_0,u_1)\right]}{\csname l\endcsname[q(x_1|x_0,u_1)\csname r\endcsname]}\) contributes \(+ \mathbb{E}_{q_{\mathrm{dyn}}} \ifblank{}{\left[\log r_{x|xu}(x_1|x_0,u_1)\right]}{\csname l\endcsname[\log r_{x|xu}(x_1|x_0,u_1)\csname r\endcsname]}\), so \(r_{x|xu}\) enters in the denominator of the dynamics kernel.

• \(+2 \mathbb{H} \ifblank{}{\left[q(y_1|x_1,\theta)\right]}{\csname l\endcsname[q(y_1|x_1,\theta)\csname r\endcsname]}\) contributes \(-2 \mathbb{E}_{q_{\mathrm{obs}}} \ifblank{}{\left[\log r_{y|x\theta}(y_1|x_1,\theta)\right]}{\csname l\endcsname[\log r_{y|x\theta}(y_1|x_1,\theta)\csname r\endcsname]}\), so \(r_{y|x\theta}^2\) enters in the numerator of the observation kernel.

• \(- \mathbb{H} \ifblank{}{\left[q(y_1|x_1)\right]}{\csname l\endcsname[q(y_1|x_1)\csname r\endcsname]}\) contributes \(+ \mathbb{E}_{q_{yx}} \ifblank{}{\left[\log r_{y|x}(y_1|x_1)\right]}{\csname l\endcsname[\log r_{y|x}(y_1|x_1)\csname r\endcsname]}\), so \(r_{y|x}\) enters in the denominator of the observation kernel.

The resulting objective is \[\label{eq:comb-detail-full-objective} \begin{align} F_{\mathrm{comb}}[q,r] & = \int q_{\mathrm{obs}} \log \frac{q_{\mathrm{obs}}}{p(y_1|x_1,\theta)}\, \mathrm{d}y_1\, \mathrm{d}x_1\, \mathrm{d}\theta - 2 \int q_{\mathrm{obs}} \log r_{y|x\theta}\, \mathrm{d}y_1\, \mathrm{d}x_1\, \mathrm{d}\theta \\ & \quad + \int q_{yx}(y_1,x_1) \log r_{y|x}(y_1|x_1)\, \mathrm{d}y_1\, \mathrm{d}x_1 \\ & \quad + \int q_{\mathrm{dyn}} \log \frac{q_{\mathrm{dyn}}}{p(x_1|x_0,\theta,u_1)}\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 \\ & \quad + \int q_{\mathrm{dyn}} \log r_{x|xu}(x_1|x_0,u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 \\ & \quad - \int q_{\mathrm{dyn}} \log r_{u|x}(u_1|x_0)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 \\ & \quad - \int q_{x_0} \log p(x_0)\, \mathrm{d}x_0 - \int q_{u_1} \log p(u_1)\, \mathrm{d}u_1 - \int q_{y_1} \log \hat{p}_y(y_1)\, \mathrm{d}y_1 \\ & \quad + (d_\theta - 1) \mathbb{H} \ifblank{}{\left[q_\theta\right]}{\csname l\endcsname[q_\theta\csname r\endcsname]} - \int q_\theta \log p(\theta)\, \mathrm{d}\theta \\ & \quad + (d_{x_1} - 1) \mathbb{H} \ifblank{}{\left[q_{x_1}\right]}{\csname l\endcsname[q_{x_1}\csname r\endcsname]} - \int q_{x_1} \log \hat{p}_x(x_1)\, \mathrm{d}x_1, \end{align}\tag{49}\]

The variable degrees are \[\label{eq:comb-detail-degrees} d_\theta = 3, \qquad d_{x_0} = 2, \qquad d_{x_1} = 3, \qquad d_{y_1} = 2, \qquad d_{u_1} = 2.\tag{50}\]

11.3 Lagrangian↩︎

We introduce Lagrange multipliers for all normalization and marginalization constraints. The factor and consistency multipliers are \[\lambda_{\mathrm{obs}}, \quad \lambda_{\mathrm{dyn}}, \quad \lambda_{y_1}(y_1), \quad \lambda_\theta^{(\mathrm{obs})}(\theta), \quad \lambda_{x_1}^{(\mathrm{obs})}(x_1), \quad \lambda_{x_1}^{(\mathrm{dyn})}(x_1), \quad \lambda_{x_0}(x_0), \quad \lambda_{u_1}(u_1), \quad \lambda_\theta^{(\mathrm{dyn})}(\theta),\] and the channel multipliers are \[\nu_{\mathrm{obs}}(x_1,\theta), \qquad \nu_{y|x}(x_1), \qquad \nu_x(x_0,u_1), \qquad \nu_{u|x}(x_0).\]

The full Lagrangian is \[\label{eq:comb-detail-lagrangian} \begin{align} \mathcal{L}_{\mathrm{comb}} & = F_{\mathrm{comb}}[q,r] \\ & \quad + \lambda_{\mathrm{obs}} \left(\int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}x_1\, \mathrm{d}\theta - 1\right) \\ & \quad + \lambda_{\mathrm{dyn}} \left(\int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 - 1\right) \\ & \quad + \int \lambda_{y_1}(y_1) \left(\int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}x_1\, \mathrm{d}\theta - q_{y_1}(y_1)\right) \mathrm{d}y_1 \\ & \quad + \int \lambda_\theta^{(\mathrm{obs})}(\theta) \left(\int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}x_1 - q_\theta(\theta)\right) \mathrm{d}\theta \\ & \quad + \int \lambda_{x_1}^{(\mathrm{obs})}(x_1) \left(\int q_{\mathrm{obs}}(y_1, x_1, \theta)\, \mathrm{d}y_1\, \mathrm{d}\theta - q_{x_1}(x_1)\right) \mathrm{d}x_1 \\ & \quad + \int \lambda_{x_1}^{(\mathrm{dyn})}(x_1) \left(\int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 - q_{x_1}(x_1)\right) \mathrm{d}x_1 \\ & \quad + \int \lambda_{x_0}(x_0) \left(\int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}\theta\, \mathrm{d}u_1 - q_{x_0}(x_0)\right) \mathrm{d}x_0 \\ & \quad + \int \lambda_{u_1}(u_1) \left(\int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta - q_{u_1}(u_1)\right) \mathrm{d}u_1 \\ & \quad + \int \lambda_\theta^{(\mathrm{dyn})}(\theta) \left(\int q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}u_1 - q_\theta(\theta)\right) \mathrm{d}\theta \\ & \quad + \iint \nu_{\mathrm{obs}}(x_1, \theta) \left(\int r_{y|x\theta}(y_1|x_1,\theta)\, \mathrm{d}y_1 - 1\right) \mathrm{d}x_1\, \mathrm{d}\theta \\ & \quad + \iint \nu_{y|x}(x_1) \left(\int r_{y|x}(y_1|x_1)\, \mathrm{d}y_1 - 1\right) \mathrm{d}x_1 \\ & \quad + \iint \nu_x(x_0, u_1) \left(\int r_{x|xu}(x_1|x_0,u_1)\, \mathrm{d}x_1 - 1\right) \mathrm{d}x_0\, \mathrm{d}u_1 \\ & \quad + \int \nu_{u|x}(x_0) \left(\int r_{u|x}(u_1|x_0)\, \mathrm{d}u_1 - 1\right) \mathrm{d}x_0. \end{align}\tag{51}\]

11.4 Stationarity Conditions↩︎

11.4.1 Variation with respect to \(q_{\mathrm{obs}}\)↩︎

The stationarity equation for the observation factor is

11.4.2 Variation with respect to \(q_{\mathrm{dyn}}\)↩︎

The dynamics-side terms now contain both channel contributions: \[\begin{align} & \int q_{\mathrm{dyn}} \log q_{\mathrm{dyn}} - \int q_{\mathrm{dyn}} \log p(x_1|x_0,\theta,u_1) + \int q_{\mathrm{dyn}} \log r_{x|xu}(x_1|x_0,u_1) \\ & \quad - \int q_{\mathrm{dyn}} \log r_{u|x}(u_1|x_0) + \text{(multiplier terms)}. \end{align}\] Taking \(\frac{\delta \mathcal{L}_{\mathrm{comb}}}{\delta q_{\mathrm{dyn}}}=0\) gives \[\log q_{\mathrm{dyn}} + 1 - \log p(x_1|x_0,\theta,u_1) + \log r_{x|xu}(x_1|x_0,u_1) - \log r_{u|x}(u_1|x_0) + \lambda_{\mathrm{dyn}} + \lambda_{x_1}^{(\mathrm{dyn})} + \lambda_{x_0} + \lambda_{u_1} + \lambda_\theta^{(\mathrm{dyn})} = 0.\]

11.4.3 Variation with respect to the observation channels↩︎

The observation-side channels satisfy \[\tag{52} \begin{align} r_{y|x\theta}^*(y_1|x_1,\theta) & = \frac{q_{\mathrm{obs}}(y_1,x_1,\theta)}{q_{\mathrm{sep}}(x_1,\theta)} = q(y_1|x_1,\theta), \tag{53} \\ r_{y|x}^*(y_1|x_1) & = \frac{q_{yx}(y_1,x_1)}{q_{x_1}(x_1)} = q(y_1|x_1). \tag{54} \end{align}\]

11.4.4 Variation with respect to \(r_{x|xu}\)↩︎

The predictive dynamics channel is obtained from \[\frac{\delta \mathcal{L}_{\mathrm{comb}}}{\delta r_{x|xu}(x_1|x_0,u_1)} = \frac{q_{\mathrm{trip}}(x_1,x_0,u_1)}{r_{x|xu}(x_1|x_0,u_1)} + \nu_x(x_0,u_1) = 0.\]

11.4.5 Variation with respect to \(r_{u|x}\)↩︎

Since \(r_{u|x}\) does not depend on \(x_1\) or \(\theta\), only the marginal \(q_{ux}(u_1,x_0)\) matters: \[- \int q_{ux}(u_1,x_0)\log r_{u|x}(u_1|x_0)\, \mathrm{d}u_1\, \mathrm{d}x_0 + \int \nu_{u|x}(x_0)\left(\int r_{u|x}(u_1|x_0)\, \mathrm{d}u_1 - 1\right)\mathrm{d}x_0.\]

Taking a pointwise derivative yields \[-\frac{q_{ux}(u_1,x_0)}{r_{u|x}(u_1|x_0)} + \nu_{u|x}(x_0) = 0.\] Normalization implies \(\nu_{u|x}(x_0)=q_{x_0}(x_0)\).

11.4.6 Variation with respect to singleton beliefs↩︎

The singleton calculations are elementary because they depend only on degree counting. Since \[d_{y_1}=d_{x_0}=d_{u_1}=2, \qquad d_{\theta}=d_{x_1}=3,\] the degree-2 and degree-3 variables behave differently. In particular:

• for degree-2 variables \(y_1\), \(x_0\), and \(u_1\), the KL term and the singleton entropy cancel, so \[\label{eq:comb-detail-deg2} \lambda_{y_1}(y_1) = -\log \hat{p}_y(y_1), \qquad \lambda_{x_0}(x_0) = -\log p(x_0), \qquad \lambda_{u_1}(u_1) = -\log p(u_1);\tag{55}\]

• for degree-3 variables \(\theta\) and \(x_1\), one obtains only constraints on sums of multipliers, \[\label{eq:comb-detail-deg3} \lambda_\theta^{(\mathrm{obs})}(\theta) + \lambda_\theta^{(\mathrm{dyn})}(\theta) = -\log q_\theta^*(\theta) - 1 - \log p(\theta),\tag{56}\] \[\label{eq:comb-detail-deg3-x1} \lambda_{x_1}^{(\mathrm{obs})}(x_1) + \lambda_{x_1}^{(\mathrm{dyn})}(x_1) = -\log q_{x_1}^*(x_1) - 1 - \log \hat{p}_x(x_1).\tag{57}\]

The extra policy channel does not alter these facts, because it modifies only the non-singleton dynamics factor and does not introduce any new local-consistency constraint involving singletons.

11.5 Solving the Stationarity Equations as Message Passing↩︎

Once the kernels ?? are identified, the rest of the derivation follows the standard Bethe logic. For a factor \(a\) with kernel \(\tilde{f}_a\) and neighboring variables \(\mathcal{E}(a)\), the factor-to-variable update is \[\label{eq:comb-detail-factor-to-var} \mu_{a\to j}(s_j) \propto \int \tilde{f}_a(\boldsymbol{s}_a) \prod_{i \in \mathcal{E}(a)\setminus j} \mu_{i\to a}(s_i)\, \mathrm{d}\boldsymbol{s}_{a\setminus j}.\tag{58}\]

11.5.1 Observation-factor messages↩︎

Using the observation kernel in ?? , the observation-factor messages are \[\label{eq:comb-detail-msgs-obs} \begin{align} \mu_{\mathrm{obs}\to\theta}(\theta) & = \iint \frac{p(y_1|x_1,\theta)\, r_{y|x\theta}^2(y_1|x_1,\theta)}{r_{y|x}(y_1|x_1)} \mu_{y_1\to\mathrm{obs}}(y_1)\mu_{x_1\to\mathrm{obs}}(x_1)\, \mathrm{d}y_1\, \mathrm{d}x_1, \\ \mu_{\mathrm{obs}\to x_1}(x_1) & = \iint \frac{p(y_1|x_1,\theta)\, r_{y|x\theta}^2(y_1|x_1,\theta)}{r_{y|x}(y_1|x_1)} \mu_{y_1\to\mathrm{obs}}(y_1)\mu_{\theta\to\mathrm{obs}}(\theta)\, \mathrm{d}y_1\, \mathrm{d}\theta, \\ \mu_{\mathrm{obs}\to y_1}(y_1) & = \iint \frac{p(y_1|x_1,\theta)\, r_{y|x\theta}^2(y_1|x_1,\theta)}{r_{y|x}(y_1|x_1)} \mu_{x_1\to\mathrm{obs}}(x_1)\mu_{\theta\to\mathrm{obs}}(\theta)\, \mathrm{d}x_1\, \mathrm{d}\theta. \end{align}\tag{59}\]

11.5.2 Dynamics-factor messages↩︎

The combined dynamics kernel yields \[\label{eq:comb-detail-msgs-dyn} \begin{align} \mu_{\mathrm{dyn}\to\theta}(\theta) & = \iiint \frac{p(x_1|x_0,\theta,u_1)\, r_{u|x}(u_1|x_0)}{r_{x|xu}(x_1|x_0,u_1)} \mu_{x_1\to\mathrm{dyn}}(x_1)\mu_{x_0\to\mathrm{dyn}}(x_0)\mu_{u_1\to\mathrm{dyn}}(u_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}u_1, \\ \mu_{\mathrm{dyn}\to x_1}(x_1) & = \iiint \frac{p(x_1|x_0,\theta,u_1)\, r_{u|x}(u_1|x_0)}{r_{x|xu}(x_1|x_0,u_1)} \mu_{x_0\to\mathrm{dyn}}(x_0)\mu_{u_1\to\mathrm{dyn}}(u_1)\mu_{\theta\to\mathrm{dyn}}(\theta)\, \mathrm{d}x_0\, \mathrm{d}u_1\, \mathrm{d}\theta, \\ \mu_{\mathrm{dyn}\to x_0}(x_0) & = \iiint \frac{p(x_1|x_0,\theta,u_1)\, r_{u|x}(u_1|x_0)}{r_{x|xu}(x_1|x_0,u_1)} \mu_{x_1\to\mathrm{dyn}}(x_1)\mu_{u_1\to\mathrm{dyn}}(u_1)\mu_{\theta\to\mathrm{dyn}}(\theta)\, \mathrm{d}x_1\, \mathrm{d}u_1\, \mathrm{d}\theta, \\ \mu_{\mathrm{dyn}\to u_1}(u_1) & = \iiint \frac{p(x_1|x_0,\theta,u_1)\, r_{u|x}(u_1|x_0)}{r_{x|xu}(x_1|x_0,u_1)} \mu_{x_1\to\mathrm{dyn}}(x_1)\mu_{x_0\to\mathrm{dyn}}(x_0)\mu_{\theta\to\mathrm{dyn}}(\theta)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta. \end{align}\tag{60}\]

11.5.3 Factor beliefs and singleton beliefs↩︎

The factor beliefs are kernel times incoming messages: \[\label{eq:comb-detail-factor-beliefs} \begin{align} q_{\mathrm{obs}}^*(y_1,x_1,\theta) & \propto \tilde{f}_{\mathrm{obs}}(y_1,x_1,\theta)\, \mu_{y_1\to\mathrm{obs}}(y_1)\mu_{x_1\to\mathrm{obs}}(x_1)\mu_{\theta\to\mathrm{obs}}(\theta), \\ q_{\mathrm{dyn}}^*(x_1,x_0,\theta,u_1) & \propto \tilde{f}_{\mathrm{dyn}}^{\mathrm{comb}}(x_1,x_0,\theta,u_1)\, \mu_{x_1\to\mathrm{dyn}}(x_1)\mu_{x_0\to\mathrm{dyn}}(x_0)\mu_{\theta\to\mathrm{dyn}}(\theta)\mu_{u_1\to\mathrm{dyn}}(u_1). \end{align}\tag{61}\]

Singleton beliefs keep the usual sum-product form: \[\label{eq:comb-detail-singletons} \begin{align} q_\theta^*(\theta) & \propto p(\theta)\, \mu_{\mathrm{obs}\to\theta}(\theta)\, \mu_{\mathrm{dyn}\to\theta}(\theta), \\ q_{x_1}^*(x_1) & \propto \hat{p}_x(x_1)\, \mu_{\mathrm{obs}\to x_1}(x_1)\, \mu_{\mathrm{dyn}\to x_1}(x_1), \\ q_{x_0}^*(x_0) & \propto p(x_0)\, \mu_{\mathrm{dyn}\to x_0}(x_0), \\ q_{u_1}^*(u_1) & \propto p(u_1)\, \mu_{\mathrm{dyn}\to u_1}(u_1), \\ q_{y_1}^*(y_1) & \propto \hat{p}_y(y_1)\, \mu_{\mathrm{obs}\to y_1}(y_1). \end{align}\tag{62}\]

11.5.4 Channel updates↩︎

At a fixed point, all four channels recover the corresponding conditionals under the current factor beliefs: \[\label{eq:comb-detail-channel-updates} \begin{align} r_{y|x\theta}^*(y_1|x_1,\theta) & = q(y_1|x_1,\theta), \\ r_{y|x}^*(y_1|x_1) & = q(y_1|x_1), \\ r_{x|xu}^*(x_1|x_0,u_1) & = q(x_1|x_0,u_1), \\ r_{u|x}^*(u_1|x_0) & = q(u_1|x_0). \end{align}\tag{63}\]

11.6 Generic scheme for arbitrary \(T\)↩︎

The passage from \(T=1\) to arbitrary horizons is immediate because all entropy corrections are additive across time, so each time step receives its own local channels \[r_{y|x\theta,t}(y_t|x_t,\theta), \quad r_{y|x,t}(y_t|x_t), \quad r_{x|xu,t}(x_t|x_{t-1},u_t), \quad r_{u|x,t}(u_t|x_{t-1}).\]

The per-time-step kernels are \[\label{eq:comb-detail-generic-kernels} \begin{align} \tilde{f}_{\mathrm{obs}_t}(y_t,x_t,\theta) & = \frac{p(y_t|x_t,\theta)\, r_{y|x\theta,t}^2(y_t|x_t,\theta)}{r_{y|x,t}(y_t|x_t)}, \\ \tilde{f}_{\mathrm{dyn}_t}^{\mathrm{comb}}(x_t,x_{t-1},\theta,u_t) & = \frac{p(x_t|x_{t-1},\theta,u_t)\, r_{u|x,t}(u_t|x_{t-1})}{r_{x|xu,t}(x_t|x_{t-1},u_t)}. \end{align}\tag{64}\]

The full multi-step scheme is obtained by applying the usual sum-product updates with the kernels in 64 .

11.6.1 Messages from Observation Factor \(f_{\mathrm{obs}_t}\)↩︎

\[\label{eq:comb-detail-gen-msgs-obs} \begin{align} \mu_{\mathrm{obs}_t\to\theta}(\theta) & = \iint \frac{p(y_t|x_t,\theta)\, r_{y|x\theta,t}^2(y_t|x_t,\theta)}{r_{y|x,t}(y_t|x_t)} \mu_{y_t\to\mathrm{obs}_t}(y_t)\mu_{x_t\to\mathrm{obs}_t}(x_t)\, \mathrm{d}y_t\, \mathrm{d}x_t, \\ \mu_{\mathrm{obs}_t\to x_t}(x_t) & = \iint \frac{p(y_t|x_t,\theta)\, r_{y|x\theta,t}^2(y_t|x_t,\theta)}{r_{y|x,t}(y_t|x_t)} \mu_{y_t\to\mathrm{obs}_t}(y_t)\mu_{\theta\to\mathrm{obs}_t}(\theta)\, \mathrm{d}y_t\, \mathrm{d}\theta, \\ \mu_{\mathrm{obs}_t\to y_t}(y_t) & = \iint \frac{p(y_t|x_t,\theta)\, r_{y|x\theta,t}^2(y_t|x_t,\theta)}{r_{y|x,t}(y_t|x_t)} \mu_{x_t\to\mathrm{obs}_t}(x_t)\mu_{\theta\to\mathrm{obs}_t}(\theta)\, \mathrm{d}x_t\, \mathrm{d}\theta. \end{align}\tag{65}\]

11.6.2 Messages from Dynamics Factor \(f_{\mathrm{dyn}_t}\)↩︎

\[\label{eq:comb-detail-gen-msgs-dyn} \begin{align} \mu_{\mathrm{dyn}_t\to\theta}(\theta) & = \iiint \frac{p(x_t|x_{t-1},\theta,u_t)\, r_{u|x,t}(u_t|x_{t-1})}{r_{x|xu,t}(x_t|x_{t-1},u_t)} \mu_{x_t\to\mathrm{dyn}_t}(x_t)\mu_{x_{t-1}\to\mathrm{dyn}_t}(x_{t-1})\mu_{u_t\to\mathrm{dyn}_t}(u_t)\, \mathrm{d}x_t\, \mathrm{d}x_{t-1}\, \mathrm{d}u_t, \\ \mu_{\mathrm{dyn}_t\to x_t}(x_t) & = \iiint \frac{p(x_t|x_{t-1},\theta,u_t)\, r_{u|x,t}(u_t|x_{t-1})}{r_{x|xu,t}(x_t|x_{t-1},u_t)} \mu_{x_{t-1}\to\mathrm{dyn}_t}(x_{t-1})\mu_{u_t\to\mathrm{dyn}_t}(u_t)\mu_{\theta\to\mathrm{dyn}_t}(\theta)\, \mathrm{d}x_{t-1}\, \mathrm{d}u_t\, \mathrm{d}\theta, \\ \mu_{\mathrm{dyn}_t\to x_{t-1}}(x_{t-1}) & = \iiint \frac{p(x_t|x_{t-1},\theta,u_t)\, r_{u|x,t}(u_t|x_{t-1})}{r_{x|xu,t}(x_t|x_{t-1},u_t)} \mu_{x_t\to\mathrm{dyn}_t}(x_t)\mu_{u_t\to\mathrm{dyn}_t}(u_t)\mu_{\theta\to\mathrm{dyn}_t}(\theta)\, \mathrm{d}x_t\, \mathrm{d}u_t\, \mathrm{d}\theta, \\ \mu_{\mathrm{dyn}_t\to u_t}(u_t) & = \iiint \frac{p(x_t|x_{t-1},\theta,u_t)\, r_{u|x,t}(u_t|x_{t-1})}{r_{x|xu,t}(x_t|x_{t-1},u_t)} \mu_{x_t\to\mathrm{dyn}_t}(x_t)\mu_{x_{t-1}\to\mathrm{dyn}_t}(x_{t-1})\mu_{\theta\to\mathrm{dyn}_t}(\theta)\, \mathrm{d}x_t\, \mathrm{d}x_{t-1}\, \mathrm{d}\theta. \end{align}\tag{66}\]

11.6.3 Factor Beliefs↩︎

\[\label{eq:comb-detail-gen-factor-beliefs} \begin{align} q_{\mathrm{obs},t}^*(y_t,x_t,\theta) & \propto \tilde{f}_{\mathrm{obs}_t}(y_t,x_t,\theta)\, \mu_{y_t\to\mathrm{obs}_t}(y_t)\mu_{x_t\to\mathrm{obs}_t}(x_t)\mu_{\theta\to\mathrm{obs}_t}(\theta), \\ q_{\mathrm{dyn},t}^*(x_t,x_{t-1},\theta,u_t) & \propto \tilde{f}_{\mathrm{dyn}_t}^{\mathrm{comb}}(x_t,x_{t-1},\theta,u_t)\, \mu_{x_t\to\mathrm{dyn}_t}(x_t)\mu_{x_{t-1}\to\mathrm{dyn}_t}(x_{t-1})\mu_{\theta\to\mathrm{dyn}_t}(\theta)\mu_{u_t\to\mathrm{dyn}_t}(u_t). \end{align}\tag{67}\]

11.6.4 Channel Updates↩︎

At a fixed point, each time step has the four local channel updates \[\label{eq:comb-detail-gen-channel-updates} \begin{align} r_{y|x\theta,t}^*(y_t|x_t,\theta) & = q_t(y_t|x_t,\theta), \\ r_{y|x,t}^*(y_t|x_t) & = q_t(y_t|x_t), \\ r_{x|xu,t}^*(x_t|x_{t-1},u_t) & = q_t(x_t|x_{t-1},u_t), \\ r_{u|x,t}^*(u_t|x_{t-1}) & = q_t(u_t|x_{t-1}). \end{align}\tag{68}\]

11.6.5 Singleton Beliefs↩︎

\[\label{eq:comb-detail-gen-singletons} \begin{align} q_{x_t}^*(x_t) & \propto \hat{p}_x(x_t)\, \mu_{\mathrm{obs}_t\to x_t}(x_t)\, \mu_{\mathrm{dyn}_t\to x_t}(x_t)\, \mu_{\mathrm{dyn}_{t+1}\to x_t}(x_t), \\ q_\theta^*(\theta) & \propto p(\theta)\prod_{\tau=1}^{T} \mu_{\mathrm{obs}_\tau\to\theta}(\theta)\, \mu_{\mathrm{dyn}_\tau\to\theta}(\theta), \\ q_{u_t}^*(u_t) & \propto p(u_t)\, \mu_{\mathrm{dyn}_t\to u_t}(u_t), \\ q_{y_t}^*(y_t) & \propto \hat{p}_y(y_t)\, \mu_{\mathrm{obs}_t\to y_t}(y_t). \end{align}\tag{69}\]

The boundary conditions follow the usual temporal-edge conventions: at \(t=1\), \(\mu_{x_0\to\mathrm{dyn}_1}(x_0)=p(x_0)\); at \(t=T\), the message \(\mu_{\mathrm{dyn}_{T+1}\to x_T}\) is absent.

11.7 Interpretation↩︎

At a fixed point, \[\label{eq:comb-detail-fixed-point-kernel} \tilde{f}_{\mathrm{dyn}_t}^{\mathrm{comb}}(x_t,x_{t-1},\theta,u_t) = \frac{p(x_t|x_{t-1},\theta,u_t)\, q(u_t|x_{t-1})}{q(x_t|x_{t-1},u_t)}.\tag{70}\] This exposes the objective transparently: the numerator implements policy commitment, while the denominator implements the risk-sensitive correction over predicted futures. The observation factor simultaneously balances the two observation-side entropy terms through \(r_{y|x\theta}\) and \(r_{y|x}\).

12 Message Passing Derivation for VBP↩︎

This appendix derives the message-passing equations for the Variational Belief Propagation (VBP) scheme that implements cross-entropy planning (4.2). The factor graph and Bethe approximation are identical to the combined-objective derivation (Appendix 11); only the entropy correction differs. VBP requires a single channel reparameterization, making the derivation substantially simpler.

12.1 Coordinate System↩︎

We use the same generative model, factor graph, Bethe coordinates, and local-polytope constraints as in Appendix 11: factor beliefs \(q_{\mathrm{obs}}(y_1, x_1, \theta)\) and \(q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\), plus singleton beliefs \(q_\theta, q_{x_0}, q_{x_1}, q_{y_1}, q_{u_1}\).

The single difference is the channel set. VBP introduces one channel: \[\label{eq:vbp-channel} r_{u|x}(u_1 | x_0), \quad \text{subject to} \quad \int r_{u|x}(u_1 | x_0)\, \mathrm{d}u_1 = 1 \quad \forall\, x_0.\tag{71}\] This channel parameterizes the conditional entropy of actions given states via: \[\label{eq:vbp-channel-identity} \mathbb{H} \ifblank{}{\left[q(u_1 | x_0)\right]}{\csname l\endcsname[q(u_1 | x_0)\csname r\endcsname]} = \min_{r_{u|x}} \mathbb{E}_{q_{\mathrm{pair}}(x_0, u_1)} \ifblank{}{\left[-\log r_{u|x}(u_1 | x_0)\right]}{\csname l\endcsname[-\log r_{u|x}(u_1 | x_0)\csname r\endcsname]},\tag{72}\] where the minimum is attained at \(r_{u|x}(u_1 | x_0) = q(u_1 | x_0)\), and \[\label{eq:vbp-q-pair} q_{\mathrm{pair}}(x_0, u_1) := \iint q_{\mathrm{dyn}}(x_1, x_0, \theta, u_1)\, \mathrm{d}x_1\, \mathrm{d}\theta\tag{73}\] is the marginal of the dynamics factor belief over \((x_0, u_1)\).

12.2 Objective Function↩︎

The VBP objective adds the cross-entropy planning correction 11 to the usual Bethe free energy: \[\label{eq:vbp-correction} \Delta F_{\mathrm{VBP}} = + \mathbb{H} \ifblank{}{\left[q(u_1 | x_0)\right]}{\csname l\endcsname[q(u_1 | x_0)\csname r\endcsname]}.\tag{74}\]

After channel reparameterization via 72 : \[\label{eq:vbp-full-objective} \begin{align} F_{\mathrm{VBP}}[q, r_{u|x}] & = \int q_{\mathrm{obs}} \log \frac{q_{\mathrm{obs}}}{p(y_1 | x_1, \theta)}\, \mathrm{d}y_1\, \mathrm{d}x_1\, \mathrm{d}\theta \\ & \quad + \int q_{\mathrm{dyn}} \log \frac{q_{\mathrm{dyn}}}{p(x_1 | x_0, \theta, u_1)}\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 \\ & \quad - \int q_{\mathrm{dyn}} \log r_{u|x}(u_1 | x_0)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta\, \mathrm{d}u_1 \\ & \quad + \text{(the usual singleton Bethe terms)}. \end{align}\tag{75}\]

12.3 Stationarity Conditions↩︎

We form the Lagrangian with the same normalization and consistency multipliers as in Appendix 11.3, except that the three observation and predictive-dynamics channel multipliers are replaced by a single multiplier \(\nu_{u|x}(x_0)\) for the channel normalization constraint 71 .

12.3.1 Variation with respect to \(q_{\mathrm{obs}}\)↩︎

The observation factor receives no channel correction. The stationarity condition is identical to standard Bethe: \[\label{eq:vbp-qobs-stationary} q_{\mathrm{obs}}^*(y_1, x_1, \theta) \propto p(y_1 | x_1, \theta)\, e^{-\lambda_{y_1}(y_1)}\, e^{-\lambda_\theta^{(\mathrm{obs})}(\theta)}\, e^{-\lambda_{x_1}^{(\mathrm{obs})}(x_1)}.\tag{76}\] The observation kernel is simply \(p(y_1 | x_1, \theta)\), as in standard belief propagation.

12.3.2 Variation with respect to \(q_{\mathrm{dyn}}\)↩︎

The \(q_{\mathrm{dyn}}\)-dependent terms include the channel correction \(-\int q_{\mathrm{dyn}} \log r_{u|x}(u_1 | x_0)\). Taking \(\frac{\delta \mathcal{L}}{\delta q_{\mathrm{dyn}}} = 0\): \[\label{eq:vbp-qdyn-stationarity-eq} \log q_{\mathrm{dyn}} + 1 - \log p(x_1 | x_0, \theta, u_1) - \log r_{u|x}(u_1 | x_0) + \lambda_{\mathrm{dyn}} + \lambda_{x_1}^{(\mathrm{dyn})} + \lambda_{x_0} + \lambda_{u_1} + \lambda_\theta^{(\mathrm{dyn})} = 0.\tag{77}\]

12.3.3 Variation with respect to \(r_{u|x}\)↩︎

The \(r_{u|x}\)-dependent terms in the Lagrangian are: \[-\int q_{\mathrm{pair}}(x_0, u_1) \log r_{u|x}(u_1 | x_0)\, \mathrm{d}u_1\, \mathrm{d}x_0 + \int \nu_{u|x}(x_0) \left(\int r_{u|x}(u_1 | x_0)\, \mathrm{d}u_1 - 1\right) \mathrm{d}x_0.\]

Pointwise derivative: \[-\frac{q_{\mathrm{pair}}(x_0, u_1)}{r_{u|x}(u_1 | x_0)} + \nu_{u|x}(x_0) = 0.\]

Imposing normalization \(\int r_{u|x}\, \mathrm{d}u_1 = 1\) yields \(\nu_{u|x}(x_0) = q_{x_0}(x_0)\), where \(q_{x_0}(x_0) = \int q_{\mathrm{pair}}(x_0, u_1)\, \mathrm{d}u_1\).

12.3.4 Singleton stationarity↩︎

The singleton stationarity conditions are identical to the corresponding degree-counting argument in Appendix 11.4: degree-2 multipliers are identified algebraically, while degree-3 multipliers satisfy the same constraint equations on sums of multipliers.

12.4 Message-Passing Equations↩︎

Using the observation kernel \(\tilde{f}_{\mathrm{obs}} = p(y_1 | x_1, \theta)\) and the VBP dynamics kernel \(\tilde{f}_{\mathrm{dyn}} = p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\), the sum-product messages follow from the standard factor-to-variable update. Variable-to-factor messages \(\mu_{i \to a}\) are the usual products of all incoming factor-to-variable messages except the one from the recipient factor.

12.4.1 Messages from Observation Factor↩︎

Standard sum-product (no channel modification): \[\tag{78} \begin{align} \mu_{\mathrm{obs} \to \theta}(\theta) & = \iint p(y_1 | x_1, \theta)\, \mu_{y_1 \to \mathrm{obs}}(y_1)\, \mu_{x_1 \to \mathrm{obs}}(x_1)\, \mathrm{d}y_1\, \mathrm{d}x_1, \tag{79} \\ \mu_{\mathrm{obs} \to x_1}(x_1) & = \iint p(y_1 | x_1, \theta)\, \mu_{y_1 \to \mathrm{obs}}(y_1)\, \mu_{\theta \to \mathrm{obs}}(\theta)\, \mathrm{d}y_1\, \mathrm{d}\theta, \tag{80} \\ \mu_{\mathrm{obs} \to y_1}(y_1) & = \iint p(y_1 | x_1, \theta)\, \mu_{\theta \to \mathrm{obs}}(\theta)\, \mu_{x_1 \to \mathrm{obs}}(x_1)\, \mathrm{d}x_1\, \mathrm{d}\theta. \tag{81} \end{align}\]

12.4.2 Messages from Dynamics Factor↩︎

Using kernel \(p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\): \[\tag{82} \begin{align} \mu_{\mathrm{dyn} \to \theta}(\theta) & = \iiint p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\, \mu_{x_0 \to \mathrm{dyn}}(x_0)\, \mu_{u_1 \to \mathrm{dyn}}(u_1)\, \mu_{x_1 \to \mathrm{dyn}}(x_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}u_1, \tag{83} \\ \mu_{\mathrm{dyn} \to x_1}(x_1) & = \iiint p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\, \mu_{x_0 \to \mathrm{dyn}}(x_0)\, \mu_{u_1 \to \mathrm{dyn}}(u_1)\, \mu_{\theta \to \mathrm{dyn}}(\theta)\, \mathrm{d}x_0\, \mathrm{d}u_1\, \mathrm{d}\theta, \tag{84} \\ \mu_{\mathrm{dyn} \to x_0}(x_0) & = \iiint p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\, \mu_{x_1 \to \mathrm{dyn}}(x_1)\, \mu_{u_1 \to \mathrm{dyn}}(u_1)\, \mu_{\theta \to \mathrm{dyn}}(\theta)\, \mathrm{d}x_1\, \mathrm{d}u_1\, \mathrm{d}\theta, \tag{85} \\ \mu_{\mathrm{dyn} \to u_1}(u_1) & = \iiint p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\, \mu_{x_0 \to \mathrm{dyn}}(x_0)\, \mu_{\theta \to \mathrm{dyn}}(\theta)\, \mu_{x_1 \to \mathrm{dyn}}(x_1)\, \mathrm{d}x_1\, \mathrm{d}x_0\, \mathrm{d}\theta. \tag{86} \end{align}\]

12.4.3 Factor Beliefs and Channel Update↩︎

The factor beliefs are the kernel times all incoming variable-to-factor messages: \[\tag{87} \begin{align} q_{\mathrm{obs}}^*(y_1, x_1, \theta) & \propto p(y_1 | x_1, \theta)\, \mu_{y_1 \to \mathrm{obs}}(y_1)\, \mu_{x_1 \to \mathrm{obs}}(x_1)\, \mu_{\theta \to \mathrm{obs}}(\theta), \tag{88} \\ q_{\mathrm{dyn}}^*(x_1, x_0, \theta, u_1) & \propto p(x_1 | x_0, \theta, u_1)\, r_{u|x}(u_1 | x_0)\, \mu_{x_1 \to \mathrm{dyn}}(x_1)\, \mu_{x_0 \to \mathrm{dyn}}(x_0)\, \mu_{\theta \to \mathrm{dyn}}(\theta)\, \mu_{u_1 \to \mathrm{dyn}}(u_1). \tag{89} \end{align}\]

The channel update follows from Proposition [15](#prop:vbp-r-ux){reference-type=“ref” reference=“prop:vbp-r-ux”}: \[\label{eq:vbp-channel-update} r_{u|x}^*(u_1 | x_0) = \frac{q_{\mathrm{pair}}(x_0, u_1)}{q_{x_0}(x_0)} = q(u_1 | x_0), \quad \text{where } q_{\mathrm{pair}}(x_0, u_1) = \iint q_{\mathrm{dyn}}^*\, \mathrm{d}x_1\, \mathrm{d}\theta.\tag{90}\]

12.4.4 Singleton Beliefs↩︎

Singleton beliefs follow the standard sum-product rule — the product of all incoming factor-to-variable messages with the prior (or goal prior): \[\tag{91} \begin{align} q_\theta^*(\theta) & \propto p(\theta)\, \mu_{\mathrm{obs} \to \theta}(\theta)\, \mu_{\mathrm{dyn} \to \theta}(\theta), \tag{92} \\ q_{x_1}^*(x_1) & \propto \hat{p}_x(x_1)\, \mu_{\mathrm{obs} \to x_1}(x_1)\, \mu_{\mathrm{dyn} \to x_1}(x_1), \tag{93} \\ q_{x_0}^*(x_0) & \propto p(x_0)\, \mu_{\mathrm{dyn} \to x_0}(x_0), \tag{94} \\ q_{u_1}^*(u_1) & \propto p(u_1)\, \mu_{\mathrm{dyn} \to u_1}(u_1), \tag{95} \\ q_{y_1}^*(y_1) & \propto \hat{p}_y(y_1)\, \mu_{\mathrm{obs} \to y_1}(y_1). \tag{96} \end{align}\]

12.5 Generic Scheme for Arbitrary \(T\)↩︎

The VBP scheme generalizes to arbitrary time horizons by introducing a time-local channel \(r_{u|x,t}(u_t | x_{t-1})\) at each time step \(t = 1, \ldots, T\). As in Appendix 11.6, the per-timestep additivity of the Lagrangian and the entropy correction \(+\sum_t \mathbb{H} \ifblank{}{\left[q(u_t | x_{t-1})\right]}{\csname l\endcsname[q(u_t | x_{t-1})\csname r\endcsname]}\) yields the multi-step scheme directly from the \(T{=}1\) derivation.

12.5.1 Messages from Observation Factor \(f_{\mathrm{obs}_t}\)↩︎

Standard sum-product (no channel modification): \[\tag{97} \begin{align} \mu_{\mathrm{obs}_t \to \theta}(\theta) & = \iint p(y_t | x_t, \theta)\, \mu_{y_t \to \mathrm{obs}_t}(y_t)\, \mu_{x_t \to \mathrm{obs}_t}(x_t)\, \mathrm{d}y_t\, \mathrm{d}x_t, \tag{98} \\ \mu_{\mathrm{obs}_t \to x_t}(x_t) & = \iint p(y_t | x_t, \theta)\, \mu_{y_t \to \mathrm{obs}_t}(y_t)\, \mu_{\theta \to \mathrm{obs}_t}(\theta)\, \mathrm{d}y_t\, \mathrm{d}\theta, \tag{99} \\ \mu_{\mathrm{obs}_t \to y_t}(y_t) & = \iint p(y_t | x_t, \theta)\, \mu_{\theta \to \mathrm{obs}_t}(\theta)\, \mu_{x_t \to \mathrm{obs}_t}(x_t)\, \mathrm{d}x_t\, \mathrm{d}\theta. \tag{100} \end{align}\]

12.5.2 Messages from Dynamics Factor \(f_{\mathrm{dyn}_t}\)↩︎

Using kernel \(p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\): \[\tag{101} \begin{align} \mu_{\mathrm{dyn}_t \to \theta}(\theta) & = \iiint p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\, \mu_{x_{t-1} \to \mathrm{dyn}_t}(x_{t-1})\, \mu_{u_t \to \mathrm{dyn}_t}(u_t)\, \mu_{x_t \to \mathrm{dyn}_t}(x_t)\, \mathrm{d}x_t\, \mathrm{d}x_{t-1}\, \mathrm{d}u_t, \tag{102} \\ \mu_{\mathrm{dyn}_t \to x_t}(x_t) & = \iiint p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\, \mu_{x_{t-1} \to \mathrm{dyn}_t}(x_{t-1})\, \mu_{u_t \to \mathrm{dyn}_t}(u_t)\, \mu_{\theta \to \mathrm{dyn}_t}(\theta)\, \mathrm{d}x_{t-1}\, \mathrm{d}u_t\, \mathrm{d}\theta, \tag{103} \\ \mu_{\mathrm{dyn}_t \to x_{t-1}}(x_{t-1}) & = \iiint p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\, \mu_{x_t \to \mathrm{dyn}_t}(x_t)\, \mu_{u_t \to \mathrm{dyn}_t}(u_t)\, \mu_{\theta \to \mathrm{dyn}_t}(\theta)\, \mathrm{d}x_t\, \mathrm{d}u_t\, \mathrm{d}\theta, \tag{104} \\ \mu_{\mathrm{dyn}_t \to u_t}(u_t) & = \iiint p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\, \mu_{x_{t-1} \to \mathrm{dyn}_t}(x_{t-1})\, \mu_{\theta \to \mathrm{dyn}_t}(\theta)\, \mu_{x_t \to \mathrm{dyn}_t}(x_t)\, \mathrm{d}x_t\, \mathrm{d}x_{t-1}\, \mathrm{d}\theta. \tag{105} \end{align}\]

12.5.3 Factor Beliefs↩︎

\[\tag{106} \begin{align} q_{\mathrm{obs},t}^*(y_t, x_t, \theta) & \propto p(y_t | x_t, \theta)\, \mu_{y_t \to \mathrm{obs}_t}(y_t)\, \mu_{x_t \to \mathrm{obs}_t}(x_t)\, \mu_{\theta \to \mathrm{obs}_t}(\theta), \tag{107} \\ q_{\mathrm{dyn},t}^*(x_t, x_{t-1}, \theta, u_t) & \propto p(x_t | x_{t-1}, \theta, u_t)\, r_{u|x,t}(u_t | x_{t-1})\, \mu_{x_t \to \mathrm{dyn}_t}(x_t)\, \mu_{x_{t-1} \to \mathrm{dyn}_t}(x_{t-1})\, \mu_{\theta \to \mathrm{dyn}_t}(\theta)\, \mu_{u_t \to \mathrm{dyn}_t}(u_t). \tag{108} \end{align}\]

12.5.4 Channel Updates↩︎

Each time step has its own channel update: \[\label{eq:vbp-gen-channel-update} r_{u|x,t}^*(u_t | x_{t-1}) = q_t(u_t | x_{t-1}), \quad \text{where } q_t(u_t | x_{t-1}) = \frac{q_{\mathrm{pair},t}(x_{t-1}, u_t)}{q_{x_{t-1}}(x_{t-1})}\tag{109}\] with \(q_{\mathrm{pair},t}(x_{t-1}, u_t) = \iint q_{\mathrm{dyn},t}^*\, \mathrm{d}x_t\, \mathrm{d}\theta\).

12.5.5 Singleton Beliefs↩︎

\[\tag{110} \begin{align} q_{x_t}^*(x_t) & \propto \hat{p}_x(x_t)\, \mu_{\mathrm{obs}_t \to x_t}(x_t)\, \mu_{\mathrm{dyn}_t \to x_t}(x_t)\, \mu_{\mathrm{dyn}_{t+1} \to x_t}(x_t), \tag{111} \\ q_\theta^*(\theta) & \propto p(\theta) \prod_{\tau=1}^{T} \mu_{\mathrm{obs}_\tau \to \theta}(\theta)\, \mu_{\mathrm{dyn}_\tau \to \theta}(\theta), \tag{112} \\ q_{u_t}^*(u_t) & \propto p(u_t)\, \mu_{\mathrm{dyn}_t \to u_t}(u_t), \tag{113} \\ q_{y_t}^*(y_t) & \propto \hat{p}_y(y_t)\, \mu_{\mathrm{obs}_t \to y_t}(y_t). \tag{114} \end{align}\]

The boundary conditions follow the usual temporal-edge conventions: at \(t=1\), \(\mu_{x_0 \to \mathrm{dyn}_1}(x_0) = p(x_0)\); at \(t=T\), the message \(\mu_{\mathrm{dyn}_{T+1} \to x_T}\) is absent.

12.6 Properties↩︎

12.6.0.1 No min-max structure.

Unlike AIF, where opposing channels create a min-max optimization over beliefs, VBP has a single channel that enters only the dynamics kernel numerator. The joint optimization over \((q, r_{u|x})\) is a pure minimization problem, avoiding the convergence difficulties of AIF’s saddle-point structure.

12.6.0.2 Convergence.

The observation factor uses standard sum-product messages with no channel modification, so standard BP convergence guarantees apply to the observation side. The dynamics channel provides a single-sided correction. While damping may improve convergence in practice, the lack of opposing channels makes it less critical than for AIF.

12.6.0.3 Reduction.

Setting \(r_{u|x,t}\) to uniform for all \(t\) removes the entropy correction and recovers standard belief propagation (marginal inference). The VBP scheme thus continuously interpolates between marginal inference (uniform channel) and full cross-entropy planning (converged channel).

12.6.0.4 Fixed-point interpretation.

At convergence, \(r_{u|x,t}^*(u_t | x_{t-1}) = q(u_t | x_{t-1})\), so the VBP dynamics kernel becomes \(p(x_t | x_{t-1}, \theta, u_t)\, q(u_t | x_{t-1})\). This reweights the state transition by the policy conditioned on the previous state, implementing the action commitment that cross-entropy planning encodes.

13 Experiment Details↩︎

This appendix provides full details for the experiments in Section 6.

13.1 Frozen Lake Environment↩︎

13.1.0.1 Grid layout.

We adapt the classic Frozen Lake environment [33], [34] to include epistemic uncertainty by treating the hole layout as unknown. A \(4{\times}4\) grid where the agent starts at the top-left cell and must reach the goal at the bottom-right cell. A subset of the remaining cells are holes; stepping into a hole terminates the episode with failure. The ice surface makes transitions stochastic: with probability \(1 - p_\text{slip}\) the agent moves in the intended direction, and with probability \(p_\text{slip}/3\) it slips to each of the three remaining directions, where \(p_\text{slip} = 0.1\).

13.1.0.2 Configurations.

The hole layout is the unknown parameter \(\theta\). We sample \(15\) hole configurations uniformly at random (with fixed seeds for reproducibility), each placing \(2\) holes on the grid (a fraction \(0.2\) of the \(14\) non-start/goal cells, truncated to an integer). The agent does not know the hole locations and must infer them from observations.

13.1.0.3 Observation model.

The observation model has two modalities. First, \(2 n_\text{pos}\) position channels observe the agent’s position and scan mode with near-deterministic precision (\(0.999/0.001\)), so the agent always knows where it is. Second, \(n_\text{pos}\) grid-cell channels each provide a binary “hole/safe” reading for the corresponding cell. Unscanned grid-cell observations are corrupted by distance-dependent noise: \(\text{noise} = \alpha_\text{base} + \alpha_\text{range} \cdot d / d_\text{max}\), where \(d\) is the Manhattan distance from the agent to the cell. A low-noise reading directly constrains which configurations \(\theta\) are consistent, making the observation model approximately unambiguous. A SCAN action switches all grid-cell observations to near-deterministic (\(0.999/0.001\)) at the cost of one time step.

13.1.0.4 State, action, and observation spaces.

States encode position and scan mode: \(2 \times n_\text{pos}\) states total (e.g., \(32\) for a \(4{\times}4\) grid). Actions are the four cardinal directions plus SCAN (\(|\mathcal{U}| = 5\)). Observations consist of \(2 n_\text{pos}\) position channels and \(n_\text{pos}\) grid-cell channels.

13.1.0.5 Priors.

The goal prior \(\hat{p}(x_T)\) is a softmax preference peaking at the bottom-right cell, with penalties for hole positions (varying per \(\theta\)). The parameter prior is uniform over the \(15\) configurations: \(p(\theta) = 1/15\). The initial state prior \(p(x_0)\) places all mass on the top-left cell in unscanned mode. The action prior assigns weight \(1\) to each movement action and weight \(c_\text{scan} = 0.1\) to SCAN, then normalizes: \(p(u_t) = w_{u_t} / \sum_u w_u\), giving \(p(\text{move}) \approx 0.244\) and \(p(\text{SCAN}) \approx 0.024\).

13.1.0.6 Planning parameters.

Planning horizon \(T = 15\), fixed across all decision steps. All methods use \(400\) iterations. We run \(1000\) episodes per method with a maximum of \(15\) steps per episode. Episode \(i\) uses seed \(i\) for reproducibility.

13.2 RockSample Environment↩︎

13.2.0.1 Grid layout.

We adapt the classic RockSample environment [35] to the epistemic planning framework by treating rock quality as the unknown parameter \(\theta\). A \(5{\times}5\) grid where the agent starts at the left edge and can exit via the right edge. Two rocks are placed at known grid positions; their quality (good or bad) is unknown.

13.2.0.2 Configurations.

Rock quality defines the unknown parameter \(\theta\). With \(2\) rocks each having binary quality, there are \(n_\theta = 4\) configurations. The agent does not know rock quality and must infer it from distance-dependent observations.

13.2.0.3 Observation model.

The observation model has two components. First, position channels observe the agent’s position with noise parameter \(\alpha_\text{pos} = 0.3\). Second, the agent passively receives a binary “good/bad” quality reading for the nearest rock, whose accuracy depends on the Euclidean distance \(d\) from the agent to that rock: \(p(\text{correct} \mid d) = \tfrac{1}{2}\bigl(1 + 2^{-d/d_{1/2}}\bigr)\), where \(d_{1/2} = 0.5\) is the half-efficiency distance. At \(d = 0\) the reading is deterministic; at \(d = d_{1/2}\) accuracy is \(75\%\); as \(d \to \infty\) the reading approaches chance. This makes the observation model locally decisive: nearby observations reliably determine rock quality, so the epistemic strategy is spatial, approach a rock before sampling.

13.2.0.4 Actions and rewards.

The agent has seven actions: four cardinal movements, CHECK, SAMPLE, and EXIT (move off the right edge). CHECK reveals the quality of the nearest rock at the cost of one time step, providing the main explicit epistemic action. SAMPLE collects the rock at the current position, yielding reward \(+2\) (good rock) or penalty \(-3\) (bad rock). EXIT gives a fixed reward \(+1\). Movement is deterministic (\(p_\text{slip} = 0\)).

13.2.0.5 State, action, and observation spaces.

States encode position: \(n_\text{pos} = 25\) states for the \(5{\times}5\) grid. Actions: four cardinal directions, CHECK, SAMPLE, and EXIT (\(|\mathcal{U}| = 7\)). Observations consist of \(n_\text{pos}\) position channels and \(2\) binary rock-quality channels.

13.2.0.6 Priors.

The goal prior \(\hat{p}(x_T)\) is a softmax preference with temperature \(\tau = 0.5\) peaking at EXIT cells, with penalties for remaining on the grid. The parameter prior is uniform over the \(4\) configurations: \(p(\theta) = 1/4\). The action prior assigns weight \(1\) to each movement action and weight \(c_\text{exit} = 0.5\) to EXIT, then normalizes.

13.2.0.7 Planning parameters.

Planning horizon \(T = 15\), fixed across all decision steps. BP uses \(50\) iterations; all other methods use \(200\) iterations. We run \(1000\) episodes per method with a maximum of \(15\) steps per episode. Episode \(i\) uses seed \(i\) for reproducibility.

13.2.0.8 Full results with confidence intervals.

3 reports RockSample results with 95% confidence intervals.

4pt

13.3 Wumpus World Environment↩︎

13.3.0.1 Grid layout.

We adapt the classic Wumpus World environment [36] to include epistemic uncertainty by treating the layout as unknown. We simplify the classic dynamics to isolate the epistemic challenge: the agent has no orientation or inventory and navigates by cardinal movement. The agent starts at cell \(0\) and must reach the gold cell. The grid contains pits and a wumpus (both absorbing hazards that terminate the episode) and a single gold cell.

13.3.0.2 Configurations.

The locations of pits, wumpus, and gold define the unknown parameter \(\theta\). We use \(25\) configurations sampled with fixed seeds. Each configuration places \(4\) pits, one wumpus, and one gold on the grid, excluding the agent’s starting cell.

13.3.0.3 Observation model.

The observation model has two components, both noisy when unscanned. Three binary feature channels detect adjacency to hazards: breeze fires if adjacent to a pit, stench fires if adjacent to the wumpus, and glitter fires if on the gold cell. Unscanned feature channels have true-positive probability \(p_\text{tp} = 1 - \alpha_\text{obs}\) and false-positive probability \(p_\text{fp} = 0.1 \, \alpha_\text{obs}\). Additionally, \(n_\text{pos}\) position channels encode the agent’s position, also noisy when unscanned. Observations are ambiguous: a breeze indicates a nearby pit but not which neighbor, and position uncertainty compounds this ambiguity. The agent must integrate evidence across multiple positions to triangulate hazard locations. A SCAN action switches all channels (feature and position) to near-deterministic (\(0.999/0.001\)) at the cost of one time step.

13.3.0.4 State, action, and observation spaces.

States encode position and scan mode: \(2 \times n_\text{pos}\) states total (e.g., \(50\) for a \(5{\times}5\) grid). Actions are the four cardinal directions plus SCAN (\(|\mathcal{U}| = 5\)). Observations consist of \(3\) binary feature channels (breeze, stench, glitter) and \(n_\text{pos}\) position channels.

13.3.0.5 Priors.

The goal prior \(\hat{p}(x_T)\) is a softmax preference peaking at the gold cell for each \(\theta\), with penalties for pits and the wumpus. The parameter prior is uniform over the \(25\) configurations. The initial state prior \(p(x_0)\) places all mass on position \(0\) in unscanned mode. The action prior assigns weight \(1\) to each movement action and weight \(c_\text{scan} = 0.7\) to SCAN, then normalizes: \(p(u_t) = w_{u_t} / \sum_u w_u\), giving \(p(\text{move}) \approx 0.213\) and \(p(\text{SCAN}) \approx 0.149\).

13.3.0.6 Planning parameters.

Planning horizon \(T = 7\), fixed across all decision steps. All methods use \(400\) iterations. SCAN costs one time step (same as Frozen Lake). We run \(1000\) episodes per method with a maximum of \(10\) steps per episode. Episode \(i\) uses seed \(i\) for reproducibility.

13.3.0.7 Representative trajectories.

[fig:wumpus_trajectory_18,fig:wumpus_trajectory_11] show two representative episodes that illustrate the qualitative behavioral separation behind the aggregate Wumpus gap reported in Section 6.2. Each panel shows one method on a fixed layout, with the agent’s path coloured by step order (light early, dark late). The header below each panel reports the terminal reward, the number of steps taken, the outcome (success or pit), and the position and reading of any SCAN action.

5 illustrates the suggestive-observation regime that drives the gap. BP, RM-MP, and VBP step into a pit on the first move without scanning. Nuijten-MP scans at the start and acquires the breeze, no-stench, no-glitter reading, but its non-variational prior update fails to translate this evidence into a safe move, so it commits to a doomed step and falls in a pit on step 2. AIF-MP scans, observes the same reading, and then reaches the gold cell in seven further steps.

6 shows a layout where one column happens to be hazard-free, so methods that move forward without scanning can succeed by luck. BP reaches the gold cell in eight steps and VBP in nine steps, both without scanning; RM-MP and Nuijten-MP still step into a pit within five steps. AIF-MP scans first, then follows the same column to the gold cell in eight steps.

13.4 Common Implementation Details↩︎

13.4.0.1 Software framework.

All tensor operations and message-passing routines use JAX [40] with JIT compilation. All planning and inference functions are decorated with jax.jit, with the planning horizon and number of iterations as compile-time constants.

13.4.0.2 Log-space computation.

All internal messages, beliefs, and channels are stored as log-probabilities. A sentinel value of \(-10^{12}\) replaces \(-\infty\) for numerical stability, and a safe logarithm floors its argument at \(10^{-30}\) before taking the log. Conversion to probability space occurs only at final output via softmax.

13.4.0.3 Message damping.

Channel-based methods (VBP, RM-MP, and AIF-MP) apply arithmetic damping 26 in probability space (implemented via logaddexp in log-space). Damping is applied to all three channels (dynamics, observation, and marginal observation) after each BP iteration. Structural zeros are preserved: a position remains at \(-10^{12}\) only if both old and new values are \(-10^{12}\). For VBP, this reduces to damping the single policy channel; BP and Nuijten-MP do not use damping (\(\lambda = 1.0\)). The damping parameter \(\lambda\) is selected per method and environment from the convergence sweep described in 13.5. 4 reports the selected values.

4pt

13.4.0.4 Message initialization.

All channels (dynamics, observation, and marginal observation) are initialized to uniform distributions over their respective domains. Parameter cavity beliefs are initialized to the prior \(p(\theta)\); state beliefs are initialized to uniform over valid states.

13.4.0.5 Action selection.

Actions are selected by greedy argmax over the action marginal at \(t = 0\) (deterministic, no sampling).

13.5 Convergence Behavior↩︎

To select damping parameters and characterize convergence, we run a systematic sweep over \(\lambda \in \{0.25, 0.4, 0.5, 0.6, 0.75, 0.9\}\) for each channel-based method (RM-MP, VBP, AIF-MP) and \(\lambda = 1.0\) for loopy BP, across all three environments. Each configuration is run with \(5\) random seeds and \(1{,}000\) BP iterations. Convergence is declared when the maximum absolute change in any channel falls below \(10^{-4}\).

13.5.0.1 Convergence rate.

7 reports the fraction of seeds that converge (color) and the median number of iterations to convergence (in parentheses) for each method–damping combination. The three environments exhibit qualitatively different convergence profiles.

On RockSample (7 (b)), all methods converge at all damping values. The deterministic dynamics make the dynamics channel update exact, removing one source of instability.

On Frozen Lake (7 (a)), the dynamics channel (RM-MP) is the most fragile: convergence drops to \(0\%\) for \(\lambda \geq 0.5\), with large VFE oscillations. This is consistent with the min-max structure identified in Section 5: the stochastic dynamics activate the opposing-sign dynamics channel, which amplifies update steps when damping is insufficient. VBP is robustly stable across all damping values (\(80\)–\(100\%\)), since it uses only the planning channel (no opposing signs). AIF-MP converges reliably at higher damping (\(100\%\) at \(\lambda = 0.9\)) but slowly at conservative settings (\(20\%\) at \(\lambda = 0.25\)).

On Wumpus World (7 (c)), most methods converge well at moderate damping. The dynamics channel fails only at \(\lambda = 0.9\) (\(40\%\), with VFE values exceeding \(10^3\) on divergent seeds). AIF-MP shows non-monotone behavior: \(100\%\) convergence at \(\lambda \in \{0.25, 0.4, 0.6\}\) but only \(20\%\) at \(\lambda = 0.5\) and \(60\%\) at \(\lambda = 0.9\). The suggestive observation structure means the observation-side channels are more active, producing a more complex optimization landscape.

13.5.0.2 VFE convergence dynamics.

8 shows VFE traces for all methods at their best damping on Frozen Lake. All four methods reach a stationary plateau within the iteration budget: loopy BP within \({\sim}5\) iterations, and the channel-augmented methods within \(15\)–\(150\) iterations, depending on the number of active channels. The absolute VFE values at the plateau are not directly comparable across methods because each method optimizes a different functional (1).

13.5.0.3 Damping selection.

The damping parameter \(\lambda\) is selected per method and environment as the value with the highest convergence rate; ties are broken by fewest median iterations. The selected values are reported in 4. The pattern is consistent across environments: RM-MP requires conservative to moderate damping (\(\lambda = 0.25\)–\(0.5\)) due to the opposing dynamics channel, while VBP tolerates higher damping (\(\lambda = 0.75\)–\(0.9\)). AIF-MP is more variable, ranging from conservative (\(\lambda = 0.25\) on Wumpus World) to aggressive (\(\lambda = 0.9\) on Frozen Lake and RockSample), reflecting the environment-dependent optimization landscape. The non-monotone convergence behavior of AIF-MP (e.g., \(20\%\) at \(\lambda = 0.5\) but \(100\%\) at \(\lambda = 0.6\) on Wumpus World) makes per-environment tuning necessary rather than using a single global value. Developing convergence theory for the channel-augmented scheme remains an open problem.

References↩︎

---

1. We write \(\mathrm{h} \ifblank{}{\left[q(y|x)\right]}{\csname l\endcsname[q(y|x)\csname r\endcsname]}\) for the entropy of the conditional \(q(y|x)\), a function of \(x\), and \(\mathbb{H} \ifblank{}{\left[q(y|x)\right]}{\csname l\endcsname[q(y|x)\csname r\endcsname]}\) for the conditional entropy, a scalar: \(\mathbb{H} \ifblank{}{\left[q(y|x)\right]}{\csname l\endcsname[q(y|x)\csname r\endcsname]} = \mathbb{E}_{q(x)} \ifblank{}{\left[ \mathrm{h} \ifblank{}{\left[q(y|x)\right]}{\csname l\endcsname[q(y|x)\csname r\endcsname]}\right]}{\csname l\endcsname[ \mathrm{h} \ifblank{}{\left[q(y|x)\right]}{\csname l\endcsname[q(y|x)\csname r\endcsname]}\csname r\endcsname]}\).↩︎

2. Kappen-style tempering \(\hat{p}(x) \propto \exp(R(x)/\lambda)\) [18], [19] parametrizes the generative model (the preference prior), whereas 1 parametrizes the objective via entropy corrections; the two axes are orthogonal.↩︎

3. Code available at https://github.com/biaslab/UAI-MP-AIF-JAX↩︎