1 Introduction↩︎

Decision-making under partial observability requires agents to balance exploiting current knowledge against gathering information to reduce uncertainty about hidden states. In standard POMDPs, information gathering has no intrinsic value. It is useful only insofar as it leads to higher expected reward. This creates a well-known difficulty: the exploration–exploitation trade-off must be resolved either by the planning horizon or by heuristic exploration bonuses.

The \(\rho\)-POMDP framework [1] addresses this by extending POMDPs with a belief-dependent utility \(\rho(b)\) that allows the agent to derive value directly from properties of its belief state. This enables explicit optimization over uncertainty reduction, information gain, or other belief-state properties alongside task reward. However, the choice of \(\rho\) remains largely heuristic and environment-specific.

Separately, the Active Inference (AIF) framework [2], [3] casts perception and action as approximate Bayesian inference, selecting policies that minimize Expected Free Energy (EFE). EFE naturally decomposes into a pragmatic term (goal-seeking) and an epistemic term (information-seeking). Because both terms come from a single variational objective, their relative scale is not free to choose. In the discrete-state formulation we use, the coefficient on expected information gain is fixed at \(w{=}1\) when both terms are measured in nats, the natural-log unit of information (Proposition [prop:equivalence]). The weight is a consequence of the derivation, not a hyperparameter.

We propose substituting EFE as \(\rho\) in \(\rho\)-POMDPs, yielding an agent whose epistemic foraging is a consequence of its objective rather than an engineered bonus.¹ Our contributions are:

1. Theory. A formal bridge between \(\rho\)-POMDPs and active inference. We prove the equivalence for observe-then-commit POMDPs (Proposition [prop:equivalence]), characterize when the canonical weight is near-optimal (Proposition [prop:nearopt]), and extend the equivalence to factored observation POMDPs, interleaved settings where information gathering preserves the hidden state (Proposition [prop:factored]).

2. Evidence. Controlled comparisons against same-horizon planning, tuned information gain, and POMCP [4] across six observe-then-commit environments and four instances of the standard RockSample benchmark [5]. A Pareto analysis shows that \(w{=}1\) Pareto-dominates same-horizon planning without per-environment search.

3. Practical guidance. A characterization of when EFE-as-\(\rho\) helps. The advantage appears when the agent must choose among multiple observation actions, and it grows with state space size (\(66.5\%\) vs.\(2.5\%\) success on Tileworld \(8{\times}8\)) and with the number of observation actions (\(+7.82\) reward on RockSample[7,8]).

2 Related work↩︎

2.0.0.1 POMDPs and solvers.

POMDPs formalize sequential decision-making under state uncertainty [6], [7]. Exact solutions are PSPACE-complete. Point-based offline methods (PBVI [8], HSVI [5], SARSOP [9]) approximate the value function on reachable beliefs [10]. Online solvers plan from the current belief: POMCP [4] uses MCTS with UCB1 and rollout evaluation, DESPOT [11] searches a regularized sparse belief tree, and POMCPOW [12] extends POMCP with progressive widening for continuous spaces. All explore implicitly through stochastic simulations rather than explicitly valuing information gain. Section 6 and Appendix 27 demonstrate the benefit of closed-form information valuation.

2.0.0.2 \(\rho\)-POMDPs.

[1] introduced \(\rho\)-POMDPs, augmenting the reward with a belief-dependent utility \(\rho : \Delta(S) \to \mathbb{R}\), so the objective becomes \(\max_\pi \mathbb{E}_\pi[\sum_t \gamma^t (R(s_t,a_t) + \rho(b_t))]\). When \(\rho\) is convex, the value function remains piecewise linear and convex (PWLC), preserving compatibility with standard solvers. [13] extended this to Lipschitz-continuous non-convex \(\rho\), and [14] developed \(\rho\)-POMCPOW for continuous-space \(\rho\)-POMDPs. Common choices for \(\rho\) (entropy reduction, KL divergence from a target belief, and information gain) each encode a different notion of epistemic value, but the choice remains heuristic and environment-specific. Our contribution is to derive a principled \(\rho\) from the variational bound of active inference, fixing the exploration weight without per-environment search.

2.0.0.3 Value of information and experimental design.

The idea that information has quantifiable decision-theoretic value predates both POMDPs and active inference. [15] formalized the value of information in decision analysis, and [16] introduced expected information gain as a criterion for optimal Bayesian experimental design. In the bandit setting, information-directed sampling (IDS) [17] explicitly trades off instantaneous regret against information gain by minimizing the information ratio \(\Gamma_t = \delta_t^2 / g_t\), where \(\delta_t\) is the expected regret and \(g_t\) is the information gain. The key structural difference from EFE is that IDS minimizes the ratio at each step (a relative weighting that adapts to the current belief), while EFE fixes the weight at \(w{=}1\) (an absolute weighting derived from the variational bound). An IDS baseline adapted to the observe-then-commit structure would be informative but is beyond the scope of this work. Our observe-then-commit structure parallels sequential Bayesian experimental design, where the agent selects experiments (observations) to maximize information about an unknown state before making a terminal decision. The \(\rho\)-POMDP framework with \(\rho = I(b)\) operationalizes this connection. Our contribution is to show that EFE derives a canonical weight for the information gain term from first principles rather than treating it as a tunable parameter.

2.0.0.4 Active inference and EFE.

The Active Inference (AIF) framework [2] casts perception and action as variational inference under the free-energy principle. [18] provide a comprehensive textbook treatment. [19] formalized the decomposition of policy value into extrinsic (goal-seeking) and epistemic (information-seeking) components, showing that curiosity-driven exploration arises automatically from expected free energy minimization. [20] synthesized discrete-state AIF from first principles, showing that the posterior over policies takes the form \(Q(\pi) \propto \exp(-\mathcal{G}(\pi))\) where \(\mathcal{G}\) is the Expected Free Energy. [3] showed that EFE decomposes into pragmatic value (divergence from preferred observations) and epistemic value (expected information gain), with both arising from a single variational bound, requiring no tunable exploration weight. Despite different constructions, EFE and Generalised Free Energy produce identical policy posteriors. The mathematical foundations of EFE have been critically examined: [21] showed that naively extending the variational free energy into the future does not yield exploratory behavior, proposing the Free Energy of the Expected Future (FEEF) as an alternative with clearer mathematical grounding. [22] addressed the “unification problem” by formalizing how multiple EFE formulations relate to a single root definition under different assumptions about prior preferences. [23] recast EFE-based planning as entropy-corrected variational inference with message-passing schemes, providing an alternative derivation that connects EFE to variational message passing.

2.0.0.5 Sophisticated inference.

Standard AIF evaluates policies myopically. [24] introduced sophisticated inference, a recursive extension implementing deep tree search over belief trajectories rather than states. Sophistication (maintaining beliefs about future beliefs) enables counterfactual reasoning about the downstream epistemic consequences of actions. [25] proved that this recursive scheme recovers Bellman-optimal policies for any finite horizon, whereas standard AIF achieves optimality only for single-step planning. Our recursive EFE agent (Equation 1 ) is derived from this framework, adapted to the \(\rho\)-POMDP commit-action structure. The recursive counterfactual reasoning over future beliefs is preserved. What changes is that our observe-then-commit setting does not involve state transitions between time steps, simplifying the belief-trajectory computation.

2.0.0.6 Scaling active inference.

Discrete-state AIF with full policy enumeration is limited to small state–action spaces. Several lines of work address scaling. [26] combined deep generative models with Monte Carlo tree search for EFE-optimal planning in continuous state spaces. [27] developed an RL-compatible objective (the free energy of the expected future) that inherits AIF’s exploration–exploitation balance while scaling to standard RL benchmarks. [28] combined active inference with Monte-Carlo tree search for large POMDPs, achieving state-of-the-art on RockSample [5]. Our MCTS-EFE variant (Section 6) follows this direction, using EFE as a leaf heuristic within MCTS to extend planning horizons beyond exact tree search.

2.0.0.7 Exploration, control as inference, and intrinsic motivation.

The control-as-inference perspective [29], [30] casts reward maximization as variational inference. Maximum-entropy RL [31] and stochastic optimal control [32] are algorithmic instances. [33] proved formal equivalence between AIF and control-as-inference. [34] showed that intrinsic motivation behaviors arise under EFE. Intrinsic motivation methods, including curiosity [35], [36], typologies of intrinsic signals [37], Bayesian surprise [38], count-based exploration [39], VIME [40], and random network distillation [41], all require tunable bonus weights. Bayes-Adaptive MDPs [42], [43] and Bayesian RL more broadly [44] address model uncertainty (unknown dynamics), distinct from our focus on state uncertainty under a known model. Our \(\rho\)-POMDP formalism makes the connection between EFE and these lines of work precise, inheriting formal properties (convexity, Lipschitz continuity) while fixing the exploration weight from first principles.

3 Methodology↩︎

3.1 The \(\rho\)-POMDP framework↩︎

We restrict attention to observe-then-commit \(\rho\)-POMDPs, in which the action set \(\mathcal{A}\) partitions into observation actions \(\mathcal{A}_\text{obs}\) (which update the belief at known cost but do not change the hidden state) and terminal commit actions \(\mathcal{A}_\text{com}\) (which end the episode with state-dependent reward). An episode consists of a variable-length sequence of observation actions followed by a single commit action. The hidden state is fixed throughout.

A \(\rho\)-POMDP extends the standard POMDP with a belief-dependent utility \(\rho : \Delta(S) \rightarrow \mathbb{R}\). The agent’s objective becomes: \[\pi^* = \arg\max_\pi \mathbb{E}_\pi \left[ \sum_{t=0}^{H} \gamma^t \left( R(s_t, a_t) + \rho(b_t) \right) \right]\] When \(\rho = 0\), we recover the standard POMDP. When \(\rho\) encodes information gain, the agent is explicitly rewarded for reducing uncertainty.

3.2 Expected Free Energy as \(\rho\)↩︎

In the standard AIF formulation, the EFE for a policy \(\pi\) at future time \(\tau\) is: \[\mathcal{G}(\pi) = \underbrace{- \mathbb{E}_{Q(o_\tau|\pi)}[\ln P(o_\tau|C)]}_{\text{Pragmatic value}} - \underbrace{\mathbb{E}_{Q(o_\tau|\pi)}\left[D_{\mathrm{KL}}\left[Q(s_\tau|o_\tau, \pi) \,\|\, Q(s_\tau|\pi)\right]\right]}_{\text{Epistemic value}}\] where \(P(o_\tau|C)\) encodes preferred outcomes and \(D_{\mathrm{KL}}\) measures expected information gain. In our observe-then-commit setting, the pragmatic and epistemic terms take concrete forms. For commit actions, the pragmatic term reduces to expected reward under the current belief: \(\mathcal{G}(\text{commit}_i) = -\mathbb{E}_b[R_i]\). We use the reward matrix directly rather than encoding rewards through preferred outcome distributions \(P(o|C)\), avoiding the preference-calibration issue flagged as a source of hidden tuning in prior AIF work. For observation actions, the pragmatic term is the known observation cost \(c_k\), and the epistemic term is the expected information gain \(I_k(b) = H(b) - \mathbb{E}_o[H(b'_o \mid \text{obs}_k)]\).

Following the sophisticated inference scheme of [24], the EFE agent evaluates actions via a recursive tree search over belief states: \[\mathcal{G}(\text{observe}_k) = c_k - I_k(b) + \mathbb{E}_{o}\!\left[\min_a \mathcal{G}(a \mid b'_o)\right] \label{eq:efe95recursive}\tag{1}\] The agent selects \(\arg\min_a \mathcal{G}(a)\). Standard AIF introduces a precision parameter \(\beta\) via a softmax policy \(Q(\pi) \propto \exp(-\beta \mathcal{G}(\pi))\). Our deterministic \(\arg\min\) corresponds to \(\beta \to \infty\), which eliminates this tunable knob. Equation 1 contains no separate exploration weight, and this is not a modeling shortcut. When EFE is written in nats, expected information gain enters on the same footing as the KL terms that define the variational objective. This shared scale is why the \(\rho\)-POMDP reduction carries \(w{=}1\) exactly (Proposition [prop:equivalence]).

3.3 Formal equivalence with \(\rho\)-POMDPs↩︎

We generalize the standard \(\rho\)-POMDP formulation to action-dependent belief utilities \(\rho(b, a)\), where the augmented reward becomes \(R(s,a) + \rho(b,a)\). This is equivalent to a standard \(\rho(b)\) formulation on an augmented belief-action space but avoids notational overhead. The structural results of [1] carry over when \(\rho(\cdot, a)\) satisfies the relevant conditions for each fixed \(a\).

Define \(V(a, b, d) \triangleq -\mathcal{G}(a, b, d)\). Then Equation 1 becomes: \[\begin{align} V(\text{observe}_k, b, d) &= -c_k + I_k(b) + \mathbb{E}_o\!\left[\max_a V(a, b'_o, d{+}1)\right] \tag{2} \\ V(\text{commit}_i, b, d) &= \mathbb{E}_b[R_i] \tag{3} \end{align}\] This is exactly the Bellman recursion for a \(\rho\)-POMDP (Equation 1) with action-dependent belief utility:

Proposition [prop:equivalence] makes precise what EFE-as-\(\rho\) means: the epistemic term of EFE functions as an action-dependent belief utility. Crucially, this is equivalent to Planning+IG with \(w{=}1\) (in nats). EFE does not eliminate the weight. It derives a canonical weight from the variational bound, fixing \(w{=}1\) without per-environment search. Whether this canonical choice is near-optimal is an empirical question we address in Section 5.2. The following result characterizes the conditions under which \(w{=}1\) is near-optimal for expected reward.

Table ¿tbl:tab:alpha95eta? validates Proposition [prop:nearopt] across our environments, computing \(\alpha\), \(\eta\), the threshold \(w^*_{\text{thresh}}\), and comparing against the observed reward-maximizing weight from the Pareto sweep (Section 5.2).

The proposition is stated for \(H{=}1\) and two states. The multi-step case is harder to analyze because the observation threshold shifts with belief, but the qualitative prediction holds: environments where \(\alpha \geq 5\) (Tiger, Diagnosis, Tileworld) have \(w^*_{\text{thresh}} \ll 0\), meaning \(w{=}1\) is far above the threshold and near-optimal. The low-asymmetry Testbed (\(\alpha{=}1\)) has \(w^*_{\text{thresh}} > 0\) but below 1, so \(w{=}1\) still induces observation but assigns more weight to information than is instrumentally optimal, consistent with our finding that EFE over-explores there (Appendix 10). On the Bandit (\(\alpha{=}1.1\)), \(w^*_{\text{thresh}}\) is slightly negative at \(H{=}1\). At \(H{>}1\), multi-step planning amplifies the value of each observation, bringing the effective reward-maximizing weight toward 1.

3.3.0.1 Extension to discounting.

With \(\gamma < 1\), the recursive EFE becomes \(\mathcal{G}(\text{obs}_k) = c_k - I_k(b) + \gamma\, \mathbb{E}_o[\min_a \mathcal{G}(a, b'_o)]\), giving \(V(\text{obs}_k) = -c_k + I_k(b) + \gamma\, \mathbb{E}_o[\max_a V(a, b'_o)]\). The information gain term \(I_k(b)\) appears undiscounted at the current step, while future values are discounted. This preserves the \(\rho\)-POMDP equivalence with \(\rho(b,a) = I_a(b)\), but the effective ratio of epistemic to pragmatic weight increases at early steps relative to late steps. In the undiscounted case (\(\gamma{=}1\)), both terms are weighted equally at every depth. With \(\gamma < 1\), the agent places relatively more value on immediate information gain compared to future reward, producing slightly more exploratory behavior at early steps. Our experiments (Appendix 24) confirm that performance is robust across \(\gamma \in \{0.9, 0.95, 0.99, 1.0\}\).

3.3.0.2 Extension to factored observation POMDPs.

Proposition [prop:equivalence] assumes the observe-then-commit structure. We now extend the equivalence to a broader class that includes interleaved observe-act POMDPs.

Observe-then-commit POMDPs are the special case where \(\mathcal{A}_{\mathrm{nav}} = \emptyset\) and \(s_{\mathrm{vis}}\) is trivial. RockSample [5] is an instance: \(s_{\mathrm{vis}}\) is the agent’s grid position (fully observable), \(s_{\mathrm{hid}}\) is the vector of rock qualities (hidden, fixed), check actions are observation actions, moves are navigation actions, and sample/exit are exploitation actions.

Proposition [prop:factored] extends the formal bridge from observe-then-commit to any POMDP where the hidden state is preserved by information-gathering and navigation actions. The agent interleaves observation, navigation, and exploitation. At each decision point, the EFE-as-\(\rho\) equivalence holds for the observation and navigation subtree. This covers RockSample, mobile sensor placement, and sequential testing with spatial access costs, where the agent must navigate to observation locations before gathering information. Section 5.4 validates this extension empirically across four RockSample instances.

The factored observation structure is common in practice whenever the quantity being measured is static or slow-changing relative to the decision horizon (Table ¿tbl:tab:taxonomy?). The key requirement, that \(T_{\mathrm{hid}}(s'_{\mathrm{hid}} | s_{\mathrm{hid}}, a) = \delta(s'_{\mathrm{hid}} = s_{\mathrm{hid}})\) for observation and navigation actions, breaks when information-gathering itself alters the hidden state. In such settings, the coupling term \(\Delta_T \neq 0\) and the canonical-weight equivalence does not hold. We discuss this further in the limitations paragraph of Section 6.

3.4 Agents↩︎

We compare six agents (Table ¿tbl:tab:agents?), all sharing the same belief-update machinery and differing only in objective function and planning depth. The Planning and Planning+IG baselines use the same recursive tree search as the EFE agent, isolating the effect of the \(\rho\) function from planning depth.

Planning+IG is the critical baseline: it uses the same tree search as the EFE agent with an additive IG bonus at the same horizon. By Proposition [prop:equivalence], the EFE agent is exactly Planning+IG with \(w{=}1\), so any advantage is attributable to the weight choice rather than a different mechanism. The Epistemic-only agent sets \(\mathcal{G}(\text{commit}) = 0\), removing reward awareness. It commits at chance on all environments, which confirms that the pragmatic term is essential. EFE computation is validated against pymdp [45] (Appendix 18).

For Info Gain and Planning+IG, \(w\) is tuned per environment via grid search over \(\{0.1, 0.5, 1, 2, 5, 10, 20, 50, 100\}\) on 200 tuning episodes. The Pareto analysis (Section 5.2) sweeps the full weight space.

4 Experiments↩︎

All environments are implemented as OpenAI Gymnasium environments following the observe-then-commit structure of Section 3. Main results use 1,000 episodes per seed across 5 random seeds (\(\{42, 123, 456, 789, 1024\}\)) for a total of 5,000 episodes. Results report the mean across all episodes. Statistical comparisons use \(t\)-tests with Holm–Bonferroni correction, with bootstrap CIs in Appendix 22. Comparison against POMCP (including compute-matched analysis with wall-clock timing at budgets 500–5,000 simulations) is in Appendix 27. Full specifications are in Appendix 15.

4.0.0.1 Tiger

[7]. Two states, one observation action (listen, accuracy 0.85), two commit actions. Rewards: correct \(+10\), incorrect \(-100\), listen cost \(-1\).

4.0.0.2 Sequential diagnosis.

\(N{=}4\) conditions, \(K{=}2\) binary tests (accuracy 0.80), \(N\) diagnose actions. Correct \(+10\), incorrect \(-50\), test cost \(-1\). The agent must choose which test to run.

4.0.0.3 Structured bandit.

\(K{=}4\) arms, \(K\) inspect actions (accuracy 0.80, cost \(-0.5\)), \(K\) pull actions. Best arm \(+10\), others \(+1\). The agent must choose which arm to inspect.

4.0.0.4 Tileworld.

An \(N{\times}N\) grid (\(N{=}6\), \(|S|{=}36\)) with a hidden target tile. \(K{=}6\) scan actions partition the grid via bit-level splits of row/column indices, each returning a noisy binary signal (accuracy \(0.80\), cost \(-1\)). \(N^2\) commit actions collect at a specific cell (correct \(+10\), incorrect \(-50\)). A spatial generalization of diagnosis that produces visually interpretable belief evolution (Figure 2).

4.0.0.5 RockSample

[5]. An \(N{\times}N\) grid with \(K\) rocks at known positions, each with hidden binary quality. Move actions change the agent’s position. Check actions produce distance-dependent noisy observations of rock quality. Sampling collects the rock at the current position (good \(+10\), bad \(-10\)), and exiting gives \(+10\). All actions cost \(-0.5\). We evaluate RS[5,3], RS[7,4], RS[7,8], and RS[11,11]. Unlike the above environments, RockSample has interleaved observe-act dynamics with state transitions, the setting addressed by Proposition [prop:factored].

4.0.0.6 Structural inspection.

\(N\) components at known spatial locations on a grid, each with a hidden binary state (nominal/faulty, prior \(p_{\text{fault}}{=}0.3\)). There are \(K{=}2\) non-destructive test types: visual (accuracy \(0.70\), cost \(-0.5\)) and detailed (accuracy \(0.90\), cost \(-2\)). The agent navigates between components, runs tests, and declares a diagnosis for each. Correct nominal \(+2\), correct fault \(+5\), missed fault \(-50\), false alarm \(-5\), move cost \(-0.5\). This is a factored observation POMDP (tests do not change fault states), mapping directly to industrial inspection, medical screening, and fault detection domains. We evaluate \(N{=}8\) (\(|S|{=}256\)) and \(N{=}16\) (\(|S|{=}65{,}536\)).

Two additional environments, a two-state testbed (Appendix 10) and navigation (Appendix 16), delimit the EFE agent’s applicability on mild-penalty and small-state-space settings.

5 Results↩︎

5.1 Core environments↩︎

On Tiger (single observation action), all multi-step agents achieve roughly \(99.5\%\) success, and EFE matches tuned alternatives without weight selection. The Epistemic-only ablation commits at chance (\(50.1\%\) on Tiger, \(25.1\%\) on Bandit, \(0.0\%\) on Tileworld \(6{\times}6\), Appendix 9). Pure information gain without reward alignment produces catastrophic exploration, so the pragmatic term is essential.

On the multi-observation-action environments, EFE Pareto-dominates Planning. It achieves substantially higher success and comparable or better reward at the same time, without any tuning. On Diagnosis, EFE outperforms same-horizon Planning in success rate (\(97.1\%\) vs.\(89.2\%\), \(+7.9\) pp) while achieving substantially better reward (\(-1.50\) vs.\(-2.37\)). On Bandit, EFE achieves both higher success (\(87.3\%\) vs.\(69.6\%\), \(+17.7\) pp) and the highest reward (\(+6.27\) vs.\(+5.65\)). Bootstrap 95% CIs (10,000 resamples over 5,000 episodes) confirm non-overlapping reward intervals: on Bandit, EFE \(+6.38\) \([+6.29, +6.47]\) vs.Planning \(+5.65\) \([+5.54, +5.76]\), and on Diagnosis, EFE \(-1.58\) \([-1.89, -1.29]\) vs.Planning \(-2.72\) \([-3.26, -2.19]\). Planning+IG at tuned weights (\(w{=}100\)) reaches near-ceiling success (\(99.3\%\) on Diagnosis, \(99.8\%\) on Bandit) but at substantial reward cost due to over-exploration: 13.21 tests on Diagnosis (\(-3.63\) reward) and 12.41 inspections on Bandit (\(+3.78\) reward). Viewed in the success–reward plane (Figure 1), the EFE agent sits at the Pareto knee on every environment, while Planning sacrifices success rate and Planning+IG sacrifices reward. The pattern is consistent: EFE’s joint pragmatic–epistemic objective knows when to stop exploring, while additive IG bonuses with tuned weights do not.

Effect sizes clarify where the differences lie. Cohen’s \(d\) on reward between EFE and same-horizon Planning is negligible on Tiger, Diagnosis, and Bandit (\(|d| < 0.2\), Table ¿tbl:tab:effect95sizes?), because both agents already achieve reasonable returns once they explore enough. The key gap is whether they explore the right observations. On success rate, where that gap appears, \(d\) is substantially larger: \(d \approx 0.33\) on Diagnosis and \(d \approx 0.47\) on Bandit (Appendix 22), in the small-to-medium range. This two-axis pattern is exactly the Pareto story: EFE moves the under-served objective (success) without sacrificing reward. Medium-to-large reward \(d\) values (\(d > 0.7\)) emerge against over-exploring baselines and on environments where Planning fails to explore sufficiently (Tileworld \(d > 2.0\)).

5.2 Pareto analysis: the canonical \(w{=}1\)↩︎

By Proposition [prop:equivalence], EFE is exactly Planning+IG with \(w{=}1\). We sweep \(w\) from \(0.01\) to \(200\) on all environments (Figure 1).

On every environment, \(w{=}1\) sits near the reward-maximizing weight \(w^*_{\text{ret}}\), while the success-maximizing weight \(w^*_{\text{succ}}\) lies at \(20\)–\(200\). We emphasize that \(w{=}1\) is near-optimal for reward, not for success rate: agents requiring near-certain accuracy (e.g., safety-critical applications) would benefit from higher weights at the cost of reduced reward. The contribution of EFE is deriving a principled weight from the variational bound that is near-optimal for reward maximization, rather than a grid search whose optimum changes by orders of magnitude across tasks.

5.3 Tileworld: spatial epistemic foraging↩︎

To test whether EFE’s advantage extends to larger state spaces, we introduce a spatial generalization: the Tileworld projects the Diagnosis partition structure onto an \(N{\times}N\) grid (\(|S|{=}N^2\)), producing spatially interpretable belief evolution (Figure 2, with step-by-step belief strips in Appendix 13).

EFE achieves the highest reward (\(-21.13\)), scanning 14.81 times, less than half of Planning+IG’s 33.38 scans, which erode reward despite reaching 98.4% success. The same over-exploration pattern from Diagnosis and Bandit recurs at this larger scale. Figure 2 visualizes the mechanism: EFE concentrates belief efficiently via partition-based narrowing and commits once confident, while Planning lacks scan-selection guidance and Info Gain continues scanning past the point of diminishing returns.

5.3.0.1 Spatial scaling.

We scale the grid from \(4{\times}4\) to \(8{\times}8\) with all agents (Figure 3). At \(8{\times}8\) (\(|S|{=}64\)), reward-only Planning collapses to \(2.5\%\) success while EFE maintains \(66.5\%\). Planning+IG at \(w{=}100\) achieves \(98.0\%\) success but at substantial reward cost due to extensive scanning. This reveals a nuanced picture: EFE (\(w{=}1\)) achieves the best reward at every scale and matches or exceeds Planning in success, but tuned Planning+IG (\(w{=}100\)) achieves higher success rates at larger grids by exploring more. The reward-maximizing weight \(w^*_{\text{ret}}\) does not shift with \(|S|\) (EFE’s reward remains highest), but its gap to the success-maximizing weight \(w^*_{\text{succ}}\) widens. Safety-critical applications at large scale would therefore benefit from higher weights.

5.3.0.2 Observation structure sensitivity.

A natural concern is whether EFE’s advantage on Tileworld depends on the highly structured bit-level partition of scan actions. We test this by replacing the deterministic row/column splits with two alternative observation structures: random partitions (each scan randomly assigns cells to two groups, breaking orthogonality) and overlapping partitions (scans use random linear combinations of coordinates, producing correlated and partially redundant observations). On \(6{\times}6\) Tileworld (\(H{=}2\), 200 episodes \(\times\) 5 seeds), EFE achieves the best reward under all three modes: bitwise \(-20.52\) (74.2% success), random \(-29.15\) (56.1%), overlapping \(-46.73\) (9.8%). Planning follows the same pattern: \(-21.44\) (73.7%), \(-30.41\) (54.5%), \(-47.61\) (8.5%). EFE’s reward advantage over Planning is consistent across modes (\(+0.9\), \(+1.3\), \(+0.9\)), confirming that the result is not an artifact of the structured observation model. As expected, random and overlapping modes reduce absolute performance because scans provide less complementary information, but the relative ranking of agents is preserved.

5.4 Interleaved observe-act: RockSample↩︎

To validate Proposition [prop:factored] beyond observe-then-commit settings, we evaluate on RockSample [5], a standard POMDP benchmark with interleaved observe-act dynamics. An agent navigates an \(N{\times}N\) grid containing \(K\) rocks at known positions, each with hidden binary quality. Actions include move (N/S/E/W, cost \(-0.5\)), check rock \(k\) (noisy observation, accuracy decays with distance), sample (collect rock at current position: good \(+10\), bad \(-10\)), and exit (\(+10\)). All agents use depth-limited belief-space tree search over factored beliefs (independent per-rock), differing only in the information gain weight \(w\).

On RS[5,3] and RS[7,4], EFE (\(w{=}1\)) achieves the highest reward. On RS[7,8], EFE (\(w{=}1\)) scores \(+19.57\) vs.Planning’s \(+11.75\) (\(+7.82\) gap), showing that the advantage grows with the number of observation actions, consistent with the observe-then-commit findings. On all instances up to RS[7,8], EFE samples dramatically fewer bad rocks than Greedy (0.00–0.05 vs.–5.67), confirming that the information gain term drives checking behavior.

On RS[11,11] (\(|S|{=}2{,}048\)), EFE and Planning achieve identical reward (\(+13.64\)), with all information-aware agents vastly outperforming Greedy (\(+13.6\) vs.\(-21.9\)). This result demonstrates tractability, since the factored belief tree search handles 2,048 states in seconds, but it does not differentiate EFE from reward-only planning. At depth 2, the tree search horizon is too shallow relative to the 11-rock environment: all informed agents converge to a conservative “check nearest rock, sample if good, exit” strategy. Depth 3 is computationally intractable at this scale, requiring orders of magnitude more search time per step. This mirrors the Tileworld scaling finding (Figure 3), where EFE’s advantage requires sufficient depth relative to the state space. The RS[7,8] result, where EFE achieves a \(+7.82\) reward gap over Planning, shows that EFE differentiation emerges when the observation action count (\(K{=}8\)) provides sufficient room for directed information gathering within the search horizon. Detailed results including POMCP baselines are in Appendix 26.

5.5 Structural inspection↩︎

To validate Proposition [prop:factored] on a domain-realistic benchmark and demonstrate scalability beyond existing environments, we implement a Structural Inspection POMDP mapping directly to industrial fault detection, medical screening, and non-destructive testing domains. The agent inspects \(N\) components arranged spatially, each with a hidden binary state (nominal/faulty). Two non-destructive test types provide accuracy–cost trade-offs: a visual check (accuracy 0.70, cost 0.5) and a detailed test (accuracy 0.90, cost 2.0). The agent navigates between components, selects tests, and declares diagnoses with asymmetric penalties (missed fault \(-50\), false alarm \(-5\)). Tests do not alter component states, so this is a factored observation POMDP and Proposition [prop:factored] applies.

EFE (\(w{=}1\)) achieves the best reward–accuracy trade-off on both instances. On \(N{=}8\) (\(|S|{=}256\)), EFE achieves \(87.9\%\) accuracy with reward \(-20.60 \pm 0.34\), gaining \(+14.9\)pp accuracy over Planning (\(73.0\%\), \(-17.85\)) at a moderate reward cost. Planning+IG (\(w{=}5\)) achieves \(94.9\%\) accuracy but at \(-22.82\) reward: it over-tests, spending resources on visual checks where a single detailed test suffices. On \(N{=}16\) (\(|S|{=}65{,}536\)), EFE outperforms Planning by \(+7.9\)pp accuracy (\(86.1\%\) vs.\(78.2\%\)) with comparable reward (\(-45.71 \pm 0.94\) vs.\(-46.09 \pm 0.94\), \(p > 0.05\)). This is the largest state space in our evaluation and confirms that the factored belief tree search scales to realistic domains. The asymmetric penalty structure (\(\alpha = 25\)) places this firmly in the regime where Proposition [prop:nearopt] predicts \(w{=}1\) is near-optimal.

5.6 Summary↩︎

The results reveal a consistent pattern across observe-then-commit, interleaved, and domain-realistic settings. EFE matches Planning+IG at \(w{=}1\), confirming Propositions [prop:equivalence] and [prop:factored]. The canonical weight sits at the Pareto knee without per-environment search. On Diagnosis and Bandit, EFE Pareto-dominates same-horizon planning with higher success and better reward. On Tileworld, RockSample, and Inspection, it matches or improves upon Planning in reward at comparable or higher accuracy. The advantage grows with state space size and observation action count, scaling to \(|S|{=}65{,}536\) on Inspection (\(N{=}16\)), and it carries over to interleaved observe-act POMDPs with state transitions (RockSample, Inspection).

6 Discussion↩︎

6.0.0.1 When does EFE-as-\(\rho\) help?

EFE’s advantage requires two conditions: (1) multiple observation actions with differential informativeness, and (2) sufficient planning horizon for recursive EFE to propagate epistemic value. When condition (1) fails, as on Tiger with its single listen action, reward-only planning matches EFE. The Tileworld adds a third axis: as the state space grows, the reward signal becomes too diffuse to guide scan selection, and EFE’s advantage increases (Figure 3). Within-episode dynamics (Appendix 19) show the mechanism: EFE concentrates belief via partition-based narrowing and commits once the value of committing exceeds the value of observing, a crossover that emerges automatically from \(w{=}1\) without tuning (Figure 7, Appendix 14).

6.0.0.2 The canonical weight and why it works.

EFE does not eliminate the exploration–exploitation weight. It derives one from the variational bound. The Pareto analysis (Figure 1) shows this weight sits near the reward-maximizing \(w^*_{\text{ret}}\) on every environment, while \(w^*_{\text{succ}}\) lies at \(20\)–\(200\). The gap between \(w^*_{\text{ret}}\) and \(w^*_{\text{succ}}\) has practical implications: \(w{=}1\) optimizes expected reward, not success probability. In safety-critical settings where near-certain accuracy is required, a higher weight may be appropriate, though such weights must still be tuned per-environment, and EFE provides a principled starting point. Increasing \(w\) beyond 1 buys marginal accuracy at substantial reward cost. Moreover, adding planning depth to a weighted IG bonus amplifies the weight’s effect (\(w{=}100\) on Bandit: Planning+IG takes 12.41 inspections vs.myopic Info Gain’s 10.99), making the tuning problem harder with depth. EFE avoids this because both terms arise from the same variational bound and share a common scale in nats. The canonical weight is \(w{=}1\) precisely because information gain is measured in the same units (nats) as the KL divergence in the variational bound. This is a consequence of the mathematical structure rather than an empirical coincidence. In bits (base 2), the corresponding weight would be \(w = 1/\ln 2 \approx 1.44\). The Pareto analysis shows that the reward-optimal weight is insensitive to this factor: the knee is broad, and \(w \in [0.5, 2.0]\) yields near-identical reward on all environments, making the nat/bit distinction practically irrelevant.

The canonical weight’s advantage requires \(\gamma \geq 0.99\) on multi-observation environments. Our discount-sensitivity analysis (Appendix 24) reveals a sharp transition: on Diagnosis (\(N{=}4\)), EFE’s success advantage over Planning is \(+7.8\)pp at \(\gamma{=}1.0\) but drops to \(+1.0\)pp at \(\gamma{=}0.95\) and reverses at \(\gamma{=}0.90\) (\(-1.6\)pp). The mechanism is that discounting truncates the effective planning horizon below the number of observations needed for confident diagnosis, erasing EFE’s advantage. On Tiger (single observation), EFE is insensitive to \(\gamma\) across the entire range \([0.90, 1.0]\). This is a meaningful practical limitation: applications with high time pressure or non-stationary environments that mandate heavy discounting will not benefit from EFE’s epistemic drive.

6.0.0.3 Zero-shot weight transfer.

The tuning problem is not merely inconvenient. It transfers catastrophically. We evaluate each environment’s success-maximizing weight \(w^*_{\text{succ}}\) on all other environments (Table ¿tbl:tab:transfer?). On Tiger, transferring \(w^*{=}100\) from Diagnosis drops reward from \(+5.02\) (EFE) to \(+4.13\), as the agent over-explores. On the Testbed, transferring \(w^*{=}20\) from Tiger drops reward from \(+0.39\) (EFE) to \(-0.16\). On Tiger alone, the native \(w^*{=}20\) achieves \(+5.42\) (slightly above EFE’s \(+5.02\)), but this weight transfers poorly to all other environments. Across all 16 environment–weight pairs, EFE (\(w{=}1\)) achieves the best or near-best reward on every environment without tuning, while every transferred weight underperforms EFE on at least one target. The success-maximizing weight varies by \(5\times\) across environments (\(w^*{=}20\) for Tiger vs.\(w^*{=}100\) for Bandit), making zero-shot deployment with a tuned weight unreliable. EFE sidesteps this entirely: its weight is derived, not tuned, and transfers robustly.

6.0.0.4 When is \(w{=}1\) near-optimal?

Proposition [prop:nearopt] formalizes the conditions for \(H{=}1\), two states: the observation threshold \(w^*_{\text{thresh}}\) depends on the reward asymmetry ratio \(\alpha = |R^-|/R^+\), the observation informativeness \(\eta = I_{\max}/c\), and the observation accuracy \(p\). Table ¿tbl:tab:alpha95eta? validates this across our environments. The key predictor is \(\alpha\): environments with high penalty asymmetry (\(\alpha \geq 5\): Tiger, Diagnosis, Tileworld) have \(w^*_{\text{thresh}} \ll 0\), meaning \(w{=}1\) is far above the threshold and near-optimal.

To assess whether the \(H{=}1\) analysis extends to multi-step planning, we conducted a Monte Carlo study over 100 randomly generated two-state environments (sampling \(\alpha \in [1, 50]\), \(p \in [0.55, 0.95]\), cost \(\in [0.1, 5]\)). For each environment and \(H \in \{1, 2, 3\}\), we compute the reward-optimal \(w^*\) by grid search and classify \(w{=}1\) as near-optimal when the reward gap from \(w^*\) is within \(\max(5\%, 0.5)\) of the best reward (Figure 12). The near-optimality rate increases from 9% at \(H{=}1\) to 22% at \(H{=}2\) and 32% at \(H{=}3\), confirming that multi-step planning amplifies the value of observation and expands the region where \(w{=}1\) is near-optimal. For high-asymmetry environments (\(\alpha \geq 10\)), the improvement is sharper: 11% \(\to\) 24% \(\to\) 32%. This is consistent with the Bandit result, where \(\alpha{=}1.1\) and the \(H{=}1\) analysis predicts marginal sufficiency, yet multi-step EFE at \(H{=}2\) achieves \(87.3\%\) success (Table ¿tbl:tab:main?).

Many real-world decision problems have \(\alpha \gg 1\): in medical diagnosis, fault detection, and security screening, a missed condition costs far more than another test. This is exactly the regime where \(w{=}1\) is well calibrated, and the multi-step analysis shows the advantage grows with planning depth. The practical consequence is that in these domains the exploration weight can be deployed as-is, with no per-task search.

6.0.0.5 Robustness to model misspecification.

Our main results assume exact knowledge of the generative model. Appendix 25 investigates robustness when the agent’s believed observation accuracy differs from the true value by up to \(\pm 0.15\). On Tiger, success rates remain above 96.7% across all mismatch levels. On Diagnosis, degradation is larger but graceful. Overestimating sensor accuracy (positive mismatch) is more harmful than underestimating it, because the agent commits prematurely on insufficient evidence. EFE’s intrinsic epistemic drive provides a partial buffer: even with a miscalibrated model, the information gain term still encourages observation, compensating for overconfident planning.

6.0.0.6 Approximate planning with MCTS-EFE.

The exact tree search cost \(\mathcal{O}(K \cdot |\mathcal{O}|^H)\) limits practical horizons to \(H{=}2\)–\(3\) with \(K \geq 3\). To push beyond this, we implement MCTS with EFE as a leaf heuristic: the tree policy uses UCB1 with observation-outcome enumeration, and leaf nodes are evaluated via greedy EFE rollouts. On Tiger at \(H{=}10\), MCTS-EFE with 500 simulations achieves 97.2% success (7.2s per 200 episodes), compared to 89.7% for POMCP at matched budget (1.7s). This demonstrates that EFE’s closed-form information valuation provides a substantially stronger leaf heuristic than semi-informed rollouts. The comparison is designed to be fair to POMCP: our implementation already uses belief-optimal commits (not purely random rollouts): observations are sampled uniformly, but the rollout terminates with the commit action maximizing expected reward under the Bayesian-updated belief (Appendix 27). The remaining performance gap therefore reflects the value of directed observation selection: EFE chooses which test to run based on information gain, while POMCP samples tests uniformly. The claim is not that EFE-based planning exceeds state-of-the-art POMDP solvers, but that the EFE leaf heuristic provides a principled, zero-tuning alternative to heuristic rollout design. POMCP with fully informed rollouts (e.g., using domain knowledge or a learned value function for observation selection) would narrow the gap further.

The approach also scales to multiple observation actions. For single-observation environments, the observation-outcome enumeration (2 outcomes per action) keeps expansion costs low. On Diagnosis (\(N{=}4\), \(K{=}2\) tests), MCTS-EFE at \(H{=}5\) achieves \(98.0\%\) success compared to POMCP’s \(71.3\%\) at matched budget (200 simulations), demonstrating that EFE scales to multi-observation environments. To validate on a larger multi-observation environment, we evaluate on Tileworld \(6{\times}6\) (\(|S|{=}36\), \(K{=}6\) scans). MCTS-EFE(50) achieves \(96.0\%\) success and \(-19.04\) reward, outperforming both Exact-EFE (75.0%, \(-20.11\)) and POMCP at matched (50) or higher (200) simulation budgets (\(2.0\%\)/\(15.0\%\) success). The improvement over exact tree search reflects MCTS’s ability to concentrate samples on promising observation sequences. The \(81\)–\(94\)pp gap over POMCP confirms that EFE’s directed observation selection is essential at this scale: POMCP’s semi-informed rollouts cannot identify which of the 6 scans to perform. For larger observation spaces, more efficient tree policies, such as progressive widening or double progressive widening [14], would further improve scalability. We leave this to future work.

6.0.0.7 When to use EFE-as-\(\rho\): summary.

Our analysis identifies the following conditions favoring EFE over tuned alternatives:

• Reward asymmetry \(\alpha \geq 5\): the penalty for a wrong commit far exceeds the observation cost, making \(w{=}1\) automatically near-optimal (Proposition [prop:nearopt]).

• Multiple observation actions: EFE’s joint pragmatic–epistemic objective selects which information to gather, producing gains over reward-only planning even at matched horizon.

• Moderate to large state spaces (\(|S| \geq 16\)): as the reward signal diffuses, directed information gathering becomes essential, and EFE’s advantage grows with \(|S|\) (Figure 3), scaling to \(|S|{=}65{,}536\) on Structural Inspection.

• Interleaved observe-act with preserved hidden state: when the hidden state does not change under observation and navigation actions (factored observation POMDPs, Table ¿tbl:tab:taxonomy?), the canonical-weight equivalence holds and EFE directs both where to go and what to check (Tables ¿tbl:tab:rocksample?, ¿tbl:tab:inspection?).

• Cross-environment deployment: when the weight cannot be tuned per-task, \(w{=}1\) transfers robustly while tuned weights fail catastrophically (Table ¿tbl:tab:transfer?).

• Planning horizon \(H \geq 2\): recursive EFE propagates epistemic value across steps. At \(H{=}1\), it reduces to myopic IG with \(w{=}1\).

EFE is not recommended when \(\alpha \approx 1\) (symmetric penalties), in navigation-style POMDPs where observations are tied to translation and a greedy mover already receives informative feedback (Table ¿tbl:tab:nav95scaling?), or when model misspecification exceeds \(\pm 0.15\) in observation accuracy.

6.0.0.8 Limitations and future work.

The formal equivalence extends from observe-then-commit (Proposition [prop:equivalence]) to factored observation POMDPs (Proposition [prop:factored]), covering settings where observation actions preserve the hidden state. POMDPs where information-gathering actions change the hidden state (e.g., destructive testing) require the full coupling term \(\Delta_T\) and are not covered. Extending the formal treatment to such settings remains future work. For slowly drifting hidden states (e.g., progressive disease), \(\Delta_T\) is small but nonzero. The approximation error scales with the per-step transition entropy \(H(s'_{\mathrm{hid}} | s_{\mathrm{hid}}, a)\), and characterizing the regime where this remains acceptable is an open question.

On the practical side, scaling MCTS-EFE to multi-observation environments requires more efficient tree policies, as discussed above. The EFE agent assumes a known generative model. The misspecification analysis (Appendix 25) shows graceful degradation with moderate model errors, but extending to full model learning (where AIF and BAMDPs converge) remains important future work. Navigation (Appendix 16) shows that scale alone does not rescue epistemic planning when the observation model is proximity-based: NavMyopic leads at \(3{\times}3\), \(5{\times}5\), and \(7{\times}7\). The practical contrast is with domains that offer explicit, choice-set observation actions (Diagnosis, Tileworld, RockSample), where \(|S| \geq 16\) marks the regime in which EFE separates from reward-only planning.

6.0.0.9 Broader impact.

This work is primarily theoretical. The principled derivation of exploration weights could benefit safety-critical decision-making by reducing reliance on ad-hoc tuning.

7 Conclusion↩︎

The \(\rho\)-POMDP framework and active inference converge on the same mathematical object: a belief-dependent utility that adds information gain at weight \(w{=}1\) to the reward signal. EFE does not eliminate the exploration–exploitation trade-off. Instead, it replaces ad hoc bonus weights with a coefficient fixed by the variational geometry (nats), which then doubles as the Planning+IG weight in Proposition [prop:equivalence]. The Pareto analysis shows this canonical weight is near-optimal for expected reward across all tested test-selection environments, sitting at the knee where further increases buy marginal accuracy at substantial reward cost. Safety-critical applications requiring near-certain accuracy may benefit from higher weights. A Monte Carlo study over randomly generated environments (Appendix 23) confirms that the near-optimality basin widens with planning horizon, extending the \(H{=}1\) analysis of Proposition [prop:nearopt] to multi-step settings.

Proposition [prop:factored] extends the formal equivalence beyond observe-then-commit to factored observation POMDPs, validated on RockSample (up to \(|S|{=}2{,}048\)) and a new Structural Inspection benchmark (\(|S|\) up to \(65{,}536\)) mapping directly to industrial fault detection and medical screening domains. On Inspection (\(N{=}16\)), EFE achieves \(86.1\%\) diagnostic accuracy where planning scores \(78.2\%\), at comparable reward (\(p > 0.05\)). Comparison against POMCP with semi-informed rollouts (random observations, belief-optimal commits, Appendix 27) shows that EFE’s advantage comes from directed observation selection, not from using a stronger solver. Domain-informed POMCP rollouts would narrow this gap, but they require per-environment engineering that EFE avoids. MCTS-EFE on Tiger achieves 97.2% success at \(H{=}10\) with 500 simulations, outperforming POMCP (89.7%) at matched compute. The advantage is most pronounced in multi-observation-action settings: on the \(8{\times}8\) Tileworld, EFE achieves \(66.5\%\) success where reward-only planning collapses to \(2.5\%\). The message for practitioners is simple: wherever an agent must pay for its observations and answer for its mistakes, the exploration weight it needs is not a hyperparameter to search over, because active inference already supplies it.

7.0.1 ↩︎

The authors have no competing interests to declare that are relevant to the content of this article.

8 Proof of Proposition [prop:equivalence]↩︎

9 Full per-environment results↩︎

10 Two-state testbed↩︎

Under mild reward asymmetry (\(+1 / {-}1\)), EFE gathers more information than is instrumentally optimal (\(+0.39\) vs.Planning’s \(+0.49\), \(p < 0.001\)) because uncertainty reduction has intrinsic value under EFE. Tuned Info Gain (\(w{=}50\)) over-explores catastrophically: 99.9% success but negative reward. This result is informative about the conditions under which \(w{=}1\) is near-optimal: the reward asymmetry ratio \(|R^-|/|R^+| = 1\) means that the penalty for guessing wrong barely exceeds the observation cost, so the reward-optimal weight \(w^*_{\text{ret}} < 1\). In this regime, EFE assigns too much weight to information gain. Many real-world decision problems, including medical diagnosis, fault detection, and security screening, have asymmetric penalties (\(|R^-|/|R^+| \gg 1\)), precisely the regime where \(w{=}1\) is well-calibrated (Tiger, Diagnosis, Tileworld).

11 Tiger reward asymmetry sweep↩︎

To test whether EFE’s Tiger parity holds across reward scales, we sweep the penalty magnitude from \(|R^-|{=}1\) to \(500\). EFE naturally adapts its exploration depth, tracking the reward-optimal strategy without reconfiguration. Info Gain collapses at low penalties where its weight (\(w{=}20\), tuned at \(|R^-|{=}100\)) drives excessive listening.

12 Observation action scaling↩︎

Holding \(N{=}8\) fixed on Diagnosis and varying \(K\) from 1 to 3 isolates the “which information” hypothesis. At \(K{=}1\), all agents face the same single test and perform similarly. As \(K\) increases, EFE’s advantage over Planning grows because the information gain term guides test selection.

13 Tileworld belief evolution↩︎

14 EFE trajectory decomposition↩︎

15 Environment specifications↩︎

All environments are implemented as OpenAI Gymnasium environments with standard reset/step API. The Diagnosis environment uses \(K = \lceil \log_2 N \rceil\) binary tests, each providing noisy information about a different partition of the state space. The Navigation environment uses proximity-based warm/cold signals with accuracy depending on Manhattan distance to the hidden goal. The Tileworld projects the Diagnosis partition structure onto a 2D grid: \(K = 2 \lceil \log_2 N \rceil\) scans partition the grid by bit-level splits of the row and column indices, enabling complete spatial disambiguation.

16 Navigation results↩︎

Navigation uses only move actions. The goal cell is hidden, and each move yields a proximity-based warm/cold observation (Section 15). Unlike Diagnosis or Tileworld, there is no separate menu of observation actions: information arrives as a side effect of translation, and greedy progress toward the belief mode already provides correlated evidence.

NavMyopic (move toward the highest-probability cell) remains the strongest baseline at every scale: extra steps devoted to exploratory detours incur the per-move cost without the sharp information returns seen when tests can be chosen explicitly. This is not a failure of \(w{=}1\) so much as a structural regime change relative to our main benchmarks. Importantly, NavEFE is competitive with one-step NavInfoGain at \(7{\times}7\) (higher mean reward, comparable success), showing that recursive epistemic valuation is not harmful once the grid is large enough that myopic IG is already costly. The contrast with Tileworld (Figure 3) underscores condition (1) in Section 6: EFE’s clearest wins appear when the agent must select among differentially informative observation actions, not only where to translate next under a smooth proximity field.

17 Scaling analysis↩︎

At \(H{=}2\), EFE and Planning perform similarly across all \(N\), both substantially outperforming Myopic and untuned Info Gain. This contrasts with the \(H{=}3\) Diagnosis result (Table ¿tbl:tab:main?), where EFE outperformed Planning by \(+7.9\) pp, confirming that EFE’s advantage requires sufficient recursive depth. EFE-over-Myopic advantage grows from \(+13.8\) pp at \(N{=}2\) to \(+39.1\) pp at \(N{=}16\), confirming that deeper planning is increasingly valuable as state spaces grow.

18 PyMDP validation↩︎

We validated our EFE computation against the pymdp library [45] by wrapping pymdp’s standard Agent class. Both implementations agree on observe-vs-commit decisions across all tested belief states (uniform, intermediate, and confident beliefs on the two-state environments). Information gain values computed by our recursive scheme and pymdp’s control module agree qualitatively, with differences attributable to our recursive multi-step evaluation vs.pymdp’s single-step policy enumeration.

19 Epistemic foraging dynamics↩︎

The aggregate statistics in Table ¿tbl:tab:main? show what EFE achieves. We now visualize how it achieves it by examining within-episode dynamics on extended Diagnosis instances (\(N{=}8\), \(K{=}3\), accuracy 0.75) that produce episodes with 8–15+ observation steps.

19.0.0.1 Belief evolution.

Figure 8 shows belief-state evolution within representative episodes for EFE, Planning, and Info Gain. EFE exhibits systematic partition-based narrowing: early tests eliminate large groups of states, and later tests refine among remaining candidates. Planning (without epistemic value) makes observations but lacks guidance on which test to run, producing less structured belief concentration. Info Gain continues testing well past the point where the agent has high confidence.

19.0.0.2 Exploration efficiency.

Figure 9 quantifies the temporal dynamics of epistemic foraging averaged across 300 episodes. EFE reduces entropy at a rate comparable to Planning+IG but commits earlier, avoiding the diminishing-returns regime. The “survival curve” of exploration shows EFE with a concentrated drop-off around steps 8–12, while Planning+IG’s survival curve extends further right.

20 Extended EFE decomposition↩︎

Figure 10 extends the EFE trajectory analysis of Figure 7 to a multi-observation-action environment where the agent must choose among \(K{=}3\) diagnostic tests at each step. Panel (b) reveals a key dynamic: as the agent gathers information, the tests become differentially informative: some partitions of the state space become irrelevant once the agent has narrowed down the true condition, while others remain critical. The EFE agent tracks this structure, selecting the most informative available test at each step (panel d). This adaptive test selection is the mechanism underlying EFE’s advantage over reward-only planning in multi-observation-action environments.

21 Stopping time analysis↩︎

Figure 11 shows the full distribution of stopping times (the number of observation steps each agent takes before committing) across three environment configurations. The shape of these distributions reveals qualitative differences in decision strategy: EFE’s distributions are concentrated, indicating consistent and confident stopping behavior. Planning’s distributions skew shorter, reflecting its tendency to commit before full disambiguation (especially visible on the \(K{=}3\) Diagnosis environment). Planning+IG shows long right tails, confirming that the additive weight drives over-exploration with high variance. The EFE agent’s concentrated stopping times are a consequence of the canonical \(w{=}1\): the crossover between observation and commit values occurs at a consistent belief-confidence level across episodes.

22 Supplementary statistics↩︎

All pairwise comparisons use independent two-sample \(t\)-tests with Holm–Bonferroni correction for family-wise error control. Bootstrap 95% confidence intervals use 10,000 resamples with fixed seed for reproducibility. Effect sizes use Cohen’s \(d\) with pooled standard deviation.

The reward table shows that EFE vs.Planning is negligible on Tiger, Diagnosis, and Bandit: once both agents take enough informative actions, mean returns are similar, even though success rates differ. The success-rate rows isolate the axis where EFE improves most relative to Planning on Diagnosis and Bandit. Against over-exploring tuned alternatives (InfoGain-Tuned, Planning+IG), EFE shows large reward effects on the Testbed (\(d > 1.3\)) and medium effects on Bandit (\(d \approx 0.7\)–\(0.8\)), reflecting the cost of over-exploration.

23 Near-optimality across planning horizons↩︎

24 Discount factor sensitivity↩︎

We test EFE and Planning agents’ sensitivity to discounting by sweeping \(\gamma \in \{0.9, 0.95, 0.99, 1.0\}\) across three environments. Including Planning at each \(\gamma\) enables comparison of the relative gap between agents under discounting.

24.0.0.1 Discounting erases EFE’s advantage on multi-observation environments.

On Tiger (single observation action), both agents achieve \({\geq}99\%\) success across all \(\gamma\) values, with negligible gaps. On Diagnosis and Bandit, heavy discounting (\(\gamma{=}0.9\)–\(0.95\)) renders both agents effectively myopic, erasing EFE’s advantage: at \(\gamma{=}0.9\), EFE’s epistemic term cannot propagate value through future observations because discounting truncates the effective horizon below the number of observations needed for disambiguation. Both agents observe \({\sim}2\) times on Bandit and \({\sim}6\) times on Diagnosis, behaving identically. The transition between \(\gamma{=}0.95\) and \(\gamma{=}0.99\) is sharp: at \(\gamma{=}0.99\), EFE’s success gap over Planning jumps to \(+7.0\) pp on Diagnosis and \(+22.0\) pp on Bandit, matching the undiscounted pattern. This confirms that EFE’s advantage depends on the discount factor preserving a sufficient effective horizon for recursive epistemic evaluation. In practical terms, \(\gamma \geq 0.99\) is needed for EFE to differentiate from reward-only planning on these environments.

25 Model misspecification sensitivity↩︎

Our main results assume the agent’s generative model matches the true environment dynamics. Here we investigate robustness when the agent’s believed observation accuracy \(p_{\text{agent}}\) differs from the true accuracy \(p_{\text{true}}\), creating a systematic model mismatch.

25.0.0.1 Key findings.

On Tiger, both EFE and Planning agents are remarkably robust: success rates remain above 96.7% even with \(\pm 0.15\) mismatch. The degradation pattern is asymmetric: underestimating accuracy (negative mismatch) causes over-observation but preserves safety, while overestimating accuracy (positive mismatch) reduces observation count and lowers success rates. EFE’s inherent epistemic drive provides a buffer against over-confident models: when the agent overestimates its sensor accuracy, the information gain term still encourages observation, partially compensating for the miscalibrated planning model.

On Diagnosis, degradation is graceful but measurable. Overestimating accuracy (the agent believes tests are more informative than they truly are) leads to premature commitment with insufficient evidence. Underestimating accuracy causes excessive testing, reducing reward through observation costs but not drastically harming success rates. The EFE agent degrades gracefully across the mismatch range, suggesting that the intrinsic epistemic value provides a degree of robustness to model misspecification that pure reward-maximizing agents lack.

25.0.0.2 Implications for learned models.

These results suggest that EFE-based agents can tolerate moderate model errors without catastrophic failure. In practice, observation models learned from data will have estimation error. The graceful degradation observed here indicates that approximate models suffice for effective EFE-driven information gathering, provided the model does not systematically overestimate sensor informativeness.

26 Interleaved observe-act: RockSample (extended)↩︎

This appendix provides extended RockSample results beyond the main text (Section 5.4), including POMCP baselines and per-instance detail. The environment and agents are described in Section 5.4.

The Greedy agent moves to rocks and samples them without checking quality first, incurring frequent bad-rock penalties (1.50 bad samples on RS[5,3], 2.00 on RS[7,4]). The EFE agent moves toward uncertain rocks, checks them to resolve quality, then samples only those confirmed good. The result is dramatically fewer bad samples (\(0.33\) and \(0.43\)) and substantially higher reward (\(+8.63\) vs.\(+4.03\) on RS[5,3] and \(+8.20\) vs.\(-0.58\) on RS[7,4]).

We additionally include Planning+IG (\(w{=}5\), \(w{=}10\)) baselines to address concerns about weak baselines. Planning+IG uses the same factored belief and decision structure as EFE but with an explicit, tunable weight. On RS[5,3], Planning+IG at \(w{=}5\) and \(w{=}10\) matches EFE (\(w{=}1\)) closely (\(+8.64\) vs.\(+8.63\) reward). On RS[7,4], EFE (\(w{=}1\)) substantially outperforms Planning+IG (\(+8.20\) vs.\(+5.57\)), suggesting that the canonical weight is better calibrated on the larger instance where information gathering requires more nuanced valuation.

These results demonstrate that the EFE principle of valuing information gain alongside reward extends to interleaved settings despite the transition–observation coupling analyzed in Section 3 (factored observation extension). The agent’s behavior mirrors the observe-then-commit pattern: check (gather information), then sample/exit (commit), even though checking and acting are interleaved with movement.

27 POMCP baseline comparison↩︎

We compare against POMCP [4], a standard online POMDP solver that handles exploration through Monte Carlo tree search with UCB1 action selection. This addresses whether EFE’s explicit information gain valuation provides benefit over the implicit exploration in POMCP’s tree search. Our implementation uses particle-based belief representation at each tree node, adapted to the observe-then-commit action structure. Crucially, our POMCP rollout policy is semi-informed: observation actions are selected uniformly at random, but the rollout terminates (stochastically, with 30% probability per step) with a belief-optimal commit, i.e., the commit action maximizing expected reward under the Bayesian-updated rollout belief. This is strictly stronger than a fully random rollout, which would also randomize over commit actions. The remaining gap between POMCP and EFE therefore reflects the value of observation selection: EFE chooses which test to run based on information gain, while POMCP samples tests uniformly. We sweep POMCP simulation budgets across \(\{500, 1000, 2000, 5000\}\) and report wall-clock timings for compute-matched analysis.

27.0.0.1 Key findings.

POMCP underperforms both EFE and Planning on all environments except Bandit success rate. On Tiger, POMCP commits prematurely (1.67 observations vs. for EFE), yielding 89.3% success. On Diagnosis, the gap is dramatic: POMCP achieves only 73.1% success (\(-10.04\) reward) while EFE reaches 96.6% (\(-1.80\) reward). POMCP’s random rollout policy cannot evaluate which of the \(K{=}2\) diagnostic tests to run. It treats all observation actions equally, whereas EFE’s information gain term directly values differential informativeness. On Bandit, POMCP’s Monte Carlo exploration achieves 96.8% success but at extreme reward cost (\(+2.51\) vs.EFE’s \(+6.42\)): 14.41 inspections vs.EFE’s 5.19. On Tileworld (\(6{\times}6\), \(|S|{=}36\)), POMCP collapses to 6.1% success, as the 36 commit actions and 6 scan actions create a branching factor that overwhelms 1,000 simulations.

27.0.0.2 Compute-matched analysis.

Increasing POMCP’s simulation budget to 5,000 does not qualitatively change the picture: on Diagnosis, POMCP(5000) reaches 71.5% success, still far short of EFE’s 96.6%, while taking substantially more wall-clock time per decision (299.5 ms/ep vs.EFE’s closed-form computation). On Tiger, POMCP(5000) reaches 89.0% but at 5\(\times\) the compute. These results demonstrate that EFE’s advantage is not merely a compute-budget artifact but reflects the fundamental benefit of closed-form information gain valuation over Monte Carlo exploration, particularly on multi-observation-action environments where the agent must choose which information to gather.

27.0.0.3 MCTS-EFE.

To disentangle EFE’s information valuation from the planning mechanism, we implement MCTS-EFE: MCTS with EFE as the leaf heuristic rather than random rollouts. On Tiger at \(H{=}10\), MCTS-EFE with 500 simulations achieves 97.2% success (7.2s per 200 episodes), compared to POMCP’s 89.7% at matched budget (1.7s) and exact EFE’s 99.5% at \(H{=}6\) (11.8s). This confirms that EFE provides a superior leaf evaluation function: the same MCTS framework with EFE rollouts substantially outperforms random rollouts.

References↩︎

---

1. Code and experiment data are available at https://github.com/PatrickAllenCooper/rho_aif.↩︎