1 Introduction↩︎

Intrinsic motivation based on prediction or compression progress appears in Schmidhuber’s work on artificial curiosity and the later compression-progress theory of interestingness [1]–[3]. Reward is paid only when the agent’s model improves. This distinguishes learnable regularity from incompressible noise and should avoid noisy-TV pathologies that trap raw prediction-error bonuses.

But the informal statement is too broad. A learning agent can improve on its own recently selected stream while becoming worse on the target distribution. It can forget and relearn the same facts if reward clips away negative progress. It can overfit a finite validation set if repeated scalar feedback leaks information. It can exploit a high-capacity model class until a nominal holdout is no longer a holdout. These are the Goodhart channels that matter for intrinsic rewards in continual learning and recursive self-improvement.

We isolate the representation under which the compression-progress claim becomes true. Let \(\mathsf Q\) be a fixed audit distribution and let \(\mathcal{E}(\theta)=\mathbb{E}_{z\sim \mathsf Q}\ell(\theta,z)\) be audit log-loss or any lower-bounded proper scoring loss. Define signed audit compression progress by \[r_t^{\rm audit}=\mathcal{E}(\theta_{t-1})-\mathcal{E}(\theta_t).\] Then the entire reward history is an endpoint identity: \[\sum_{t=1}^T r_t^{\rm audit}=\mathcal{E}(\theta_0)-\mathcal{E}(\theta_T).\] Thus any apparent long-run reward must be paid for by a genuine reduction in audit loss. Goodhart resistance here is a property of the measurement frame: it holds because progress is scored against a fixed audit loss.

1.0.0.1 Contributions.

We make four contributions. First, we define budgeted Goodhart resistance: a progress signal is credible up to a finite false-positive budget \(\Gamma\) if cumulative reward cannot exceed true audit improvement by more than \(\Gamma\). Exact sealed-audit compression progress has \(\Gamma=0\); finite panels have \(\Gamma=2\Delta_n\). Second, we mechanize the structural core in Lean 4: exact telescoping, finite-audit Goodhart resistance under a uniform-deviation event, finite Gibbs nonnegativity, and an entropy-floor theorem for incompressible components. Third, we separate the reward signal from the scheduler: audit compression progress supplies the credible reward, while multiplicative weights / EXP3 provides allocation. Fourth, we run a focused experiment suite using ARC-TGI task generators, RND [4], ICM [5], prediction-error curiosity, finite-audit concentration checks, stream leakage, clipping cycles, reusable-panel memorization, and black-box scalar-feedback holdout attacks.

2 Related work↩︎

2.0.0.1 Goodhart’s law and reward hacking.

Optimizing a proxy until it diverges from the target it stands for is Goodhart’s law [6], and its learned-agent form is reward hacking or specification gaming [7]–[10]. Most of this literature characterizes when a fixed proxy is unsafe to optimize. We hold the measurement frame fixed and ask a quantitative question: when can a progress signal exceed true improvement, and by how much? Budgeted Goodhart resistance answers this for compression-progress rewards with a finite false-positive budget.

2.0.0.2 Intrinsic motivation and compression progress.

Compression progress as a driver of curiosity and creativity originates with Schmidhuber [1]–[3]. Prediction-error and feature-prediction bonuses [5] and random network distillation [4] are the standard deep reinforcement learning realizations. These bonuses score error or novelty on the agent’s own stream. Audit compression progress scores signed error reduction on a sealed distribution, which is what produces the endpoint accounting identity and the entropy floor.

2.0.0.3 Adaptive data analysis and holdout reuse.

Repeated queries against a finite validation set erode its guarantees; the reusable holdout and the Ladder mechanism bound this erosion [11], [12], and empirical studies measure it on real leaderboards and benchmarks [13], [14]. The finite-panel budget \(2\Delta_n\) is the audit-CP form of the same phenomenon, and our adaptive scalar-feedback attack instantiates it together with the standard release defenses.

2.0.0.4 Proper scoring rules.

Log-loss is a strictly proper scoring rule [15], so its population minimizer is the true conditional distribution. The entropy floor (Theorem 3) is the statement that this minimum equals the conditional entropy, which is why a purely random component carries only a finite compression-progress budget. The calibration probe separates this proper-scoring signal from hard accuracy.

3 Setup: audit compression progress↩︎

Let \(\Theta\) be a class of model states and let \(\ell:\Theta\times\mathcal{Z}\to\mathbb{R}\) be a bounded or lower-bounded predictive loss. In the log-loss case, \(\ell(\theta,(x,y))=-\log p_\theta(y\mid x)\). For experiments we use probability-floored cross-entropy, \[\ell_\varepsilon(\theta,(x,y))=-\log \max\{\varepsilon,p_\theta(y\mid x)\},\] which is bounded by \(R=-\log\varepsilon\).

This condition is stronger than correlation with learning: it caps the apparent reward obtainable without true audit improvement.

4 Theorems↩︎

4.1 Exact sealed audits: zero false-positive budget↩︎

Each hypothesis of Theorem 1 is necessary: a fixed audit loss provides a single potential to telescope, signed accounting lets negative terms cancel, and a lower bound yields a finite budget.

4.2 Finite audits: the \(2\Delta_n\) false-positive budget↩︎

A reusable finite audit panel is not automatically sealed in the population sense; the relevant condition is a uniform-deviation event.

There is no union bound over \(T\): after signed telescoping, the adaptive history reduces to endpoint control. The cost of adaptivity appears in proving that the panel realizes a uniform-deviation event for the reachable class; the theorem itself is horizon-free.

For infinite classes, replace \(N\) by the appropriate covering number, Rademacher complexity, or PAC-Bayesian radius. In particular, bounded linear balls and bounded RKHS balls produce the same form: a finite false-positive budget whenever the effective audit capacity is finite. General neural networks enter the theory only through their effective class size or stability. If the reachable class is large enough to memorize the reusable audit panel, the bound becomes vacuous; this delimits where finite-audit resistance ceases to hold.

4.3 Entropy floor: why noisy TV cannot pay forever↩︎

Prediction-error bonuses pay for error itself and are therefore attracted to irreducible noise. Compression progress pays for error reduction. For log-loss, the distinction is formalized by the following entropy floor.

5 Where the theorem breaks↩︎

Each assumption above has a failure construction and a corresponding experiment.

5.0.0.1 Clipping destroys accounting.

If reward is \(r_t^+=\max\{0,\mathcal{E}(\theta_{t-1})-\mathcal{E}(\theta_t)\}\), then a two-state cycle \(a,b,a,b,\ldots\) with \(\mathcal{E}(a)<\mathcal{E}(b)\) accumulates positive reward on every \(b\to a\) transition while returning to the same endpoint every two steps. Thus \(\sum_t r_t^+\) can grow linearly at zero net improvement. Signed progress cancels exactly.

5.0.0.2 Stream scoring destroys the fixed potential.

If the agent is rewarded by \(\mathcal{E}_{\mathsf P_t}(\theta_{t-1})-\mathcal{E}_{\mathsf P_t}(\theta_t)\) on its own selected stream \(\mathsf P_t\), then there is no fixed \(\mathcal{E}\) to telescope. A policy can select or distort streams where local loss decreases while sealed-audit loss does not.

5.0.0.3 High-capacity reusable panels destroy uniform deviation.

If \(\mathcal{F}\) can interpolate the finite audit panel, a policy can drive \(\widehat{\mathcal{E}}_n\) down while \(\mathcal{E}\) stagnates or rises. This is ordinary adaptive holdout overfitting [11], [12], now expressed as false audit-CP. The \(2\Delta_n\) theorem remains correct, but \(\Delta_n\) is no longer small.

6 Algorithmic separation: reward signal versus scheduler↩︎

We distinguish the reward signal from the scheduler.

6.0.0.1 Audit-CP is the reward/accounting signal.

For each candidate training source \(j\), temporarily train or estimate the update effect, evaluate the sealed-audit loss before and after, and release the signed reward \[r_{t,j}=\widehat{\mathcal{E}}_n(\theta_t)-\widehat{\mathcal{E}}_n(U_j(\theta_t)),\] where \(U_j\) denotes the update produced by source \(j\). This reward is meaningful because Theorem 2 controls its cumulative false positives.

6.0.0.2 EXP3/MWU is the scheduler.

We use multiplicative weights / EXP3 to allocate training among sources. With weights \(w_{j,t}\) and exploration parameter \(\gamma\), \[p_{j,t}=(1-\gamma)\frac{w_{j,t}}{\sum_k w_{k,t}}+\frac{\gamma}{K},\qquad w_{j,t+1}=w_{j,t}\exp\left(\eta\frac{r_{t,j}\mathbf{1}\{j=j_t\}}{p_{j,t}}\right).\] This is the standard adversarial-bandit use of multiplicative weights [16], providing exploration and exploitation. The credibility of the cumulative reward comes from the reward signal: if the payoff fed to MWU is a hackable stream reward, MWU efficiently optimizes the hack; if the payoff is signed audit-CP, Theorem 2 supplies the credibility certificate.

7 Experiments↩︎

7.1 Experimental substrate↩︎

All reported numbers come from the accompanying experiment artifact. The curriculum substrate is ARC-TGI, a generator framework for ARC-style grid transformation tasks; its released generator inventory includes ARC-Mini, ARC-AGI-1, and ARC-AGI-2 families [17]. Curriculum experiments require repeated fresh samples from a stable latent rule; a single static puzzle instance is insufficient. The broader ARC-AGI-2 benchmark is motivated by few-shot abstraction and refinement-loop behavior [18].

The experiment suite uses \(30\times 30\) padded grids, 24 learnable ARC-TGI grid-to-grid families, 8 i.i.d. uniform distractor families, probability-floored cross-entropy with \(\varepsilon=0.01\) and cap \(R\approx 4.6\), audit panels of size 512 per family unless otherwise stated, 5000 training steps, and 20 seeds for every experiment except the adaptive attack. The adaptive scalar-feedback attack reports its single-cell power calibration and panel-size scaling curve over 20 seeds each; the full-scale grid attack, in which panel capacity far exceeds the query budget, uses 12 seeds. Baselines include prediction-error curiosity, real RND with a fixed random target network and learned predictor [4], real ICM with inverse-model features and forward prediction error [5], uniform sampling, round-robin, and an oracle that samples only learnable families.

7.2 Main result table↩︎

7.3 Finite-audit concentration↩︎

The finite-audit theorem is useful only if \(\Delta_n\) is small for the reachable class. We measure empirical uniform deviation over finite model families and vary panel size.

The observed slope is \(-0.527\) over 20 seeds, matching the predicted \(n^{-1/2}\) rate; the Hoeffding constant itself is loose. The false-positive budget is therefore an empirically measurable quantity.

7.4 Reward-signal ablation under noise distractors↩︎

The ablation holds the scheduler fixed and varies only the reward signal. The win condition is downstream active-cell reconstruction accuracy on a sealed audit panel, plus low allocation to i.i.d. noise distractors.

Prediction error is Goodharted by noise: it spends 50.4% of its sampling budget on distractors. Audit-CP spends 21.9%, remains above uniform and round-robin, and approaches the oracle. Audit-CP allocates toward tasks whose error compresses, independent of task difficulty.

7.5 Adaptive scalar-feedback holdout overfitting↩︎

In the black-box scalar-feedback setting, the attacker sees only released audit scores and proposes adaptive model updates. This is the holdout-reuse problem applied to audit-CP.

This attack maps the boundary of finite audit-CP: reusing a panel and releasing too much information lets an adaptive agent overfit it, whereas defenses that reduce information leakage keep the scalar-feedback gap below the validity threshold. The \(2\Delta_n\) threshold is therefore operational.

7.6 Boundary and metric tests↩︎

Figure 2 collects the remaining assumption tests. Signed accounting is load-bearing: signed cumulative progress equals endpoint improvement \(-0.072\pm0.042\), while clipped reward accumulates \(1.417\pm0.088\). The stream/audit separation holds: stream-scored CP is \(29.0\) to \(30.0\) while sealed-audit CP is \(0.72\) to \(0.74\) for \(k\in\{0,0.5\}\), and at \(k=1\) the sealed audit goes slightly negative. The capacity boundary is real: direct training on a reusable panel gives apparent CP around \(+2.39\) while true fresh-audit CP is negative. Calibration and discrimination decouple: log-loss and ECE can change with no change in hard accuracy, so compression progress should be read as probabilistic predictive improvement, not merely 0-1 discrimination.

8 Lean 4 mechanization↩︎

The formal artifact is a self-contained Lean 4 / Mathlib development. The core declarations are listed in Table 4. The two structural theorems are stated for an arbitrary potential \(\mathcal{E}:\iota\to\mathbb{R}\); cross-entropy and finite PMFs instantiate the entropy-floor theorem.

The finite-audit bound is mechanized as follows:

theorem finite_audit_goodhart {iota : Type*} {F : Set iota}
    {Ehat E : iota -> Real} {Delta : Real}
    (hUD : UniformDev F Ehat E Delta) {g : Nat -> iota}
    (hg : Admissible g F) (T : Nat) :
    cumCP Ehat g T <= cumCP E g T + 2 * Delta := by
  rw [cumCP_telescope Ehat g T, cumCP_telescope E g T]
  have h0 := abs_le.1 (hUD (g 0) (hg 0))
  have hT := abs_le.1 (hUD (g T) (hg T))
  linarith [h0.1, h0.2, hT.1, hT.2]

Once signed empirical CP telescopes, only the endpoints matter: the uniform-deviation event is used exactly twice, at \(g(0)\) and \(g(T)\). The bound is therefore horizon-free.

9 Discussion↩︎

9.0.0.1 Relation to compression-progress folklore.

The informal compression-progress principle rewards improvements in the compressor. Our theorem states the conditions under which this is credible: the compressor must be evaluated by a fixed audit loss, the reward must be signed, and finite reuse must be controlled by a uniform-deviation budget.

9.0.0.2 Multiplicative weights versus the audit certificate.

Multiplicative weights has regret and allocation guarantees, but its potential is not a predictive-performance metric: it can reweight tasks without reducing audit loss. The credibility certificate is therefore audit-CP, and MWU/EXP3 is a scheduler that consumes it.

9.0.0.3 Log-loss versus hard accuracy.

Compression is probabilistic. Proper scoring losses reward calibrated beliefs, not only argmax decisions. Calibration can improve at fixed hard accuracy. For reconstruction tasks we report active-cell accuracy for interpretability; the certified signal is audit cross-entropy.

10 Limitations↩︎

The guarantee is conditional: it assumes a sealed audit, a reachable model class of finite effective capacity, and an audit the agent cannot alter, the boundaries probed by the stream-scoring, panel-memorization, and holdout-reuse experiments. The deterministic and information-theoretic core is mechanized in Lean; the finite-class concentration corollary is the standard Hoeffding-plus-union argument, stated but not mechanized. The empirical claims are scoped to match: audit-CP leads the non-oracle baselines only narrowly in mean active-cell accuracy, so its robust separation is the distractor-allocation budget; the unchanged accuracy in the calibration probe is an identity of temperature rescaling, with the calibration signal carried by NLL and ECE; and the scalar-feedback attack crosses threshold in the single-cell calibration while the deployed-scale panel stays below it.

11 Conclusion↩︎

Compression progress becomes a Goodhart-resistant RSI signal only after a precise measurement choice: score signed progress on a sealed audit distribution. The resulting reward has an endpoint accounting identity and, for finite panels, a \(2\Delta_n\) false-positive budget. The same budget accounts for the failure modes: prediction-error curiosity is attracted to irreducible noise; clipped progress farms forgetting/relearning cycles; stream progress optimizes the wrong distribution; high-capacity reusable audits can be overfit. Audit-CP avoids these failures exactly when the audit remains sealed and its uniform-deviation budget is finite.

The resulting design rule for intrinsic-motivation and self-improvement systems is: use compression progress as the audited reward, use a separate scheduler for exploration, retain negative progress, and treat the audit as a protected measurement instrument. Under these conditions, apparent improvement is bounded by true audit improvement plus the panel’s deviation budget.

Ethics statement↩︎

This work studies an accounting signal for genuine capability improvement, motivated by reward design for continual learning and recursive self-improvement. The contribution is a measurement condition, not a safety guarantee. The telescoping identity and the \(2\Delta_n\) budget certify that signed compression progress on a sealed audit cannot overstate true audit improvement by more than a finite amount. They do not certify that the audit distribution captures everything a deployer cares about, nor that an agent which satisfies the budget is safe. We state the boundary conditions explicitly, namely clipping, stream scoring, high-capacity reusable panels, and adaptive feedback, so that the signal is not read as a stronger guarantee than it is.

The adaptive scalar-feedback attack reproduces a known adaptive-data-analysis vulnerability: repeated scalar feedback from a reusable panel can be overfit. We include it to calibrate the test and to show that standard release defenses suppress it, not to introduce a new exploit. All experiments use synthetic ARC-TGI grid-transformation tasks and contain no human-subject or personal data. The principal societal risk is overclaiming: deploying audit-CP without a sealed audit and a finite uniform-deviation budget could create false assurance of genuine improvement. The boundary results are included to make that failure mode visible.

Reproducibility statement↩︎

The numerical claims in this paper are computed from the accompanying experiment artifact containing scripts, logs, per-seed JSON shards, summary tables, and vector figures. The formal claims in Table 4 are represented by the accompanying Lean 4 artifact with a pinned Mathlib dependency and an axiom-audit target. The public source package for this paper uses only these two artifacts as inputs for results and proof declarations.

12 Additional theorem details↩︎

12.1 Finite-class concentration↩︎

Let \(X_i^\theta=\ell(\theta,z_i)\) for \(z_i\sim \mathsf Q\), with \(X_i^\theta\in[0,R]\). For a fixed \(\theta\), Hoeffding gives \[\Pr\left(|\widehat{\mathcal{E}}_n(\theta)-\mathcal{E}(\theta)|>\epsilon\right) \le 2\exp\left(-\frac{2n\epsilon^2}{R^2}\right).\] Union bounding over \(N\) experts gives \[\Pr\left(\sup_{\theta\in\mathcal{F}}|\widehat{\mathcal{E}}_n(\theta)-\mathcal{E}(\theta)|>\epsilon\right) \le 2N\exp\left(-\frac{2n\epsilon^2}{R^2}\right).\] Setting the right-hand side to \(\delta\) yields Corollary 1. Combining with Theorem 2 gives the finite-expert false-positive budget.

12.2 Covering-number extension↩︎

For a bounded Lipschitz loss family \(\ell_\theta\) and an \(\epsilon\)-cover of size \(N(\epsilon,\mathcal{F},d)\) in a metric that controls loss uniformly, the same argument yields \[\Delta_n(\mathcal{F})\lesssim \epsilon + R\sqrt{\frac{\log N(\epsilon,\mathcal{F},d)+\log(1/\delta)}{n}}.\] For bounded linear classes in \(d\) dimensions, \(\log N(\epsilon)\) scales as \(d\log(B/\epsilon)\) under standard norm constraints. For bounded RKHS balls, the analogous statement is controlled by the corresponding metric entropy or effective dimension. The paper’s claims do not require a novel concentration theorem; they require recognizing that any such theorem supplies the exact same false-positive budget in Theorem 2.

12.3 Boundary constructions↩︎

13 Lean declaration excerpts↩︎

13.1 Telescoping and finite budget↩︎

theorem cumCP_telescope {iota : Type*} (E : iota -> Real)
    (g : Nat -> iota) (T : Nat) :
    cumCP E g T = E (g 0) - E (g T) := by
  induction T with
  | zero => simp [cumCP]
  | succ n ih =>
      simp only [cumCP, Finset.sum_range_succ] at ih |- 
      rw [ih]; ring

theorem cumCP_le_of_lb {iota : Type*} (E : iota -> Real)
    (g : Nat -> iota) (T : Nat) {Emin : Real}
    (hlb : Emin <= E (g T)) :
    cumCP E g T <= E (g 0) - Emin := by
  rw [cumCP_telescope]; linarith

13.2 Entropy floor↩︎

theorem klDivFinitePMF_nonneg {H : Type*} [Fintype H]
    (Q P : FinitePMF H) [HasPositivePrior P] :
    0 <= klDivFinitePMF Q P := by
  ...

-- T3 as stated in the paper: the CONDITIONAL (input-averaged) entropy floor. The floor
-- H(Y|X) <= E_x H(Q, P (g T)) is DISCHARGED by conditional Gibbs
-- (condEntropy_le_condCrossEntropy), not taken as a hypothesis. The unconditional
-- entropy_floor_budget_crossEntropy is the single-input special case.
theorem cond_entropy_floor_budget_crossEntropy {iota X H : Type*} [Fintype X] [Fintype H]
    (mu : FinitePMF X) (Q : X -> FinitePMF H) (P : iota -> X -> FinitePMF H)
    [forall t, forall x, HasPositivePrior (P t x)] (g : Nat -> iota) (T : Nat) :
    cumCP (fun t => condCrossEntropyFinitePMF mu Q (P t)) g T
      <= condCrossEntropyFinitePMF mu Q (P (g 0)) - condEntropyFinitePMF mu Q :=
  cumCP_le_of_lb _ g T (condEntropy_le_condCrossEntropy mu Q (P (g T)))

14 Experiment protocol summary↩︎

14.0.0.1 Reward-signal ablation.

The task source consists of 24 learnable ARC-TGI grid-to-grid generator families and 8 i.i.d. uniform distractor families. The model is trained for 5000 steps. The scheduler is held fixed across reward variants. Reward variants are signed audit-CP, prediction error, RND, ICM, uniform, round-robin, and learnable-only oracle. Metrics are active-cell audit accuracy, floored audit cross-entropy, and distractor allocation fraction.

14.0.0.2 Finite-audit concentration.

Sample finite families of predictors and audit panels at several \(n\). Compute empirical uniform deviation between panel loss and a larger fresh-audit estimate. Fit a log-log slope of deviation against \(n\).

14.0.0.3 Scalar-feedback attack.

An adaptive attacker receives scalar audit feedback from a reusable panel and selects candidate updates. We compare naive reusable release against fresh subsampling, ladder release, rounded release, and one-shot release. The reported gap is reusable CP minus fresh CP; success is crossing \(2\Delta_n\).

14.0.0.4 Boundary tests.

The stream-versus-audit comparison contrasts stream-scored CP with sealed-audit CP under increasing stream bias. The clipping comparison cycles between states to compare signed and clipped progress. The panel-memorization test trains directly on a reusable panel to probe high-capacity memorization. The calibration probe applies temperature scaling and checks that hard accuracy is invariant while NLL/ECE move.

References↩︎