Proof. Let \(h(t)=\log t\). Fix \(x \geq y\) without loss of generality. By the mean-value theorem, there is \(t \in (y, x)\) such that

\[h(x) - h(y) = h'(t) \cdot (x - y) = \frac{x - y}{t} \leq \frac{x - y}{\zeta}\]

Repeating the above with the role of \(x\) and \(y\) reversed completes the proof. ◻

Proof. The spectral norm of \(A\) is given by \(\opnorm{A} = \sup_{u: \norm{u} = 1} \abs{\ip{u, Au}}\). For any such \(u\), \[\begin{align} \abs{\ip{u, Au}} & = \abs[\Big]{\sum_{i = 1}^{n} \ip{u_{i}, \sum_{j = 1}^{n} A_{ij} u_{j}}} \\ & \leq \sum_{i = 1}^{n} \norm{u_{i}} \norm[\Big]{\sum_{j = 1}^{n} A_{ij} u_{j}} \\ & \leq \sqrt{\sum_{i = 1}^{n} \norm[\Big]{\sum_{j = 1}^{n} A_{ij} u_j}^2} \\ & \leq \sqrt{\sum_{i = 1}^{n} \Big(\sum_{j = 1}^{n} \opnorm{A_{ij}} \norm{u_j}\Big)^2} \\ & \leq \sqrt{\sum_{i = 1}^{n} n \cdot \max_{j} \opnorm{A_{ij}}^2} \\ & \leq n \cdot \max_{i, j} \opnorm{A_{ij}}, \end{align}\] by repeatedly applying the Cauchy-Schwarz and Hölder inequalities. ◻

Proof of Proposition 7. Below, we write \(H_{ij}(\theta) := \grad^2_{\mu_i \mu_j} \mathcal{L}(\theta)\). By 2, we have that \[\nabla_{\mu_i}\mathcal{L}(\theta) = -\mathbb{E}_{X\sim p^\star}\!\big[\psi_i(X;\theta)(X-\mu_i)\big].\] To differentiate the responsibility, write \[\psi_i(x;\theta)=\frac{N_i(x)}{Z(x)}, \qquad N_i(x):=\pi_i\phi(x\mid \mu_i), \qquad Z(x):=\sum_{k=1}^n N_k(x).\] Then \[\nabla_{\mu_j}N_i(x)=\mathbf{1}_{\{i=j\}}N_i(x)(x-\mu_j), \qquad \nabla_{\mu_j}Z(x)=N_j(x)(x-\mu_j).\] Hence, by the quotient rule, \[\begin{align} \nabla_{\mu_j}\psi_i(x;\theta) & = \frac{\nabla_{\mu_j}N_i(x)}{Z(x)} -\frac{N_i(x)\nabla_{\mu_j}Z(x)}{Z(x)^2} \\ & = \mathbf{1}_{\{i=j\}}\frac{N_i(x)}{Z(x)}(x-\mu_j) -\frac{N_i(x)}{Z(x)}\frac{N_j(x)}{Z(x)}(x-\mu_j) \\ & = \psi_i(x;\theta)\big(\mathbf{1}_{\{i=j\}}-\psi_j(x;\theta)\big)(x-\mu_j). \end{align}\] Continuing with the calculation of the second derivative, we obtain \[\begin{align} H_{ij}(\theta) & = \nabla_{\mu_j}(\nabla_{\mu_i}\mathcal{L}(\theta)) \\ & = -\mathbb{E}_{X\sim p^\star}\!\Big[ \nabla_{\mu_j}\big(\psi_i(X;\theta)(X-\mu_i)\big) \Big] \\ & = -\mathbb{E}_{X\sim p^\star}\!\Big[ (X-\mu_i)\big(\nabla_{\mu_j}\psi_i(X;\theta)\big)^\T -\mathbf{1}_{\{i=j\}}\psi_i(X;\theta)I_d \Big] \\ & = -\,\mathbb{E}_{X\sim p^\star}\!\Big[ \psi_i(X;\theta)\big(\mathbf{1}_{\{i=j\}}-\psi_j(X;\theta)\big) (X-\mu_i)(X-\mu_j)^\T \Big] \\ & \qquad +\,\mathbf{1}_{\{i=j\}}\mathbb{E}_{X\sim p^\star}\!\big[\psi_i(X;\theta)\big]I_d. \end{align}\] We now carry out the calculation for \(\theta \equiv \theta^\star\in\Theta_\star\). Since \(p_{\theta^\star}=p^\star\), \[\begin{align} \mathbb{E}_{X\sim p^\star}\!\Big[\psi_i(X;\theta^\star)(X-\mu_i^\star)(X-\mu_i^\star)^\T\Big] & = \int \pi_i\phi(x\mid \mu_i^\star)(x-\mu_i^\star)(x-\mu_i^\star)^\T\,dx \\ & = \mathbb{E}_{X\sim p^\star}\!\big[\psi_i(X;\theta^\star)\big]I_d. \end{align}\] Substituting this identity into the diagonal blocks yields \[H_{ij}(\theta^\star) = \mathbb{E}_{X\sim p^\star}\!\Big[ \psi_i(X;\theta^\star)\psi_j(X;\theta^\star) (X-\mu_i^\star)(X-\mu_j^\star)^\T \Big].\] Finally, if \(i\in S_\ell\) and \(j\in S_k\), the exact clustered representation gives \(\mu_i^\star=\mu_\ell^\star\) and \(\mu_j^\star=\mu_k^\star\). Hence \[\psi_i(x;\theta^\star)=\frac{\pi_i}{\pi_\ell^\star}\gamma_\ell(x), \qquad \psi_j(x;\theta^\star)=\frac{\pi_j}{\pi_k^\star}\gamma_k(x),\] which gives the factorized expression. ◻

Proof. A standard statistical result states that the Hessian of the KL divergence evaluated at the optimal parameters \(\nu^\star\) coincides with the Fisher Information Matrix (FIM) of this model [37]. Therefore, proving \(\nabla^2 \mathcal{L}_{\mathrm{coll}}(\nu^\star) \succ 0\) is equivalent to proving that the FIM is non-singular. For parametric models satisfying standard smoothness and regularity conditions (which the Gaussian family easily satisfies), the FIM is strictly positive definite if and only if the model is locally identifiable [38].

A classic result by [39] establishes that finite mixtures of Gaussians are strictly identifiable. Under Item [assm:item:nondegeneracy], the mapping from the means \(\nu\) to the density \(p_\nu\) is unique. This strict identifiability guarantees that the Fisher Information Matrix is positive definite, concluding the proof. ◻

Proof. We introduce the following notation for simplicity: \[H:=\nabla^2_{\mu\mu}\mathcal{L}(\theta^\star), \qquad \nu^\star:=(\mu_1^\star,\dots,\mu_m^\star), \qquad \bar H:=\nabla^2 \mathcal{L}_{\mathrm{coll}}(\nu^\star).\] By Lemma 5, we have \(\bar H\succ 0\). For \(u=(u_1,\dots,u_n)\in(\mathbb{R}^d)^n\), define \[v_\ell:=\frac{1}{\pi_\ell^\star}\sum_{i\in S_\ell}\pi_i u_i = \frac{1}{\pi_\ell^\star}\mathsf{Avg}_{\ell}(u), \qquad v=(v_1,\dots,v_m)\in(\mathbb{R}^d)^m.\] Fix \(i\in S_\ell\). Then Proposition 7 gives \[\begin{align} (Hu)_i & = \sum_{k=1}^m\sum_{j\in S_k}H_{ij}u_j \\ & = \sum_{k=1}^m\sum_{j\in S_k} \frac{\pi_i\pi_j}{\pi_\ell^\star\pi_k^\star}\Gamma_{\ell k}u_j \\ & = \frac{\pi_i}{\pi_\ell^\star}\sum_{k=1}^m \Gamma_{\ell k} \left( \frac{1}{\pi_k^\star}\sum_{j\in S_k}\pi_j u_j \right) \\ & = \frac{\pi_i}{\pi_\ell^\star}\sum_{k=1}^m \Gamma_{\ell k}v_k. \end{align}\] Clearly, if \(u \in \ker(\mathsf{Avg})\), we have \(v = 0\). Consequently, \(Hu = 0\) and thus \(\mathcal{T} \subset \ker(H)\). Conversely, \[u \in \ker(H) \implies 0 = (Hu)_i = \frac{\pi_i}{\pi_\ell^\star}\sum_{k=1}^m \Gamma_{\ell k}v_k \qquad \text{for every }i\in S_\ell.\] Since \(\pi_i>0\) by assumption, this implies \[\sum_{k=1}^m \Gamma_{\ell k}v_k=0 \qquad \text{for every }\ell\in[m].\] Applying Proposition 7 to the exact-parametrized \(m\)-component model shows that the \((\ell,k)\) block of \(\bar H\) is exactly \(\Gamma_{\ell k}\), so \(\bar H v=0\). Because \(\bar H\succ0\), we must have \(v=0\), which is equivalent to \[\sum_{i\in S_\ell}\pi_i u_i=0 \qquad \text{for every }\ell\in[m].\] Hence \(u\in\mathcal{T}\), proving the reverse inclusion.

Finally, we prove that \(\mathsf{Avg}: \mathbb{R}^{dn}\to \mathbb{R}^{dm}\) has rank \(md\): for each \(\ell\), the map \((u_i)_{i\in S_\ell}\mapsto \sum_{i\in S_\ell}\pi_i u_i\) is surjective onto \(\mathbb{R}^d\) because \(\pi_i>0\) and \(S_\ell\neq\varnothing\). Therefore \[\dim \mathcal{T} = nd-md = (n-m)d.\] ◻

Proof of Corollary 3. For each \(\ell\in[m]\), write \[\mu_i-\mu_\ell^\star=(\mu_i-\bar\mu_\ell)+(\bar\mu_\ell-\mu_\ell^\star).\] Expanding the squared norm, multiplying by \(\pi_i\), and summing over \(i\in S_\ell\) gives \[\sum_{i\in S_\ell}\pi_i\norm{\mu_i-\mu_\ell^\star}^2 = \widehat\pi_\ell\norm{\bar\mu_\ell-\mu_\ell^\star}^2 + \sum_{i\in S_\ell}\pi_i\norm{\mu_i-\bar\mu_\ell}^2,\] because the cross term vanishes: \[\sum_{i\in S_\ell}\pi_i(\mu_i-\bar\mu_\ell)=0.\] Summing over \(\ell\in[m]\) yields the claimed decomposition. ◻

Proof. Because \(\theta^\star\) is a global minimizer, \(f_\pi(\mu^\star)=\mathcal{L}^\star\). By continuity of \(f_\pi\) and of the student means, we may choose a neighborhood \(U\) of \(\mu^\star\) such that for every \(\mu\in U\), the partition \([n]=\bigsqcup_{\ell=1}^m S_\ell\) remains valid and \[f_\pi(\mu)-f_\pi(\mu^\star)\le \varepsilon_0,\] where \(\varepsilon_0\) is the threshold in Theorem 4. Then, for any \(\mu \in U\), we have \[\sum_{\ell=1}^m\sum_{i\in S_\ell}\pi_i\|\mu_i-\mu_\ell^\star\|_2^2 \le C\sqrt{\mathcal{L}(\mu, \pi)}.\] On the other hand, we have the following lower bound: \[\sum_{\ell=1}^m\sum_{i\in S_\ell}\pi_i\|\mu_i-\mu_\ell^\star\|_2^2 \ge \pi_{\min}\|\mu-\mu^\star\|_2^2, \qquad \pi_{\min}:=\min_{i\in[n]}\pi_i>0.\] Combining the last two displays and squaring both sides yields \[\mathcal{L}(\mu, \pi) \gtrsim \pi_{\min}^2 \norm{\mu-\mu^\star}^4.\] This proves the claim. ◻

Proof. Theorem 9 implies that \(\mu^\star\) is the unique minimizer of \(f_\pi\) in a sufficiently small neighborhood of \(\mu^\star\). Since \(f_\pi\) is \(C^\infty\), the local solution set is the singleton \(\{\mu^\star\}\), so the Hessian rank is constant on the solution set. Corollary 1 therefore applies. ◻

Proof. At \(\theta^\star\), every student mean equals the teacher mean. Consequently, \[\psi_i(x;\theta^\star) = \frac{\pi_i\phi(x\mid 0)}{\sum_{k=1}^n \pi_k\phi(x\mid 0)} = \pi_i.\] Applying Proposition 7 with \(m=1\) therefore gives \[H_{ij}(\theta^\star) = \mathbb{E}_{X\sim p^\star}\!\big[\pi_i\pi_j XX^\T\big] = \pi_i\pi_j I_d,\] because \(X\sim\mathcal{N}(0,I_d)\) under \(p^\star\). Hence \[\nabla^2_{\mu\mu}\mathcal{L}(\theta^\star) = (\pi\pi^\T)\otimes I_d = (\pi \otimes I_d)(\pi \otimes I_d)^{\T}.\] By standard properties of the Kronecker product, for any \(u \in \mathbb{R}^{dn}\), we have \[(\pi \otimes I_d)^{\T} u = \sum_{i = 1}^{n} \pi_{i} u_{i} \implies \ker(\grad_{\mu\mu}^2 \mathcal{L}(\theta^{\star})) = \ker(\pi^{\T} \otimes I_d) = \set*{u \in \mathbb{R}^{dn} \mid \sum_{i = 1}^{n} \pi_{i} u_{i} = 0}.\] The rank property is immediate from \(\rank(A \otimes B) = \rank(A) \cdot \rank(B)\). ◻

Proof of Proposition 11. From [6], which is applicable as long as the weights are bounded, there exists a constant \(D_{\max}>0\) such that for every \(v\in\mathbb{R}^d\) with \(\norm{v}_2=1\), \[\mathcal{L}(\mu,\pi) \geq D_{\max}^{-4} \left( \sum_{i=1}^n \pi_i \langle \mu_i,v\rangle^2 \right)^2.\] If \(\mu_{\max}=0\), then every \(\mu_i=0\), so both sides of the desired inequality vanish and there is nothing to prove. Hence we may assume \(\mu_{\max}\neq 0\) in what follows. Choosing \[v:=\frac{\mu_{\max}}{\norm{\mu_{\max}}_2},\] we deduce the following inequality: \[\sum_{i=1}^n \pi_i \langle \mu_i,v\rangle^2 \geq \pi_{\min}\max_{i\in[n]}\langle \mu_i,v\rangle^2 \geq \pi_{\min}\langle \mu_{\max},v\rangle^2 = \pi_{\min}\norm{\mu_{\max}}_2^2.\] Substituting this bound into the previous display gives \[\mathcal{L}(\mu,\pi) \geq D_{\max}^{-4} \bigl(\pi_{\min}\norm{\mu_{\max}}_2^2\bigr)^2.\] The claim follows with \(C:=D_{\max}^{-4}\). ◻

Proof. Expanding the definition of \(\bar{\pi}\), we obtain \[\begin{align} \norm{\bar{\pi}-\pi}_1 & = \sum_{\ell = 1}^{m} \sum_{i \in S_{\ell}} \abs{\bar{\pi}_i - \pi} \\ & = \sum_{\ell = 1}^{m} \sum_{i \in S_{\ell}} \abs[\Big]{\pi_{i} \Big(\tfrac{\pi_{\ell}^{\star}}{\widehat{\pi}_{\ell}} - 1\Big)} \\ & = \sum_{\ell = 1}^{m} \abs[\big]{\tfrac{\pi_{\ell}^{\star}}{\widehat{\pi}_{\ell}} - 1} \sum_{i \in S_{\ell}} \pi_{i} \\ & = \sum_{\ell = 1}^{m} \abs{\widehat{\pi}_{\ell} - \pi_{\ell}^{\star}} \\ & = \sum_{\ell = 1}^{m} \Delta_{\pi}^{(\ell)} \cdot \pi_{\ell}^{\star} \\ & \leq \Delta_{\pi}, \end{align}\] where the second equality follows from the fact that \(\frac{\widehat{\pi}_{\ell}}{\pi_{\ell}^{\star}}\) is constant throughout \(S_{\ell}\), the fourth equality follows from 26 , and the last two lines follow from 27 28 . ◻

Proof. Since \(\mathcal{L}(\mu, \pi) = \expec[X \sim p^{\star}]{\log(p^{\star}(X) / p_{\mu, \pi}(X))}\), we control deviations by comparing the densities induced by \(\pi\) and \(\bar{\pi}\). For \(\mu \in \mathbb{R}^{dn}\) and corresponding partition \([n] = \bigsqcup_{\ell} S_{\ell}\), let \[q_\ell(x;\mu):=\frac{1}{\widehat\pi_\ell}\sum_{i\in S_\ell}\pi_i\phi(x\mid \mu_i). \;\; \text{for \ell \in [m]} \label{eq:group-partition}\tag{31}\] With 31 at hand, we rewrite both densities as \[\begin{align} p_{\mu,\pi}(x) & = \sum_{\ell=1}^m \widehat\pi_\ell q_\ell(x;\mu), \tag{32} \\ p_{\mu,\bar{\pi}}(x) & = \sum_{\ell=1}^m \pi_\ell^\star q_\ell(x;\mu). \tag{33} \end{align}\] Using the preceding decomposition and Jensen’s inequality, we obtain \[\abs{\mathcal{L}(\mu, \pi) - \mathcal{L}(\mu, \bar{\pi})} = \abs[\big]{\expec[X \sim p^{\star}]{\log\left(\tfrac{p_{\mu, \bar{\pi}}(X)}{p_{\mu, \pi}(X)}\right)}} \leq \expec[X \sim p^{\star}]{\abs*{\log\left(\tfrac{p_{\mu, \bar{\pi}}(X)}{p_{\mu, \pi}(X)}\right)}}, \label{eq:log-diff}\tag{34}\] We now bound the expression inside the expectation in 34 . For any finite \(X\), 3 yields

\[\abs{\log(\tfrac{p_{\mu, \bar{\pi}}(X)}{p_{\mu, \pi}(X)})} \leq \frac{\abs{p_{\mu, \pi}(X) - p_{\mu, \bar{\pi}}(X)}}{\min\set*{ p_{\mu, \pi}(X), p_{\mu, \bar{\pi}}(X) }} \label{eq:log-density-ratio-bound}\tag{35}\]

To further bound the right-hand side in 35 , we proceed as follows:

• We lower bound both densities uniformly. Indeed, since \(\Delta_{\pi} < \tfrac{1}{2}\) by assumption, \[\widehat{\pi}_{\ell} \geq (1 - \Delta_{\pi}^{\ell}) \cdot \pi_{\ell}^{\star} \geq \frac{1}{2} \pi_{\ell}^{\star}. \label{eq:adjusted-density-bound}\tag{36}\] From the above and 32 33 , we deduce the lower bounds \[\begin{align} p_{\mu, \bar{\pi}}(X) & = \sum_{\ell = 1}^{m} \pi_{\ell}^{\star} q_{\ell}(X; \mu) \geq \pi_{\min}^{\star} \sum_{\ell=1}^{m} q_{\ell}(X; \mu); \tag{37} \\ p_{\mu, \pi}(X) & = \sum_{\ell = 1}^{m} \widehat{\pi}_{\ell} q_{\ell}(X; \mu) \geq \frac{1}{2} \pi_{\min}^{\star} \sum_{\ell=1}^{m} q_{\ell}(X; \mu). \tag{38} \end{align}\]

• We upper bound their difference for a fixed \(X\). Indeed, 6 supplies the bound \[\begin{align} \abs{p_{\mu,\pi}(X)-p_{\mu,\bar\pi}(X)} & = \abs[\Big]{\sum_{\ell=1}^m (\widehat\pi_\ell-\pi_\ell^\star)q_\ell(X;\mu)} \notag \\ & \le \sum_{\ell=1}^m \abs{\widehat{\pi}_\ell-\pi_{\ell}^\star} q_\ell(X;\mu) \notag \\ & \leq \Delta_{\pi} \sum_{\ell=1}^m q_\ell(X;\mu). \label{eq:density-difference-under-weight-mismatch} \end{align}\tag{39}\]

Plugging the previous estimates into 35 and taking expectations, noting \(\norm{X} < \infty\) a.s., yields \[\expec[X \sim p^{\star}]{\abs{\log(p_{\mu, \pi}(X)) - \log(p_{\mu, \bar{\pi}}(X))}} \leq \frac{2 \Delta_{\pi}}{\pi_{\min}^{\star}}.\] Substituting into 34 completes the proof of ?? .

We now turn to the gradient bound. Fix \(i \in S_{\ell}\); by the definition of \(\bar{\pi}_i\) and 6 , we have \[\psi_i(x;\mu, \pi) = \frac{\pi_i\phi(x\mid \mu_i)}{p_{\mu,\pi}(x)}, \quad \psi_i(x;\mu,\bar{\pi}) = \frac{\bar\pi_i\phi(x\mid \mu_i)}{p_{\mu,\bar\pi}(x)} = \frac{\pi_\ell^\star}{\widehat\pi_\ell} \cdot \frac{p_{\mu, \pi}(x)}{p_{\mu,\bar\pi}(x)} \cdot \psi_{i}(x; \mu, \pi).\] This implies the following bound on the difference between responsibilities: \[\begin{align} \abs{\psi_{i}(X; \mu, \pi) - \psi_{i}(X; \mu, \bar{\pi})} & = \psi_{i}(X; \mu, \pi) \cdot \abs[\Big]{ 1 - \frac{\pi_{\ell}^{\star}}{\widehat{\pi}_{\ell}} \cdot \frac{p_{\mu, \pi}(X)}{p_{\mu, \bar{\pi}}(X)} } \\ & = \psi_{i}(X; \mu, \pi) \cdot \abs[\Big]{ \frac{\widehat{\pi}_{\ell} p_{\mu, \bar{\pi}}(X) - \pi_{\ell}^{\star} p_{\mu, \pi}(X)}{ \widehat{\pi}_{\ell} p_{\mu, \bar{\pi}}(X) } } \\ & \leq 2\,\psi_{i}(X; \mu, \pi) \cdot \abs[\Big]{ \frac{\widehat{\pi}_{\ell} p_{\mu, \bar{\pi}}(X) - \pi_{\ell}^{\star} p_{\mu, \pi}(X)}{ {\pi}_{\ell}^{\star} p_{\mu, \bar{\pi}}(X) } } \\ & \leq 2\, \psi_{i}(X; \mu, \pi) \cdot \left( \Delta_{\pi}^{(\ell)} + \frac{\abs{p_{\mu, \pi}(X) - p_{\mu, \bar{\pi}}(X)}}{p_{\mu, \bar{\pi}}(X)} \right) \\ & \leq 2\, \psi_{i}(X; \mu, \pi) \cdot \left( \Delta_{\pi}^{(\ell)} + \frac{\Delta_{\pi}}{\pi_{\min}^{\star}} \right) \\ & \leq 2\,\Delta_{\pi} \left(1 + \frac{1}{\pi_{\min}^{\star}}\right) \\ & = C_{0} \Delta_{\pi}, \end{align}\] where the first inequality follows from the bound 36 , the second inequality follows from the definition of \(\Delta_{\pi}^{(\ell)}\), the penultimate inequality follows from 39 and 37 , and the last inequality follows from \(\Delta_{\pi}^{(\ell)} \leq \Delta_{\pi}\) and the fact that \(\set{\psi_{i}(X; \mu, \pi)}_{i=1}^n \in \Delta^{n-1}\).

By the gradient identity furnished by 2, we have \[\begin{align} \norm{\grad_{\mu_i}\mathcal{L}(\mu,\pi) - \grad_{\mu_{i}}\mathcal{L}(\mu,\bar{\pi})} & = \norm{\expec[X\sim p^\star]{(\psi_i(X;\mu, \pi) - \psi_{i}(X; \mu, \bar{\pi})) \cdot (X-\mu_i)}} \\ & \leq C_{0} \Delta_{\pi} \cdot \Big(\expec[X \sim p^{\star}]{\norm{X - \mu_{i}}^2}\Big)^{1/2} \\ & \leq C_{0} \Delta_{\pi} \sqrt{2(m_{2} + M_{U}^2)}, \end{align}\] where the penultimate inequality follows from Cauchy-Schwarz and the fact that \[\begin{align} \big(\expec[X \sim p^{\star}]{\norm{X - \mu_{i}}^2})^{1/2} & \leq \big(2 \expec[X \sim p^{\star}]{\norm{X}^2} + 2 \norm{\mu_{i}}^2\big)^{1/2} \\ & \leq 2^{1/2} \cdot \big(m_{2} + M_{U}^2\big)^{1/2}. \end{align}\] Taking the Euclidean norm over the \(n\) blocks of the gradient proves ?? .

Finally, let \(H^\pi_{ij}(\mu)\) and \(H^{\bar\pi}_{ij}(\mu)\) denote the \((i,j)\) blocks of \(\grad^2_{\mu\mu}\mathcal{L}(\mu,\pi)\) and \(\grad^2_{\mu\mu}\mathcal{L}(\mu,\bar\pi)\). By the derivation in the proof of Proposition 7, we obtain \[H^\pi_{ij}(\mu) = -\expec{\psi_i(X; \mu, \pi)(\delta_{ij}-\psi_j^\pi) (X-\mu_i)(X-\mu_j)^\T } + \delta_{ij}\expec{\psi_i(X; \mu, \pi)}I_d.\] The formula for \(H^{\bar\pi}_{ij}(\mu)\) follows mutatis-mutandis. Writing \(\psi_{i}^{\pi} := \psi_{i}(X; \mu, \pi)\) for brevity, we have \[\begin{align} \abs{\psi_i^{\pi}(\delta_{ij}-\psi_j^\pi)-\psi_i^{\bar\pi}(\delta_{ij}-\psi_j^{\bar\pi})} & \leq \delta_{ij} \abs{\psi_{i}^{\pi} - \psi_{i}^{\bar{\pi}}} + \abs{ \psi_{i}^{\pi} \psi_{j}^{\pi} - \psi_{i}^{\bar{\pi}} \psi_{j}^{\bar{\pi}} } \\ & \leq \abs{\psi_{i}^{\pi} - \psi_{i}^{\bar{\pi}}} + \psi_j^{\pi} \abs{\psi_{i}^{\pi} - \psi_{i}^{\bar{\pi}}} + \psi_{i}^{\bar{\pi}} \abs{\psi_{j}^{\pi} - \psi_{j}^{\bar{\pi}}} \\ & \leq C_{0} \Delta_{\pi} \left( 1 + \psi_{j}^{\bar{\pi}} + \psi_{i}^{\bar{\pi}} + \psi_{j}^{\bar{\pi}} - \psi_{j}^{\pi} \right) \\ & \leq C_{0} \Delta_{\pi} \left(2 + C_{0} \Delta_{\pi}\right) =: C_{1} \Delta_{\pi}, \end{align}\] using the fact that \(\set{\psi_{i}^{\bar{\pi}}}_{i=1}^n \in \Delta^{n-1}\) in the last inequality. Therefore, we deduce \[\begin{align} \opnorm{H^\pi_{ij}(\mu)-H^{\bar\pi}_{ij}(\mu)} & \leq C_{1} \Delta_{\pi} \expec{\norm{X-\mu_i}\norm{X-\mu_j}} + \delta_{ij} \expec{\abs{\psi_{i}^{\pi} - \psi_{i}^{\bar{\pi}}}} \\ & \leq C_{1} \Delta_{\pi} (\expec{\norm{X - \mu_{i}}^2})^{1/2} (\expec{\norm{X - \mu_j}^2})^{1/2} + C_{0} \Delta_{\pi} \\ & \leq C_{0} \Delta_{\pi} + 2C_{1} \Delta_{\pi} \left(m_{2} + M_{U}^2\right) \\ &= C_0 \Delta_{\pi} \left( 1 + 2(m_2 + M_{U}^2) (2 + C_{0} \Delta_{\pi}) \right), \end{align}\] again using Cauchy-Schwarz. Invoking 4 supplies the bound for the full Hessian. ◻

Proof. Recall the collapsed model loss from 5; for any \(\nu \in \mathbb{R}^{dm}\) and \(\rho \in \Delta^{m-1}\), let \[\mathcal{L}_{\mathrm{coll}}(\nu, \rho) := D_{\mathrm{KL}}\!\left(p^\star\,\|\,\sum_{\ell=1}^m \rho_{\ell}\, \cN(\nu_{\ell}, I_{d})\right).\] The function \(\rho \mapsto \mathcal{L}_{\mathrm{coll}}(\nu^{\star}, \rho)\) is minimized at \(\rho = \pi^{\star} \in \intr{\Delta^{m-1}}\). As a result, we have \[\left.\grad_{\rho}\mathcal{L}_{\mathrm{coll}}(\nu^{\star}, \rho) \right|_{\rho = \pi^{\star}} = 0. \label{eq:collapsed-weight-gradient-vanishes}\tag{41}\] Now, let \(\nu^{\star}: U \to \mathbb{R}^{dn}\) be the following map: \[\begin{align} \nu^{\star}(\mu) = \bmx{ \nu_{1}^{\star}(\mu) \\ \vdots \\ \nu_{n}^{\star}(\mu) }, \quad [\nu^{\star}(\mu)]_{i} = \nu^{\star}_{\ell}, \;\; \text{for any i \in S_{\ell}}. \end{align}\] For any \(\pi\) such that \(\norm{\widehat{\pi} - \pi^{\star}}\) is sufficiently small, we have that \[\begin{align} \mathcal{L}_{\pi}^{\dagger} - \mathcal{L}(\theta^{\star}) & = \min_{\mu \in U} \mathcal{L}(\mu, \pi) - \mathcal{L}(\theta^{\star}) \notag \\ & \leq \mathcal{L}(\nu^{\star}(\mu), \pi) - \mathcal{L}(\theta^{\star}) \notag \\ & = D_{\mathrm{KL}}\!\left(p^{\star}\,\|\,\sum_{\ell = 1}^{m} \sum_{i \in S_{\ell}} \pi_{i} \mathcal{N}(\nu_{\ell}^{\star}, I_d)\right) - \mathcal{L}(\theta^{\star}) \tag{42} \\ & = \mathcal{L}_{\mathrm{coll}}(\nu^{\star}, \widehat{\pi}) - \mathcal{L}(\theta^{\star}) \notag \\ & = \underbrace{\mathcal{L}_{\mathrm{coll}}(\nu^{\star}, \pi^{\star})}_{\mathcal{L}(\theta^{\star})} + \ip{\grad_{\rho}\mathcal{L}(\nu^{\star}, \rho)|_{\rho = \pi^{\star}}, \widehat{\pi} - \pi^{\star}} + O(\norm{\widehat{\pi} - \pi^{\star}}^2) - \mathcal{L}(\theta^{\star}) \notag \\ &\lesssim \norm{\widehat{\pi} - \pi^{\star}}_1^2 \notag \\ &= \Delta_{\pi}^2, \tag{43} \end{align}\] where the last inequality follows from 41 and norm equivalence. ◻

Proof. The function \(\mathsf{D}(\cdot)\) is \(C^{\infty}\), thus Lipschitz on any compact set. It follows that \[\abs{\mathsf{D}(y)-\mathsf{D}(\mu)} = \abs{\mathsf{D}(P_{\mathcal{M}}(\mu)) - \mathsf{D}(\mu)} \leq C \cdot \norm{P_{\mathcal{M}}(\mu) - \mu} = C \cdot \dist_{\mathcal{M}}(\mu).\] Therefore, from the inclusion \(\mu \in \mathcal{T}_{\tau}\), it follows that \[\begin{align} \abs{\mathsf{D}(y)-\mathsf{D}(\mu)} \leq C\tau \mathsf{D}(y)^{3/2} \Rightarrow \bigl(1 - C \tau \sqrt{\mathsf{D}(y)}\bigr) \mathsf{D}(y) \leq \mathsf{D}(\mu) \leq \mathsf{D}(y) \bigl(1 + C \tau \sqrt{\mathsf{D}(y)}\bigr). \end{align}\] By shrinking \(U\) if necessary, we have \(\sqrt{\mathsf{D}(y)} \leq \tfrac{1}{2 C \tau}\), proving the claim for \(c_1 = \tfrac{1}{2}\), \(c_2 = \tfrac{3}{2}\). ◻

Proof. For \(\mu\) near \(\mu^{\star} \in \mathcal{M}\), the projection \(P_{\mathcal{M}}\) is \(C^{1}\). In particular, we have that \[\begin{align} y - \mu^{\star} & = P_{\mathcal{M}}(\mu) - \mu^{\star} \notag \\ & = P_{\mathcal{M}}(\mu^{\star}) - \mu^{\star} + \grad P_{\mathcal{M}}(\mu^{\star})(\mu - \mu^{\star}) + O(\norm{\mu - \mu^{\star}}^2) \notag \\ & = P_{\mathcal{T}_{\mu^{\star}}\mathcal{M}}(\mu - \mu^{\star}) + O(\norm{\mu - \mu^{\star}}^2), \label{eq:distance-along-ravine-taylor-expansion} \end{align}\tag{51}\] where the second equality follows by smoothness of the projection and the third equality follows from the fact that \(\grad P_{\mathcal{M}}(\bar{\mu}) = P_{\mathcal{T}_{\bar{\mu}} \mathcal{M}}\) for any \(\bar{\mu} \in \mathcal{M}\). Extracting the constant from the remainder term and labeling at \(C_{U}\) completes the proof. ◻

Proof. The objective \(\mathcal{L}(\mu, \bar{\pi})\) admits a local solution set \(\set{\mu^{\star}}\), which is a singleton. The conclusion follows immediately by applying 6 with \(f^{\star} = \mathcal{L}(\theta^{\star}) = 0\). ◻

Proof. We first identify the tangent space of the reference loss ravine at \(\mu^\star\). Since \[\bar{\pi}_{i} = \pi_{i} \cdot \frac{\pi_{\ell}^{\star}}{\widehat{\pi}_{\ell}}, \;\; \text{for all i \in S_{\ell}},\] the following linear subspaces are equal: \[\begin{align} \mathcal{T} & := \set{u \in \mathbb{R}^{dn} \mid \sum_{i \in S_{\ell}} \pi_{i} u_{i} = 0, \; \text{for all \ell \in [m]}} \\ & = \set{u \in \mathbb{R}^{dn} \mid \sum_{i \in S_{\ell}} \bar{\pi}_{i} u_{i} = 0, \; \text{for all \ell \in [m]}}. \label{eq:equal-tangent-space} \end{align}\tag{58}\] By 8 4 applied to \(\mu \mapsto \mathcal{L}(\mu, \bar{\pi})\), \(\mathcal{T}\) is the tangent space of \(\mathcal{M}\) at \(\mu^{\star}\). Therefore, by 1, we deduce that \(y \in \mathcal{M}\) near \(\mu^{\star}\) can be written as \[y = \mu^{\star} + u + O(\norm{y - \mu^{\star}}^2), \;\; \text{where u \in \mathcal{T}_{\mu^{\star}} \mathcal{M}}.\] In particular, the dispersion \(\mathsf{D}(y)\) does not change up to first-order: indeed, \[\begin{align} \mathsf{D}(\mu^{\star} + u) & = \sum_{\ell = 1}^{m} \sum_{i \in S_{\ell}} \pi_{i} \norm[\Big]{ \mu^{\star}_{i} + u_{i} - \frac{1}{\widehat{\pi}_{\ell}} \sum_{j \in S_{\ell}} \pi_{j} (\mu^{\star}_j + u_{j}) }^2 = \sum_{\ell = 1}^{m} \sum_{i \in S_{\ell}} \pi_{i} \norm{u_{i}}^2, \end{align}\] where the second equality follows from 58 . Consequently, \[\mathsf{D}(y) = \sum_{i = 1}^{n} \pi_{i} \norm{u_{i}}^2 + O(\norm{u}^3). \label{eq:dispersion-near-mustar}\tag{59}\] Therefore, after shrinking the local neighborhood if necessary, we have \[c \cdot \norm{u}^2 \leq \mathsf{D}(y) \leq C \cdot \norm{u}^2.\] Finally, since \(\norm{y - \mu^\star} \asymp \norm{u}\), this proves 52 .

We now apply Item [item:tangent-part-p-growth] from 6 with \(p = 4\), which yields \[\begin{align} c \cdot \norm{y - \mu^{\star}}^4 & \leq \mathcal{L}_{T}(y) \leq C \cdot \norm{y - \mu^{\star}}^4, \\ c \cdot \norm{y - \mu^{\star}}^3 & \leq \norm{\grad \mathcal{L}_{T}(y)} \leq C \cdot \norm{y - \mu^{\star}}^3, \end{align}\] for all \(y \in V \cap \mathcal{M}\). Since \(\mathsf{D}(y) \asymp \norm{y - \mu^{\star}}^2\) by 52 , this proves 53 54 .

We now turn to the proof of Item [item:reference-gradient-pl-like]. In particular, we argue that the gradient of the tangent part dominates near \(\mu^{\star}\) via the decomposition furnished by 13. To that end, note that \(\mathcal{L}_{N}\) is \(C^{1}\) and vanishes along \(\mathcal{M}\). Therefore, its gradient is Lipschitz near \(\mathcal{M}\) and satisfies \[\norm{\grad \mathcal{L}_{N}(\mu)} \leq C \cdot \dist_{\mathcal{M}}(\mu) \leq C \tau \mathsf{D}(y)^{3/2}, \label{eq:normal-grad-ub}\tag{60}\] where the second inequality follows from the inclusion \(\mu \in \mathcal{T}_{\tau}\); this proves 56 .

On the other hand, 54 suggests that \[\norm{\grad \mathcal{L}_{T}(y)} \geq c \cdot \mathsf{D}(y)^{3/2}. \label{eq:tangent-grad-lb}\tag{61}\] Finally, using the tangent-normal decomposition from 13, we obtain \[\begin{align} \norm{\grad \mathcal{L}(\mu, \bar{\pi})} & \geq \norm{\grad \mathcal{L}_{T}(\mu)} - \norm{\grad \mathcal{L}_{N}(\mu)} \\ & \geq \norm{\grad \mathcal{L}_{T}(y)}(1 - C \dist_{\mathcal{M}}(\mu)) - C \tau \dist_{\mathcal{M}}(\mu) \\ & \gtrsim \norm{\grad \mathcal{L}_{T}(y)} - C' \tau \norm{\grad \mathcal{L}_{T}(y)} \\ & \gtrsim \norm{\grad \mathcal{L}_{T}(y)}, \end{align}\] where the first step follows from the reverse triangle inequality, the second inequality follows from [34] and 60 , and the last two inequalities follow from 61 and shrinking \(\tau\) if necessary. This proves 55 and Item [item:reference-gradient-pl-like].

Finally, we turn to the proof of Item [item:improved-gradient-perturbation-in-annulus]. Indeed, we have \[\begin{align} \norm{\grad_{\mu} \mathcal{L}(\mu,\pi)-\grad_{\mu} \mathcal{L}(\mu,\bar{\pi})} & \leq C_{U}^{(1)} \Delta_{\pi} \\ & \leq C_{U}^{(1)} \kappa^{-{3}/{2}} \mathsf{D}(y)^{3/2} \\ & \leq \frac{C_{U}^{(1)}}{c \cdot \kappa^{{3}/{2}}} \norm{\grad_{\mu} \mathcal{L}(\mu, \bar{\pi})}. \end{align}\] Here the first inequality follows from ?? , the second inequality follows from the inclusion \(\mu \in \mathcal{A}_{\tau, \kappa}\) and the last inequality follows from Item [item:reference-gradient-pl-like]. This proves Item [item:improved-gradient-perturbation-in-annulus]. ◻

Proof. Define \(h(\mu) := \mathcal{L}(\mu, \pi) - \mathcal{L}(\mu, \bar{\pi})\); at \(\mu^{\star}\), \(h(\mu) = \mathcal{L}(\mu^{\star}, \pi)\) since the second loss term vanishes. We now analyze the growth of \(h\) near \(\mu^{\star}\). For any \(y \in \mathcal{M}\) near \(\mu^{\star}\), 1 yields \[y = \mu^{\star} + u + O(\norm{y - \mu^{\star}}^2), \;\; \text{where u \in \mathcal{T}_{\mu^{\star}}\mathcal{M}.} \label{eq:local-C2-expansion-again}\tag{65}\] Using this estimate in a second-order Taylor expansion of \(h(\mu)\), we deduce that \[\begin{align} h(y) - h(\mu^{\star}) & = \ip{\grad h(\mu^{\star}), u} + \frac{1}{2} \ip{u, \grad^2 h(\mu^{\star}), u} + O(\Delta_{\pi} \norm{y - \mu^{\star}}^2), \label{eq:function-gap-second-order} \end{align}\tag{66}\] where the last term in 66 follows from the following claim:

We now simplify the terms on the RHS of 66 . We focus on the gradient first.

With 2 at hand, we turn to the quadratic form induced by the Hessian. We have \[\begin{align} \abs{\ip{u, \grad^2 h(\mu^{\star}) u}} & = \abs{\ip{u, (\grad^2 \mathcal{L}(\mu^{\star}, \pi) - \grad^2 \mathcal{L}(\mu^{\star}, \bar{\pi})) u}} \\ & \leq \norm{u}^2 \cdot C_{U}^{(2)} \Delta_{\pi} \\ & \lesssim \norm{y - \mu^{\star}}^2 C_{U}^{(2)} \Delta_{\pi} \\ & \lesssim \mathsf{D}(y) \cdot \Delta_{\pi}, \end{align}\] where the penultimate inequality follows from the tangent-normal decomposition of \(y\) and the last inequality follows from 52 . We conclude that, for \(y \in \mathcal{M} \cap V\), where \(V\) is defined in 9, \[h(y) - h(\mu^{\star}) = O(\Delta_{\pi} \cdot \mathsf{D}(y)) \label{eq:grad-diff-on-manifold}\tag{67}\] Now, suppose that \(\mu \notin \mathcal{M}\) with \(y = P_{\mathcal{M}}(\mu)\). By the mean value theorem, we have for \(t\in[0,1]\) \[\begin{align} \abs{h(\mu) - h(y)} & = \abs{\ip{\grad h(y + t(\mu - y)), \mu - y}} \\ & \leq \norm{\grad h(y + t(\mu - y))} \cdot \dist_{\mathcal{M}}(\mu) \\ & \leq C_{U}^{(1)} \Delta_{\pi} \cdot \dist_{\mathcal{M}}(\mu), \end{align}\] where the last inequality follows from the estimate ?? . From this and 67 , we deduce \[\abs{h(\mu) - h(\mu^{\star})} \leq C \Delta_{\pi} \left(\dist_{\mathcal{M}}(\mu) + \mathsf{D}(y)\right). \label{eq:h-diff}\tag{68}\] Finally, the derivation used between 42 and 43 in 6 supplies the bound \[h(\mu^{\star}) = \mathcal{L}(\mu^{\star}, \pi) - \mathcal{L}(\theta^{\star}) \lesssim \Delta_{\pi}^2,\] from which 62 immediately follows: \[\abs{h(\mu)} \leq C\Delta_{\pi} \left(\Delta_{\pi}+\mathsf{D}(y)+\dist_{\mathcal{M}}(\mu)\right).\]

To prove 63 , recall \(\mu \in \mathcal{A}_{\tau,\kappa}\) yields \[\mathsf{D}(y) \geq \kappa \Delta_{\pi}^{2/3}, \;\; \text{and} \;\; \dist_{\mathcal{M}}(\mu) \leq \tau \mathsf{D}(y)^{3/2}.\] Plugging these into the right-hand side of 62 leads to \[\begin{align} \abs{h(\mu)} & \lesssim \kappa^{-{3}/{2}}\mathsf{D}(y)^{{5}/{2}} \left( 1 + \kappa^{-{3}/{2}}\mathsf{D}(y)^{{1}/{2}} + \tau \mathsf{D}(y)^{{1}/{2}} \right) \\ & \leq C \cdot \kappa^{-3/2} \mathsf{D}(y)^{5/2}, \end{align}\] assuming \(\mathsf{D}(y)\) is sufficiently small. From proximity to \(\mu^{\star}\), the growth bound from 4 and the bias-dispersion decomposition from 3, it follows that \[\mathsf{D}(y)^2 \lesssim \frac{1}{m \pi_{\min}} \left( \mathcal{L}(\mu, \bar{\pi}) - \mathcal{L}(\theta^{\star}) \right) \implies \abs{h(\mu)} \lesssim C \cdot \frac{(\mathcal{L}(\mu, \bar{\pi}) - \mathcal{L}(\theta^{\star}))^{5/4}}{\kappa^{3/2}}.\] This proves 63 , from which 64 easily follows. ◻

Proof. Fix \(\mu \in V \cap \mathcal{A}_{\tau, \kappa}\) and denote \(h(\mu) := \mathcal{L}(\mu, \pi) - \mathcal{L}(\mu, \bar{\pi})\). We have \[\begin{align} \abs{h(\mu)} & \leq \frac{C}{\kappa^{3/2}} \left(\mathcal{L}(\mu, \bar{\pi}) - \mathcal{L}(\theta^{\star})\right)^{5/4}, \tag{72} \\ \norm{\grad h(\mu)} & \leq \frac{C}{\kappa^{3/2}} \norm{\grad \mathcal{L}(\mu, \bar{\pi})}, \tag{73} \end{align}\] with 72 following from 63 and 73 following from Item ([item:improved-gradient-perturbation-in-annulus]) of 9.

We now compare the Polyak steps themselves. Since \(\mathcal{L}(\theta^{\star}) = 0\), we have \[\begin{align} \norm{\mu^{+} - \bar{\mu}^{+}} & = \norm[\Big]{ \underbrace{\frac{\mathcal{L}(\mu, \pi)}{\norm{\grad \mathcal{L}(\mu, \pi)}^2}}_{\zeta_{\pi}} \cdot \grad \mathcal{L}(\mu, \pi) - \underbrace{\frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}^2}}_{\zeta_{\bar{\pi}}} \cdot \grad \mathcal{L}(\mu, \bar{\pi}) } \notag \\ & \leq \abs{\zeta_{\pi} - \zeta_{\bar{\pi}}} \norm{\grad \mathcal{L}(\mu, \bar{\pi})} + \zeta_{\pi} \cdot \norm{\grad h(\mu)}. \label{eq:polyak-step-difference} \end{align}\tag{74}\] We handle each term in 74 separately. First, we argue that \[\begin{align} \zeta_{\pi} \norm{\grad h(\mu)} & = \left[ \frac{h(\mu)}{\norm{\grad \mathcal{L}(\mu, \pi)}^2} + \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \pi)}^2} \right] \cdot \norm{\grad h(\mu)} \notag \\ & \leq \left[ \frac{C \kappa^{-3/2} \mathcal{L}(\mu, \bar{\pi})^{5/4} + \mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \pi)}^2} \right] \cdot \norm{\grad h(\mu)} \notag \\ & \leq (1 + \epsilon) \cdot \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \pi)}^2} \cdot \frac{C}{\kappa^{3/2}} \norm{\grad \mathcal{L}(\mu, \bar{\pi})} \notag \\ & \leq \frac{(1 + \epsilon) C \kappa^{-3/2}}{(1 - C \kappa^{-3/2})^2} \cdot \zeta_{\bar{\pi}} \norm{\grad \mathcal{L}(\mu, \bar{\pi})} \notag \\ & \leq \frac{C}{\kappa^{3/2}} \cdot \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}}, \label{eq:polyak-step-diff-i} \end{align}\tag{75}\] where the first inequality follows from 72 , the second inequality follows from 73 and the fact that \(\mathcal{L}(\mu, \bar{\pi})\) dominates \(\mathcal{L}(\mu, \bar{\pi})^{5/4}\) when \(\mu\) is sufficiently close to \(\mu^{\star}\), the penultimate inequality again follows from 73 , and the last inequality follows from adjusting \(\kappa\) and \(C\) and relabeling.

Now, we bound \(\abs{\zeta_{\pi} - \zeta_{\bar{\pi}}}\). Indeed, we obtain the difference \[\begin{align} \abs{\zeta_{\pi} - \zeta_{\bar{\pi}}} & = \abs*{\frac{\mathcal{L}(\mu, \pi)}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}^2 (1 + O(C\kappa^{-3/2}))^2} - \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}^2}} \notag \\ & = \abs*{ \frac{h(\mu)}{\norm{\grad \mathcal{L}(\mu,\bar{\pi})}^2 (1 + O(C \kappa^{-3/2}))^2} - \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}^2} \left[ 1 - \frac{1}{(1 + O(C \kappa^{-3/2}))^2} \right] } \notag \\ & \lesssim \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}^2} \abs*{ \frac{(1 + O(C \kappa^{-3/2}))^2 - 1}{(1 + O(C \kappa^{-3/2}))^2} } \notag \\ & \leq \frac{C}{\kappa^{3/2}} \cdot \zeta_{\bar{\pi}}, \label{eq:polyak-step-diff-ii} \end{align}\tag{76}\] again using 72 and 73 , as well as adjusting \(\kappa\) and \(C\) and relabeling if necessary. Finally, we plug 75 76 into 74 to obtain the first inequality in 71 : \[\begin{align} \norm{\mu^{+} - \bar{\mu}^{+}} & \leq \frac{2C}{\kappa^{3/2}} \cdot \frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}}. \end{align}\]

We now prove the second inequality in 71 . From the tangent-normal expansion, we deduce \[\begin{align} \mathcal{L}(\mu, \bar{\pi}) & = \mathcal{L}_{T}(y) + \mathcal{L}_{N}(\mu) \lesssim \mathsf{D}(y)^2 + \dist^2_{\mathcal{M}}(\mu) \lesssim \mathsf{D}(y)^2 + \tau^2 \mathsf{D}(y)^3, \end{align}\] where the first inequality follows from Item ([item:reference-ravine-tangent-growth]) of 9 and the fact that \(\mathcal{L}_{N}\) is \(C^{\infty}\) and minimized on \(\mathcal{M}\), and the second inequality follows from \(\mu \in \mathcal{T}_{\tau}\). After shrinking \(V\) if necessary, the term \(\tau^2\mathsf{D}(y)^3\) is absorbed into \(\mathsf{D}(y)^2\). On the other hand, Item ([item:reference-gradient-pl-like]) of 9 gives \[\norm{\grad \mathcal{L}(\mu, \bar{\pi})} \gtrsim \mathsf{D}(y)^{3/2}.\] using Item ([item:reference-gradient-pl-like]) of 9. Combining the two inequalities yields \[\frac{\mathcal{L}(\mu, \bar{\pi})}{\norm{\grad \mathcal{L}(\mu, \bar{\pi})}} \lesssim \mathsf{D}(y)^{1/2} \lesssim \norm{P_{\mathcal{M}}(\mu) - \mu^{\star}},\] where the last inequality follows from Item ([item:reference-ravine-tangent-growth]) of 9. This proves 71 , after potentially enlarging the constant \(C > 0\). ◻

Proof. We first analyze one step of gradient descent for the reference objective. Since \(\mathcal{M}\) is a local \(C^\infty\) Morse ravine \(\mathcal{L}(\mu, \bar{\pi})\), there is a neighborhood \(U\) of \(\mu^{\star}\) on which \[\mu = y + \xi, \;\; \text{where} \;\; y = P_{\mathcal{M}}\mu, \; \xi \perp \mathcal{T}_{y}\mathcal{M} \;\; \text{and} \;\; \norm{\xi} \leq \dist_{\mathcal{M}}(\mu). \label{eq:tangent-normal-expansion}\tag{78}\] Consequently, Corollary 2, with the coefficient \(\rho>0\), shows that there exists \(c>0\) such that \[\begin{align} \label{eq:reference-ravine-normal-splitting} \norm{\xi-\eta\grad_{\mu} \mathcal{L}(\mu,\bar{\pi})} \leq (1 - c \eta) \dist_{\mathcal{M}}(\mu) + \rho \eta \cdot \dist(y, \mathcal{S}^{\star})^{3}, \end{align}\tag{79}\] for all \(\mu \in U\) and \(\eta \in (0, \eta_{\mathsf{ub}})\); the exponent \(p-1 = 3\) since \(\mathcal{L}(\mu, \bar{\pi})\) has \(4^{\text{th}}\)-order growth near \(\mu_{\star}\).

We now analyze a step of gradient descent on the mismatched objective. We obtain \[\begin{align} \dist_{\mathcal{M}}(\mu^{+}) & = \inf_{u \in \mathcal{M}} \norm{\mu^{+} - u} \notag \\ & \leq \norm{\mu^{+} - P_{\mathcal{M}} \mu} \notag \\ & = \norm{\mu - P_{\mathcal{M}}\mu - \eta \grad_{\mu} \mathcal{L}(\mu, \pi)} \notag \\ & \leq \norm{\xi - \eta \grad_{\mu} \mathcal{L}(\mu, \bar{\pi})} + \eta C_{U}^{(1)} \Delta_{\pi} \notag \\ & \leq (1 - c \eta) \dist_{\mathcal{M}}(\mu) + \rho \eta \cdot \dist(y, \mathcal{S}^{\star})^{3} + \eta C_{U}^{(1)} \Delta_{\pi}, \label{eq:ravine-normal-splitting-master-bound} \end{align}\tag{80}\] where the penultimate inequality follows from 78 and Item (ii) of 12 and the last inequality follows from 79 . It remains to show that the last two terms on the RHS are controlled by \(\mathsf{D}(y)^{3/2}\).

We focus on \(\Delta_{\pi}\) first. Since \(\mu \notin \mathcal{C}_{\kappa}\), we have \[\Delta_{\pi}^{2/3} \leq \frac{1}{\kappa} \mathsf{D}(y) \quad\Longrightarrow\quad \eta C_{U}^{(1)} \Delta_{\pi} \leq \eta C_{U}^{(1)}\kappa^{-3/2}\mathsf{D}(y)^{3/2}. \label{eq:delta-pi-residual-dominated}\tag{81}\] At the same time, Item ([item:reference-ravine-tangent-growth]) of 9 shows that \[\dist(y,\mathcal{S}^\star)^3 \leq C\mathsf{D}(y)^{3/2}.\] Plugging 81 and the preceding inequality into 80 , and enlarging \(C\) if necessary, we obtain \[\dist_{\mathcal{M}}(\mu^+) \leq (1 - c \eta)\dist_{\mathcal{M}}(\mu) + C\eta(\rho+\kappa^{-3/2})\mathsf{D}(y)^{3/2}. \label{eq:ravine-normal-splitting-master-bound-simplified}\tag{82}\] Now write \(y^{(j)} = P_{\mathcal{M}}(\mu^{(j)})\) and suppose that \(\mu^{(0)}, \dots, \mu^{(k-1)} \notin \mathcal{T}_{\tau}\). This means \[\mathsf{D}(y^{(j)})^{3} \leq \frac{\dist_{\mathcal{M}}^2(\mu^{(j)})}{\tau^2}.\] Plugging this back into 82 , we obtain the one-step improvement \[\dist_{\mathcal{M}}(\mu^{(j+1)}) \leq \left[ 1 - c \eta + \frac{C \eta}{\tau} \left(\rho + \frac{1}{\kappa^{3/2}}\right) \right] \cdot \dist_{\mathcal{M}}(\mu^{(j)}).\] By the first condition in 77 , the last term inside the contraction factor is less than \(\tfrac{c}{2}\). Therefore, \[\dist_{\mathcal{M}}(\mu^{(j+1)}) \le (1-\tfrac c2\eta) \dist_{\mathcal{M}}(\mu^{(j)}).\] Applying the preceding estimate successively for \(j=0,\dots,k-1\), we obtain \[\begin{align} \dist_{\mathcal{M}}(\mu^{(k)}) & \leq (1-\tfrac c2\eta)^k\dist_{\mathcal{M}}(\mu^{(0)}). \end{align}\] By the second condition in 77 , \[\dist_{\mathcal{M}}(\mu^{(k)}) \le \tau\mathsf{D}(y^{(k)})^{3/2}.\] We conclude that \(\mu^{(k)}\in\mathcal{T}_\tau\), whence \(\mu^{(k)} \in \mathcal{A}_{\tau, \kappa}\) as \(\mu^{(k)} \in \mathcal{C}_{\kappa}\) by assumption. ◻

Proof. Fix \(\kappa\ge \kappa_{0}\), \(\mu\in W\cap \mathcal{A}_{\tau,\kappa}\), and \(y:=P_{\mathcal{M}}(\mu),\) and suppose that \(\mu \notin \mathcal{C}_{\kappa}.\)

We first verify the antecedent condition in ?? . Since \(\mu\in \mathcal{A}_{\tau,\kappa}\subset \mathcal{T}_\tau\), we have \[\dist_{\mathcal{M}}(\mu)\le \tau \mathsf{D}(y)^{3/2}.\] Moreover, 56 combined with 54 shows \[\norm{\grad \mathcal{L}_N(\mu)} \lesssim \dist_{\mathcal{M}}(\mu) \leq C\tau \mathsf{D}(y)^{3/2} \lesssim \tau \norm{\grad \mathcal{L}_{T}(y)} \leq \frac{1}{100} \norm{\grad \mathcal{L}_{T}(y)},\] after shrinking \(\tau\) if necessary. We have the following chain of inequalities: \[\begin{align} \norm{P_{\mathcal{M}}(\mu^+) - \mu^{\star}} & \leq \norm{P_{\mathcal{M}}(\bar{\mu}^+) - \mu^{\star}} + \norm{P_{\mathcal{M}}(\mu^{+}) - P_{\mathcal{M}}(\bar{\mu}^+)} \\ & \leq q \cdot \norm{P_{\mathcal{M}}(\mu) - \mu^{\star}} + C \norm{\mu^{+} - \bar{\mu}^{+}} \\ & \leq \left(q + \frac{C}{\kappa}\right) \norm{P_{\mathcal{M}}(\mu) - \mu^{\star}}, \end{align}\] where the second inequality follows from 13 and local Lipschitz continuity of \(P_{\mathcal{M}}\) and the last inequality follows from 11 and relabeling. Enlarging \(\kappa_{0}\) if necessary gives the desired contraction. ◻

Proof. Let \(y^{(j)}:=P_{\mathcal{M}}(\mu^{(j)})\), \(y := P_{\mathcal{M}}(\mu)\), and \(r:=\norm{y-\mu^\star}\); moreover, assume \(\mu^{(0)} \notin \mathcal{C}_{\kappa}\) (since otherwise the first alternative holds trivially). By 14, we have \[\label{eq:epoch-polyak-projected-contraction} \norm{P_{\mathcal{M}}(\mu^{(0)})-\mu^\star} \le \left(q+\frac{C}{\kappa}\right) \norm{y - \mu^{\star}}.\tag{84}\] After possibly enlarging \(C\), we have the inequality \[\label{eq:epoch-polyak-offshoot-bound} \dist_{\mathcal{M}}(\mu^{(0)}) \le \dist_{\mathcal{M}}(\bar\mu^{(0)}) + \norm{\mu^{(0)}-\bar\mu^{(0)}} \le \left(C + \frac{C}{\kappa}\right) \norm{y - \mu^{\star}} \leq C \norm{y - \mu^{\star}},\tag{85}\] where \(\bar{\mu}^{(0)}\) denotes the corresponding reference Polyak step, the second inequality follows from ?? applied to the reference Polyak step and 71 , and the last inequality uses \(\kappa\ge 1\).

If \(\mu^{(K)}\in\mathcal{C}_\kappa\), then the first alternative holds; henceforth, let \(\mu^{(K)}\notin\mathcal{C}_\kappa\). By definition of \(K\), this implies that \(\mu^{(K)} \in \mathcal{T}_{\tau} \setminus \mathcal{C}_{\kappa} = \mathcal{A}_{\tau, \kappa}\). It remains to show that gradient descent retains the progress made by the Polyak step towards the solution.

We now prove the contraction estimate. Combining 86 with 84 , we obtain \[\norm{P_{\mathcal{M}}(\mu^{(K)})-\mu^\star} \le \left(q+\frac{C}{\kappa}+\varepsilon\right) \norm{P_{\mathcal{M}}(\mu)-\mu^\star}.\] Choosing \(\varepsilon > 0\) sufficiently small and \(\kappa_0\) sufficiently large so that \[q+\frac{C}{\kappa}+\varepsilon =: q' <1 \qquad \text{for all }\kappa\ge\kappa_0,\] we obtain a contraction towards \(\mu^{\star}\) along the manifold for \(\mu^{(K)}\): \[\norm{P_{\mathcal{M}}(\mu^{(K)})-\mu^\star} \le q' \norm{P_{\mathcal{M}}(\mu)-\mu^\star}. \label{eq:almost-ambient-contraction}\tag{90}\] Finally, since \(\mu^{(K)}\in\mathcal{T}_\tau\) by definition of the hitting time \(K\), we have \[\dist_{\mathcal{M}}(\mu^{(K)}) \le \tau \mathsf{D}^{3/2}(y^{(K)}) \lesssim \tau \norm{y^{(K)} - \mu^{\star}}^{3}, \label{eq:last-iterate-distance-from-M}\tag{91}\] where the second inequality follows from Item [item:reference-ravine-tangent-growth] of 9. Consequently, \[\begin{align} \norm{\mu^{(K)} - \mu^{\star}} & \leq \norm{y^{(K)} - \mu^{\star}} + \dist_{\mathcal{M}}(\mu^{(K)}) \\ & \leq (1 + C \tau \norm{y^{(K)} - \mu^{\star}}^2) \norm{y^{(K)} - \mu^{\star}} \\ & \leq (1 + \varepsilon') \cdot q' \cdot \norm{y - \mu^{\star}} \\ &\leq (1 + \varepsilon') \cdot q' \cdot \big[ 1 + C_{W} \norm{\mu - \mu^{\star}} \big] \cdot \norm{\mu - \mu^{\star}}, \end{align}\] where the first inequality follows from the triangle inequality, the second inequality follows from 91 , the third inequality follows from 90 , and the last inequality follows from 8. We label \(q_{\mathrm{amb}} = (1 + \varepsilon') q' \cdot \big[1 + C_{W} \norm{\mu - \mu^{\star}}\big] < 1\) by suitably shrinking \(W\) if necessary. ◻

Proof. Let \(W_{\text{out}} \equiv W\) and \(\tau_{W}\) denote the following stopping time: \[\tau_{W} := \inf\set{k \in \mathbb{N} \mid \mu^{(k)} \notin W_{\text{out}} },\] and suppose that \(\tau_{W} < K < \infty\), where \(K\) is the stopping time defined in 7. By definition, \(\mu^{(0)}, \dots, \mu^{(\tau_W - 1)} \in W_{\mathrm{out}}\). For the iterate \(\mu^{(\tau_W)}\) itself, we have \[\begin{align} \norm{\mu^{(\tau_W)} - \mu^{\star}} &\leq \norm{P_{\mathcal{M}}(\mu^{(\tau_W)}) - \mu^{\star}} + \dist_{\mathcal{M}}(\mu^{(\tau_W)}) \\ &\leq \norm{y^{(\tau_W)} - \mu^{\star}} + \left(1 - \frac{c \eta}{2}\right)^{(\tau_W)} \dist_{\mathcal{M}}(\mu^{(0)}) \\ &\leq \norm{y^{(0)} - \mu^{\star}} + \dist_{\mathcal{M}}(\mu^{(0)}) + \varepsilon \norm{y - \mu^{\star}} \\ &\leq (1 + \varepsilon) \norm{y - \mu^{\star}} + \dist_{\mathcal{M}}(\mu^{(0)}) \\ &\leq (1 + \varepsilon + C) \norm{y - \mu^{\star}} \\ &\leq (1 + \varepsilon + C) \left[1 + C_{W} \norm{\mu - \mu^{\star}} \right]\norm{\mu - \mu^{\star}} \end{align}\] where the second inequality follows from 12, the third inequality holds for any \(\varepsilon > 0\) by suitably shrinking \(W\) (following the proof of 3), the fourth inequality follows from 14, the penultimate inequality follows from 85 , and the last inequality follows from 8. By suitably shrinking \(W\) and defining \(W_{\text{in}} = (1 + \varepsilon + C)^{-1} W_{\text{out}} \subset W\), we arrive at a contradiction; therefore, \(\mu^{(j)} \in W_{\text{out}}\) for all \(j \leq K\). ◻