1 Introduction↩︎

Associative memory—the ability to retrieve patterns from noisy or partial inputs—is a fundamental brain function, enabling the retrieval of memories from imperfect sensory stimuli. This type of content-addressable memory can be modeled by recurrent neural networks called “Hopfield networks”, for which their inventor John Hopfield was recognized with the Nobel Prize in physics in 2024 [1], [2]. Recently, continuous-valued extensions of Hopfield networks have also found renewed application in machine learning [3].

Despite decades of development since their first introduction in 1982, the principles underlying associative memory formation remain incompletely understood. Originally, Hopfield networks were trained using the biologically-inspired Hebbian learning rule based on firing coincidences. Since then, new learning rules have been introduced that display improved memory capacity and stability  [4], [5]. Nevertheless, a key question remains: Is there an underlying principle that governs the formation of associative memory? And, if so, can it be exploited directly to improve performance?

To answer these questions, we propose to analyze Hopfield networks from an information processing perspective. Hopfield networks store patterns as attractors of their neural dynamics, created by training the network’s weights using the patterns as a teaching signal. How the information of the recurrent dynamics and the teaching signal together predicts the neuron’s firing thus becomes pivotal to the network’s performance. Describing this relation in an abstract, implementation-independent manner can be achieved using the framework of information theory [6], [7]. In particular, the mutual information quantifies general statistical dependencies between two variables, enabling the quantification of the amount of information in a neuron’s output that is contained in the input from other neurons (recurrent input) or in the teaching signal (target input). However, using only mutual information it is impossible to differentiate how that information is contributed in unique, redundant, or synergistic ways. We can differentiate these input contributions using a recent extension of information theory called Partial Information Decomposition (PID, [8]).

In this work, we use PID as a tool to describe the individual neuron’s function in a Hopfield network in both an analytic and constructive manner. The analysis of classical Hebbian Hopfield networks reveals that neurons show high redundancy in their firing between recurrent and teaching inputs. Building on this insight, we then use PID to construct novel local goal functions which optimize for redundancy and other information processing goals directly.

Information theory and the concept of PID have been used to analyze information processing in artificial neural networks in the past [9]–[12]. Furthermore, it has been shown that “infomorphic” neurons which train by directly optimizing a PID goal function are in principle capable of solving tasks from different learning paradigms [13] and have been demonstrated to achieve high performance and local interpretability for supervised learning tasks in particular [14].

The main contributions of this work are (i) an information-theoretic analysis of a classical Hebbian Hopfield network which reveals that redundancy between recurrent input and target (teaching signal) dominates the neurons’ output when operating below capacity, (ii) the construction of “infomorphic” associative memory neurons that directly maximize a local PID goal function, building on and improving the proof of concept provided by [13], and (iii) experimental results showcasing that infomorphic neurons that maximize redundancy significantly outperform the original Hebbian learning rule and other previous learning methods.

2 Methodological Approach↩︎

2.1 Hopfield Networks↩︎

A Hopfield network consists of \(N\) recurrently connected neurons. Initialized with a noisy or incomplete version of a stored pattern, the network iteratively updates neuron states to minimize an energy function, converging to a stable state that corresponds to the stored pattern. This allows for pattern completion and error correction through memory retrieval (see 1).

The neurons are updated based on the activity of other neurons in the network. Given a network state vector \(\{y_i(t)\}_i\) at time \(t\), the state of neuron \(i\) at the next timestep is computed as \[\label{eq:NeuronUpdate} y_i(t+1) = \sgn\left( r_i \right),\tag{1}\] where \(\sgn(x)\) denotes the sign function and the recurrent input \(r_i\) is defined as a weighted sum of the activities of all neurons \(y_j\) as \[r_i = \sum_{j=1}^N w_{ij} y_j(t).\]

The originally proposed training method for the weights \(\{w_{ij}\}_{ij}\) was Hebbian learning. In this learning rule, neurons which co-activate as part of the same pattern become more strongly connected, making the patterns attractors in the energy landscape. For bipolar patterns \(\boldsymbol{\xi}^{(p)}\) (see 1A for example patterns) with pattern index \(p\) (i.e., with pattern elements \(\xi_i^{(p)} = \pm1\)) the resulting weights using the Hebbian learning rule are given by [15] \[\label{eq:HebbBinary} w_{ij} = \sum_{p=1}^m \xi_i^{(p)}\xi_j^{(p)},\tag{2}\] where \(m\) is the total number of patterns. There are typically no self-connections in the network and therefore \(w_{ii} = 0\). When training on uncorrelated patterns, the amount of memories (patterns) \(m_c\) that a Hopfield network can successfully store scales linearly with network size \(N\). The corresponding memory capacity is defined as \(\alpha_c = \frac{m_c}{N}\) and depends on the learning method. The Hebbian rule has a capacity of \(\alpha_\mathrm{H} \approx 0.14\) [16], lower than the theoretical upper bound of \(\alpha_\mathrm{c} = 2\) [17], [18].

2.2 Partial Information Decomposition↩︎

To understand how individual neurons contribute to associative memory function, we investigate the information processing at a single neuron. From an information-theoretic viewpoint, each neuron can be viewed as a channel that receives information about the outputs of other neurons via the recurrent input \(R\) and produces a bipolar output signal \(Y\). The total entropy of a neuron’s output \(H(Y)\) can be decomposed into two parts, namely the information \(I(Y:R)\) explained by \(R\), and the residual entropy \(H(Y\mid R)\), such that \(H(Y) = I(Y:R)+H(Y\mid R)\).

To evaluate whether this output information is relevant to the memory task, we introduce an auxiliary input variable \(T\) that corresponds to the targeted output for a given pattern and acts as a teaching signal for the neuron (2A). While the total information which the recurrent and the new target input together hold about the neuron output \(Y\) is quantified by the joint mutual information \(I(Y:R,T)\), classical information theory cannot explain how this information is provided by the two source variables \(R\) and \(T\): Some parts of the neuron’s firing might be explainable uniquely by the recurrent input (denoted by \(\Pi_{\unq, {R}}\)) and thus be unrelated to the target input, or uniquely by the target input, i.e., relevant but not encoded in \(R\) (\(\Pi_{\unq, T}\)). Other parts of the output information may be carried redundantly (\(\Pi_{\red}\)) in both sources, meaning they are both relevant and encoded in \(R\), while yet others may be carried synergistically (\(\Pi_{\syn}\)), meaning that both information sources are necessary to uncover this piece of information. Enumerating and quantifying these information atoms is the subject of Partial Information Decomposition (PID) [8], [19].

The four PID atoms are related to the three classical mutual information terms via the so-called consistency equations [8] (see  2C.) \[\begin{align} \label{eq:consistencyeq} I(Y:R) &= \Pi_{\unq,R} + \Pi_{\red} , \\ I(Y:T) &= \Pi_{\unq,T} + \Pi_{\red} , \\ I(Y:R,T) &= \Pi_{\unq,R} + \Pi_{\unq,T} + \Pi_{\red} + \Pi_{\syn}. \end{align}\tag{3}\]

This set of linear equations is underdetermined and leaves open a degree of freedom in the values of atoms \(\Pi\). To resolve this underdetermination, an additional quantity needs to be defined, which is usually a measure for redundancy. Throughout the literature, a plethora of different redundancy measures have been devised, which fulfill distinct requirements and have different operational interpretations e.g. [20] and references therein. Throughout this work, we use the \(I_\cap^\mathrm{sx}\) measure introduced by [21], due to its differentiability, which is essential for optimization (see 5.3). Using PID now allows for a more fine-grained decomposition of the entropy of \(Y\) into five components \[H(Y) = \Pi_{\unq,R} + \Pi_{\unq,T} +\Pi_{\red} + \Pi_{\syn} + H_\text{res},\] where \(H_\text{res} = H(Y\mid R,T)\) denotes the residual entropy of the output \(Y\) not explained by either source variable \(R\) or \(T\) (see Appendix 5.1 for explicit calculations of the information atoms).

2.3 Information-theoretic Learning↩︎

PID provides a valuable framework to analyze the information processing of an individual neuron in a Hopfield network. However, to investigate whether this information processing view provides a sufficient description of the neurons’ functions, PID can also be employed to build novel “infomorphic neurons” [13], which optimize a PID-based goal function directly.

During evaluation, these infomorphic networks operate analogously to classical Hebbian Hopfield networks with the neuron’s activity given by 1 . The network state is initialized in a target pattern, and neurons synchronously update their output based on both inputs until convergence or up to 100 time steps using the same update rule (1 ).

For training only, the true pattern element is introduced as an additional target input \(T\) to each neuron to guide the learning. The recurrent dynamics are updated only once, this time utilizing a sigmoid activation function to produce a firing probability and stochastically assigning \(+1\) or \(-1\) to \(Y\) according to this probability. After this, the joint input distribution is estimated using a differentiable “soft” binning approach (see 8), enabling computation of PID atoms using the discrete \(I^\text{sx}_\cap\) PID measure. Given these PID atoms, a general infomorphic objective function of the form \[\label{eq:loss} G = \gamma_{\unq,R} \Pi_{\unq,R} + \gamma_{\unq,T} \Pi_{\unq,T} + \gamma_{\red} \Pi_{\red} + \gamma_{\syn} \Pi_{\syn} + \gamma_{\res} H_{\res},\tag{4}\] can be evaluated, where \(\gamma_i\) are fixed parameters that control the maximization and minimization of individual information atoms. Finally, the weights of each infomorphic neuron are locally updated by gradient ascent on \(G\) using automatic differentiation.

Performance is assessed using cosine similarity between recalled and target patterns, with accuracy quantified both as an average over all patterns (\(a_\mathrm{cos}\)) and via a threshold measure (\(a_\theta\)) to obtain the fraction of patterns recalled with above \(\theta = 95 \%\) accuracy. The memory capacity (\(\alpha_c\)) is determined by evaluating accuracy at different memory loads using different random seeds, and stability is quantified by measuring the maximum fraction of corrupted bits \(f_\mathrm{max}\) in the initial network state. More details can be found in 5.

3 Results↩︎

3.1 Redundancy in Hebbian Learning↩︎

To understand the information-theoretic footprint of Hebbian Hopfield networks, we first investigate a network with \(500\) neurons trained using Hebbian learning. When computing the PID between the recurrent inputs \(r_i\) to the neuron and the target patterns \(t_i\) introduced as an additional variable with no direct influence on the output (see 2A,B), one can see that the information contributions of the inputs change for different memory loads \(\alpha\): For memory loads below and close to the memory capacity of \(\alpha_\mathrm{H} \approx 0.14\), redundant information is the largest information contribution (see 2D), indicating that high redundant information \(\Pi_{\red}\) coincides with successful learning. Above the memory capacity, redundant information begins to fall while the unique information \(\Pi_{\unq,R}\) begins to rise, indicating that the neurons start to encode more non-relevant information from the recurrent inputs. In Appendix 10 we provide a formal connection between high redundancy and high recall accuracy in Hebbian Hopfield networks as well as empirical results using other PID measures.

Overall, the results suggest that high redundant information may play an important role in successful associative memory function.

3.2 Redundancy as a Computational Goal↩︎

To evaluate whether the information-theoretic description found in 3.1 is sufficient to describe associative memory function, this section introduces a constructive approach to show how maximization of redundancy can be used as a computational goal for associative memory learning and shows that it outperforms the classical Hopfield network.

To this end, we employ “infomorphic neurons” in a training procedure described in 5.3 (see also 3A and B). Motivated by the results of the previous section, we first use the simple neuron goal function \(G = \Pi_{\red}\), i.e., we train each neuron to maximize the redundant information between its inputs about the output.

To evaluate the performance of these infomorphic Hopfield networks, we train networks with \(N=100\) neurons for different numbers of patterns and apply the testing procedure described in 5.4. We find that the network memorizes all patterns up to a memory load of \(\alpha_\mathrm{c}^{\red} = 1.59 \;[1.56, 1.61]\) followed by a sharp drop in recall accuracy above this memory capacity (see 3C). The angular brackets indicate bootstrapped \(95 \%\) confidence intervals of the capacity \(\alpha_c\).

The transition above \(\alpha_\mathrm{c}^{\red}\) can be explained by performing the same PID analysis as in 3.1. As shown in 3D, redundancy is uniformly high below \(\alpha_\mathrm{c}^{\red}\) apart from some finite size effects for low numbers of patterns discussed in 9.2. However, when the threshold is crossed, the information encoded by the neurons is no longer purely redundant. Instead, as the neurons fail to encode only task-relevant redundant information, small amounts of other information atoms, particularly synergistic information \(\Pi_{\syn}\), start to arise.

3.3 Mutual Information Goals↩︎

Besides optimizing redundancy or other PID-based goal functions directly, the infomorphic approach also allows us to compare the former to goal functions based on classical information theory alone. To this end, we consider two main candidate goal functions (4A), with two further variants discussed below: (i) the mutual information between the target inputs and the neuron’s output \(I(Y:T)\) and (ii) the co-information, \(I(Y: R : T)\), which is sometimes used as a classical measure of the tradeoff between redundancy and synergy [22]. Co-information is formally defined as \[\begin{align} I(Y:R:T) &= I(Y:R)+I(Y:T)-I(Y:R,T)\\ &= \Pi_{\red}-\Pi_{\syn}. \end{align}\]

As before, we first evaluate the capacity of the networks trained with the two classical goal functions (4B). While maximizing co-information fails to store any patterns, maximizing target mutual information \(I(Y:T)\) achieves very similar performance to maximizing redundancy and reaches a memory capacity of \(\alpha_{\mathrm{c}} = 1.62 \;[1.61, 1.64]\). In addition, the joint mutual information \(I(Y:T,R)\) reaches a capacity of \(\alpha_{\mathrm{c}} = 0.85 \;[0.84,0.86]\), while the recurrent mutual information \(I(Y:R)\) fails to learn any patterns.

The high performance of the mutual information optimization can be understood by looking at the PID profile shown in 4: The mutual information \(I(Y:T)\) can be decomposed into \(\Pi_{\red}\) and \(\Pi_{\unq,T}\), i.e., into a redundant and a unique contribution. Below the maximum capacity \(\alpha_\mathrm{c}\), the mutual information is maximized only by maximizing the redundant information \(\Pi_{\red}\) between the two inputs, explaining the similar performance to optimizing this redundancy alone. Above that point, the mutual information remains high, but redundancy decreases, matched by an increase in unique information of the target input \(\Pi_{\unq,T}\). The corresponding decrease in accuracy suggests that the mutual information goal works only as long as it results in a maximization of redundancy, further suggesting redundancy as the underlying principle of Hopfield memory function. The effect of varying the unique contribution \(\gamma_{\unq,T}\) in the goal function \(G = \gamma_{\unq,T} \Pi_{\unq,T} + \Pi_{\red}\) indicates that additional maximization of target unique information hardly impacts the memory capacity while negative values lead to a decrease in performance (see 4D).

3.4 Improving the Capacity of the Infomorphic Goal↩︎

While optimizing redundancy has proven sufficient for achieving high associative memory performance, it remains open whether this simple intuitive goal is optimal or whether admixtures of other PID atoms can improve performance even further. To investigate this, we performed a parameter sweep over the \(\gamma\) parameters in the general goal function \(G=\gamma_{\unq,R}\Pi_{\unq,R}+\gamma_{\unq,T}\Pi_{\unq,T}+\gamma_{\red}\Pi_{\red}+\gamma_{\syn}\Pi_{\syn}\), with a fixed parameter for residual entropy \(\gamma_{\res}=0\). We then measured the network capacities for the different optimization goals (see 5).

The resulting performance landscape reveals that the highest capacities are indeed achieved in proximity of the heuristic redundancy goal \(G = \Pi_{\red}\) (5A), consistent with the intuition that in order to memorize a pattern, each neuron’s recurrent activation must align with its target activation. The performance remains mostly stable for positive additions of \(\Pi_{\unq,T}\), as long as the redundant contribution dominates. Suppressing the target-unrelated information of each neuron by choosing negative parameters for both recurrent unique information and synergy, \(\gamma_{\unq,R}\) and \(\gamma_{\syn}\), appears to slightly increase performance, but only if both parameters are approximately equal (5B). In line with this result, the capacity decreases when either or both of them are positive.

Similar to the approach of [14], we further used a hyperparameter optimization technique to find optimal hyperparameter combinations. We used the CMA-ES sampler [23] to find the goal parameters that maximize the capacity of the model. Two of the best goals found with this approach reach capacities of \(\alpha^\mathrm{(i)}_\mathrm{c} = 1.68 \; [1.65, 1.72]\) and \(\alpha^\mathrm{(ii)}_\mathrm{c} = 1.7 \;[1.66,1.71]\), slightly outperforming the heuristic redundancy maximization (\(\alpha_\mathrm{c}^{\red} = 1.59 \;[1.56, 1.61]\)) (see 5C). The two goal functions are \[\begin{align} G^{(i)} &= -0.27\,\Pi_{\unq,T} -0.68\,\Pi_{\unq,R} + 0.68\,\Pi_{\red} - 0.77\,\Pi_{\syn} - 0.80\,H_{\res} \text{\quad and}\\ G^{(ii)} &= 0.48\,\Pi_{\unq,T} - 0.16\,\Pi_{\unq,R} + 0.25\,\Pi_{\red} + 0.04\,\Pi_{\syn} - 0.63\,H_{\res}. \end{align}\] While both goals maximize redundancy, the first suppresses all other contributions and the second only suppresses \(\Pi_{\unq,R}\) and \(H_{\res}\) while it maximizes \(\Pi_{\unq,T}\), similar to the mutual information goal \(I(Y:T)\). The results of the other optimizations can be found in [app:Optimizationresults]. The landscapes around the optimized goals have a similar qualitative structure in the subspace of \(\gamma_{\unq, R}\) and \(\gamma_{\red}\), while the optima are at different values (see 5D).

3.5 Performance Comparison↩︎

Infomorphic Hopfield networks achieve competitive performance compared to other state-of-the-art learning rules for Hopfield networks. To demonstrate this, we selected two learning methods which are also based on goal functions—the minimum probability flow goal (MPF) [4] and descending the L2 norm (Descent L2) [5], [24]). Initially introduced by [26], MPF demonstrates that by minimizing the probability of entering neighboring states of the pattern states \(\xi\), many patterns can be encoded in a robust way. Our approach differs from MPF by introducing local goal functions at the level of each neuron and by explicitly connecting the probabilistic approach to information theory. [5] demonstrate that learning in a Hopfield network can often be implemented as descent of different loss functions that describe the stability of the training patterns. They compare different norms, revealing that minimizing the L2 norm between the pattern and the next evolved time step achieves high memory capacity. Both MPF and Descent L2 reach state-of-the-art capacities above the Hebbian learning rule, but are outperformed by the infomorphic redundancy maximization (see 6A,B).

In addition, we also evaluate the stability of the memories encoded in different networks (6C). The stability is measured by introducing progressively more noise \(f\) into the patterns and testing whether recall of memories is still successful. The area under the curve \(f_\mathrm{max}\) is a proxy for the overall stability of memories and indicates the point up to which \(95\,\%\) of the pattern elements are reconstructed. For more details, see 5.3. The redundancy goal’s profile for stability follows an overall trajectory similar to that of the other learning methods. Note that above \(\alpha \approx 0.8\) stability is reduced to almost zero for all methods.

4 Discussion↩︎

4.0.0.1 Summary

This work presents a new perspective on associative memories by examining the information-processing at the scale of individual neurons. Using PID, we quantify how recurrent and target inputs predict neuronal output. We demonstrate that effective pattern storage requires the recurrent and target inputs to redundantly determine firing, highlighting redundancy as a key local information-processing principle underlying associative memory function. Analyzing classical Hopfield networks (3.1), we find that below the memory capacity, neurons predominantly process information redundantly, while above capacity, redundancy degrades and unique recurrent contributions increase, indicating a loss of task-relevant encoding (2D).

This connection is not coincidental: as we show analytically in Appendix 10, when using a Heaviside activation function, successful recall requires the conditional input distributions \(p(R\mid T{=}\pm w_T)\) to have non-overlapping support across the decision boundary—a condition that is mathematically equivalent to maximal redundancy regardless of how the weights are obtained. We corroborate this finding with a closed-form derivation for Hebbian networks that reproduces the empirical profile of 2D, and holds across alternative PID measures and network sizes (Appendices 10.4 and 9.2).

Building on this insight, we construct infomorphic Hopfield networks where each neuron directly maximizes redundant information between its inputs. These networks significantly outperform classical Hebbian Hopfield networks in memory capacity and stability (3.2). Further optimizing the redundancy-based objective enhances performance by suppressing irrelevant information contributions (3.4, 5C). Using only classical information-theoretic measures, we find that maximizing mutual information between target pattern elements and neuron output yields comparable performance to PID-based models, which can, however, be ascribed to an implicit optimization of redundancy (4).

Across varying memory loads, the observed correspondence between high redundancy and recall accuracy supports our claim that it underlies successful associative memory learning (4C). Finally, benchmarking against state-of-the-art methods confirms that infomorphic Hopfield networks achieve superior memory loads, up to \(\alpha_\mathrm{c} = 1.7 \;[1.66, 1.71]\), while maintaining competitive stability (6).

4.0.0.2 Related Work

The approach of constructing local information-theoretic goal functions to perform associative memory learning has previously been suggested by [13]. In this work, we improve upon these results by replacing the complex custom activation function used by [13] by a sign function during testing, which is replaced by a differentiable sigmoid for training, as well as a novel “soft” binning procedure, and demonstrate that \(G=\Pi_{\red}\) and the classical goal \(I(Y:T)\) both achieve a memory capacity that is fourfold the capacity reported in [13]. Furthermore, we show that a direct optimization of the goal parameters further increases the memory capacity.

In addition, information theory has previously been employed to analyze Hopfield networks. Notably, the mutual information of the target pattern and the internal state \(I(\boldsymbol{\xi}: \boldsymbol{Y})\) has been prominently utilized. [27] initially established \(I(\boldsymbol{\xi}: \boldsymbol{Y})\) as a suitable measure for evaluating the quality of pattern retrieval. [28], [29] subsequently demonstrated that \(I(\boldsymbol{\xi}: \boldsymbol{Y})\) was optimal for a sparsely connected network, which ultimately enhanced the memory capacity. Additionally, information-theoretic concepts have been used to train (deep) neural networks [13], [22], [30]. Previous research has demonstrated the significance of maximizing \(I(\boldsymbol{\xi}: \boldsymbol{Y})\) in enhancing the memory capacity, which in this work we explain by its implicit maximization of redundancy.

4.0.0.3 Limitations and Outlook

Our results demonstrate that maximizing redundancy directly can significantly improve memory capacity for Hopfield networks. However, promoting redundancy alone does not suffice to reach the theoretical optimum \(\alpha_\mathrm{c} = 2\) of twice the number of neurons established by [17]. Beyond classical associative memories, it has been shown that this bound can be exponentially exceeded by dense associative memories, which incorporate a layered architecture to improve performance [31], [32]. Similarly, [14] constructed infomorphic networks with multiple layers, achieving substantial improvements in supervised learning tasks compared to single-layer networks. These findings open an intriguing avenue for future research: leveraging the framework of infomorphic networks to identify the information-processing mechanisms required to realize dense associative memories with hidden units. We hypothesize that the general principle of redundancy will continue to play a central role.

Furthermore, since the infomorphic update rule requires the estimation of the full joint probabilities, the approach is computationally expensive and biologically implausible on its own. However, the main advantage of the infomorphic approach lies in its flexible and interpretable goal functions. In future research, approximations to these learning rules may be devised which more implicitly achieve the principle of redundancy maximization.

A further important direction concerns correlated patterns. While it has been shown in classical Hebbian Hopfield networks that spatial correlation is detrimental to storage capacity [33], we expect that infomorphic Hopfield networks could perform better in this setting, since infomorphic networks can better differentiate which target pattern elements are truly informative of the output. Promising first results in this direction are shown in 11.

Another interesting research direction is to apply the principles of memory found here to complex objectives. For example, curiosity has been theorized as the seeking of novel patterns or information based on memorized patterns [34]. Infomorphic networks could reproduce this behavior by following a mixed goal, maximizing redundancy for memorization and unique information from the target for exploration.

In summary, Partial Information Decomposition (PID) and infomorphic networks form a powerful pair of tools for analysis and construction of associative memory networks. Together, they offer a new level of interpretability at the local level that can be used to assess the strengths and weaknesses of existing learning methods and to inspire the design of novel network architectures with state-of-the-art performance in associative memory tasks.

Code Availability↩︎

The code to reproduce the results of this work is submitted with the paper and will be made public at the date of publication.

Acknowledgements↩︎

We would like to thank the members of the Priesemann and Wibral groups for valuable feedback and discussions. M.B. was supported by the Max Planck Society. A.S., V.N. and V.P. were funded via the MBExC by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy-EXC 2067/1-390729940. A.S., V.N., M.W., and V.P. were supported and funded by the DFG – GRK2906 – project number 502807174 and acknowledge support from the Max Planck Society. V.N. was partly supported by the Else Kröner Fresenius Foundation via the Else Kröner Fresenius Center for Optogenetic Therapies. D.A.E. and M.W. were supported by a funding from the Ministry for Science and Education of Lower Saxony and the Volkswagen Foundation through the “Niedersächsisches Vorab” under the program “Big Data in den Lebenswissenschaften” – project “Deep learning techniques for association studies of transcriptome and systems dynamics in tissue morphogenesis”. M.W. and A.M. are employed at the Campus Institute for Dynamics of Biological Networks (CIDBN) funded by the Volkswagenstiftung. M.W., V.P. and A.M. received funding from the DFG via the SFB 1528 “Cognition of Interaction” - project-ID 454648639. M.W. was supported by the flagship science initiative of the European Commission’s Future and Emerging Technologies program under the Human Brain project, HBP-SP3.1-SGA1-T3.6.1. M.G. received a PhD scholarship by the Champalimaud Foundation.

5 Extended Methods↩︎

5.1 Calculating PID atoms↩︎

In the case of two source variables, the calculation of all PID atoms follows from the three consistency equations (see 3 ) \[\begin{align} I(Y:R) &= \Pi_{\unq,R} + \Pi_{\red} , \\ I(Y:T) &= \Pi_{\unq,T} + \Pi_{\red} , \\ I(Y:R,T) &= \Pi_{\unq,R} + \Pi_{\unq,T} + \Pi_{\red} + \Pi_{\syn}. \end{align}\] and one additional axiom. In our case, we define redundant information using the \(I_\cap^\mathrm{sx}\) measure introduced by [21], obtaining \[\begin{align} \label{eq:Isx} \Pi_{\red} := I_\cap^\mathrm{sx}(Y:R;T) &:= \sum_{r,t,y} p(r,t,y) i_\cap^\mathrm{sx}(y:r;t)\\ &:= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r\cup t)}{p(y)p(r\cup t)}\\ &:= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r) + p(y,t) - p(y,r,t)}{p(y)[p(r)+p(t)-p(r,t)]}. \end{align}\tag{5}\] Given this definition, the other atoms are calculated via Moebius inversion: \[\begin{align} \label{eq:Moebius} \Pi_{\unq,R} &= I(Y:R) - \Pi_{\red} , \\ \Pi_{\unq,T} &= I(Y:T) - \Pi_{\red} , \\ \Pi_{\syn} &= I(Y:R,T) - \Pi_{\unq,R} - \Pi_{\unq,T} - \Pi_{\red}. \end{align}\tag{6}\]

5.2 Infomorphic Hopfield Network Set-up↩︎

The architecture of infomorphic Hopfield networks is similar to that of the classical Hopfield network introduced in 2.1. The number of infomorphic neurons \(N\) corresponds to the number of pattern elements of the target patterns to be stored in the recurrent weights of the network. During training, each neuron receives two different inputs. The first input is a weighted sum of the outputs of the recurrently connected neurons \[\label{eq:lateralInput} r_i = \sum_j^N w_{ij}^R y_j.\tag{7}\] The second, target input \(T\) is given by the (weighted) element of the target pattern that the neuron encodes, i.e., \[t_i = w_{i}^T \xi_i^{(p)}.\] Given these two inputs, the probability of neuron \(i\) returning state \(+1\) is given by \[\begin{align} p(y_i=+1) = \sigma\left(r_i + t_i\right), \label{eq:stochastic} \end{align}\tag{8}\] with \(\sigma\) indicating the sigmoid function, while the probability of state \(-1\) is given by \(1-p(y_i=+1)\).

5.3 Training↩︎

By treating the inputs and the output as random variables, we apply PID to optimize specific information-processing objectives. Before training, the network and optimizer are initialized according to 1 in 5.5.

The training procedure itself begins by initializing the network state in the target pattern \(\xi^{(p)}\) and presenting single elements \(\xi^{(p)}_i\) of the same pattern via the target input \(t\) as teaching signal to each corresponding neuron \(i\). The network is then run for one time step in a synchronous manner to obtain the recurrent inputs \(r\) and the output probabilities \(p(Y = y \mid R = r, T = t)\) according to 8 . Over all patterns, we estimate the empirical joint probability mass functions \(p(R = r, T = t)\) of \(R\) and \(T\) for each neuron using the (soft) plug-in binning method described in 8.

Given these probability estimates, the PID terms are computed as described in Appendix 5.1. Then, the objective function \(G\) is evaluated according to 4 . Finally, the recurrent weights \(\boldsymbol{w}_R\) of each neuron are updated via gradient ascent on \(G\), using automatic differentiation. While in principle the target weight \(w_T\) of each neuron could also be updated, in practice a constant weight of order 1 resulted in fastest learning and best performance. Specifically, we chose a constant \(w_T = 2.3\).

5.4 Testing↩︎

To evaluate the performance of the trained network, we use a procedure similar to Hopfield. First, the neurons are initialized in the pattern \(\xi^{(p)}\). As before, the recurrent inputs are given by 7 , but the neurons do not receive target input signals. The bipolar output of the neurons is then given by \[y_i(t+1) = \sgn(r(t))\] which is identical to the update rule in 1 . This step is repeated until the network converges to a fixed point or limit cycle, or for a maximum of 100 iterations. The network’s performance is then evaluated based on the cosine similarity between the final network state \(\mbf y(t_f)\) and the target pattern \(\boldsymbol{\xi}_i^{(p)}\). We use two measures to aggregate the individual similarities and arrive at an overall accuracy score. The first is the mean accuracy \[\begin{align} a_{\mathrm cos} = \frac{1}{P}\sum_p \langle(\mbf y(t_f),\boldsymbol{\xi}^{(p)}) \rangle, \end{align}\] where we average alignment both across neurons and across patterns. The second is the threshold accuracy \(a_\theta\) \[\begin{align} a_\theta = \frac{1}{P}\sum_p \mathcal{H}(\langle(\mbf y(t_f),\boldsymbol{\xi}^{(p)}) \rangle -\theta), \end{align}\] where \(\mathcal{H}\) is the Heaviside function and \(\theta\) the threshold parameter, which we set to \(\theta =0.95\). For this measure to be non-zero, at least some patterns must be restored with high accuracy.

To arrive at the memory capacity of a given learning method, we validate it on 20 different seeds. On each seed, we train networks with memory loads \(\alpha\) increasing with a step size of \(0.02\). We take as the memory capacity the value \(\alpha_\mathrm{c}\), at which for all \(\alpha < \alpha_\mathrm{c}\), the network encodes memories with \(a_\mathrm{cos}(\alpha)>0.95\) (excluding finite size effects). Unless mentioned otherwise, we take the median over all validation seeds to arrive at a memory capacity.

5.4.1 Stability↩︎

In addition to estimating the performance, we also estimate the stability of the stored memories. We test the stability by introducing an increasing fraction of flips \(f\) to a pattern \(\xi^{(p)}\) to initialize the network state and then measuring the accuracy \(a_\mathrm{cos}(f)\). The stability \(f_\mathrm{max}\) is given by the maximum fraction of flips under which the original memory can still be recovered with an accuracy \(a_\mathrm{cos}(f_{max}) \geq \epsilon\) of at least \(\epsilon=0.95\). By varying the memory load \(\alpha\) before measuring the stability, we can obtain a full stability profile \(f_\mathrm{max}(\alpha)\). The area under the curve \(f_\mathrm{max}(\alpha)\) can be viewed as a proxy for the overall level of stability achieved by a given rule. The choice of parameters is taken as in [5].

5.5 Model Parameters↩︎

In 1, we explain the parameters of infomorphic networks and their values that were used during training unless specified otherwise.

6 Pseudocode for Training↩︎

7 Optimization Results↩︎

Each run of the optimization algorithm takes place on a single seed, producing a set of best goals which cluster close together. We have taken one candidate goal function from each of the 10 optimization results and verified them on a set of 20 different seeds. The results are depicted in 5. Out of these ten candidate goals, the best two were chosen for 6.

8 Binning↩︎

8.1 Differentiable Binning↩︎

The PID terms are estimated using a discrete PID measure due to the lack of a differentiable PID measure for mixed discrete-continuous variables [35]. Thus, the continuous-valued inputs \(r\) and \(t\) are binned to allow estimating a discrete probability distribution \(p(r,t)\). Since this operation is not differentiable, previous approaches to estimating the PID terms assumed that \(p(r,t)\) is constant with respect to the weights \(w_R\) and \(\boldsymbol{w}_T\), resorting to estimating \(p(r,t)\) from a static histogram [36]. Taking into consideration the small changes in \(p(r,t)\) with respect to the weights, however, might improve the PID estimates. Therefore, we implemented a kernel-based ‘soft binning’ approach that smooths the histogram at the bin edges providing estimates of \(p(r,t)\) gradients (11).

8.2 Soft Binning Procedure↩︎

The soft binning procedure is composed of three steps. First, for each bin the distances \(d\) between its edges and the coordinates of every sample \((r,t)\) in all dimensions is calculated. Then, each bin is assigned weights according to a kernel \(K\) for every sample. The choice of \(K\) and its properties are discussed in 8.3. Finally, the bin weights obtained for every sample are normalized, so that the total count of the histogram adds up to the number of samples.

8.3 Kernel choice↩︎

The kernel \(K(d)\) should be differentiable, monotonically fall to zero as distance of the bin to the sample increases and approach a delta function as its width parameter decreases. We chose a sigmoid kernel \(K(d)= \Pi_{i \in \{x,y\}}\sigma((c_i/2-d)/l)\) that satisfies all of these constraints, where \(c_x\) and \(c_y\) are the the histogram sizes and \(l\) is the width parameter. As \(l\) tends to zero, the non-differentiable binning values are recovered (11).

Note that the normalization step in the binning procedure ensures that each sample has an equal impact on the total results. Otherwise, especially when \(l\) is smaller than half a bin size, samples that by chance lie closer to the center of a bin are more impactful than those that lie closer to an edge.

Since the size of the histogram scales throughout learning, \(K(d)\) is also scaled in proportion to the bin size. This is done in each epoch by resetting the width parameter \(l = f_{l}\sqrt{(c_x/2)^2+(c_y/2)^2}\) for a fixed fraction \(f_{l}\) and with side lengths of a bin \(c_x\) and \(c_y\).

9 Scaling Behavior↩︎

9.1 Scaling of Capacity and Stability↩︎

One of the fundamental properties of Hopfield networks is that the memory capacity \(\alpha_c\) scales linearly with the network size \(N\). To support this claim for infomorphic Hopfield networks, we repeated the experiments from 3 and 6 at different network sizes (12). Both the memory load \(\alpha\) and the stability \(f_\mathrm{max}\) are defined as fractions of the network size \(N\) and should thus produce the same curve at different \(N\). For the memory capacity, we compare at sizes \(N \in [50,100,200,400,800]\). For sizes \(50-400\), all hyperparameters are unchanged, and the transition around \(\alpha_c^\mathrm{red} \approx 1.59\) stays the same. For \(N=800\), we repeated the experiments twice with different values for \(|\boldsymbol{w}_T| \in \{2.3,3\}\). For \(|\boldsymbol{w}_T| = 2.3\), the capacity falls to \(\alpha_c \approx 1.41\), while it increases to \(\alpha_c \approx 1.66\) for \(|\boldsymbol{w}_T| = 3\). These moderate changes indicate that important parameters may have different optimal values at different scales, and that in particular the external target weights have to be adapted to the amount of internal recurrent input a neuron receives.

We find that stability also scales linearly with network size. There are small increases in \(f_\mathrm{max}\) as a function of \(\alpha\) across most memory loads. However, only the maximum memory load with \(f_\mathrm{max}>0\) keeps increasing with network size. The other increases in \(f_\mathrm{max}\) saturate for large networks.

In addition, the redundancy estimates for Hebbian networks are also independent of network size (see 13).

9.2 Finite Size Effects↩︎

At small network sizes and very small memory loads, a small portion of neurons will receive constant target inputs with an entropy of \(H(T)=0\) and thus \(I(Y:T)=0\) and \(\Pi_\mathrm{red} = 0\), leading to reduced recall accuracy (14). This effect happens if neurons receive the same target input \(T\in\{-w_T,w_T\}\) across all patterns. Writing the absolute number of memory patterns as \(m=\alpha N\), and considering uncorrelated patterns, the expected number of neurons that receive only one target label can be written as \(\langle N_\mathrm{const}\rangle(\alpha) = \frac{N}{2^{\alpha N-1}}\), which goes to 0 for large \(N\).

At finite sizes, however, the number of atoms without any entropy can become significant. In particular, for the low memory loads below the Hebbian capacity \(\alpha_H\) this leads to an underestimation of the average redundancy of the network. This effect can be reduced by increasing the number of neurons while keeping \(\alpha\) fixed (14A). This explains why the finite size effects can only be observed for small memory loads and can be countered by increasing the number of neurons as in 2. For infomorphic neurons, the finite size effects also prevent learning successfully at small memory loads \(\alpha\) in small networks (14 B). Because of the higher capacity of infomorphic networks, they are less important at the memory capacity \(\alpha_c^\mathrm{red}\), such as in 3.

10 Derivation of Redundancy for Associative Memory↩︎

10.1 Redundancy as a Necessary Requirement for Associative Memory↩︎

In the following, we provide further intuition for why high redundancy is found in Hebbian Hopfield networks as well as other learning algorithms.

For successful recall of memories, each neuron’s activity needs to be aligned with its respective target value, i.e. \(Y=1\) iff \(T>0\). Both in the Hebbian and infomorphic Hopfield networks we use a Heaviside activation function in the test phase, implying that \(Y=1\) iff \(R\geq 0\). The goal of training thus becomes to change the recurrent weights such that the conditional distributions for each target value become disjoint and restricted to one quadrant of the \(R\)-\(T\)-plane each, with \(p(r\mid T\geq 0)>0\) only if \(r>0\), and vice versa \(p(r\mid T<0)>0\) only if \(r<0\) (see 16A).

We now show that this separation condition directly implies maximal redundancy. We start from the \(I^\mathrm{sx}_\cap\) redundancy measure given in 5 , which can be written as \[\label{eq:isx95repeat} \begin{align} I_\cap^\mathrm{sx} &= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r\cup t)}{p(y)p(r\cup t)}\\ &= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r) + p(y,t) - p(y,r,t)}{p(y)[p(r)+p(t)-p(r,t)]}. \end{align}\tag{9}\]

We first inspect the second line of 9 . If the two conditional distributions \(p(R\mid T=w_T)\) and \(p(R\mid T=-w_T)\) are restricted to the first and third quadrant, respectively and the output \(Y\) is predicting the target \(T\) correctly, then \(\forall y,r,t: p(y,r,t) > 0 \implies p(y,r,-t) = p(y,-r,t) = 0\). As a result, the union probability in the denominator (indicated by a purple cross in 16) collapses to \[\begin{align} p(r\cup t) &= p(r) + p(t) - p(r,t)\\ &= p(r,t) + p(r,-t) + p(t) - p(r,t)\\ &= p(r\mid -t)p(-t) + p(t)\\ &= p(t) \end{align}\] for all \((r,t)\) with non-zero probability mass \(p(r,t)\). By the same logic, the union probability in the numerator collapses to \(p(y, r\cup t) = p(y, t)=p(t)\). Combining these effects leads to redundant information taking on its maximum value given by the entropy of the target signal as \[\begin{align} I_\cap^\mathrm{sx} &= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r\cup t)}{p(y)p(r\cup t)}\\ &= \sum_{r,t,y} p(r,t,y) \log \frac{1}{p(y)} = H(Y) = H(T) = 1 \,\mathrm{bit}. \end{align}\]

In the case of imperfect prediction, \(p(y\mid t) < 1\) and \(I_\cap^\mathrm{sx} = I(Y:T)\). This mutual information is maximized if and only if the neuron always fires when \(T>0\) and never fires when \(T<0\) or vice versa. Note that information theory is agnostic to signs, such that the case of \(p(\sgn R \neq \sgn T) = 1\) also maximizes redundancy. [13] dealt with this case by training a separate decoder, here we preclude it through positive target weights \(w_T\).

Next, we show that any deviation from the separated supports necessarily decreases the redundancy. If the target weights \(w_T\) are close to zero, as is the case in Hebbian learning, the Heaviside activation function \(p(y|r,t)=\delta_{y, \sgn(r+t)}\) reduces to \(\delta_{y, \sgn(r)}\) for almost all inputs, which produces \[\begin{align} I_\cap^\mathrm{sx} &= \sum_{r,t,y} p(r,t,y) \log \frac{p(y,r\cup t)}{p(y)p(r\cup t)}\\ &= \sum_{r,t,y} \delta_{y,\sgn(r)}p(r,t) \log \frac{p(y,r) + p(y,t) - p(y,r,t)}{p(y)[p(r)+p(t)-p(r,t)]}\\ &= \sum_{r,t}p(r, t) \log \frac{p(Y = \sgn r,r) + p(Y = \sgn r,t) - p(Y = \sgn r,r,t)}{p(y)[p(r)+p(t)-p(r,t)]}\\ &= \sum_{r,t}p(r, t) \log \left[\frac{p(r) + \sum_{\sgn r'=\sgn r}p(r',t) - p(r,t)}{p(y)[p(r)+p(t)-p(r,t)]}\right]. \end{align}\]

In general, we have \(\sum_{\sgn r'=\sgn r}p(r',t) \leq p(t)\). In particular, this turns into a strict inequality if the conditional probability \(p(R\mid T=t)\) has support for both positive and negative values of \(r\). In this case, the logarithm will become negative and consequently the redundant information will not be maximal, \(I_\cap^\mathrm{sx}<1=H(Y)\). From this, it follows that redundancy can only be maximal when the two conditional distributions \(p(R\mid T=t)\) have support on opposite sides of the \(R=0\) line, and thus are non-overlapping. Since the disjoint probability mass functions are the only way to achieve perfect recall accuracy, redundancy is therefore necessary for associative memory function.

10.2 Optimizing Redundancy in Infomorphic Networks↩︎

Having established redundancy as necessary for associative memory function, it remains to be shown that it can also function as a goal to be optimized directly in infomorphic networks.

An important distinction to the Hopfield setup discussed so far is that during training, infomorphic networks use a sigmoid activation function. Thus, the probability of an output \(Y\) is dependent on the magnitude of the activation function’s inputs \(R\) and \(T\). For our case of fixed target weights \(w_T\), gradient-based learning can only change the recurrent weights and thus move the conditional probability distributions along the r-axis in 16. Under the sigmoid activation function a global maximum is obtained for the support of \(p(R\mid T)\) going to \(\pm\infty\). The sigmoid thus provides smooth gradients towards the optimal weight configuration, making gradient-based learning possible.

In practice, networks trained on redundancy learn to separate the two conditional distributions \(p(R\mid T= \pm1)\). Instead of the normal distributions found in Hebbian learning, recurrent inputs \(R\) are skewed towards larger absolute values (15).

10.3 Analytical Derivation of Redundancy in Hebbian Hopfield Networks↩︎

Hebbian Hopfield networks in particular allow for an analytical derivation of the redundancy for a given memory load.

First, we take the definition of the joint probability distribution \(p(y,r,t)=p(t) p(r\mid t) p(y\mid r,t)\), where \(y\in\{-1,+1\}\) is the neuron output, \(r\) the recurrent input and \(t\in\{-1\cdot w_T,+1 \cdot w_T\}\) the pattern element. In 3.1, the pattern element was defined as \(t=w_T\xi_i^{(q)}\). Since \(w_T\) only scales the label of the pattern input but does not enter the network dynamics, the joint distribution \(p(y,r,t)\) is invariant under the choice of \(w_T\). We therefore set \(w_T =1\) without loss of generality. Using that \(y = \sgn (r)\) is a deterministic function of r, we can rewrite the joint probability distribution as \[p(y,r,t) = \frac{1}{2}\cdot p(r\mid t) \cdot p(y\mid r),\] where \(p(y \mid r) = \delta_{y,\sgn (r)}\). So we only have to define \(p(r \mid t)\). Following the derivation of [37], we decompose the local field using the Hebbian weights as defined in 2 . For a network with \(N\) neurons storing \(m\) patterns and assuming perfect recurrent input \(r\) of a pattern \(\xi^{(q)}\) (\(y_j = \xi^{(q)}_j \;\forall j \neq i\)), the local field of a single neuron \(i\) becomes \[\label{eq:decomp} r_i^{(q)} = \sum_{j\neq i}^N w_{ij}\, \xi_j^{(q)} = (N-1)\, \xi^{(q)}_i + \sum_{j\neq i}^N \sum_{{p} \neq q}^m \xi_i^{(p)}\, \xi_j^{(p)}\, \xi_j^{(q)}.\tag{10}\] The assumption holds, if we look at the first iteration of the Hopfield network as done in the main part of the paper. Each of the \(\xi_i^{(p)} \xi_j^{(p)} \xi_j^{(q)}\) in the double sum is a Rademacher random variable (i.e.,\(\pm1\) with equal probability), and there are \(L=(N-1)(m-1)\) such independent terms. If \(k\) of the \(L\) terms equal \(+1\), the sum evaluates to \(2k-L\), with \(k\sim\mathrm{Binomial}(L,\frac{1}{2})\). Using our definition of the \(t\) input (\(t = \xi_i^{(q)}\)), the conditional distribution can be written as \[\label{eq:rcond} p\left(r \mid t\right) = \binom{L}{\frac{r-(N-1)\,t+L}{2}}\frac{1}{2^L}.\tag{11}\] It is important to note that \(R\) is not continuous as it is a sum of \(L+1\) terms that are each \(\pm1\) with the allowed values being \(r\in \{(N-1)\,t-L,(N-1)\,t-L+2,\cdots,(N-1)\,t+L\}\), i.e.integers spaced by \(2\). With this, we get the discrete joint distribution \[\label{eq:joint} p(y,r,t) = \frac{1}{2} \binom{L}{\frac{r-(N-1) t+L}{2}} \frac{1}{2^L}\delta_{y,\mathrm{sgn}(r)}.\tag{12}\]

Now, we can use the formulation of \(I^\mathrm{sx}_\cap\) derived above given by \[I^\mathrm{sx}_\cap = \sum_{r,t,y} p(r,t,y) \log \frac{p(y\mid t) + p(y,r\mid -t)}{p(y)[1 + p(r\mid -t)]}\] to find an expression for the redundancy. From 12 follows directly that \(p(y)=\frac{1}{2}\) which simplifies the logarithm to \[\log_2 \frac{2(p(r \mid -t)+p(y\mid t))}{p(r\mid -t)+1}\] in all cases of non-zero join probability \(p(y,r,t)\neq 0\).

Due to the symmetry of the neurons we can use
\(p(r \mid t=+1)=p(-r\mid t=-1)\) to combine the \(t= \pm1\) contributions to derive at \[\label{eq:RedHopf} \begin{align} I_\cap^\mathrm{sx}(Y:R;T) &= \sum_r p(r\mid t = +1) \log_2 \frac{2 \left(p(r \mid t=-1)+p(\sgn (r) \mid t=+1\right)}{p(r\mid t = -1)+1}\\ &= \sum_r p(r\mid t = +1) \log_2 \frac{2\;p(r \mid t=-1)+2+2p(\sgn (r) \mid t=+1)-2}{p(r\mid t = -1)+1}\\ &= \sum_r p(r\mid t = +1) \log_2 \left(2 + \frac{2\;p(\sgn (r) \mid t=+1)-2}{p(r\mid t = -1)+1}\right) \end{align}\tag{13}\] where the sum runs over all allowed values of \(r\) as defined above. As the definition of 13 depends on the probability \(p(r \mid t)\), which is defined based on a discrete grid spacing, the redundancy depends on both \(N\) and \(m\) separately.

11 directly gives us the expression for \(p(r\mid t=-1)\) and \(p(r\mid t=+1)\) while the probability \(p(\sgn(r)\mid t=+1)\) can be written as \[\begin{align} p(\sgn(r)=+1\mid t=+1) &= \sum_{r \geq 0} p(r\mid t=+1) = \sum_{k=k^*}^L \binom{L}{k}\frac{1}{2^L}\\ p(\sgn(r)=-1\mid t=+1) &= \sum_{r < 0} p(r \mid t=+1) = \sum_{k=0}^{k^*-1} \binom{L}{k}\frac{1}{2^L}, \end{align}\] where \(k^*\) is given by \(\left\lceil \frac{L-N+1}{2}\right\rceil\) which corresponds to the value of \(k\) where \(r\) switches sign. The qualitative behavior of the different distributions are shown in 17.

13 now analytically shows that redundancy is high if the memory load is low: the distributions \(p(r \mid t=+1)\) and \(p(r\mid t=-1)\) have essentially no overlap (see 17A). So for any \(r\) where \(p(r\mid t=+1)\) has a significant mass, \(p(r\mid t=-1)\) is negligible. At the same time, \(p(\sgn (r) \mid t=+1) \approx 1\) since the neuron is almost always correct for low \(\alpha\). On the other hand, if the memory load increases, the probability that the neuron fires correctly \(p(\sgn(r)\mid t=+1)\), goes down (see 17B) while the overlap between the \(p(r \mid t=+1)\) and \(p(r\mid t=-1)\) goes up. This results in a decrease of the redundancy for an increase in \(\alpha\) (see 17C). This result supports the empirical result shown in 3.1.

10.4 Robustness of PID Estimation↩︎

To validate the empirical results of the analysis of Hebbian Hopfield networks discussed in 3.1 and the analytical results above, which are based on the \(I_\cap^\mathrm{sx}\) measure, we performed the same analysis for different PID measures. In particular, we applied \(I_\cap^\mathrm{BROJA}\) [38], \(I_\cap^\mathrm{min}\)[8], \(I^\mathrm{PM}\)[39] and \(I^\mathrm{MMI}\)[40] (see 18). While there are small numerical differences between the results of the different measures, the overall qualitative results agree with the results based on the \(I_\cap^\mathrm{sx}\) measure. We used the dit toolbox [41] to compute all additional measures except \(I_\cap^\mathrm{BROJA}\), for which we used the optimized implementation provided by [42].

As an additional check of the stability of the estimation in 3.1, we also compare the results for different number of bins used in the dimension of the recurrent input \(n_r\) (see 19). As before, the results are not identical which is due to an estimation bias depending on the number of bins. However, the qualitative results are consistent with the results reported in 2. With an increase of \(n_r\), the empirical results approach the analytical results derived on the calculation above. The choice of \(n_r = 60\) for our experiments strikes a balance between computational efficiency and estimation accuracy.

Note that in all of the results, the sample size used to estimate the probability distributions is given by \(N \alpha\), which is dependent on the memory load \(\alpha\) and the number of neurons \(N\). However, the results do not change if we limit the number of samples to be constant for different memory loads.

11 Correlations↩︎

The capacity results presented here assume that patterns are created i.i.d. and in particular without spatial correlation. In first experiments, we find that for the redundancy method, more correlation leads to improved capacity (20A). This is in contrast to Hebbian learning, where the same correlation decreases capacity (20B).

To create spatial correlations, each pattern \(\xi^{(p)}\) is procedurally generated in a stepwise manner, where the probability of flipping the next element of a pattern \(p(\xi^{(p)}_{i+1}=1 |\xi^{(p)}_{i}=-1 )=p(\xi^{(p)}_{i+1}=-1 |\xi^{(p)}_{i}=1 )p\) is controlled by the parameter \(p\). The first element of each pattern is chosen randomly with \(p(\xi^{(p)}_1=1)=0.5\). Increasing \(p\) increases the average size of blocks within a pattern without changing the balance of \(+1\) and \(-1\) in patterns. While capacity improves for the redundancy method, further investigation could explore other methods of generating correlated patterns, e.g.between patterns, or compare changes in capacity with other learning methods or theoretical bounds.

12 Ablation Experiments↩︎

Our infomorphic Hopfield networks build on those introduced in [13], but employ multiple changes to the network and training procedure. Through these changes, the resulting memory capacity increased more than fourfold, from 0.35 to above 1.59. To quantify the effect of the most relevant changes, we conducted a series of ablation experiments. Starting from a ‘fully ablated’ condition that most closely mirrors the setup of [13], features of the training procedure are successively added in while measuring their effect on memory capacity (21).

The five features varied in the ablation are:

• Testing procedure: stochastic neuron outputs over 20 recall time steps (as in [13]) → deterministic outputs with up to 100 recall time steps.

• Target weights \(|\boldsymbol{w}_T|\): learnable, initialized at scale \(1.0\) → frozen at scale \(1.0\) → frozen at scale \(2.3\).

• Optimizer: SGD → Adam.

• PID estimation: hard (non-differentiable) binning → soft binning (see 8).

• Goal function: \(G = \Pi_\mathrm{red}\) → optimized goal \(G^{(i)}\) from 3.4.

Out of these, the most effective changes are the deterministic testing procedure, the fixed scale of target weights of \(2.3\) and the switch to the more sophisticated Adam optimizer.

Further differences to the original were omitted from the ablation study because they were not needed to match the 0.35 capacity reported in [13]. First, the goal function was simplified from \(G = \Pi_\mathrm{red} + 0.1\, I(Y; R, T)\) in [13] to pure redundancy, \(G = \Pi_\mathrm{red}\), while retaining high capacity and stable learning. Second, the activation function was simplified from \(A(r, t) = \frac{r^8 \, \mathrm{sign}(r) + t^8 \, \mathrm{sign}(t)}{r^8 + t^8} \cdot \left(r^8 + t^8\right)^{1/8}\) to the linear form \(A(r, t) = r + t\). Additionally, we note that the target weights in [13] differ from our setup in ways beyond initialization scale: there, they are drawn from a normal distribution (with both positive and negative values). The Hopfield layer was also followed by a simple decoder layer to compensate for inverse neuron representations. These structural differences are outside the scope of the ablation.

References↩︎