Neurons "attract or repel each other" in state space

The weight between two units has a powerful impact upon the values of the neurons. Consider the connection weight ${\displaystyle w_{ij}}$ between two neurons i and j. If ${\displaystyle w_{ij}>0}$, the updating rule implies that:

• when ${\displaystyle s_{j}=1}$, the contribution of j in the weighted sum is positive. Thus, ${\displaystyle s_{i}}$ is pulled by j towards its value ${\displaystyle s_{i}=1}$
• when ${\displaystyle s_{j}=-1}$, the contribution of j in the weighted sum is negative. Then again, ${\displaystyle s_{i}}$ is pushed by j towards its value ${\displaystyle s_{i}=-1}$

Thus, the values of neurons i and j will converge if the weight between them is positive. Similarly, they will diverge if the weight is negative.

Learning rules

There are various different learning rules that can be used to store information in the memory of the Hopfield network. It is desirable for a learning rule to have both of the following two properties:

• Local: A learning rule is local if each weight is updated using information available to neurons on either side of the connection that is associated with that particular weight.
• Incremental: New patterns can be learned without using information from the old patterns that have been also used for training. That is, when a new pattern is used for training, the new values for the weights only depend on the old values and on the new pattern.^[20]

These properties are desirable, since a learning rule satisfying them is more biologically plausible. For example, since the human brain is always learning new concepts, one can reason that human learning is incremental. A learning system that was not incremental would generally be trained only once, with a huge batch of training data.

Hebbian learning rule for Hopfield networks

Hebbian theory was introduced by Donald Hebb in 1949 in order to explain "associative learning," in which simultaneous activation of neuron cells leads to pronounced increases in synaptic strength between those cells.^[21] It is often summarized as "Neurons that fire together wire together. Neurons that fire out of sync fail to link".

The Hebbian rule is both local and incremental. For the Hopfield networks, it is implemented in the following manner when learning ${\displaystyle n}$ binary patterns:

${\displaystyle w_{ij}={\frac {1}{n}}\sum _{\mu =1}^{n}\epsilon _{i}^{\mu }\epsilon _{j}^{\mu }}$

where ${\displaystyle \epsilon _{i}^{\mu }}$ represents bit i from pattern ${\displaystyle \mu }$.

If the bits corresponding to neurons i and j are equal in pattern ${\displaystyle \mu }$, then the product ${\displaystyle \epsilon _{i}^{\mu }\epsilon _{j}^{\mu }}$ will be positive. This would, in turn, have a positive effect on the weight ${\displaystyle w_{ij}}$ and the values of i and j will tend to become equal. The opposite happens if the bits corresponding to neurons i and j are different.

Storkey learning rule

This rule was introduced by Amos Storkey in 1997 and is both local and incremental. Storkey also showed that a Hopfield network trained using this rule has a greater capacity than a corresponding network trained using the Hebbian rule.^[22] The weight matrix of an attractor neural network is said to follow the Storkey learning rule if it obeys:

${\displaystyle w_{ij}^{\nu }=w_{ij}^{\nu -1}+{\frac {1}{n}}\epsilon _{i}^{\nu }\epsilon _{j}^{\nu }-{\frac {1}{n}}\epsilon _{i}^{\nu }h_{ji}^{\nu }-{\frac {1}{n}}\epsilon _{j}^{\nu }h_{ij}^{\nu }}$

where ${\displaystyle h_{ij}^{\nu }=\sum _{k=1~:~i\neq k\neq j}^{n}w_{ik}^{\nu -1}\epsilon _{k}^{\nu }}$ is a form of local field^[20] at neuron i.

This learning rule is local, since the synapses take into account only neurons at their sides. The rule makes use of more information from the patterns and weights than the generalized Hebbian rule, due to the effect of the local field.

Discrete variables

A simple example^[10] of the modern Hopfield network can be written in terms of binary variables ${\displaystyle V_{i}}$ that represent the active ${\displaystyle V_{i}=+1}$ and inactive ${\displaystyle V_{i}=-1}$ state of the model neuron ${\displaystyle i}$.

$${\displaystyle E=-\sum \limits _{\mu =1}^{N_{\text{mem}}}F{\Big (}\sum \limits _{i=1}^{N_{f}}\xi _{\mu i}V_{i}{\Big )}}$$

In this formula the weights ${\textstyle \xi _{\mu i}}$ represent the matrix of memory vectors (index ${\displaystyle \mu =1...N_{\text{mem}}}$ enumerates different memories, and index ${\displaystyle i=1...N_{f}}$ enumerates the content of each memory corresponding to the ${\displaystyle i}$-th feature neuron), and the function ${\displaystyle F(x)}$ is a rapidly growing non-linear function. The update rule for individual neurons (in the asynchronous case) can be written in the following form

$${\displaystyle V_{i}^{(t+1)}=Sign{\bigg [}\sum \limits _{\mu =1}^{N_{\text{mem}}}{\bigg (}F{\Big (}\xi _{\mu i}+\sum \limits _{j\neq i}\xi _{\mu j}V_{j}^{(t)}{\Big )}-F{\Big (}-\xi _{\mu i}+\sum \limits _{j\neq i}\xi _{\mu j}V_{j}^{(t)}{\Big )}{\bigg )}{\bigg ]}}$$

which states that in order to calculate the updated state of the ${\textstyle i}$-th neuron the network compares two energies: the energy of the network with the ${\displaystyle i}$-th neuron in the ON state and the energy of the network with the ${\displaystyle i}$-th neuron in the OFF state, given the states of the remaining neuron. The updated state of the ${\displaystyle i}$-th neuron selects the state that has the lowest of the two energies.^[10]

In the limiting case when the non-linear energy function is quadratic ${\displaystyle F(x)=x^{2}}$ these equations reduce to the familiar energy function and the update rule for the classical binary Hopfield Network.^[1]

The memory storage capacity of these networks can be calculated for random binary patterns. For the power energy function ${\displaystyle F(x)=x^{n}}$ the maximal number of memories that can be stored and retrieved from this network without errors is given by^[10]

$${\displaystyle N_{\text{mem}}^{max}\approx {\frac {1}{2(2n-3)!!}}{\frac {N_{f}^{n-1}}{\ln(N_{f})}}}$$

For an exponential energy function ${\textstyle F(x)=e^{x}}$ the memory storage capacity is exponential in the number of feature neurons^[11]

$${\displaystyle N_{\text{mem}}^{max}\approx 2^{N_{f}/2}}$$

Fig. 1: An example of a continuous modern Hopfield network with ${\textstyle N_{f}=5}$ feature neurons and ${\displaystyle N_{\text{mem}}=11}$ memory (hidden) neurons with symmetric synaptic connections between them.

Continuous variables

Modern Hopfield networks or dense associative memories can be best understood in continuous variables and continuous time.^[12][13] Consider the network architecture, shown in Fig.1, and the equations for neuron's states evolution^[13]

$${\displaystyle {\begin{cases}\tau _{f}{\frac {dx_{i}}{dt}}=\sum \limits _{\mu =1}^{N_{h}}\xi _{i\mu }f_{\mu }-x_{i}+I_{i}\\\tau _{h}{\frac {dh_{\mu }}{dt}}=\sum \limits _{i=1}^{N_{f}}\xi _{\mu i}g_{i}-h_{\mu }\end{cases}}}$$

(1)

where the currents of the feature neurons are denoted by ${\textstyle x_{i}}$, and the currents of the memory neurons are denoted by ${\displaystyle h_{\mu }}$ (${\displaystyle h}$ stands for hidden neurons). There are no synaptic connections among the feature neurons or the memory neurons. A matrix ${\displaystyle \xi _{\mu i}}$ denotes the strength of synapses from a feature neuron ${\displaystyle i}$ to the memory neuron ${\displaystyle \mu }$. The synapses are assumed to be symmetric, so that the same value characterizes a different physical synapse from the memory neuron ${\displaystyle \mu }$ to the feature neuron ${\displaystyle i}$. The outputs of the memory neurons and the feature neurons are denoted by ${\displaystyle f_{\mu }}$ and ${\displaystyle g_{i}}$, which are non-linear functions of the corresponding currents. In general these outputs can depend on the currents of all the neurons in that layer so that ${\displaystyle f_{\mu }=f(\{h_{\mu }\})}$ and ${\textstyle g_{i}=g(\{x_{i}\})}$. It is convenient to define these activation functions as derivatives of the Lagrangian functions for the two groups of neurons

$${\displaystyle f_{\mu }={\frac {\partial L_{h}}{\partial h_{\mu }}},\ \ \ \ {\text{and}}\ \ \ \ g_{i}={\frac {\partial L_{x}}{\partial x_{i}}}}$$

(2)

This way the specific form of the equations for neuron's states is completely defined once the Lagrangian functions are specified. Finally, the time constants for the two groups of neurons are denoted by ${\displaystyle \tau _{f}}$ and ${\displaystyle \tau _{h}}$, ${\displaystyle I_{i}}$ is the input current to the network that can be driven by the presented data. 

Fig. 2: Effective theory on the feature neurons for various common choices of the Lagrangian functions. Model A reduces to the models studied in^[10][11] depending on the choice of the activation function, model B reduces to the model studied in,^[12] model C reduces to the model of.^[13] F is a "smooth enough" function.^[10]

General systems of non-linear differential equations can have many complicated behaviors that can depend on the choice of the non-linearities and the initial conditions. For Hopfield Networks, however, this is not the case - the dynamical trajectories always converge to a fixed point attractor state. This property is achieved because these equations are specifically engineered so that they have an underlying energy function^[13]

$${\displaystyle E(t)={\Big [}\sum \limits _{i=1}^{N_{f}}(x_{i}-I_{i})g_{i}-L_{x}{\Big ]}+{\Big [}\sum \limits _{\mu =1}^{N_{h}}h_{\mu }f_{\mu }-L_{h}{\Big ]}-\sum \limits _{\mu ,i}f_{\mu }\xi _{\mu i}g_{i}}$$

(3)

The terms grouped into square brackets represent a Legendre transform of the Lagrangian function with respect to the states of the neurons. If the Hessian matrices of the Lagrangian functions are positive semi-definite, the energy function is guaranteed to decrease on the dynamical trajectory^[13]

$${\displaystyle {\frac {dE(t)}{dt}}=-\tau _{f}\sum \limits _{i,j=1}^{N_{f}}{\frac {dx_{i}}{dt}}{\frac {\partial ^{2}L_{x}}{\partial x_{i}\partial x_{j}}}{\frac {dx_{j}}{dt}}-\tau _{h}\sum \limits _{\mu ,\nu =1}^{N_{h}}{\frac {dh_{\mu }}{dt}}{\frac {\partial ^{2}L_{h}}{\partial h_{\mu }\partial h_{\nu }}}{\frac {dh_{\nu }}{dt}}\leq 0}$$

(4)

This property makes it possible to prove that the system of dynamical equations describing temporal evolution of neurons' activities will eventually reach a fixed point attractor state.

In certain situations one can assume that the dynamics of hidden neurons equilibrates at a much faster time scale compared to the feature neurons, ${\textstyle \tau _{h}\ll \tau _{f}}$. In this case the steady state solution of the second equation in the system (1) can be used to express the currents of the hidden units through the outputs of the feature neurons. This makes it possible to reduce the general theory (1) to an effective theory for feature neurons only. The resulting effective update rules and the energies for various common choices of the Lagrangian functions are shown in Fig.2. In the case of log-sum-exponential Lagrangian function the update rule (if applied once) for the states of the feature neurons is the attention mechanism^[12] commonly used in many modern AI systems (see Ref.^[13] for the derivation of this result from the continuous time formulation).

Relationship to classical Hopfield network with continuous variables

Classical formulation of continuous Hopfield Networks^[6] can be understood^[13] as a special limiting case of the modern Hopfield networks with one hidden layer. Continuous Hopfield Networks for neurons with graded response are typically described^[6] by the dynamical equations

$${\displaystyle \tau _{f}{\frac {dx_{i}}{dt}}=\sum \limits _{j=1}^{N_{f}}T_{ij}V_{j}-x_{i}+I_{i}}$$

(5)

and the energy function

$${\displaystyle E=-{\frac {1}{2}}\sum \limits _{i,j=1}^{N_{f}}T_{ij}V_{i}V_{j}-\sum \limits _{i=1}^{N_{f}}V_{i}I_{i}+\sum \limits _{i=1}^{N_{f}}\int \limits ^{V_{i}}g^{-1}(z)\,dz}$$

(6)

where ${\textstyle V_{i}=g(x_{i})}$, and ${\displaystyle g^{-1}(z)}$ is the inverse of the activation function ${\displaystyle g(x)}$. This model is a special limit of the class of models that is called models A,^[13] with the following choice of the Lagrangian functions

$${\displaystyle L_{v}=\sum \limits _{i=1}^{N_{f}}\int \limits ^{x_{i}}g(x)dx,\ \ \ \ \ {\text{and}}\ \ \ \ \ L_{h}={\frac {1}{2}}\sum \limits _{\mu =1}^{N_{h}}h_{\mu }^{2}}$$

(7)

that, according to the definition (2), leads to the activation functions

$${\displaystyle V_{i}=g(x_{i}),\ \ \ \ \ {\text{and}}\ \ \ \ \ f_{\mu }=h_{\mu }}$$

(8)

If we integrate out the hidden neurons the system of equations (1) reduces to the equations on the feature neurons (5) with ${\displaystyle T_{ij}=\sum \limits _{\mu =1}^{N_{h}}\xi _{\mu i}\xi _{\mu j}}$, and the general expression for the energy (3) reduces to the effective energy

$${\displaystyle E=-{\frac {1}{2}}\sum \limits _{i,j=1}^{N_{f}}T_{ij}V_{i}V_{j}-\sum \limits _{i=1}^{N_{f}}V_{i}I_{i}+\sum \limits _{i=1}^{N_{f}}{\Big (}x_{i}V_{i}-\int \limits ^{x_{i}}g(x)dx{\Big )}}$$

(9)

While the first two terms in equation (6) are the same as those in equation (9), the third terms look superficially different. In equation (9) it is a Legendre transform of the Lagrangian for the feature neurons, while in (6) the third term is an integral of the inverse activation function. Nevertheless, these two expressions are in fact equivalent, since the derivatives of a function and its Legendre transform are inverse functions of each other. The easiest way to see that these two terms are equal explicitly is to differentiate each one with respect to ${\displaystyle x_{i}}$. The results of these differentiations for both expressions are equal to ${\displaystyle x_{i}g(x_{i})'}$. Thus, the two expressions are equal up to an additive constant. This completes the proof^[13] that the classical Hopfield Network with continuous states^[6] is a special limiting case of the modern Hopfield network (1) with energy (3).

General formulation of the modern Hopfield network

Fig. 3: The connectivity diagram of the fully-connected modern Hopfield network consisting of five neurons. The synaptic weights are described by a symmetric matrix ${\displaystyle W_{IJ}}$.

Biological neural networks have a large degree of heterogeneity in terms of different cell types. This section describes a mathematical model of a fully connected modern Hopfield network assuming the extreme degree of heterogeneity: every single neuron is different.^[28] Specifically, an energy function and the corresponding dynamical equations are described assuming that each neuron has its own activation function and kinetic time scale.  The network is assumed to be fully connected, so that every neuron is connected to every other neuron using a symmetric matrix of weights ${\displaystyle W_{IJ}}$, indices ${\displaystyle I}$ and ${\displaystyle J}$ enumerate different neurons in the network, see Fig.3. The easiest way to mathematically formulate this problem is to define the architecture through a Lagrangian function ${\displaystyle L(\{x_{I}\})}$ that depends on the activities of all the neurons in the network. The activation function for each neuron is defined as a partial derivative of the Lagrangian  with respect to that neuron's activity

$${\displaystyle g_{I}={\frac {\partial L}{\partial x_{I}}}}$$

(10)

From the biological perspective one can think about ${\displaystyle g_{I}}$ as an axonal output of the neuron ${\displaystyle I}$. In the simplest case, when the Lagrangian is additive for different neurons, this definition results in the activation that is a non-linear function of that neuron's activity. For non-additive Lagrangians this activation function can depend on the activities of a group of neurons. For instance, it can contain contrastive (softmax) or divisive normalization. The dynamical equations describing temporal evolution of a given neuron are given by^[28]

$${\displaystyle \tau _{I}{\frac {dx_{I}}{dt}}=\sum \limits _{J=1}^{N}W_{IJ}g_{J}-x_{I}}$$

(11)

This equation belongs to the class of models called firing rate models in neuroscience. Each neuron ${\displaystyle I}$ collects the axonal outputs ${\displaystyle g_{J}}$ from all the neurons, weights them with the synaptic coefficients ${\displaystyle W_{IJ}}$ and produces its own time-dependent activity ${\displaystyle x_{I}}$. The temporal evolution has a time constant ${\displaystyle \tau _{I}}$, which in general can be different for every neuron. This network has a global energy function^[28]

$${\displaystyle E=\sum \limits _{I=1}^{N}x_{I}g_{I}-L-{\frac {1}{2}}\sum \limits _{I,J=1}^{N}g_{I}W_{IJ}g_{J}}$$

(12)

where the first two terms represent the Legendre transform of the Lagrangian function with respect to the neurons' currents ${\displaystyle x_{I}}$. The temporal derivative of this energy function can be computed on the dynamical trajectories leading to (see ^[28] for details)

$${\displaystyle {\frac {dE}{dt}}=-\sum \limits _{I,K=1}^{N}{\frac {dx_{I}}{dt}}M_{IK}{\frac {dx_{K}}{dt}}\leq 0,\ \ \ \ {\text{where}}\ \ \ \ M_{IK}=\tau _{I}{\frac {\partial ^{2}L}{\partial x_{I}\partial x_{K}}}}$$

(13)

The last inequality sign holds provided that the matrix ${\displaystyle M_{IK}}$ (or its symmetric part) is positive semi-definite. If, in addition to this, the energy function is bounded from below the non-linear dynamical equations are guaranteed to converge to a fixed point attractor state. The advantage of formulating this network in terms of the Lagrangian functions is that it makes it possible to easily experiment with different choices of the activation functions and different architectural arrangements of neurons. For all those flexible choices the conditions of convergence are determined by the properties of the matrix ${\displaystyle M_{IJ}}$ and the existence of the lower bound on the energy function.

Fig. 4: The connectivity diagram of the layered Hierarchical Associative Memory network.^[28] Each layer can have different number of neurons, different activation function, and different time scales. The feedforward weights and feedback weights are equal.

Hierarchical associative memory network

The neurons can be organized in layers so that every neuron in a given layer has the same activation function and the same dynamic time scale. If we assume that there are no horizontal connections between the neurons within the layer (lateral connections) and there are no skip-layer connections, the general fully connected network (11), (12) reduces to the architecture shown in Fig.4. It has ${\displaystyle N_{\text{layer}}}$ layers of recurrently connected neurons with the states described by continuous variables ${\displaystyle x_{i}^{A}}$ and the activation functions ${\displaystyle g_{i}^{A}}$, index ${\displaystyle A}$ enumerates the layers of the network, and index ${\displaystyle i}$ enumerates individual neurons in that layer. The activation functions can depend on the activities of all the neurons in the layer. Every layer can have a different number of neurons ${\displaystyle N_{A}}$. These neurons are recurrently connected with the neurons in the preceding and the subsequent layers. The matrices of weights that connect neurons in layers ${\displaystyle A}$ and ${\displaystyle B}$ are denoted by ${\displaystyle \xi _{ij}^{(A,B)}}$ (the order of the upper indices for weights is the same as the order of the lower indices, in the example above this means that the index ${\displaystyle i}$ enumerates neurons in the layer ${\displaystyle A}$, and index ${\displaystyle j}$ enumerates neurons in the layer ${\displaystyle B}$). The feedforward weights and the feedback weights are equal. The dynamical equations for the neurons' states can be written as^[28]

$${\displaystyle \tau _{A}{\frac {dx_{i}^{A}}{dt}}=\sum \limits _{j=1}^{N_{A-1}}\xi _{ij}^{(A,A-1)}g_{j}^{A-1}+\sum \limits _{j=1}^{N_{A+1}}\xi _{ij}^{(A,A+1)}g_{j}^{A+1}-x_{i}^{A}}$$

(14)

with boundary conditions

$${\displaystyle g_{i}^{0}=0,\ \ \ \ \ {\text{and}}\ \ \ \ \ g_{i}^{N_{\text{layer}}+1}=0}$$

(15)

The main difference between these equations and those from the conventional feedforward networks is the presence of the second term, which is responsible for the feedback from higher layers. These top-down signals help neurons in lower layers to decide on their response to the presented stimuli. Following the general recipe it is convenient to introduce a Lagrangian function ${\displaystyle L^{A}(\{x_{i}^{A}\})}$ for the ${\displaystyle A}$-th hidden layer, which depends on the activities of all the neurons in that layer.^[28] The activation functions in that layer can be defined as partial derivatives of the Lagrangian

$${\displaystyle g_{i}^{A}={\frac {\partial L^{A}}{\partial x_{i}^{A}}}}$$

(16)

With these definitions the energy (Lyapunov) function is given by^[28]

$${\displaystyle E=\sum \limits _{A=1}^{N_{\text{layer}}}{\Big [}\sum \limits _{i=1}^{N_{A}}x_{i}^{A}g_{i}^{A}-L^{A}{\Big ]}-\sum \limits _{A=1}^{N_{\text{layer}}-1}\sum \limits _{i=1}^{N_{A+1}}\sum \limits _{j=1}^{N_{A}}g_{i}^{A+1}\xi _{ij}^{(A+1,A)}g_{j}^{A}}$$

(17)

If the Lagrangian functions, or equivalently the activation functions, are chosen in such a way that the Hessians for each layer are positive semi-definite and the overall energy is bounded from below, this system is guaranteed to converge to a fixed point attractor state. The temporal derivative of this energy function is given by^[28]

$${\displaystyle {\frac {dE}{dt}}=-\sum \limits _{A=1}^{N_{\text{layer}}}\tau _{A}\sum \limits _{i,j=1}^{N_{A}}{\frac {dx_{j}^{A}}{dt}}{\frac {\partial ^{2}L^{A}}{\partial x_{j}^{A}\partial x_{i}^{A}}}{\frac {dx_{i}^{A}}{dt}}\leq 0}$$

(18)

Thus, the hierarchical layered network is indeed an attractor network with the global energy function. This network is described by a hierarchical set of synaptic weights that can be learned for each specific problem.