Literature
Display
← All papers

Hopfield Networks is All You Need

Hubert Ramsauer · Bernhard Schäfl · Johannes Lehner · Philipp Seidl · Michael Widrich · Thomas Adler · Lukas Gruber · Markus Holzleitner · Milena Pavlović · Geir Kjetil Sandve · Victor Greiff · David Kreil · Michael Kopp · Günter Klambauer · Johannes Brandstetter · Sepp Hochreiter

Preprint first posted · ICLR 2021 ↗

Language models20 min text · CondensedOriginal paper ↗
Contents

Overview

This edition follows the complete 94-page arXiv v3, including the convergence and storage proofs, mathematical background, experimental protocols and PyTorch interface. The authors provide Hopfield-layer code and an explanatory project page. All original research figures, tables and displayed formulas remain.

At a glance

Intuition. Associative memory retrieves a stored pattern from an incomplete or noisy cue, like recognizing a saved vector after some coordinates have changed. Here retrieval forms a similarity-weighted average of stored patterns. A strong match pulls the result toward one pattern; ambiguous matches can produce a mixture.

Reading the math. The query is the current state, stored vectors are the memories, and softmax supplies their mixing weights. The inverse-temperature parameter β controls how sharply the update favors strong matches. An energy function is a mathematical score used to analyze this update and its fixed points. The attention connection applies to this retrieval construction; it does not make an entire Transformer a physical system in thermal equilibrium.

A continuous-state associative memory updates a query to a softmax-weighted average of stored patterns. Under appropriate separation and norm conditions, it has fixed points near individual patterns and can retrieve them accurately in one step; random spherical patterns admit an exponential-in-dimension capacity bound. Other regimes yield global or cluster averages. The update gives the mathematical core of attention and motivates layers for association, pooling and learned memories. Experiments cover language-model attention, multiple-instance learning, small tabular datasets and molecular prediction.

1 Introduction

Classical Hopfield memories and later dense associative memories trade different energy functions for greater storage capacity. This paper makes the exponential-interaction construction continuous and differentiable, so it can be used inside neural networks. The energy-based results concern a fixed memory and its iterative retrieval map. Their implications for a trained transformer depend on its projections, values, residual blocks and changing inputs; they are not a convergence theorem for the entire transformer. The proposed layers expose useful choices of query sets, stored sets and learned prototypes.

Complete original Figure 1: We generalize
the energy
of binary modern Hopfield networks to continuous states
while keeping fast convergence and
storage capacity properties.
We also propose a new update
rule that minimizes the energy.
The new update rule is the attention mechanism of the transformer.
Formulae are modified to express softmax \mathrm{softmax} as row vector.
“ = = ”-sign means “keeps the properties”.
Figure 1: We generalize the energy of binary modern Hopfield networks to continuous states while keeping fast convergence and storage capacity properties. We also propose a new update rule that minimizes the energy. The new update rule is the attention mechanism of the transformer. Formulae are modified to express softmax\mathrm{softmax} as row vector. “==”-sign means “keeps the properties”.

2 Modern Hopfield Nets with Continuous States

The construction uses a quadratic state penalty minus log-sum-exp similarity to stored patterns. The quadratic term bounds the energy from below; the corresponding update returns a convex combination of memory vectors.

New energy function for continuous state Hopfield networks.

For memory matrix X and inverse temperature β, the update is X·softmax(βXᵀξ). It is a concave–convex minimization step. Theorems 1–2 guarantee decreasing/convergent energy and stationary limit points; convergence of the actual iterates is conditional when the stationary level set is not finite. “Globally convergent” does not mean finding a global energy minimum. The stated upper energy bound applies to the bounded state region used in the appendix, not arbitrarily large queries before the first update.

A stored pattern means a unique attracting fixed point in its own disjoint neighborhood, not exact equality between the vector and the fixed point. The random-pattern capacity theorem assumes independent spherical patterns with radius proportional to √(d−1), a failure probability and sufficient dimension/separation. The one-step and exponentially small-error statements likewise require a query close enough to a well-separated pattern. All exact hypotheses and bounds remain below.

Equations 1–3
lse⁡(β,𝒙)\displaystyle\mathrm{lse}(\beta,\bm{x})\ =β−1​log⁡(∑i=1Nexp⁡(β​xi)),\displaystyle=\ \beta^{-1}\log\left(\sum_{i=1}^{N}\exp(\beta x_{i})\right)\ , (1)
E\displaystyle\mathrm{E}\ =−lse⁡(β,𝑿T​𝝃)+12​𝝃T​𝝃+β−1​log⁡N+12​M2.\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\log N\ +\ \frac{1}{2}M^{2}\ . (2)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =f⁡(𝝃)=𝑿​𝒑=𝑿​softmax​(β​𝑿T​𝝃).\displaystyle=\ f(\bm{\xi})\ =\ \bm{X}\bm{p}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ . (3)
Theorem 1.

The update rule Eq. (3) converges globally: For 𝛏t+1=f⁡(𝛏t)\bm{\xi}^{t+1}=f(\bm{\xi}^{t}), the energy E⁡(𝛏t)→E⁡(𝛏∗)\mathrm{E}(\bm{\xi}^{t})\to\mathrm{E}(\bm{\xi}^{*}) for t→∞t\to\infty and a fixed point 𝛏∗\bm{\xi}^{*}.

Theorem 2.

For the iteration Eq. (3) we have E⁡(𝛏t)→E⁡(𝛏∗)=E∗\mathrm{E}\left(\bm{\xi}^{t}\right)\to\mathrm{E}\left(\bm{\xi}^{*}\right)=\mathrm{E}^{*} as t→∞t\to\infty, for some stationary point 𝛏∗\bm{\xi}^{*}. Furthermore, ‖𝛏t+1−𝛏t‖→0{{\left\|\bm{\xi}^{t+1}-\bm{\xi}^{t}\right\|}}\to 0 and either {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} converges or, in the other case, the set of limit points of {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} is a connected and compact subset of ℒ⁡(E∗)\mathcal{L}\left(\mathrm{E}^{*}\right), where ℒ⁡(a)={𝛏∈ℒ∣E⁡(𝛏)=a}\mathcal{L}\left(a\right)=\{\bm{\xi}\in\mathcal{L}\mid\mathrm{E}\left(\bm{\xi}\right)=a\} and ℒ\mathcal{L} is the set of stationary points of the iteration Eq. (3). If ℒ⁡(E∗)\mathcal{L}\left(\mathrm{E}^{*}\right) is finite, then any sequence {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} generated by the iteration Eq. (3) converges to some 𝛏∗∈ℒ⁡(E∗)\bm{\xi}^{*}\in\mathcal{L}\left(\mathrm{E}^{*}\right).

Definition 1 (Pattern Stored and Retrieved).

We assume that around every pattern 𝐱i\bm{x}_{i} a sphere Si\mathrm{S}_{i} is given. We say 𝐱i\bm{x}_{i} is stored if there is a single fixed point 𝐱i∗∈Si\bm{x}_{i}^{*}\in\mathrm{S}_{i} to which all points 𝛏∈Si\bm{\xi}\in\mathrm{S}_{i} converge, and Si∩Sj=∅\mathrm{S}_{i}\cap\mathrm{S}_{j}=\emptyset for i≠ji\not=j. We say 𝐱i\bm{x}_{i} is retrieved for a given ϵ\epsilon if iteration (update rule) Eq. (3) gives a point 𝐱~i\tilde{\bm{x}}_{i} that is at least ϵ\epsilon-close to the single fixed point 𝐱i∗∈Si\bm{x}_{i}^{*}\in\mathrm{S}_{i}. The retrieval error is ‖𝐱~i−𝐱i‖{{\left\|\tilde{\bm{x}}_{i}-\bm{x}_{i}\right\|}}.

Theorem 3.

We assume a failure probability 0<p⩽10<p\leqslant 1 and randomly chosen patterns on the sphere with radius M:=K​d−1M:=K\sqrt{d-1}. We define a:=2d−1​(1+ln⁡(2​β​K2​p​(d−1)))a:=\frac{2}{d-1}(1+\ln(2\beta K^{2}p(d-1))), b:=2​K2​β5b:=\frac{2K^{2}\beta}{5}, and c:=bW0​(exp⁡(a+ln⁡(b))𝐶𝐿𝑂𝑆𝐸c:=\frac{b}{W_{0}(\exp(a+\ln(b))}, where W0W_{0} is the upper branch of the Lambert WW function (Olver et al., 2010, (4.13)), and ensure c≥(2p)4d−1c\geq\left(\frac{2}{\sqrt{p}}\right)^{\frac{4}{d-1}}. Then with probability 1−p1-p, the number of random patterns that can be stored is

Equation 4
N\displaystyle N\ ≥p​cd−14.\displaystyle\geq\ \sqrt{p}\ c^{\frac{d-1}{4}}\ . (4)

Therefore it is proven for c≥3.1546c\geq 3.1546 with β=1\beta=1, K=3K=3, d=20d=20 and p=0.001p=0.001 (a+ln⁡(b)>1.27a+\ln(b)>1.27) and proven for c≥1.3718c\geq 1.3718 with β=1\beta=1, K=1K=1, d=75d=75, and p=0.001p=0.001 (a+ln⁡(b)<−0.94a+\ln(b)<-0.94).

Equation 5
Δi\displaystyle\Delta_{i}\ :=minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle:=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (5)
Theorem 4.

With query 𝛏\bm{\xi}, after one update the distance of the new point f⁡(𝛏)f(\bm{\xi}) to the fixed point 𝐱i∗\bm{x}_{i}^{*} is exponentially small in the separation Δi\Delta_{i}. The precise bounds using the Jacobian J=∂f⁡(𝛏)∂𝛏\mathrm{J}=\frac{\partial f(\bm{\xi})}{\partial\bm{\xi}} and its value Jm\mathrm{J}^{m} in the mean value theorem are:

Equations 6–7
‖f⁡(𝝃)−𝒙i∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒙i∗‖,\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ , (6)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (7)

For given ϵ\epsilon and sufficient large Δi\Delta_{i}, we have ‖f⁡(𝛏)−𝐱i∗‖<ϵ{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}<\epsilon, that is, retrieval with one update.

Theorem 5 (Exponentially Small Retrieval Error).

The retrieval error ‖f⁡(𝛏)−𝐱i‖{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}\right\|}} of pattern 𝐱i\bm{x}_{i} is bounded by

Equation 8
‖f⁡(𝝃)−𝒙i‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}\right\|}}\ ⩽ 2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))​M\displaystyle\leqslant\ 2\ (N-1)\ \exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ M (8)

and for ‖𝐱i−𝐱i∗‖⩽12​β​M{{\left\|\bm{x}_{i}-\bm{x}_{i}^{*}\right\|}}\leqslant\frac{1}{2\ \beta\ M} together with ‖𝐱i−𝛏‖⩽12​β​M{{\left\|\bm{x}_{i}-\bm{\xi}\right\|}}\leqslant\frac{1}{2\ \beta\ M} by

Equation 9
‖𝒙i−𝒙i∗‖\displaystyle{{\left\|\bm{x}_{i}\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽ 2​e​(N−1)​M​exp⁡(−β​Δi).\displaystyle\leqslant\ 2\ e\ (N-1)\ M\ \exp(-\ \beta\ \Delta_{i})\ . (9)

Metastable states and one global fixed point.

When all patterns form a sufficiently tight cluster, a contraction argument gives one fixed point near their mean. A separated subgroup can instead have a fixed point near its own mean. These global-average, cluster-average and individual-pattern regimes are established under the appendix’s quantitative conditions; absence of an individually isolated pattern alone is not a general uniqueness proof.

Hopfield update rule is attention of the transformer.

Project raw queries and keys into a common associative space and use β=1/√dₖ. A retrieval followed by a value projection gives softmax(QKᵀ/√dₖ)V. The derivation first factors V through the key representation; standard transformer attention allows a separate value map. Thus the attention weights and associative update coincide in the stated construction, while the fixed-memory energy proof applies directly in the associative space rather than automatically to arbitrary output/value dynamics.

Equation 10
𝒁=softmax⁡(1/dk​𝑸​𝑲T)​𝑽=softmax⁡(β​𝑹​𝑾𝑸​𝑾𝑲T​𝒀T)​𝒀​𝑾𝑲​𝑾𝑽.\displaystyle\bm{Z}\ =\ \mathrm{softmax}\left(1/\sqrt{d_{k}}\ \bm{Q}\ \bm{K}^{T}\right)\ \bm{V}\ =\ \mathrm{softmax}\left(\beta\ \bm{R}\ \bm{W}_{\bm{Q}}\ \bm{W}_{\bm{K}}^{T}\bm{Y}^{T}\right)\ \bm{Y}\ \bm{W}_{\bm{K}}\bm{W}_{\bm{V}}\ . (10)

3 New Hopfield Layers for Deep Learning

Differentiability and useful single-step retrieval make the memory map suitable as a neural layer. Its proposed uses include set operations, attention, multiple-instance learning and access to reference data or learned prototypes; not every listed application is experimentally evaluated here.

Complete original Figure 2: Left: A standard deep network with layers
( ■ \blacksquare ) propagates
either a vector or a set of vectors from the input to the output.
Right: A deep network, where layers ( ■ \blacksquare )
are equipped with associative memories
via Hopfield layers ( ■ \blacksquare ).
Figure 2: Left: A standard deep network with layers ( ■\blacksquare) propagates either a vector or a set of vectors from the input to the output. Right: A deep network, where layers ( ■\blacksquare) are equipped with associative memories via Hopfield layers ( ■\blacksquare).

Types of neural networks.

The layer can operate in networks propagating one activation vector or a set of vectors. It supplies an associative memory at selected layers while retaining the surrounding feed-forward or recurrent computation.

Types of new Hopfield layers.

Hopfield associates a query set with a stored set, both supplied by inputs or previous layers. HopfieldPooling uses learned static queries to summarize an input set into a fixed number of outputs; one query gives a bag representation for multiple-instance learning. HopfieldLayer keeps a fixed or learnable memory and accepts changing queries, enabling repeated access to training examples, prototypes or reference patterns. Memory inputs and target values can be separated so similarity-weighted retrieval predicts labels. These are architectural analogies to nearest-neighbor, prototype and support-vector methods, not a claim that every such algorithm is identical to one layer.

Complete original Figure 3: The layer Hopfield allows the association of two sets 𝑹 \bm{R} ( ■ \blacksquare ) and 𝒀 \bm{Y} ( ■ \blacksquare ).
It can be integrated into deep networks that propagate sets of vectors.
The Hopfield memory
is filled
with a set from either the input or previous layers.
The output is a set of vectors 𝒁 \bm{Z} ( ■ \blacksquare ).
Figure 3: The layer Hopfield allows the association of two sets 𝑹\bm{R} ( ■\blacksquare) and 𝒀\bm{Y} ( ■\blacksquare). It can be integrated into deep networks that propagate sets of vectors. The Hopfield memory is filled with a set from either the input or previous layers. The output is a set of vectors 𝒁\bm{Z} ( ■\blacksquare).
Complete original Figure 4: The layer HopfieldPooling enables
pooling or summarization of sets,
which are obtained from the input or from previous layers.
The input 𝒀 \bm{Y} ( ■ \blacksquare )
can be either a set or a sequence.
The query patterns of each layer are static and can be learned.
The output is a set of vectors 𝒁 \bm{Z} ( ■ \blacksquare ),
where the number of vectors equals the number of query patterns.
The layer HopfieldPooling can realize multiple instance learning.
Figure 4: The layer HopfieldPooling enables pooling or summarization of sets, which are obtained from the input or from previous layers. The input 𝒀\bm{Y} ( ■\blacksquare) can be either a set or a sequence. The query patterns of each layer are static and can be learned. The output is a set of vectors 𝒁\bm{Z} ( ■\blacksquare), where the number of vectors equals the number of query patterns. The layer HopfieldPooling can realize multiple instance learning.
Complete original Figure 5: The layer HopfieldLayer enables
multiple queries of the training set,
a reference set, prototype set, or a learned set (a learned matrix).
The queries for each layer are computed from the results of previous layers.
The input is a set of vectors 𝑹 \bm{R} ( ■ \blacksquare ). The output is also a set of vectors 𝒁 \bm{Z} ( ■ \blacksquare ),
where the number of output vectors equals the number of input vectors.
The layer HopfieldLayer can realize
SVM models, k k -nearest neighbor, and LVQ.
Figure 5: The layer HopfieldLayer enables multiple queries of the training set, a reference set, prototype set, or a learned set (a learned matrix). The queries for each layer are computed from the results of previous layers. The input is a set of vectors 𝑹\bm{R} ( ■\blacksquare). The output is also a set of vectors 𝒁\bm{Z} ( ■\blacksquare), where the number of output vectors equals the number of input vectors. The layer HopfieldLayer can realize SVM models, kk-nearest neighbor, and LVQ.

Additional functionality of new Hopfield layers.

Repeated updates refine retrieval without adding parameters. Temperature β controls concentration, associative dimension affects potential capacity, and normalization changes pattern norms and fixed-point geometry. These settings interact with learning rate and training dynamics; larger capacity does not by itself ensure useful learned memories.

4 Experiments

The experiments test the layer family across attention analysis, bag classification, small tabular data and molecular prediction. The appendix supplies the protocols and additional drug-design results.

Analysis of transformer and BERT models.

Sort attention weights and count the smallest k carrying 90% of their mass. Classify a head by the median k relative to sequence length: above N/2, (N/8,N/2], (N/32,N/8], or at most N/32. Lower BERT layers often average broadly, middle layers include both focused and broad heads, and higher layers favor intermediate concentration. This attention-concentration diagnostic is evidence about operating behavior; it is not a direct numerical verification that every head has converged to an energy minimum.

Multiple Instance Learning Datasets.

DeepRC pools receptor-sequence features from bags with roughly 300,000 instances. On the reported real repertoire dataset it achieves AUC .832±.022, versus .825±.022 for the MinMax-kernel SVM; overlapping variation cautions against interpreting the small gap as conclusive superiority on that dataset. Across the reported repertoire tasks its average is strongest. On the four smaller MIL benchmarks, HopfieldPooling leads Tiger, Elephant and UCSB but performs poorly on Fox (.6405 versus .738 for MILES). Table 1 uses previously published baseline results, choosing the higher reported AUC where two sources differ.

Method tiger fox elephant UCSB
Hopfield (ours) 91.3±0.5\mathbf{91.3\pm 0.5} 64.05±0.464.05\pm 0.4 94.9±0.3\mathbf{94.9\pm 0.3} 89.5±0.8\mathbf{89.5\pm 0.8}
Path encoding (Küçükaşcı & Baydoğan, 2018) 91.0±1.091.0\pm 1.0a 71.2±1.471.2\pm 1.4a 94.4±0.794.4\pm 0.7a 88.0±2.288.0\pm 2.2a
MInD (Cheplygina et al., 2016) 85.3±1.185.3\pm 1.1a 70.4±1.670.4\pm 1.6a 93.6±0.993.6\pm 0.9a 83.1±2.783.1\pm 2.7a
MILES (Chen et al., 2006) 87.2±1.787.2\pm 1.7b 73.8±1.6\mathbf{73.8\pm 1.6}a 92.7±0.792.7\pm 0.7a 83.3±2.683.3\pm 2.6a
APR (Dietterich et al., 1997) 77.8±0.777.8\pm 0.7b 54.1±0.954.1\pm 0.9b 55.0±1.055.0\pm 1.0b —
Citation-kNN (Wang, 2000) 85.5±0.985.5\pm 0.9b 63.5±1.563.5\pm 1.5b 89.6±0.989.6\pm 0.9b 70.6±3.270.6\pm 3.2a
DD (Maron & Lozano-Pérez, 1998) 84.184.1b 63.163.1b 90.790.7b —
Table 1: Results for MIL datasets Tiger, Fox, Elephant, and UCSB Breast Cancer in terms of AUC. Results for all methods except the first are taken from either a(Küçükaşcı & Baydoğan, 2018) or b(Carbonneau et al., 2016), depending on which reports the higher AUC.

UCI Benchmark Collection.

On 75 UCI datasets with fewer than 1,000 samples, an SNN-based representation plus Hopfield retrieval has the best aggregate rank. The top three runner-up differences are not statistically significant by the reported Wilcoxon comparison; this is not dominance on every dataset. The appendix also tests training-set memories on HIV, BACE, BBBP and SIDER, reporting the best displayed results on BACE (.902±.023) and SIDER (.672±.019). These task-specific findings support useful associative layers without making the capacity theorem a generalization guarantee.

Method avg. rank diff. pp-value
Hopfield (ours) −3.92\mathbf{-3.92} —
SVM −3.23-3.23 0.150.15
SNN −2.85-2.85 0.100.10
RandomForest −2.79-2.79 0.050.05
…       … …
Stacking 8.738.73 1.21.2e−11-11
Table 2: Results on 75 small datasets of the UCI benchmarks given as difference to average rank.

Acknowledgments

The authors acknowledge institutional, public and industrial support for the Linz research groups and IARAI.

Appendix A Appendix

The appendix develops continuous-memory theory, mathematical tools, prior binary-memory models, the attention correspondence, experiment details and the implementation interface. Its original formal statements and displayed derivations are retained alongside the condensed explanations.

A.1 Continuous State Modern Hopfield Networks (A New Concept)

This section establishes global energy behavior, local contraction regimes, storage/retrieval conditions and extensions for learned associations and changing memories.

A.1.1 Introduction

The theory proceeds from energy descent to local contraction, then uses separation to derive storage and retrieval bounds. Later subsections distinguish learning the association geometry from retrieval with a fixed memory, and discuss temporal weighting.

A.1.2 New Energy Function

The energy can also be written as a negative logarithm of weighted Gaussian similarities. A nearest-pattern comparison proves nonnegativity. For states in the convex hull of the memory vectors, Jensen’s inequality and norm bounds give both β⁻¹ ln N+M²/2 and 2M² upper bounds. Every retrieval enters this hull, but those upper bounds do not apply to an arbitrary unbounded initial query.

Equations 11–15
𝑿\displaystyle\bm{X}\ =(𝒙1,…,𝒙N).\displaystyle=\ \left(\bm{x}_{1},\ldots,\bm{x}_{N}\right)\ . (11)
M\displaystyle M\ =maxi⁡‖𝒙i‖.\displaystyle=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ . (12)
E\displaystyle\mathrm{E}\ =−lse⁡(β,𝑿T​𝝃)+12​𝝃T​𝝃+β−1​ln⁡N+12​M2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}M^{2} (13)
=−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+β−1​ln⁡N+12​𝝃T​𝝃+12​M2\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}M^{2} (14)
=−β−1​ln⁡(1N​∑i=1Nexp⁡(−12​β​(M2−‖𝒙i‖2))​exp⁡(−12​β​‖𝒙i−𝝃‖2)).\displaystyle=\ -\ \beta^{-1}\ln\left(\frac{1}{N}\ \sum_{i=1}^{N}\exp\left(-\ \frac{1}{2}\ \beta\ \left(M^{2}\ -\ {{\left\|\bm{x}_{i}\right\|}}^{2}\right)\right)\ \exp\left(-\ \frac{1}{2}\ \beta\ {{\left\|\bm{x}_{i}\ -\ \bm{\xi}\right\|}}^{2}\right)\right)\ . (15)
Lemma A1.

The energy E\mathrm{E} is larger than zero:

Equation 16
0\displaystyle 0\ ⩽E.\displaystyle\leqslant\ \mathrm{E}\ . (16)

For 𝛏\bm{\xi} in the simplex defined by the patterns, the energy E\mathrm{E} is upper bounded by:

Equations 17–18
E\displaystyle\mathrm{E}\ ⩽β−1​ln⁡N+12​M2,\displaystyle\leqslant\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2}\ , (17)
E\displaystyle\mathrm{E}\ ⩽ 2​M2.\displaystyle\leqslant\ 2\ M^{2}\ . (18)
Equations 19–27
𝒙𝝃\displaystyle\bm{x}_{\bm{\xi}}\ =𝒙k,k=arg⁡maxi​𝝃T​𝒙i.\displaystyle=\ \bm{x}_{k}\ ,\quad k\ =\ \arg\max_{i}\bm{\xi}^{T}\bm{x}_{i}\ . (19)
E\displaystyle\mathrm{E}\ =−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+β−1​ln⁡N+12​𝝃T​𝝃+12​M2\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ M^{2} (20)
=−β−1​ln⁡(1N​∑i=1Nexp⁡(β​𝒙iT​𝝃))+12​𝝃T​𝝃+12​M2\displaystyle=\ -\ \beta^{-1}\ln\left(\frac{1}{N}\ \sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ M^{2}
≥−β−1​ln⁡(1N​∑i=1Nexp⁡(β​𝒙iT​𝝃))+12​𝝃T​𝝃+12​𝒙𝝃T​𝒙𝝃\displaystyle\geq\ -\ \beta^{-1}\ln\left(\frac{1}{N}\ \sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{x}_{\bm{\xi}}^{T}\bm{x}_{\bm{\xi}}
≥−β−1​ln⁡(exp⁡(β​𝒙𝝃T​𝝃))+12​𝝃T​𝝃+12​𝒙𝝃T​𝒙𝝃\displaystyle\geq\ -\ \beta^{-1}\ln\left(\exp(\beta\bm{x}_{\bm{\xi}}^{T}\bm{\xi})\right)\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{x}_{\bm{\xi}}^{T}\bm{x}_{\bm{\xi}}
=−𝒙𝝃T​𝝃+12​𝝃T​𝝃+12​𝒙𝝃T​𝒙𝝃\displaystyle=\ -\ \bm{x}_{\bm{\xi}}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{x}_{\bm{\xi}}^{T}\bm{x}_{\bm{\xi}}
=12​(𝝃−𝒙𝝃)T​(𝝃−𝒙𝝃)=12​‖𝝃−𝒙𝝃‖2≥ 0.\displaystyle=\frac{1}{2}\ \left(\bm{\xi}\ -\ \bm{x}_{\bm{\xi}}\right)^{T}\left(\bm{\xi}\ -\ \bm{x}_{\bm{\xi}}\right)\ =\ \frac{1}{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{\bm{\xi}}\right\|}}^{2}\ \geq\ 0\ .
𝝃=∑i=1Npi​𝒙i,∑i=1Npi= 1,∀i: 0⩽pi.\displaystyle\bm{\xi}\ =\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\ ,\quad\sum_{i=1}^{N}p_{i}\ =\ 1\ ,\quad\forall_{i}:\ 0\ \leqslant\ p_{i}\ . (21)
E=−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+12​𝝃T​𝝃+β−1​ln⁡N+12​M2\displaystyle\mathrm{E}\ =\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2} (22)
⩽−∑i=1Npi(𝒙iT𝝃)+12𝝃T𝝃+β−1lnN+12M2\displaystyle\leqslant\ -\sum_{i=1}^{N}p_{i}\ (\bm{x}_{i}^{T}\bm{\xi})\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2}
=−12​𝝃T​𝝃+β−1​ln⁡N+12​M2⩽β−1​ln⁡N+12​M2.\displaystyle=\ -\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2}\ \leqslant\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2}\ .
−lse⁡(β,𝑿T​𝝃)\displaystyle-\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ ⩽−∑i=1Npi(𝒙iT𝝃)+β−1∑i=1Npilnpi⩽−∑i=1Npi(𝒙iT𝝃),\displaystyle\leqslant\ -\ \sum_{i=1}^{N}p_{i}\ (\bm{x}_{i}^{T}\bm{\xi})\ +\ \beta^{-1}\sum_{i=1}^{N}p_{i}\ln p_{i}\ \leqslant\ -\ \sum_{i=1}^{N}p_{i}\ (\bm{x}_{i}^{T}\bm{\xi})\ , (23)
𝒎𝒙\displaystyle\bm{m}_{\bm{x}}\ =1N​∑i=1N𝒙i.\displaystyle=\ \frac{1}{N}\ \sum_{i=1}^{N}\bm{x}_{i}\ . (24)
E=−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+12​𝝃T​𝝃+β−1​ln⁡N+12​M2\displaystyle\mathrm{E}\ =\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\ M^{2} (25)
⩽−∑i=1N1N𝒙iT𝝃+12𝝃T𝝃+12M2\displaystyle\leqslant\ -\sum_{i=1}^{N}\frac{1}{N}\ \bm{x}_{i}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ M^{2}
=−𝒎𝒙T​𝝃+12​𝝃T​𝝃+12​M2\displaystyle=\ -\ \bm{m}_{\bm{x}}^{T}\bm{\xi}\ +\ \frac{1}{2}\ \bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}\ M^{2}
⩽‖𝒎𝒙‖​‖𝝃‖+12​‖𝝃‖2+12​M2\displaystyle\leqslant\ {{\left\|\bm{m}_{\bm{x}}\right\|}}\ {{\left\|\bm{\xi}\right\|}}\ +\ \frac{1}{2}\ {{\left\|\bm{\xi}\right\|}}^{2}\ +\ \frac{1}{2}\ M^{2}
⩽ 2​M2,\displaystyle\leqslant\ 2\ M^{2}\ ,
‖𝝃‖\displaystyle{{\left\|\bm{\xi}\right\|}}\ =‖∑ipi​𝒙i‖⩽∑ipi​‖𝒙i‖⩽∑ipi​M=M\displaystyle=\ {{\left\|\sum_{i}p_{i}\ \bm{x}_{i}\right\|}}\ \leqslant\ \sum_{i}p_{i}\ {{\left\|\bm{x}_{i}\right\|}}\ \leqslant\ \sum_{i}p_{i}M\ =\ M (26)
‖𝒎𝒙‖\displaystyle{{\left\|\bm{m}_{\bm{x}}\right\|}}\ =‖∑i(1/N)​𝒙i‖⩽∑i(1/N)​‖𝒙i‖⩽∑i(1/N)​M=M.\displaystyle=\ {{\left\|\sum_{i}(1/N)\ \bm{x}_{i}\right\|}}\ \leqslant\ \sum_{i}(1/N)\ {{\left\|\bm{x}_{i}\right\|}}\ \leqslant\ \sum_{i}(1/N)\ M\ =\ M\ . (27)

A.1.3 New Update Rule

The update replaces the state by a softmax-weighted convex combination of stored patterns. This immediately confines subsequent states to their convex hull. Unlike the sign-based binary update, it remains continuous and differentiable.

Equations 28–30
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑿​𝒑=𝑿​softmax​(β​𝑿T​𝝃),\displaystyle=\ \bm{X}\bm{p}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ , (28)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ . (29)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =sgn(𝑿​𝑿T​𝝃).\displaystyle=\ \mathop{\mathrm{sgn}\,}(\bm{X}\bm{X}^{T}\bm{\xi})\ . (30)

A.1.4 Global Convergence of the Update Rule

Split the energy into a convex quadratic and concave negative log-sum-exp. Linearizing the concave term produces a tight quadratic upper bound whose unique minimizer is the retrieval update. The bound decreases strictly whenever the state changes. Compactness and the cited convergence results imply stationary limit points, vanishing step sizes and either convergence or a connected compact stationary limit set. A finite stationary set at the limiting energy ensures convergence to one point. These results establish energy descent, not global optimality or a uniform convergence rate.

Equations 31–34
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =f⁡(𝝃)=𝑿​𝒑=𝑿​softmax​(β​𝑿T​𝝃),\displaystyle=\ f(\bm{\xi})\ =\ \bm{X}\bm{p}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ , (31)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ . (32)
E\displaystyle\mathrm{E}\ =−lse⁡(β,𝑿T​𝝃)+12​𝝃T​𝝃+β−1​ln⁡N+12​M2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}M^{2} (33)
=−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+β−1​ln⁡N+12​𝝃T​𝝃+12​M2.\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}M^{2}\ . (34)
Theorem A1 (Global Convergence (Zangwill): Energy).

The update rule Eq. (31) converges globally: For 𝛏t+1=f⁡(𝛏t)\bm{\xi}^{t+1}=f(\bm{\xi}^{t}), the energy E⁡(𝛏t)→E⁡(𝛏∗)\mathrm{E}(\bm{\xi}^{t})\to\mathrm{E}(\bm{\xi}^{*}) for t→∞t\to\infty and a fixed point 𝛏∗\bm{\xi}^{*}.

Equations 35–55
E⁡(𝝃)\displaystyle\mathrm{E}(\bm{\xi})\ =E1​(𝝃)+E2​(𝝃),\displaystyle=\ \mathrm{E}_{1}(\bm{\xi})\ +\ \mathrm{E}_{2}(\bm{\xi})\ , (35)
E1​(𝝃)\displaystyle\mathrm{E}_{1}(\bm{\xi})\ =12​𝝃T​𝝃+β−1​ln⁡N+12​M2=12​𝝃T​𝝃+C1,\displaystyle=\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}M^{2}\ =\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ C_{1}\ , (36)
E2​(𝝃)\displaystyle\mathrm{E}_{2}(\bm{\xi})\ =−lse⁡(β,𝑿T​𝝃),\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ , (37)
∇ξE1​(𝝃t+1)\displaystyle\nabla_{\xi}\mathrm{E}_{1}\left(\bm{\xi}^{t+1}\right)\ =−∇ξE2​(𝝃t),\displaystyle=\ -\ \nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ , (38)
∇ξ(12​𝝃T​𝝃+C1)​(𝝃t+1)\displaystyle\nabla_{\xi}\left(\frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ C_{1}\right)\left(\bm{\xi}^{t+1}\right)\ =∇ξlse​(β,𝑿T​𝝃t).\displaystyle=\ \nabla_{\xi}\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi}^{t})\ . (39)
𝝃t+1\displaystyle\bm{\xi}^{t+1}\ =𝑿​𝒑t=𝑿​softmax​(β​𝑿T​𝝃t)\displaystyle=\ \bm{X}\bm{p}^{t}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}^{t}) (40)
𝒑t\displaystyle\bm{p}^{t}\ =softmax⁡(β​𝑿T​𝝃t).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}^{t})\ . (41)
min𝝃\displaystyle\min_{\bm{\xi}}  ​E1+E2\displaystyle{\mbox{\ ~}}\mathrm{E}_{1}\ +\ \mathrm{E}_{2} (42)
s.t.  ​𝒄​(𝝃)⩽𝟎,𝒅⁡(𝝃)= 0\displaystyle{\mbox{\ ~}}\bm{c}(\bm{\xi})\leqslant\bm{0}\ ,\quad\bm{d}(\bm{\xi})\ =\ \bm{0}
𝝃t+1∈arg⁡min𝝃\displaystyle\bm{\xi}^{t+1}\ \in\ \arg\min_{\bm{\xi}}  ​E1​(𝝃)+𝝃T​∇ξE2​(𝝃t)\displaystyle{\mbox{\ ~}}\mathrm{E}_{1}\left(\bm{\xi}\right)\ +\ \bm{\xi}^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right) (43)
s.t.  ​𝒄​(𝝃)⩽𝟎,𝒅⁡(𝝃)= 0.\displaystyle{\mbox{\ ~}}\bm{c}(\bm{\xi})\leqslant\bm{0}\ ,\quad\bm{d}(\bm{\xi})\ =\ \bm{0}\ .
EC​(𝝃,𝝃t)\displaystyle\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)\ :=E1​(𝝃)+E2​(𝝃t)+(𝝃−𝝃t)T​∇ξE2​(𝝃t).\displaystyle:=\ \mathrm{E}_{1}\left(\bm{\xi}\right)\ +\ \mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ +\ \left(\bm{\xi}\ -\ \bm{\xi}^{t}\right)^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ . (44)
EC​(𝝃t,𝝃t)\displaystyle\mathrm{E}_{\mathrm{C}}\left(\bm{\xi}^{t},\bm{\xi}^{t}\right)\ =E1​(𝝃t)+E2​(𝝃t)=E⁡(𝝃t).\displaystyle=\ \mathrm{E}_{1}\left(\bm{\xi}^{t}\right)\ +\ \mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ =\ \mathrm{E}\left(\bm{\xi}^{t}\right)\ . (45)
−E2​(𝝃)\displaystyle-\ \mathrm{E}_{2}\left(\bm{\xi}\right)\ ≥−E2​(𝝃t)−(𝝃−𝝃t)T​∇ξE2​(𝝃t),\displaystyle\geq\ -\ \mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ -\ \left(\bm{\xi}\ -\ \bm{\xi}^{t}\right)^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ , (46)
E2​(𝝃)\displaystyle\mathrm{E}_{2}\left(\bm{\xi}\right)\ ⩽E2​(𝝃t)+(𝝃−𝝃t)T​∇ξE2​(𝝃t).\displaystyle\leqslant\ \mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ +\ \left(\bm{\xi}\ -\ \bm{\xi}^{t}\right)^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ . (47)
E⁡(𝝃)\displaystyle\mathrm{E}\left(\bm{\xi}\right)\ ⩽EC​(𝝃,𝝃t)=E1​(𝝃)+E2​(𝝃t)+(𝝃−𝝃t)T​∇ξE2​(𝝃t)\displaystyle\leqslant\ \mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)\ =\ \mathrm{E}_{1}\left(\bm{\xi}\right)\ +\ \mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ +\ \left(\bm{\xi}\ -\ \bm{\xi}^{t}\right)^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right) (48)
=E1​(𝝃)+𝝃T​∇ξE2​(𝝃t)+C2,\displaystyle=\ \mathrm{E}_{1}\left(\bm{\xi}\right)\ +\ \bm{\xi}^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ +\ C_{2}\ ,
𝝃t+1∈arg⁡min𝝃\displaystyle\bm{\xi}^{t+1}\ \in\ \arg\min_{\bm{\xi}}  ​EC​(𝝃,𝝃t),\displaystyle{\mbox{\ ~}}\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)\ , (49)
E⁡(𝝃t+1)\displaystyle\mathrm{E}\left(\bm{\xi}^{t+1}\right)\ ⩽EC​(𝝃t+1,𝝃t)⩽EC​(𝝃t,𝝃t)=E⁡(𝝃t).\displaystyle\leqslant\ \mathrm{E}_{\mathrm{C}}\left(\bm{\xi}^{t+1},\bm{\xi}^{t}\right)\ \leqslant\ \mathrm{E}_{\mathrm{C}}\left(\bm{\xi}^{t},\bm{\xi}^{t}\right)\ =\ \mathrm{E}\left(\bm{\xi}^{t}\right)\ . (50)
EC​(𝝃,𝝃t)\displaystyle\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)\ =E1​(𝝃)+𝝃T​∇ξE2​(𝝃t)+C2\displaystyle=\ \mathrm{E}_{1}\left(\bm{\xi}\right)\ +\ \bm{\xi}^{T}\nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ +\ C_{2} (51)
∂EC​(𝝃,𝝃t)∂𝝃\displaystyle\frac{\partial\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)}{\partial\bm{\xi}}\ =𝝃+∇ξE2​(𝝃t)=𝝃−𝑿​softmax​(β​𝑿T​𝝃t)= 0\displaystyle=\ \bm{\xi}\ +\ \nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ =\ \bm{\xi}\ -\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}^{t})\ =\ \bm{0} (52)
∂2EC​(𝝃,𝝃t)∂𝝃2\displaystyle\frac{\partial^{2}\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)}{\partial\bm{\xi}^{2}}\ =𝑰.\displaystyle=\ \bm{I}\ . (53)
EC​(𝝃,𝝃t)\displaystyle\mathrm{E}_{\mathrm{C}}\left(\bm{\xi},\bm{\xi}^{t}\right)\ =12​(𝝃+∇ξE2​(𝝃t))T​(𝝃+∇ξE2​(𝝃t))+C3,\displaystyle=\ \frac{1}{2}\ \left(\bm{\xi}\ +\ \nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\right)^{T}\left(\bm{\xi}\ +\ \nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\right)\ +\ C_{3}\ , (54)
𝝃t+1\displaystyle\bm{\xi}^{t+1}\ =−∇ξE2​(𝝃t)=𝑿​softmax​(β​𝑿T​𝝃t)\displaystyle=\ -\ \nabla_{\xi}\mathrm{E}_{2}\left(\bm{\xi}^{t}\right)\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}^{t}) (55)
Theorem A2 (Global Convergence: Stationary Points).

For the iteration Eq. (55) we have E⁡(𝛏t)→E⁡(𝛏∗)=E∗\mathrm{E}\left(\bm{\xi}^{t}\right)\to\mathrm{E}\left(\bm{\xi}^{*}\right)=\mathrm{E}^{*} as t→∞t\to\infty, for some stationary point 𝛏∗\bm{\xi}^{*}. Furthermore ‖𝛏t+1−𝛏t‖→0{{\left\|\bm{\xi}^{t+1}-\bm{\xi}^{t}\right\|}}\to 0 and either {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} converges or, in the other case, the set of limit points of {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} is a connected and compact subset of ℒ⁡(E∗)\mathcal{L}\left(\mathrm{E}^{*}\right), where ℒ⁡(a)={𝛏∈ℒ∣E⁡(𝛏)=a}\mathcal{L}\left(a\right)=\{\bm{\xi}\in\mathcal{L}\mid\mathrm{E}\left(\bm{\xi}\right)=a\} and ℒ\mathcal{L} is the set of stationary points of the iteration Eq. (55). If ℒ⁡(E∗)\mathcal{L}\left(\mathrm{E}^{*}\right) is finite, then any sequence {𝛏t}t=0∞\{\bm{\xi}^{t}\}_{t=0}^{\infty} generated by the iteration Eq. (55) converges to some 𝛏∗∈ℒ⁡(E∗)\bm{\xi}^{*}\in\mathcal{L}\left(\mathrm{E}^{*}\right).

Equation 56
E⁡(𝝃t+1)\displaystyle\mathrm{E}\left(\bm{\xi}^{t+1}\right)\ ⩽EC​(𝝃t+1,𝝃t)<EC​(𝝃t,𝝃t)=E⁡(𝝃t).\displaystyle\leqslant\ \mathrm{E}_{\mathrm{C}}\left(\bm{\xi}^{t+1},\bm{\xi}^{t}\right)\ <\ \mathrm{E}_{\mathrm{C}}\left(\bm{\xi}^{t},\bm{\xi}^{t}\right)\ =\ \mathrm{E}\left(\bm{\xi}^{t}\right)\ . (56)

A.1.5 Local Convergence of the Update Rule: Fixed Point Iteration

Local convergence uses Banach’s fixed-point theorem: first find a closed invariant neighborhood, then bound the retrieval Jacobian below one throughout it. A small Jacobian also yields a fast rate.

A.1.5.1 General Bound on the Jacobian of the Iteration.

The symmetric Jacobian is βX(diag(p)−ppᵀ)Xᵀ, equivalently β times the attention-weighted covariance of stored patterns. Its norm is bounded using pattern norms and max pᵢ(1−pᵢ); concentration near one pattern makes it small. Separation Δᵢ measures the self-dot-product advantage over competing memories. For equal-norm patterns it equals half the nearest squared Euclidean distance. Figure A.1 illustrates the three regimes, while the following lemmas give their sufficient conditions.

Equations 57–59
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =f⁡(𝝃)=𝑿​𝒑=𝑿​softmax​(β​𝑿T​𝝃)\displaystyle=\ f(\bm{\xi})\ =\ \bm{X}\bm{p}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}) (57)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ . (58)
J\displaystyle\mathrm{J}\ =∂f⁡(𝝃)∂𝝃=β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T=𝑿​Js​𝑿T,\displaystyle=\ \frac{\partial f(\bm{\xi})}{\partial\bm{\xi}}\ =\ \beta\ \bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T}\ =\ \bm{X}\mathrm{J}_{s}\bm{X}^{T}\ , (59)
Complete original Figure A.1: The three cases of fixed points. a) Stored patterns (fixed point is single pattern) :
patterns are stored if they are well separated.
Each pattern 𝒙 i \bm{x}_{i} has a single fixed point 𝒙 i ∗ \bm{x}_{i}^{*} close to it.
In the sphere S i \mathrm{S}_{i} , pattern 𝒙 i \bm{x}_{i} is the only pattern and 𝒙 i ∗ \bm{x}_{i}^{*} the only fixed point. b) Metastable state (fixed point is average of similar patterns) : 𝒙 i \bm{x}_{i} and 𝒙 j \bm{x}_{j} are similar to each other and not
well separated. The fixed point 𝒎 𝒙 ∗ \bm{m}_{\bm{x}}^{*} is a metastable state
that is close to the mean 𝒎 𝒙 \bm{m}_{\bm{x}} of the similar patterns. c) Global fixed point (fixed point is average of all patterns) :
no pattern is well separated from the others.
A single global fixed point 𝒎 𝒙 ∗ \bm{m}_{\bm{x}}^{*} exists that is close to
the arithmetic mean 𝒎 𝒙 \bm{m}_{\bm{x}} of all patterns.
Figure A.1: The three cases of fixed points. a) Stored patterns (fixed point is single pattern): patterns are stored if they are well separated. Each pattern 𝒙i\bm{x}_{i} has a single fixed point 𝒙i∗\bm{x}_{i}^{*} close to it. In the sphere Si\mathrm{S}_{i}, pattern 𝒙i\bm{x}_{i} is the only pattern and 𝒙i∗\bm{x}_{i}^{*} the only fixed point. b) Metastable state (fixed point is average of similar patterns): 𝒙i\bm{x}_{i} and 𝒙j\bm{x}_{j} are similar to each other and not well separated. The fixed point 𝒎𝒙∗\bm{m}_{\bm{x}}^{*} is a metastable state that is close to the mean 𝒎𝒙\bm{m}_{\bm{x}} of the similar patterns. c) Global fixed point (fixed point is average of all patterns): no pattern is well separated from the others. A single global fixed point 𝒎𝒙∗\bm{m}_{\bm{x}}^{*} exists that is close to the arithmetic mean 𝒎𝒙\bm{m}_{\bm{x}} of all patterns.
Lemma A2.

For NN patterns 𝐗=(𝐱1,…,𝐱N)\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N}), 𝐩=softmax⁡(β​𝐗T​𝛏)\bm{p}=\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}), M=maxi⁡‖𝐱i‖M=\max_{i}{{\left\|\bm{x}_{i}\right\|}}, and m=maxi⁡pi​(1−pi)m=\max_{i}p_{i}(1-p_{i}), the spectral norm of the Jacobian J\mathrm{J} of the fixed point iteration is bounded:

Equation 60
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽ 2​β​‖𝑿‖22​m⩽ 2​β​N​M2​m.\displaystyle\leqslant\ 2\ \beta\ {{\left\|\bm{X}\right\|}}_{2}^{2}\ m\ \leqslant\ 2\ \beta\ N\ M^{2}\ m\ . (60)

If pmax=maxi⁡pi≥1−ϵp_{\max}=\max_{i}p_{i}\geq 1-\epsilon, then for the spectral norm of the Jacobian holds

Equation 61
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽ 2​β​N​M2​ϵ− 2​ϵ2​β​N​M2< 2​β​N​M2​ϵ.\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ \epsilon\ -\ 2\ \epsilon^{2}\ \beta\ N\ M^{2}\ <\ 2\ \beta\ N\ M^{2}\ \epsilon\ . (61)
Equations 62–69
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃),\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ , (62)
J\displaystyle\mathrm{J}\ =∂f⁡(𝝃)∂𝝃=β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T=𝑿​Js​𝑿T,\displaystyle=\ \frac{\partial f(\bm{\xi})}{\partial\bm{\xi}}\ =\ \beta\ \bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T}\ =\ \bm{X}\mathrm{J}_{s}\bm{X}^{T}\ , (63)
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ =β​‖diag⁡(𝒑)−𝒑​𝒑T‖2⩽ 2​m​β.\displaystyle=\ \beta\ {{\left\|\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right\|}}_{2}\ \leqslant\ 2\ m\ \beta\ . (64)
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β​‖𝑿T‖2​‖Js‖2​‖𝑿‖2⩽ 2​m​β​‖𝑿‖22.\displaystyle\leqslant\ \beta\ {{\left\|\bm{X}^{T}\right\|}}_{2}\ {{\left\|\mathrm{J}_{s}\right\|}}_{2}\ {{\left\|\bm{X}\right\|}}_{2}\ \leqslant\ 2\ m\ \beta\ {{\left\|\bm{X}\right\|}}_{2}^{2}\ . (65)
‖𝑿‖2\displaystyle{{\left\|\bm{X}\right\|}}_{2}\ ⩽‖𝑿‖F=∑i‖𝒙i‖2.\displaystyle\leqslant\ {{\left\|\bm{X}\right\|}}_{F}\ =\ \sqrt{\sum_{i}{{\left\|\bm{x}_{i}\right\|}}^{2}}\ . (66)
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽ 2​β​‖𝑿‖22​m⩽ 2​β​N​M2​m.\displaystyle\leqslant\ 2\ \beta\ {{\left\|\bm{X}\right\|}}_{2}^{2}\ m\ \leqslant\ 2\ \beta\ N\ M^{2}\ m\ . (67)
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ ⩽ 2​β​ϵ− 2​ϵ2​β< 2​β​ϵ.\displaystyle\leqslant\ 2\ \beta\ \epsilon\ -\ 2\ \epsilon^{2}\ \beta\ <\ 2\ \beta\ \epsilon\ . (68)
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽ 2​β​N​M2​ϵ− 2​ϵ2​β​N​M2< 2​β​N​M2​ϵ.\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ \epsilon\ -\ 2\ \epsilon^{2}\ \beta\ N\ M^{2}\ <\ 2\ \beta\ N\ M^{2}\ \epsilon\ . (69)
Definition 2 (Separation of Patterns).

We define Δi\Delta_{i}, i.e. the separation of pattern 𝐱i\bm{x}_{i} from data 𝐗=(𝐱1,…,𝐱N)\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N}) as:

Equation 70
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (70)

The pattern is separated from the other data if 0<Δi0<\Delta_{i}. Using the parallelogram identity, Δi\Delta_{i} can also be expressed as

Equation 71
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡12​(‖𝒙i‖2−‖𝒙j‖2+‖𝒙i−𝒙j‖2)\displaystyle=\ \min_{j,j\not=i}\frac{1}{2}\ \left({{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{j}\right\|}}^{2}\ +\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right) (71)
=12​‖𝒙i‖2−12​maxj,j≠i⁡(‖𝒙j‖2−‖𝒙i−𝒙j‖2).\displaystyle=\ \frac{1}{2}{{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ \frac{1}{2}\ \max_{j,j\not=i}\left({{\left\|\bm{x}_{j}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right)\ .

For ‖𝐱i‖=‖𝐱j‖{{\left\|\bm{x}_{i}\right\|}}={{\left\|\bm{x}_{j}\right\|}} we have Δi=1/2​minj,j≠i​‖𝐱i−𝐱j‖2\Delta_{i}=1/2\min_{j,j\not=i}{{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}.

Analog we say for a query 𝛏\bm{\xi} and data 𝐗=(𝐱1,…,𝐱N)\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N}), that 𝐱i\bm{x}_{i} is least separated from 𝛏\bm{\xi} while being separated from other 𝐱j\bm{x}_{j} with j≠ij\not=i if

Equations 72–73
i\displaystyle i\ =arg⁡maxk​minj,j≠k⁡(𝝃T​𝒙k−𝝃T​𝒙j)=arg⁡maxk⁡(𝝃T​𝒙k−maxj,j≠k⁡𝝃T​𝒙j)\displaystyle=\ \arg\max_{k}\min_{j,j\not=k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ =\ \arg\max_{k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \max_{j,j\not=k}\bm{\xi}^{T}\bm{x}_{j}\right) (72)
0\displaystyle 0\ ⩽c=maxk⁡minj,j≠k⁡(𝝃T​𝒙k−𝝃T​𝒙j)=maxk⁡(𝝃T​𝒙k−maxj,j≠k⁡𝝃T​𝒙j).\displaystyle\leqslant\ c\ =\ \max_{k}\min_{j,j\not=k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ =\ \max_{k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \max_{j,j\not=k}\bm{\xi}^{T}\bm{x}_{j}\right)\ . (73)

A.1.5.2 One Stable State: Fixed Point Near the Mean of the Patterns.

Let m be the pattern mean and r=maxᵢ‖xᵢ−m‖. The covariance bound gives ‖J‖₂≤βr², so βr²<1 makes retrieval contractive on its invariant convex hull and yields a unique fixed point. If the data are centered, zero is exactly a fixed point with uniform weights. Further estimates relate near-uniform attention to spread, centering and the Gram matrix. Uniform-looking weights alone do not imply small covariance for arbitrarily large pattern norms. The source’s phrase “in each compact set” also needs the map-to-itself condition required by Banach’s theorem.

Equations 74–81
𝑿⁡(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T\displaystyle\bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T} (74)
𝑿⁡(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T=𝑿​diag​(𝒑)​𝑿T−𝑿​𝒑​𝒑T​𝑿T\displaystyle\bm{X}\left(\mathrm{diag}(\bm{p})\ -\ \bm{p}\bm{p}^{T}\right)\bm{X}^{T}\ =\ \bm{X}\mathrm{diag}(\bm{p})\bm{X}^{T}\ -\ \bm{X}\bm{p}\bm{p}^{T}\bm{X}^{T} (75)
=∑i=1Npi​𝒙i​𝒙iT−(∑i=1Npi​𝒙i)​(∑i=1Npi​𝒙i)T\displaystyle=\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)^{T} (76)
=E𝒑​[𝒙​𝒙T]−E𝒑​[𝒙]​E𝒑​[𝒙]T=Var𝒑​[𝒙],\displaystyle=\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}\ \bm{x}^{T}]\ -\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}]\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}]^{T}\ =\ \mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\ , (77)
J\displaystyle\mathrm{J}\ =β​Var𝒑​[𝒙].\displaystyle=\ \beta\ \mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\ . (78)
𝒎𝒙\displaystyle\bm{m}_{\bm{x}}\ =1N​∑i=1N𝒙i,\displaystyle=\ \frac{1}{N}\ \sum_{i=1}^{N}\ \bm{x}_{i}\ , (79)
mmax\displaystyle m_{\max}\ =max1⩽i⩽N⁡‖𝒙i−𝒎𝒙‖2.\displaystyle=\ \max_{1\leqslant i\leqslant N}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}_{2}\ . (80)
Var𝒑​[𝒙]\displaystyle\mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\ =∑i=1Npi​𝒙i​𝒙iT−(∑i=1Npi​𝒙i)​(∑i=1Npi​𝒙i)T\displaystyle=\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)^{T} (81)
=∑i=1Npi​(𝒙i−∑i=1Npi​𝒙i)​(𝒙i−∑i=1Npi​𝒙i)T.\displaystyle=\ \sum_{i=1}^{N}p_{i}\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{N}p_{i}\bm{x}_{i}\right)\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{N}p_{i}\bm{x}_{i}\right)^{T}\ .
Lemma A3.

The following bound on the norm ‖J‖2{{\left\|\mathrm{J}\right\|}}_{2} of the Jacobian of the fixed point iteration ff holds independent of 𝐩\bm{p} or the query 𝛏\bm{\xi}.

Equation 82
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β​mmax2.\displaystyle\leqslant\ \beta\ m_{\max}^{2}\ . (82)

For β​mmax2<1\beta\ m_{\max}^{2}<1 there exists a unique fixed point (global fixed point) of iteration ff in each compact set.

Equations 83–129
f⁡(𝒂)\displaystyle f(\bm{a})\ =∑i=1Npi​‖𝒙i−𝒂‖2=∑i=1Npi​(𝒙i−𝒂)T​(𝒙i−𝒂).\displaystyle=\ \sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{a}\right\|}}^{2}\ =\ \sum_{i=1}^{N}p_{i}(\bm{x}_{i}\ -\ \bm{a})^{T}(\bm{x}_{i}\ -\ \bm{a})\ . (83)
∂f⁡(𝒂)∂𝒂\displaystyle\frac{\partial f(\bm{a})}{\partial\bm{a}}\ = 2​∑i=1Npi​(𝒂−𝒙i)= 0\displaystyle=\ 2\ \sum_{i=1}^{N}p_{i}(\bm{a}\ -\ \bm{x}_{i})\ =\ 0 (84)
𝒂\displaystyle\bm{a}\ =∑i=1Npi​𝒙i.\displaystyle=\ \sum_{i=1}^{N}p_{i}\bm{x}_{i}\ . (85)
∂2f⁡(𝒂)∂𝒂2\displaystyle\frac{\partial^{2}f(\bm{a})}{\partial\bm{a}^{2}}\ = 2​∑i=1Npi​𝑰= 2​𝑰\displaystyle=\ 2\ \sum_{i=1}^{N}p_{i}\ \bm{I}\ =\ 2\ \bm{I} (86)
𝒙¯\displaystyle\bar{\bm{x}}\ :=∑i=1Npi​𝒙i\displaystyle:=\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i} (87)
∑i=1Npi​‖𝒙i−𝒙¯‖2\displaystyle\sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ ⩽∑i=1Npi​‖𝒙i−𝒎𝒙‖2⩽mmax2.\displaystyle\leqslant\ \sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}^{2}\ \leqslant\ m_{\max}^{2}\ . (88)
‖𝒂​𝒃T‖2\displaystyle{{\left\|\bm{a}\bm{b}^{T}\right\|}}_{2}\ =λmax​(𝒃​𝒂T​𝒂​𝒃T)=‖𝒂‖​λmax​(𝒃​𝒃T)=‖𝒂‖​‖𝒃‖,\displaystyle=\ \sqrt{\lambda_{\max}(\bm{b}\bm{a}^{T}\bm{a}\bm{b}^{T})}\ =\ {{\left\|\bm{a}\right\|}}\ \sqrt{\lambda_{\max}(\bm{b}\bm{b}^{T})}\ =\ {{\left\|\bm{a}\right\|}}\ {{\left\|\bm{b}\right\|}}\ , (89)
‖Var𝒑​[𝒙]‖2\displaystyle{{\left\|\mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\right\|}}_{2}\ ⩽∑i=1Npi​‖(𝒙i−𝒙¯)​(𝒙i−𝒙¯)T‖2\displaystyle\leqslant\ \sum_{i=1}^{N}p_{i}{{\left\|\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right\|}}_{2} (90)
=∑i=1Npi‖𝒙i−𝒙¯‖2⩽∑i=1Npi‖𝒙i−𝒎𝒙‖2⩽mmax2.\displaystyle=\ \sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ \leqslant\ \sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}^{2}\ \leqslant\ \ m_{\max}^{2}\ .
‖Var𝒑​[𝒙]‖2\displaystyle{{\left\|\mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\right\|}}_{2}\ =Tr⁡(∑i=1Npi​(𝒙i−𝒙¯)​(𝒙i−𝒙¯)T)−∑k=2dek\displaystyle=\ \mathbf{\mathrm{Tr}}\left(\sum_{i=1}^{N}p_{i}\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right)\ -\ \sum_{k=2}^{d}e_{k} (91)
=∑i=1Npi​Tr​((𝒙i−𝒙¯)​(𝒙i−𝒙¯)T)−∑k=2dek\displaystyle=\ \sum_{i=1}^{N}p_{i}\mathbf{\mathrm{Tr}}\left(\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right)\ -\ \sum_{k=2}^{d}e_{k}
=∑i=1Npi​‖𝒙i−𝒙¯‖2−∑k=2dek.\displaystyle=\ \sum_{i=1}^{N}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ -\ \sum_{k=2}^{d}e_{k}\ .
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =f⁡(𝝃)=𝑿​𝒑=𝑿​softmax​(β​𝑿T​𝝃)\displaystyle=\ f(\bm{\xi})\ =\ \bm{X}\bm{p}\ =\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi}) (92)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ . (93)
𝒑\displaystyle\bm{p}\ = 1/N​ 1\displaystyle=\ 1/N\ \bm{1} (94)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ = 1/N​𝑿​ 1=𝒎𝒙=𝝃.\displaystyle=\ 1/N\ \bm{X}\ \bm{1}\ =\ \bm{m}_{\bm{x}}\ =\ \bm{\xi}\ . (95)
|𝝃T​(𝒙i−𝒙j)|\displaystyle\left|\bm{\xi}^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽‖𝝃‖​‖𝒙i−𝒙j‖⩽‖𝝃‖​(‖𝒙i−𝒎𝒙‖+‖𝒙j−𝒎𝒙‖)\displaystyle\leqslant\ {{\left\|\bm{\xi}\right\|}}\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\bm{\xi}\right\|}}\ ({{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}\ +\ {{\left\|\bm{x}_{j}\ -\ \bm{m}_{\bm{x}}\right\|}}) (96)
⩽ 2​mmax​‖𝝃‖.\displaystyle\leqslant\ 2\ m_{\max}\ {{\left\|\bm{\xi}\right\|}}\ .
|𝝃T​(𝒙i−𝒙j)|\displaystyle\left|\bm{\xi}^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽ 2​mmax​(mmax+‖𝒎𝒙‖),\displaystyle\leqslant\ 2\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}})\ , (97)
|𝝃T​(𝒙i−𝒙j)|\displaystyle\left|\bm{\xi}^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽ 2​mmax​M,\displaystyle\leqslant\ 2\ m_{\max}\ M\ ,
β​|𝒎𝒙T​(𝒙i−𝒙j)|\displaystyle\beta\ \left|\bm{m}_{\bm{x}}^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽ 2​β​mmax​‖𝒎𝒙‖,\displaystyle\leqslant\ 2\ \beta\ m_{\max}\ {{\left\|\bm{m}_{\bm{x}}\right\|}}\ , (98)
β​|(𝒎𝒙∗)T​(𝒙i−𝒙j)|\displaystyle\beta\ \left|(\bm{m}_{\bm{x}}^{*})^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽ 2​β​mmax​‖𝒎𝒙∗‖⩽ 2​β​mmax​(mmax+‖𝒎𝒙‖),\displaystyle\leqslant\ 2\ \beta\ m_{\max}\ {{\left\|\bm{m}_{\bm{x}}^{*}\right\|}}\ \leqslant\ 2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}})\ , (99)
β​|𝒙iT​(𝒙i−𝒙j)|\displaystyle\beta\ \left|\bm{x}_{i}^{T}(\bm{x}_{i}\ -\ \bm{x}_{j})\right|\ ⩽ 2​β​mmax​‖𝒙i‖⩽ 2​β​mmax​(mmax+‖𝒎𝒙‖).\displaystyle\leqslant\ 2\ \beta\ m_{\max}\ {{\left\|\bm{x}_{i}\right\|}}\ \leqslant\ 2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}})\ . (100)
[softmax⁡(β​𝑿T​𝝃)]i=11+∑j≠iexp⁡(−β⁡(𝝃T​𝒙i−𝝃T​𝒙j))\displaystyle[\mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\xi})]_{i}\ =\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(-\ \beta\ (\bm{\xi}^{T}\bm{x}_{i}\ -\ \bm{\xi}^{T}\bm{x}_{j}))} (101)
⩽11+∑j≠iexp⁡(− 2​β​mmax​(mmax+‖𝒎𝒙‖))\displaystyle\leqslant\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(-\ 2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))}
=11+(N−1)​exp⁡(− 2​β​mmax​(mmax+‖𝒎𝒙‖))\displaystyle=\ \frac{1}{1\ +\ (N-1)\exp(-\ 2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))}
=exp⁡(2​β​mmax​(mmax+‖𝒎𝒙‖))exp⁡(2​β​mmax​(mmax+‖𝒎𝒙‖))+(N−1)\displaystyle=\ \frac{\exp(2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))}{\exp(2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))\ +\ (N-1)}
⩽ 1/N​exp⁡(2​β​mmax​(mmax+‖𝒎𝒙‖)).\displaystyle\leqslant\ 1/N\ \exp(2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))\ .
[softmax⁡(β​𝑿T​𝒎𝒙)]i\displaystyle[\mathrm{softmax}(\beta\ \bm{X}^{T}\bm{m}_{\bm{x}})]_{i}\ ⩽ 1/N​exp⁡(2​β​mmax​‖𝒎𝒙‖).\displaystyle\leqslant\ 1/N\ \exp(2\ \beta\ m_{\max}\ {{\left\|\bm{m}_{\bm{x}}\right\|}})\ . (102)
[softmax⁡(β​𝑿T​𝒎𝒙∗)]i\displaystyle[\mathrm{softmax}(\beta\ \bm{X}^{T}\bm{m}_{\bm{x}}^{*})]_{i}\ ⩽ 1/N​exp⁡(2​β​mmax​‖𝒎𝒙∗‖)\displaystyle\leqslant\ 1/N\ \exp(2\ \beta\ m_{\max}\ {{\left\|\bm{m}_{\bm{x}}^{*}\right\|}}) (103)
⩽ 1/N​exp⁡(2​β​mmax​(mmax+‖𝒎𝒙‖)).\displaystyle\leqslant\ 1/N\ \exp(2\ \beta\ m_{\max}\ (m_{\max}\ +\ {{\left\|\bm{m}_{\bm{x}}\right\|}}))\ .
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ < 1.\displaystyle<\ 1\ . (104)
𝒑new\displaystyle\bm{p}^{\mathrm{new}}\ =g⁡(𝒑)=softmax⁡(β​𝑿T​𝑿​𝒑).\displaystyle=\ g(\bm{p})\ =\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{X}\bm{p})\ . (105)
J⁡(𝒑)\displaystyle\mathrm{J}(\bm{p})\ =∂g⁡(𝒑)∂𝒑=𝑿T​𝑿​Js\displaystyle=\ \frac{\partial g(\bm{p})}{\partial\bm{p}}\ =\ \bm{X}^{T}\bm{X}\ \mathrm{J}_{s} (106)
Js​(𝒑new)\displaystyle\mathrm{J}_{s}(\bm{p}^{\mathrm{new}})\ =β⁡(diag⁡(𝒑new)−𝒑new​(𝒑new)T).\displaystyle=\ \beta\left(\mathrm{diag}(\bm{p}^{\mathrm{new}})\ -\ \bm{p}^{\mathrm{new}}(\bm{p}^{\mathrm{new}})^{T}\right)\ . (107)
𝒑new\displaystyle\bm{p}^{\mathrm{new}}\ =g⁡(𝒑)=g⁡(𝟎)+(Jm)T​𝒑=g⁡(𝟎)+Jsm​𝑿T​𝑿​𝒑= 1/N​ 1+Jsm​𝑿T​𝑿​𝒑.\displaystyle=\ g(\bm{p})\ =\ g(\bm{0})\ +\ (\mathrm{J}^{m})^{T}\bm{p}\ =\ g(\bm{0})\ +\ \mathrm{J}_{s}^{m}\ \bm{X}^{T}\bm{X}\ \bm{p}\ =\ 1/N\ \bm{1}\ +\ \mathrm{J}_{s}^{m}\ \bm{X}^{T}\bm{X}\ \bm{p}\ . (108)
‖Js​(𝒑)‖2\displaystyle{{\left\|\mathrm{J}_{s}(\bm{p})\right\|}}_{2}\ =β​‖diag⁡(𝒑)−𝒑​𝒑T‖2⩽ 2​m​β.\displaystyle=\ \beta\ {{\left\|\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right\|}}_{2}\ \leqslant\ 2\ m\ \beta\ . (109)
‖Jsm‖2\displaystyle{{\left\|\mathrm{J}_{s}^{m}\right\|}}_{2}\ ⩽ 2​m​β.\displaystyle\leqslant\ 2\ m\ \beta\ . (110)
‖Jsm‖2\displaystyle{{\left\|\mathrm{J}_{s}^{m}\right\|}}_{2}\ ⩽ 0.5​β.\displaystyle\leqslant\ 0.5\ \beta\ . (111)
‖Jsm‖2\displaystyle{{\left\|\mathrm{J}_{s}^{m}\right\|}}_{2}\ ⩽ 2​min⁡{0.25,pmax}​β.\displaystyle\leqslant\ 2\ \min\{0.25,p_{\max}\}\ \beta\ . (112)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽N​M2​ 2​min⁡{0.25,pmax}​β,\displaystyle\leqslant\ N\ M^{2}\ 2\ \min\{0.25,p_{\max}\}\ \beta\ , (113)
N​M2​ 2​min⁡{0.25,pmax}​β< 1.\displaystyle N\ M^{2}\ 2\ \min\{0.25,p_{\max}\}\ \beta\ <\ 1\ . (114)
𝒑new\displaystyle\bm{p}^{\mathrm{new}}\ =g⁡(𝒑)= 1/N​ 1+Jm​𝒑,\displaystyle=\ g(\bm{p})\ =\ 1/N\ \bm{1}\ +\ \mathrm{J}^{m}\ \bm{p}\ , (115)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽N​M2​ 2​min⁡{0.25,pmax}​β=c< 1.\displaystyle\leqslant\ N\ M^{2}\ 2\ \min\{0.25,p_{\max}\}\ \beta\ =\ c\ <\ 1\ . (116)
‖𝒑‖2\displaystyle{{\left\|\bm{p}\right\|}}^{2}\ =𝒑T​𝒑⩽‖𝒑‖1​‖𝒑‖∞=‖𝒑‖∞=pmax.\displaystyle=\ \bm{p}^{T}\bm{p}\ \leqslant\ {{\left\|\bm{p}\right\|}}_{1}{{\left\|\bm{p}\right\|}}_{\infty}\ =\ {{\left\|\bm{p}\right\|}}_{\infty}\ =\ p_{\max}\ . (117)
‖𝒑‖2\displaystyle{{\left\|\bm{p}\right\|}}^{2}\ =∑ipi2=pmax​∑ipipmax​pi⩽pmax​∑ipi=pmax.\displaystyle=\ \sum_{i}p_{i}^{2}\ =\ p_{\max}\sum_{i}\frac{p_{i}}{p_{\max}}\ p_{i}\ \leqslant\ p_{\max}\sum_{i}p_{i}\ =\ p_{\max}\ . (118)
‖𝒑new‖\displaystyle{{\left\|\bm{p}^{\mathrm{new}}\right\|}}\ ⩽ 1/N+‖Jm‖2​‖𝒑‖⩽ 1/N+pmax.\displaystyle\leqslant\ 1/\sqrt{N}\ +\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{p}\right\|}}\ \leqslant\ 1/\sqrt{N}\ +\ \sqrt{p_{\max}}\ . (119)
‖𝒑new− 1/N​ 1‖1\displaystyle{{\left\|\bm{p}^{\mathrm{new}}\ -\ 1/N\ \bm{1}\right\|}}_{1}\ ⩽‖Jm‖1⩽ 2​β​m​‖𝑿‖∞​M1,\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{1}\ \leqslant\ 2\ \beta\ m\ {{\left\|\bm{X}\right\|}}_{\infty}\ M_{1}\ , (120)
‖𝒑new− 1/N​ 1‖1\displaystyle{{\left\|\bm{p}^{\mathrm{new}}\ -\ 1/N\ \bm{1}\right\|}}_{1}\ ⩽‖Jm‖1⩽ 2​β​m​N​M∞​M1,\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{1}\ \leqslant\ 2\ \beta\ m\ N\ M_{\infty}\ M_{1}\ , (121)
‖𝒑new− 1/N​ 1‖1\displaystyle{{\left\|\bm{p}^{\mathrm{new}}\ -\ 1/N\ \bm{1}\right\|}}_{1}\ ⩽‖Jm‖1⩽ 2​β​m​N​M2,\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{1}\ \leqslant\ 2\ \beta\ m\ N\ M^{2}\ , (122)
‖𝑿T​𝑿‖1\displaystyle{{\left\|\bm{X}^{T}\bm{X}\right\|}}_{1}\ =max⁡∑j=1Ni⁡|𝒙iT​𝒙j|⩽max⁡∑j=1Ni⁡‖𝒙i‖∞​‖𝒙j‖1\displaystyle=\ \max_{i}\sum_{j=1}^{N}\left|\bm{x}_{i}^{T}\bm{x}_{j}\right|\ \leqslant\ \max_{i}\sum_{j=1}^{N}{{\left\|\bm{x}_{i}\right\|}}_{\infty}\ {{\left\|\bm{x}_{j}\right\|}}_{1} (123)
⩽M∞​∑j=1NM1=N​M∞​M1,\displaystyle\leqslant\ M_{\infty}\ \sum_{j=1}^{N}M_{1}\ =\ N\ M_{\infty}\ M_{1}\ ,
‖𝑿T​𝑿‖1\displaystyle{{\left\|\bm{X}^{T}\bm{X}\right\|}}_{1}\ =max⁡∑j=1Ni⁡|𝒙iT​𝒙j|⩽max⁡∑j=1Ni⁡‖𝒙i‖​‖𝒙j‖\displaystyle=\ \max_{i}\sum_{j=1}^{N}\left|\bm{x}_{i}^{T}\bm{x}_{j}\right|\ \leqslant\ \max_{i}\sum_{j=1}^{N}{{\left\|\bm{x}_{i}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}} (124)
⩽M​∑j=1NM=N​M2,\displaystyle\leqslant\ M\ \sum_{j=1}^{N}M\ =\ N\ M^{2}\ ,
min𝒑\displaystyle\min_{\bm{p}}  ​‖𝒑‖2\displaystyle{\mbox{\ ~}}{{\left\|\bm{p}\right\|}}^{2} (125)
s.t.  ​∑ipi=1\displaystyle{\mbox{\ ~}}\sum_{i}p_{i}=1
 ∀i:pi≥ 0.\displaystyle{\mbox{\ ~}}\forall_{i}:\ \ p_{i}\ \geq\ 0\ .
1/N\displaystyle 1/\sqrt{N}\ ⩽‖𝒑new‖⩽ 1/N+pmax.\displaystyle\leqslant\ {{\left\|\bm{p}^{\mathrm{new}}\right\|}}\ \leqslant\ 1/\sqrt{N}\ +\ \sqrt{p_{\max}}\ . (126)
‖𝒑new‖2\displaystyle{{\left\|\bm{p}^{\mathrm{new}}\right\|}}^{2}\ =(𝒑new)T​𝒑new= 1/N+(𝒑new)T​Jm​𝒑⩽ 1/N+‖Jm‖2​‖𝒑‖\displaystyle=\ (\bm{p}^{\mathrm{new}})^{T}\bm{p}^{\mathrm{new}}\ =\ 1/N\ +\ (\bm{p}^{\mathrm{new}})^{T}\mathrm{J}^{m}\ \bm{p}\ \leqslant\ 1/N\ +\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{p}\right\|}} (127)
⩽ 1/N+‖Jm‖2,\displaystyle\leqslant\ 1/N\ +\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ,
‖𝒑∗‖2\displaystyle{{\left\|\bm{p}^{*}\right\|}}^{2}\ =(𝒑∗)T​𝒑∗= 1/N+(𝒑∗)T​Jm​𝒑∗⩽ 1/N+‖Jm‖2​‖𝒑∗‖2,\displaystyle=\ (\bm{p}^{*})^{T}\bm{p}^{*}\ =\ 1/N\ +\ (\bm{p}^{*})^{T}\mathrm{J}^{m}\ \bm{p}^{*}\ \leqslant\ 1/N\ +\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{p}^{*}\right\|}}^{2}\ , (128)
1/N\displaystyle 1/N\ ⩽‖𝒑∗‖2⩽ 1/N​11−‖Jm‖2= 1/N⁡(1+‖Jm‖21−‖Jm‖2).\displaystyle\leqslant\ {{\left\|\bm{p}^{*}\right\|}}^{2}\ \leqslant\ 1/N\frac{1}{1\ -\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}}\ =\ 1/N\ (1\ +\ \frac{{{\left\|\mathrm{J}^{m}\right\|}}_{2}}{1\ -\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}})\ . (129)

A.1.5.3 Many Stable States: Fixed Points Near Stored Patterns.

For a query whose preferred pattern has a positive score margin, total competing attention is at most (N−1)e^(−βmargin). This bounds the updated state’s distance from that pattern. Lemmas A5–A7 use a ball of radius 1/(βNM), sufficient pattern separation, invariance and a Jacobian bound to obtain a unique attracting fixed point. The state generally converges near the stored vector, not exactly to it. A mean-value bound then makes the one-step error small when the separation overwhelms the competing-pattern count and query distance.

Lemma A4.

For a query 𝛏\bm{\xi} and data 𝐗=(𝐱1,…,𝐱N)\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N}), there exists a 𝐱i\bm{x}_{i} that is least separated from 𝛏\bm{\xi} while being separated from other 𝐱j\bm{x}_{j} with j≠ij\not=i:

Equations 130–131
i\displaystyle i\ =arg⁡maxk​minj,j≠k⁡(𝝃T​𝒙k−𝝃T​𝒙j)=arg⁡maxk⁡(𝝃T​𝒙k−maxj,j≠k⁡𝝃T​𝒙j)\displaystyle=\ \arg\max_{k}\min_{j,j\not=k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ =\ \arg\max_{k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \max_{j,j\not=k}\bm{\xi}^{T}\bm{x}_{j}\right) (130)
0\displaystyle 0\ ⩽c=maxk⁡minj,j≠k⁡(𝝃T​𝒙k−𝝃T​𝒙j)=maxk⁡(𝝃T​𝒙k−maxj,j≠k⁡𝝃T​𝒙j).\displaystyle\leqslant\ c\ =\ \max_{k}\min_{j,j\not=k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ =\ \max_{k}\left(\bm{\xi}^{T}\bm{x}_{k}\ -\ \max_{j,j\not=k}\bm{\xi}^{T}\bm{x}_{j}\right)\ . (131)

For 𝐱i\bm{x}_{i}, the following holds:

Equation 132
‖𝒙i−f⁡(𝝃)‖\displaystyle{{\left\|\bm{x}_{i}\ -\ f(\bm{\xi})\right\|}}\ ⩽ 2​ϵ​M,\displaystyle\leqslant\ 2\ \epsilon\ M\ , (132)

where

Equations 133–134
M\displaystyle M\ =maxi⁡‖𝒙i‖,\displaystyle=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ , (133)
ϵ\displaystyle\epsilon\ =(N−1)​exp⁡(−β​c).\displaystyle=\ (N-1)\ \exp(-\ \beta\ c)\ . (134)
Equations 135–140
[softmax⁡(β​𝑿T​𝝃)]i\displaystyle[\mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\xi})]_{i}\ =11+∑j≠iexp⁡(β⁡(𝝃T​𝒙j−𝝃T​𝒙i))≥11+∑j≠iexp⁡(−β​c)\displaystyle=\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(\beta\ (\bm{\xi}^{T}\bm{x}_{j}\ -\ \bm{\xi}^{T}\bm{x}_{i}))}\ \geq\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(-\ \beta\ c)} (135)
=11+(N−1)​exp⁡(−β​c)= 1−(N−1)​exp⁡(−β​c)1+(N−1)​exp⁡(−β​c)\displaystyle=\ \frac{1}{1\ +\ (N-1)\exp(-\ \beta\ c)}\ =\ 1\ -\ \frac{(N-1)\exp(-\ \beta\ c)}{1\ +\ (N-1)\exp(-\ \beta\ c)}
≥ 1−(N−1)​exp⁡(−β​c)= 1−ϵ\displaystyle\geq\ 1\ -\ (N-1)\exp(-\ \beta\ c)\ =\ 1\ -\ \epsilon\
[softmax⁡(β​𝑿T​𝝃)]k\displaystyle[\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{k}\ =exp⁡(β⁡(𝝃T​𝒙k−𝝃T​𝒙i))1+∑j≠iexp⁡(β⁡(𝝃T​𝒙j−𝝃T​𝒙i))⩽exp⁡(−β​c)=ϵN−1.\displaystyle=\ \frac{\exp(\beta\ (\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{i}))}{1\ +\ \sum_{j\not=i}\exp(\beta\ (\bm{\xi}^{T}\bm{x}_{j}\ -\ \bm{\xi}^{T}\bm{x}_{i}))}\ \leqslant\ \exp(-\ \beta\ c)\ =\ \frac{\epsilon}{N-1}\ . (136)
f⁡(𝝃)\displaystyle f(\bm{\xi})\ =𝑿​softmax​(β​𝑿T​𝝃)=∑j=1N𝒙j​[softmax⁡(β​𝑿T​𝝃)]j.\displaystyle=\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ =\ \sum_{j=1}^{N}\bm{x}_{j}\ [\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{j}\ . (137)
‖𝒙i−f⁡(𝝃)‖\displaystyle{{\left\|\bm{x}_{i}\ -\ f(\bm{\xi})\right\|}}\ =‖𝒙i−∑j=1N[softmax⁡(β​𝑿T​𝝃)]j​𝒙j‖\displaystyle=\ {{\left\|\bm{x}_{i}\ -\ \sum_{j=1}^{N}[\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{j}\ \bm{x}_{j}\right\|}} (138)
=‖(1−[softmax⁡(β​𝑿T​𝝃)]i)​𝒙i−∑j=1,j≠iN[softmax⁡(β​𝑿T​𝝃)]j​𝒙j‖\displaystyle=\ {{\left\|(1-[\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{i})\ \bm{x}_{i}\ -\ \sum_{j=1,j\not=i}^{N}[\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{j}\ \bm{x}_{j}\right\|}}
⩽ϵ⁡‖𝒙i‖+ϵN−1​∑j=1,j≠iN‖𝒙j‖\displaystyle\leqslant\ \epsilon\ {{\left\|\bm{x}_{i}\right\|}}\ +\ \frac{\epsilon}{N-1}\ \sum_{j=1,j\not=i}^{N}{{\left\|\bm{x}_{j}\right\|}}
⩽ϵ​M+ϵN−1​∑j=1,j≠iNM= 2​ϵ​M.\displaystyle\leqslant\ \epsilon\ M\ +\ \frac{\epsilon}{N-1}\ \sum_{j=1,j\not=i}^{N}M\ =\ 2\ \epsilon\ M\ .
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (139)
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡12​(‖𝒙i‖2−‖𝒙j‖2+‖𝒙i−𝒙j‖2)\displaystyle=\ \min_{j,j\not=i}\frac{1}{2}\ \left({{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{j}\right\|}}^{2}\ +\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right) (140)
=12​‖𝒙i‖2−12​maxj,j≠i⁡(‖𝒙j‖2−‖𝒙i−𝒙j‖2).\displaystyle=\ \frac{1}{2}{{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ \frac{1}{2}\ \max_{j,j\not=i}\left({{\left\|\bm{x}_{j}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right)\ .
Definition 3 (Sphere Si\mathrm{S}_{i}).

The sphere Si\mathrm{S}_{i} is defined as

Equation 141
Si\displaystyle\mathrm{S}_{i}\ :={𝝃∣‖𝝃−𝒙i‖⩽1β​N​M}.\displaystyle:=\ \left\{\bm{\xi}\mid{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ \leqslant\ \frac{1}{\beta\ N\ M}\right\}\ . (141)
Lemma A5.

With 𝛏\bm{\xi} given, if the assumptions

  1. A1:

    𝝃\bm{\xi} is inside sphere: 𝝃∈Si\bm{\xi}\in\mathrm{S}_{i},

  2. A2:

    data point 𝒙i\bm{x}_{i} is well separated from the other data:

    Equation 142
    Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​(N−1)​N​β​M2)\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right) (142)

hold, then f⁡(𝛏)f(\bm{\xi}) is inside the sphere: f⁡(𝛏)∈Sif(\bm{\xi})\in\mathrm{S}_{i}. Therefore, with assumption (A2), ff is a mapping from Si\mathrm{S}_{i} into Si\mathrm{S}_{i}.

Equations 143–148
Δ~i\displaystyle\tilde{\Delta}_{i}\ =minj,j≠i⁡(𝝃T​𝒙i−𝝃T​𝒙j).\displaystyle=\ \min_{j,j\not=i}\left(\bm{\xi}^{T}\bm{x}_{i}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ . (143)
|𝝃T​𝒙j−𝒙iT​𝒙j|\displaystyle\left|\bm{\xi}^{T}\bm{x}_{j}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right| ⩽‖𝝃−𝒙i‖​‖𝒙j‖⩽‖𝝃−𝒙i‖​M.\displaystyle\leqslant\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\ . (144)
Δ~i\displaystyle\tilde{\Delta}_{i}\ ≥minj,j≠i⁡((𝒙iT​𝒙i−‖𝝃−𝒙i‖​M)−(𝒙iT​𝒙j+‖𝝃−𝒙i‖​M))\displaystyle\geq\ \min_{j,j\not=i}\left(\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\right)\ -\ \left(\bm{x}_{i}^{T}\bm{x}_{j}\ +\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right) (145)
=− 2​‖𝝃−𝒙i‖​M+minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=Δi− 2​‖𝝃−𝒙i‖​M\displaystyle=\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\ +\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \Delta_{i}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M
≥Δi−2β​N,\displaystyle\geq\ \Delta_{i}\ -\ \frac{2}{\beta\ N}\ ,
pmax\displaystyle p_{\max}\ =[softmax⁡(β​𝑿T​𝝃)]i≥ 1−(N−1)​exp⁡(−β​Δ~i)= 1−ϵ~.\displaystyle=\ [\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{i}\ \geq\ 1\ -\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i})\ =\ 1\ -\ \tilde{\epsilon}\ . (146)
‖𝒙i−f⁡(𝝃)‖\displaystyle{{\left\|\bm{x}_{i}\ -\ f(\bm{\xi})\right\|}}\ ⩽ 2​ϵ~​M= 2​(N−1)​exp⁡(−β​Δ~i)​M\displaystyle\leqslant\ 2\ \tilde{\epsilon}\ M\ =\ 2\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i})\ M (147)
⩽ 2​(N−1)​exp⁡(−β⁡(Δi−2β​N))​M.\displaystyle\leqslant\ 2\ (N-1)\ \exp(-\ \beta\ (\Delta_{i}\ -\ \frac{2}{\beta\ N}))\ M\ .
‖𝒙i−f⁡(𝝃)‖\displaystyle{{\left\|\bm{x}_{i}\ -\ f(\bm{\xi})\right\|}} (148)
⩽ 2​(N−1)​exp⁡(−β⁡(2β​N+1β​ln⁡(2​(N−1)​N​β​M2)−2β​N))​M\displaystyle\leqslant\ 2\ (N-1)\ \exp(-\ \beta\ (\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ -\ \frac{2}{\beta\ N}))\ M
= 2​(N−1)​exp⁡(−ln⁡(2​(N−1)​N​β​M2))​M\displaystyle=\ 2\ (N-1)\ \exp(-\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right))\ M
=1N​β​M,\displaystyle=\ \frac{1}{N\ \beta\ M}\ ,
Lemma A6.

Assume that

  1. A1:
    Equation 149
    Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​(N−1)​N​β​M2),\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ , (149)

then ff is a contraction mapping in Si\mathrm{S}_{i}.

Equations 150–164
f⁡(𝝃)\displaystyle f(\bm{\xi})\ =f⁡(𝒙i)+Jm​(𝝃−𝒙i).\displaystyle=\ f(\bm{x}_{i})\ +\ \mathrm{J}^{m}\ (\bm{\xi}\ -\ \bm{x}_{i})\ . (150)
‖f⁡(𝝃)−f⁡(𝒙i)‖\displaystyle{{\left\|f(\bm{\xi})\ -\ f(\bm{x}_{i})\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒙i‖.\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ . (151)
pmax​(𝝃~)\displaystyle p_{\max}(\tilde{\bm{\xi}})\ =[softmax⁡(β​𝑿T​𝝃~)]i≥ 1−(N−1)​exp⁡(−β​Δ~i)= 1−ϵ~,\displaystyle=\ [\mathrm{softmax}(\beta\ \bm{X}^{T}\ \tilde{\bm{\xi}})]_{i}\ \geq\ 1\ -\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i})\ =\ 1\ -\ \tilde{\epsilon}\ , (152)
ϵ~\displaystyle\tilde{\epsilon}\ =(N−1)​exp⁡(−β​Δ~i),\displaystyle=\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i})\ , (153)
Δ~i\displaystyle\tilde{\Delta}_{i}\ =minj,j≠i⁡(𝝃~T​𝒙i−𝝃~T​𝒙j).\displaystyle=\ \min_{j,j\not=i}\left(\tilde{\bm{\xi}}^{T}\bm{x}_{i}\ -\ \tilde{\bm{\xi}}^{T}\bm{x}_{j}\right)\ . (154)
|𝝃~T​𝒙j−𝒙iT​𝒙j|\displaystyle\left|\tilde{\bm{\xi}}^{T}\bm{x}_{j}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right| ⩽‖𝝃~−𝒙i‖​‖𝒙j‖⩽‖𝝃~−𝒙i‖​M.\displaystyle\leqslant\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\ . (155)
Δ~i\displaystyle\tilde{\Delta}_{i}\ ≥minj,j≠i⁡((𝒙iT​𝒙i−‖𝝃~−𝒙i‖​M)−(𝒙iT​𝒙j+‖𝝃~−𝒙i‖​M))\displaystyle\geq\ \min_{j,j\not=i}\left(\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\ -\ \left(\bm{x}_{i}^{T}\bm{x}_{j}\ +\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right) (156)
=− 2​‖𝝃~−𝒙i‖​M+minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=Δi− 2​‖𝝃~−𝒙i‖​M\displaystyle=\ -\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\ +\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \Delta_{i}\ -\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M
≥Δi− 2​‖𝝃−𝒙i‖​M,\displaystyle\geq\ \Delta_{i}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\ ,
ϵ~\displaystyle\tilde{\epsilon}\ =(N−1)​exp⁡(−β​Δ~i)\displaystyle=\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i}) (157)
⩽(N−1)​exp⁡(−β⁡(Δi− 2​‖𝝃−𝒙i‖​M))\displaystyle\leqslant\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right)
⩽(N−1)​exp⁡(−β⁡(Δi−2β​N)),\displaystyle\leqslant\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ -\ \frac{2}{\beta\ N}\right)\right)\ ,
Δ~i\displaystyle\tilde{\Delta}_{i}\ ⩽minj,j≠i⁡((𝒙iT​𝒙i+‖𝝃~−𝒙i‖​M)−(𝒙iT​𝒙j−‖𝝃~−𝒙i‖​M))\displaystyle\leqslant\ \min_{j,j\not=i}\left(\left(\bm{x}_{i}^{T}\bm{x}_{i}\ +\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\ -\ \left(\bm{x}_{i}^{T}\bm{x}_{j}\ -\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right) (158)
= 2​‖𝝃~−𝒙i‖​M+minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=Δi+ 2​‖𝝃~−𝒙i‖​M\displaystyle=\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M\ +\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \Delta_{i}\ +\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{x}_{i}\right\|}}\ M
⩽Δi+ 2​‖𝝃−𝒙i‖​M,\displaystyle\leqslant\ \Delta_{i}\ +\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\ ,
ϵ~\displaystyle\tilde{\epsilon}\ =(N−1)​exp⁡(−β​Δ~i)\displaystyle=\ (N-1)\ \exp(-\ \beta\ \tilde{\Delta}_{i}) (159)
≥(N−1)​exp⁡(−β⁡(Δi+ 2​‖𝝃−𝒙i‖​M))\displaystyle\geq\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ +\ 2\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right)
≥(N−1)​exp⁡(−β⁡(Δi+2β​N)),\displaystyle\geq\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ ,
‖J⁡(𝝃~)‖2\displaystyle{{\left\|\mathrm{J}(\tilde{\bm{\xi}})\right\|}}_{2}\ ⩽ 2​β​N​M2​ϵ~− 2​ϵ~2​β​N​M2\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ \tilde{\epsilon}\ -\ 2\ \tilde{\epsilon}^{2}\ \beta\ N\ M^{2} (160)
⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi−2β​N))−\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ -\frac{2}{\beta\ N}\right)\right)\ -
2​(N−1)2​exp⁡(− 2​β​(Δi+2β​N))​β​N​M2.\displaystyle 2\ (N-1)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ \beta\ N\ M^{2}\ .
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi−2β​N))−\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\ \exp\left(-\ \beta\ \left(\Delta_{i}\ -\ \frac{2}{\beta\ N}\right)\right)\ - (161)
2​(N−1)2​exp⁡(− 2​β​(Δi+2β​N))​β​N​M2.\displaystyle 2\ (N-1)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ \beta\ N\ M^{2}\ .
Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​(N−1)​N​β​M2),\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ , (162)
Δi−2β​N\displaystyle\Delta_{i}\ -\ \frac{2}{\beta\ N}\ ≥1β​ln⁡(2​(N−1)​N​β​M2),\displaystyle\geq\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ , (163)
‖Jm‖2⩽ 2​β​(N−1)​exp⁡(−β​1β​ln⁡(2​(N−1)​N​β​M2))​N​M2−\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ \leqslant\ 2\ \beta\ (N-1)\ \exp\left(-\ \beta\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\right)\ N\ M^{2}\ - (164)
2​(N−1)2​exp⁡(− 2​β​(Δi+2β​N))​β​N​M2\displaystyle 2\ (N-1)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ \beta\ N\ M^{2}
= 2​β​(N−1)​12​(N−1)​N​β​M2​N​M2−\displaystyle=\ \ 2\ \beta\ (N-1)\ \frac{1}{2\ (N-1)\ N\ \beta\ M^{2}}\ N\ M^{2}\ -
2​(N−1)2​exp⁡(− 2​β​(Δi+2β​N))​β​N​M2\displaystyle 2\ (N-1)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ \beta\ N\ M^{2}
= 1−2​(N−1)2​exp⁡(− 2​β​(Δi+2β​N))​β​N​M2<1.\displaystyle=\ 1\ -2\ (N-1)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{i}\ +\ \frac{2}{\beta\ N}\right)\right)\ \beta\ N\ M^{2}\ <1\ .
Lemma A7.

Assume that

  1. A1:
    Equation 165
    Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​(N−1)​N​β​M2),\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ , (165)

then ff has a fixed point in Si\mathrm{S}_{i}.

Equations 166–186
‖f⁡(𝝃)−𝒙i∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ =‖f⁡(𝝃)−f⁡(𝒙i∗)‖⩽‖Jm‖2​‖𝝃−𝒙i∗‖\displaystyle=\ {{\left\|f(\bm{\xi})\ -\ f(\bm{x}_{i}^{*})\right\|}}\ \leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}} (166)
‖Js​(𝒙~)‖2\displaystyle{{\left\|\mathrm{J}_{s}(\tilde{\bm{x}})\right\|}}_{2}\ < 2​ϵ​β.\displaystyle<\ 2\ \epsilon\ \beta\ . (167)
‖J⁡(𝒙~)‖2\displaystyle{{\left\|\mathrm{J}(\tilde{\bm{x}})\right\|}}_{2}\ ⩽ 2​β​‖𝑿‖22​ϵ⩽ 2​β​N​M2​ϵ.\displaystyle\leqslant\ 2\ \beta\ {{\left\|\bm{X}\right\|}}_{2}^{2}\ \epsilon\ \leqslant\ 2\ \beta\ NM^{2}\ \epsilon\ . (168)
‖𝑿‖2\displaystyle{{\left\|\bm{X}\right\|}}_{2}\ ⩽‖𝑿‖F=∑i‖𝒙i‖2.\displaystyle\leqslant\ {{\left\|\bm{X}\right\|}}_{F}\ =\ \sqrt{\sum_{i}{{\left\|\bm{x}_{i}\right\|}}^{2}}\ . (169)
‖𝑿‖22\displaystyle{{\left\|\bm{X}\right\|}}_{2}^{2}\ ⩽N​M2.\displaystyle\leqslant\ N\ M^{2}\ . (170)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​‖𝑿‖22​ϵ⩽ 2​β​N​M2​ϵ.\displaystyle\leqslant\ 2\ \beta\ {{\left\|\bm{X}\right\|}}_{2}^{2}\ \epsilon\ \leqslant\ 2\ \beta\ NM^{2}\ \epsilon\ . (171)
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (172)
Δ~i\displaystyle\tilde{\Delta}_{i}\ =minj,j≠i⁡(𝒙~T​𝒙i−𝒙~T​𝒙j).\displaystyle=\ \min_{j,j\not=i}\left(\tilde{\bm{x}}^{T}\bm{x}_{i}\ -\ \tilde{\bm{x}}^{T}\bm{x}_{j}\right)\ . (173)
|𝒙~T​𝒙j−𝒙iT​𝒙j|\displaystyle\left|\tilde{\bm{x}}^{T}\bm{x}_{j}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right| ⩽‖𝒙~−𝒙i‖​‖𝒙j‖⩽‖𝒙~−𝒙i‖​M.\displaystyle\leqslant\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ M\ . (174)
Δ~i\displaystyle\tilde{\Delta}_{i}\ ≥minj,j≠i⁡((𝒙iT​𝒙i−‖𝒙~−𝒙i‖​M)−(𝒙iT​𝒙j+‖𝒙~−𝒙i‖​M))\displaystyle\geq\ \min_{j,j\not=i}\left(\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\ -\ \left(\bm{x}_{i}^{T}\bm{x}_{j}\ +\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ M\right)\right) (175)
=− 2​‖𝒙~−𝒙i‖​M+minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=Δi− 2​‖𝒙~−𝒙i‖​M.\displaystyle=\ -\ 2\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ M\ +\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \Delta_{i}\ -\ 2\ {{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ M\ .
‖𝒙~−𝒙i‖\displaystyle{{\left\|\tilde{\bm{x}}\ -\ \bm{x}_{i}\right\|}}\ =‖λ​𝝃+(1−λ)​𝒙i∗−𝒙i‖\displaystyle=\ {{\left\|\lambda\bm{\xi}+(1-\lambda)\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}} (176)
⩽λ⁡‖𝝃−𝒙i‖+(1−λ)​‖𝒙i∗−𝒙i‖\displaystyle\leqslant\ \lambda\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ +\ (1-\lambda)\ {{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}
⩽max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖},\displaystyle\leqslant\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ ,
Δ~i\displaystyle\tilde{\Delta}_{i}\ ≥Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M.\displaystyle\geq\ \Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M\ . (177)
[softmax⁡(β​𝑿T​𝝃~)]i=11+∑j≠iexp⁡(β⁡(𝝃~T​𝒙j−𝝃~T​𝒙i))\displaystyle[\mathrm{softmax}(\beta\ \bm{X}^{T}\tilde{\bm{\xi}})]_{i}\ =\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(\beta\ (\tilde{\bm{\xi}}^{T}\bm{x}_{j}\ -\ \tilde{\bm{\xi}}^{T}\bm{x}_{i}))} (178)
≥11+∑j≠iexp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))\displaystyle\geq\ \frac{1}{1\ +\ \sum_{j\not=i}\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))}
=11+(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))\displaystyle=\ \frac{1}{1\ +\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))}
= 1−(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))1+(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))\displaystyle=\ 1\ -\ \frac{(N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))}{1\ +\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))}
≥ 1−(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))\displaystyle\geq\ 1\ -\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))
= 1−ϵ.\displaystyle=\ 1\ -\ \epsilon\ .
ϵ\displaystyle\epsilon\ =(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle=\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (179)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (180)
‖Jm‖2< 1,\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ <\ 1\ , (181)
2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))< 1.\displaystyle 2\ \beta\ NM^{2}\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ <\ 1\ . (182)
Δi\displaystyle\Delta_{i}\ > 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M+1β​ln⁡(2​(N−1)​N​β​M2).\displaystyle>\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ . (183)
‖f⁡(𝝃)−𝒙i∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒙i∗‖<‖𝝃−𝒙i∗‖.\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ <\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ . (184)
‖f⁡(𝝃)−𝒙i∗‖⩽‖Jm‖2​‖𝝃−𝒙i∗‖⩽‖Jm‖2​‖𝝃−f⁡(𝝃)‖+‖Jm‖2​‖f⁡(𝝃)−𝒙i∗‖,\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ \leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ \leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ f(\bm{\xi})\right\|}}\ +\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ , (185)
‖f⁡(𝝃)−𝒙i∗‖⩽‖Jm‖21−‖Jm‖2​‖𝝃−f⁡(𝝃)‖.\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ \leqslant\ \frac{{{\left\|\mathrm{J}^{m}\right\|}}_{2}}{1\ -\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}}\ {{\left\|\bm{\xi}\ -\ f(\bm{\xi})\right\|}}\ . (186)

A.1.5.4 Metastable States: Fixed Points Near Mean of Similar Patterns.

For a cluster of l similar patterns, split attention into within-cluster mass 1−γ and leakage γ. The Jacobian bound combines within-cluster spread with a term of order βγM² from outside patterns. Lemmas A9–A12 require a compact neighborhood around the cluster mean, sufficient separation from the remaining memories and a sufficiently small cluster radius. These ensure invariance and contraction toward a cluster fixed point. The final bounds show why a tightly packed, well-isolated cluster can converge rapidly even when no individual member is separately retrieved.

Equations 187–200
J\displaystyle\mathrm{J}\ =β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T=𝑿​Js​𝑿T.\displaystyle=\ \beta\ \bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T}\ =\ \bm{X}\mathrm{J}_{s}\bm{X}^{T}\ . (187)
𝑿⁡(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T\displaystyle\bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T} (188)
𝑿⁡(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T=𝑿​diag​(𝒑)​𝑿T−𝑿​𝒑​𝒑T​𝑿T\displaystyle\bm{X}\left(\mathrm{diag}(\bm{p})\ -\ \bm{p}\bm{p}^{T}\right)\bm{X}^{T}\ =\ \bm{X}\mathrm{diag}(\bm{p})\bm{X}^{T}\ -\ \bm{X}\bm{p}\bm{p}^{T}\bm{X}^{T} (189)
=∑i=1Npi​𝒙i​𝒙iT−(∑i=1Npi​𝒙i)​(∑i=1Npi​𝒙i)T\displaystyle=\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{N}p_{i}\ \bm{x}_{i}\right)^{T} (190)
=E𝒑​[𝒙​𝒙T]−E𝒑​[𝒙]​E𝒑​[𝒙]T=Var𝒑​[𝒙],\displaystyle=\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}\ \bm{x}^{T}]\ -\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}]\ \mathbf{\mathrm{E}}_{\bm{p}}[\bm{x}]^{T}\ =\ \mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\ , (191)
J\displaystyle\mathrm{J}\ =β​Var𝒑​[𝒙].\displaystyle=\ \beta\ \mathbf{\mathrm{Var}}_{\bm{p}}[\bm{x}]\ . (192)
M\displaystyle M\ :=maxi⁡‖𝒙i‖,\displaystyle:=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ , (193)
γ\displaystyle\gamma\ =∑i=l+1Npi⩽ϵ,\displaystyle=\ \sum_{i=l+1}^{N}p_{i}\ \leqslant\ \epsilon\ , (194)
1−γ\displaystyle 1-\gamma\ =∑i=1lpi≥ 1−ϵ,\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \geq\ 1\ -\ \epsilon\ , (195)
p~i\displaystyle\tilde{p}_{i}\ :=pi1−γ⩽pi/(1−ϵ),\displaystyle:=\ \frac{p_{i}}{1-\gamma}\ \leqslant\ p_{i}/(1-\epsilon)\ , (196)
∑i=1lp~i\displaystyle\sum_{i=1}^{l}\tilde{p}_{i}\ = 1,\displaystyle=\ 1\ , (197)
𝒎𝒙\displaystyle\bm{m}_{\bm{x}}\ =1l​∑i=1l𝒙i,\displaystyle=\ \frac{1}{l}\ \sum_{i=1}^{l}\ \bm{x}_{i}\ , (198)
mmax\displaystyle m_{\max}\ =max1⩽i⩽l⁡‖𝒙i−𝒎𝒙‖.\displaystyle=\ \max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}\ . (199)
Varp~[𝒙1:l]\displaystyle\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\ =∑i=1lp~i​𝒙i​𝒙iT−(∑i=1lp~i​𝒙i)​(∑i=1lp~i​𝒙i)T\displaystyle=\ \sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\right)^{T} (200)
=∑i=1lp~i​(𝒙i−∑i=1lp~i​𝒙i)​(𝒙i−∑i=1lp~i​𝒙i)T.\displaystyle=\ \sum_{i=1}^{l}\tilde{p}_{i}\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{l}\tilde{p}_{i}\bm{x}_{i}\right)\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{l}\tilde{p}_{i}\bm{x}_{i}\right)^{T}\ .
Lemma A8.

With the definitions in Eq. (193) to Eq. (200), the following bounds on the norm ‖J‖2{{\left\|\mathrm{J}\right\|}}_{2} of the Jacobian of the fixed point iteration hold. The γ\gamma-bound for ‖J‖2{{\left\|\mathrm{J}\right\|}}_{2} is

Equation 201
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β⁡((1−γ)​mmax2+γ​ 2​(2−γ)​M2)\displaystyle\leqslant\ \beta\left((1-\gamma)\ m_{\max}^{2}\ +\ \gamma\ 2\ (2\ -\ \gamma)\ M^{2}\right) (201)

and the ϵ\epsilon-bound for ‖J‖2{{\left\|\mathrm{J}\right\|}}_{2} is:

Equation 202
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β⁡(mmax2+ϵ​ 2​(2−ϵ)​M2).\displaystyle\leqslant\ \beta\left(\ m_{\max}^{2}\ +\ \epsilon\ 2\ (2\ -\ \epsilon)\ M^{2}\right)\ . (202)
Equations 203–226
(1−γ)Varp~[𝒙1:l]=∑i=1lpi(𝒙i−11−γ∑i=1lpi𝒙i)(𝒙i−11−γ∑i=1lpi𝒙i)T\displaystyle(1-\gamma)\ \mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\ =\ \sum_{i=1}^{l}p_{i}\ \left(\bm{x}_{i}\ -\ \frac{1}{1-\gamma}\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\bm{x}_{i}\ -\ \frac{1}{1-\gamma}\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T} (203)
=∑i=1lpi​𝒙i​𝒙iT−(∑i=1lpi​𝒙i)​11−γ​(∑i=1lpi​𝒙i)T−11−γ​(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \frac{1}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ -\ \frac{1}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}
+∑i=1lpi(1−γ)2​(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T=∑i=1lpi​𝒙i​𝒙iT−11−γ​(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T\displaystyle+\ \frac{\sum_{i=1}^{l}p_{i}}{(1-\gamma)^{2}}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ =\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \frac{1}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}
=∑i=1lpi​𝒙i​𝒙iT−(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T+(1−11−γ)​(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ +\ \left(1\ -\ \frac{1}{1-\gamma}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}
=∑i=1lpi​𝒙i​𝒙iT−(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T−γ1−γ​(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T.\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ -\ \frac{\gamma}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ .
∑i=1lpi​𝒙i​𝒙iT−(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T\displaystyle\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T} (204)
=(1−γ)Varp~[𝒙1:l]+γ1−γ(∑i=1lpi𝒙i)(∑i=1lpi𝒙i)T.\displaystyle=\ (1-\gamma)\ \mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\ +\ \frac{\gamma}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\ .
J\displaystyle\mathrm{J}\ =β⁡(∑i=1lpi​𝒙i​𝒙iT+∑i=l+1Npi​𝒙i​𝒙iT\displaystyle=\ \beta\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ +\ \sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\right. (205)
−(∑i=1lpi​𝒙i+∑i=l+1Npi​𝒙i)​(∑i=1lpi​𝒙i+∑i=l+1Npi​𝒙i)T)\displaystyle-\ \left.\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ +\ \sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ +\ \sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\right)
=β⁡(∑i=1lpi​𝒙i​𝒙iT−(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T\displaystyle=\ \beta\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right.
+∑i=l+1Npi𝒙i𝒙iT−(∑i=l+1Npi𝒙i)(∑i=l+1Npi𝒙i)T\displaystyle+\ \left.\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\right.
−(∑i=1lpi​𝒙i)​(∑i=l+1Npi​𝒙i)T−(∑i=l+1Npi​𝒙i)​(∑i=1lpi​𝒙i)T)\displaystyle-\ \left.\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\ -\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right)
=β((1−γ)Varp~[𝒙1:l]+γ1−γ(∑i=1lpi𝒙i)(∑i=1lpi𝒙i)T\displaystyle=\ \beta\ \left((1-\gamma)\ \mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\ +\ \frac{\gamma}{1-\gamma}\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right.
+∑i=l+1Npi𝒙i𝒙iT−(∑i=l+1Npi𝒙i)(∑i=l+1Npi𝒙i)T\displaystyle+\ \left.\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\right.
−(∑i=1lpi​𝒙i)​(∑i=l+1Npi​𝒙i)T−(∑i=l+1Npi​𝒙i)​(∑i=1lpi​𝒙i)T).\displaystyle-\ \left.\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\ -\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right)\ .
‖𝒂​𝒃T‖2\displaystyle{{\left\|\bm{a}\bm{b}^{T}\right\|}}_{2}\ =λmax​(𝒃​𝒂T​𝒂​𝒃T)=‖𝒂‖​λmax​(𝒃​𝒃T)=‖𝒂‖​‖𝒃‖,\displaystyle=\ \sqrt{\lambda_{\max}(\bm{b}\bm{a}^{T}\bm{a}\bm{b}^{T})}\ =\ {{\left\|\bm{a}\right\|}}\ \sqrt{\lambda_{\max}(\bm{b}\bm{b}^{T})}\ =\ {{\left\|\bm{a}\right\|}}\ {{\left\|\bm{b}\right\|}}\ , (206)
‖∑i=1lpi​𝒙i‖\displaystyle{{\left\|\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right\|}}\ ⩽∑i=1lpi​‖𝒙i‖⩽(1−γ)​M,\displaystyle\leqslant\ \sum_{i=1}^{l}p_{i}\ {{\left\|\bm{x}_{i}\right\|}}\ \leqslant\ (1-\gamma)\ M\ , (207)
‖∑i=l+1Npi​𝒙i‖\displaystyle{{\left\|\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right\|}}\ ⩽∑i=l+1Npi​‖𝒙i‖⩽γ​M,\displaystyle\leqslant\ \sum_{i=l+1}^{N}p_{i}\ {{\left\|\bm{x}_{i}\right\|}}\ \leqslant\ \gamma\ M\ , (208)
‖∑i=l+1Npi​𝒙i​𝒙iT‖2\displaystyle{{\left\|\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\right\|}}_{2}\ ⩽∑i=l+1Npi​‖𝒙i​𝒙iT‖2=∑i=l+1Npi​‖𝒙i‖2⩽∑i=l+1Npi​M2=γ​M2.\displaystyle\leqslant\ \sum_{i=l+1}^{N}p_{i}\ {{\left\|\bm{x}_{i}\ \bm{x}_{i}^{T}\right\|}}_{2}\ =\ \sum_{i=l+1}^{N}p_{i}\ {{\left\|\bm{x}_{i}\right\|}}^{2}\ \leqslant\ \sum_{i=l+1}^{N}p_{i}\ M^{2}\ =\ \gamma\ M^{2}\ . (209)
f⁡(𝒂)\displaystyle f(\bm{a})\ =∑i=1lpi​‖𝒙i−𝒂‖2=∑i=1lpi​(𝒙i−𝒂)T​(𝒙i−𝒂).\displaystyle=\ \sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{a}\right\|}}^{2}\ =\ \sum_{i=1}^{l}p_{i}(\bm{x}_{i}\ -\ \bm{a})^{T}(\bm{x}_{i}\ -\ \bm{a})\ . (210)
∂f⁡(𝒂)∂𝒂\displaystyle\frac{\partial f(\bm{a})}{\partial\bm{a}}\ = 2​∑i=1Npi​(𝒂−𝒙i)= 0\displaystyle=\ 2\ \sum_{i=1}^{N}p_{i}(\bm{a}\ -\ \bm{x}_{i})\ =\ 0 (211)
𝒂\displaystyle\bm{a}\ =∑i=1Npi​𝒙i.\displaystyle=\ \sum_{i=1}^{N}p_{i}\bm{x}_{i}\ . (212)
∂2f⁡(𝒂)∂𝒂2\displaystyle\frac{\partial^{2}f(\bm{a})}{\partial\bm{a}^{2}}\ = 2​∑i=1Npi​𝑰= 2​𝑰\displaystyle=\ 2\ \sum_{i=1}^{N}p_{i}\ \bm{I}\ =\ 2\ \bm{I} (213)
𝒙¯\displaystyle\bar{\bm{x}}\ :=∑i=1Npi​𝒙i\displaystyle:=\ \sum_{i=1}^{N}p_{i}\ \bm{x}_{i} (214)
∑i=1lpi​‖𝒙i−𝒙¯‖2\displaystyle\sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ ⩽∑i=1lpi​‖𝒙i−𝒎𝒙‖2⩽(1−γ)​mmax2.\displaystyle\leqslant\ \sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}^{2}\ \leqslant\ (1\ -\ \gamma)\ m_{\max}^{2}\ . (215)
(1−γ)‖Varp~[𝒙1:l]‖2\displaystyle(1-\gamma)\ {{\left\|\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\right\|}}_{2}\ ⩽∑i=1lpi​‖(𝒙i−𝒙¯)​(𝒙i−𝒙¯)T‖2\displaystyle\leqslant\ \sum_{i=1}^{l}p_{i}{{\left\|\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right\|}}_{2} (216)
=∑i=1lpi​‖𝒙i−𝒙¯‖2⩽∑i=1lpi​‖𝒙i−𝒎𝒙‖2⩽(1−γ)​mmax2.\displaystyle=\ \sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ \leqslant\ \sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}^{2}\ \leqslant\ (1\ -\ \gamma)\ m_{\max}^{2}\ .
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β((1−γ)‖Varp~[𝒙1:l]‖2\displaystyle\leqslant\ \beta\left((1-\gamma)\ {{\left\|\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\right\|}}_{2}\right. (217)
+γ1−γ​‖(∑i=1lpi​𝒙i)​(∑i=1lpi​𝒙i)T‖2\displaystyle+\ \left.\frac{\gamma}{1-\gamma}\ {{\left\|\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right\|}}_{2}\right.
+‖∑i=l+1Npi​𝒙i​𝒙iT‖2+‖(∑i=l+1Npi​𝒙i)​(∑i=l+1Npi​𝒙i)T‖2\displaystyle+\ \left.{{\left\|\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\right\|}}_{2}\ +\ {{\left\|\left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\right\|}}_{2}\right.
+‖(∑i=1lpi​𝒙i)​(∑i=l+1Npi​𝒙i)T‖2+‖(∑i=l+1Npi​𝒙i)​(∑i=1lpi​𝒙i)T‖2)\displaystyle+\ \left.{{\left\|\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)^{T}\right\|}}_{2}\ +\ {{\left\|\left(\sum_{i=l+1}^{N}p_{i}\ \bm{x}_{i}\right)\left(\sum_{i=1}^{l}p_{i}\ \bm{x}_{i}\right)^{T}\right\|}}_{2}\right)
⩽β((1−γ)‖Varp~[𝒙1:l]‖2+γ(1−γ)M2+γM2+γ2M2+\displaystyle\leqslant\ \beta\left((1-\gamma)\ {{\left\|\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\right\|}}_{2}\ +\ \gamma\ (1-\gamma)\ M^{2}\ +\ \gamma\ M^{2}\ +\ \gamma^{2}\ M^{2}\ +\right.
γ⁡(1−γ)​M2+γ⁡(1−γ)​M2)\displaystyle\left.\gamma\ (1-\gamma)\ M^{2}\ +\ \gamma\ (1-\gamma)\ M^{2}\right)
=β((1−γ)‖Varp~[𝒙1:l]‖2+γ 2(2−γ)M2).\displaystyle=\ \beta\left((1-\gamma)\ \ {{\left\|\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\right\|}}_{2}\ +\ \gamma\ 2\ (2\ -\ \gamma)\ M^{2}\right)\ .
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β⁡(mmax2+ϵ​ 2​(2−ϵ)​M2).\displaystyle\leqslant\ \beta\left(\ m_{\max}^{2}\ +\ \epsilon\ 2\ (2\ -\ \epsilon)\ M^{2}\right)\ . (218)
‖Varp~[𝒙1:l]‖2\displaystyle{{\left\|\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\right\|}}_{2}\ =Tr⁡(∑i=1lpi​(𝒙i−𝒙¯)​(𝒙i−𝒙¯)T)−∑k=2dek\displaystyle=\ \mathbf{\mathrm{Tr}}\left(\sum_{i=1}^{l}p_{i}\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right)\ -\ \sum_{k=2}^{d}e_{k} (219)
=∑i=1lpi​Tr​((𝒙i−𝒙¯)​(𝒙i−𝒙¯)T)−∑k=2dek\displaystyle=\ \sum_{i=1}^{l}p_{i}\mathbf{\mathrm{Tr}}\left(\left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)\ \left(\bm{x}_{i}\ -\ \bar{\bm{x}}\right)^{T}\right)\ -\ \sum_{k=2}^{d}e_{k}
=∑i=1lpi​‖𝒙i−𝒙¯‖2−∑k=2dek.\displaystyle=\ \sum_{i=1}^{l}p_{i}{{\left\|\bm{x}_{i}\ -\ \bar{\bm{x}}\right\|}}^{2}\ -\ \sum_{k=2}^{d}e_{k}\ .
M\displaystyle M\ :=maxi⁡‖𝒙i‖,\displaystyle:=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ , (220)
γ\displaystyle\gamma\ =∑i=l+1Npi⩽ϵ,\displaystyle=\ \sum_{i=l+1}^{N}p_{i}\ \leqslant\ \epsilon\ , (221)
1−γ\displaystyle 1-\gamma\ =∑i=1lpi≥ 1−ϵ,\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \geq\ 1\ -\ \epsilon\ , (222)
p~i\displaystyle\tilde{p}_{i}\ :=pi1−γ⩽pi/(1−ϵ),\displaystyle:=\ \frac{p_{i}}{1-\gamma}\ \leqslant\ p_{i}/(1-\epsilon)\ , (223)
∑i=1lp~i\displaystyle\sum_{i=1}^{l}\tilde{p}_{i}\ = 1,\displaystyle=\ 1\ , (224)
𝒎𝒙\displaystyle\bm{m}_{\bm{x}}\ =1l​∑i=1l𝒙i,\displaystyle=\ \frac{1}{l}\ \sum_{i=1}^{l}\ \bm{x}_{i}\ , (225)
mmax\displaystyle m_{\max}\ =max1⩽i⩽l⁡‖𝒙i−𝒎𝒙‖.\displaystyle=\ \max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}\ . (226)
Lemma A9.

For a query 𝛏\bm{\xi} and data 𝐗=(𝐱1,…,𝐱N)\bm{X}=(\bm{x}_{1},\ldots,\bm{x}_{N}), we define

Equation 227
0\displaystyle 0\ ⩽c=minj,l<j⁡(𝝃T​𝒎𝒙−𝝃T​𝒙j)=𝝃T​𝒎𝒙−maxj,l<j⁡𝝃T​𝒙j.\displaystyle\leqslant\ c\ =\ \min_{j,l<j}\left(\bm{\xi}^{T}\bm{m}_{\bm{x}}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ =\ \bm{\xi}^{T}\bm{m}_{\bm{x}}\ -\ \max_{j,l<j}\bm{\xi}^{T}\bm{x}_{j}\ . (227)

The following holds:

Equation 228
‖𝒎𝒙−f⁡(𝝃)‖\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ f(\bm{\xi})\right\|}}\ ⩽mmax+ 2​γ​M⩽mmax+ 2​ϵ​M,\displaystyle\leqslant\ m_{\max}\ +\ 2\ \gamma\ M\ \leqslant\ m_{\max}\ +\ 2\ \epsilon\ M\ , (228)

where

Equations 229–230
M\displaystyle M\ =maxi⁡‖𝒙i‖,\displaystyle=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ , (229)
ϵ\displaystyle\epsilon\ =(N−l)​exp⁡(−β​c).\displaystyle=\ (N-l)\ \exp(-\ \beta\ c)\ . (230)
Equations 231–237
[softmax⁡(β​𝑿T​𝝃)]j\displaystyle[\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{j}\ =exp⁡(β⁡(𝝃T​𝒙j−𝝃T​𝒙s))1+∑k,k≠sexp⁡(β⁡(𝝃T​𝒙k−𝝃T​𝒙s))⩽exp⁡(−β​c)=ϵN−l,\displaystyle=\ \frac{\exp(\beta\ (\bm{\xi}^{T}\bm{x}_{j}\ -\ \bm{\xi}^{T}\bm{x}_{s}))}{1\ +\ \sum_{k,k\not=s}\exp(\beta\ (\bm{\xi}^{T}\bm{x}_{k}\ -\ \bm{\xi}^{T}\bm{x}_{s}))}\ \leqslant\ \exp(-\ \beta\ c)\ =\ \frac{\epsilon}{N-l}\ , (231)
f⁡(𝝃)\displaystyle f(\bm{\xi})\ =𝑿​softmax​(β​𝑿T​𝝃)=∑j=1N𝒙j​[softmax⁡(β​𝑿T​𝝃)]j.\displaystyle=\ \bm{X}\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ =\ \sum_{j=1}^{N}\bm{x}_{j}\ [\mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})]_{j}\ . (232)
‖𝒎𝒙−∑j=1lpj1−γ​𝒙j‖2=‖∑j=1lpj1−γ​(𝒎𝒙−𝒙j)‖2\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\right\|}}^{2}\ =\ {{\left\|\sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\left(\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right)\right\|}}^{2} (233)
=∑j=1,k=1lpj1−γ​pk1−γ​(𝒎𝒙−𝒙j)T​(𝒎𝒙−𝒙k)\displaystyle=\ \sum_{j=1,k=1}^{l}\frac{p_{j}}{1-\gamma}\frac{p_{k}}{1-\gamma}\left(\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right)^{T}\left(\bm{m}_{\bm{x}}\ -\ \bm{x}_{k}\right)
=12​∑j=1,k=1lpj1−γ​pk1−γ​(‖𝒎𝒙−𝒙j‖2+‖𝒎𝒙−𝒙k‖2−‖𝒙j−𝒙k‖2)\displaystyle=\ \frac{1}{2}\ \sum_{j=1,k=1}^{l}\frac{p_{j}}{1-\gamma}\frac{p_{k}}{1-\gamma}\left({{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right\|}}^{2}\ +\ {{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{k}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{j}\ -\ \bm{x}_{k}\right\|}}^{2}\right)
=∑j=1lpj1−γ​‖𝒎𝒙−𝒙j‖2−12​∑j=1,k=1lpj1−γ​pk1−γ​‖𝒙j−𝒙k‖2\displaystyle=\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ {{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right\|}}^{2}\ -\ \frac{1}{2}\ \sum_{j=1,k=1}^{l}\frac{p_{j}}{1-\gamma}\frac{p_{k}}{1-\gamma}{{\left\|\bm{x}_{j}\ -\ \bm{x}_{k}\right\|}}^{2}
⩽∑j=1lpj1−γ​‖𝒎𝒙−𝒙j‖2⩽mmax2.\displaystyle\leqslant\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ {{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right\|}}^{2}\ \leqslant\ m_{\max}^{2}\ .
‖𝒎𝒙−∑j=1lpj1−γ​𝒙j‖\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\right\|}}\ ⩽mmax\displaystyle\leqslant\ m_{\max} (234)
‖𝒎𝒙−f⁡(𝝃)‖\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ f(\bm{\xi})\right\|}}\ =‖𝒎𝒙−∑j=1Npj​𝒙j‖\displaystyle=\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{N}p_{j}\ \bm{x}_{j}\right\|}} (235)
=‖𝒎𝒙−∑j=1lpj​𝒙j−∑j=l+1Npj​𝒙j‖\displaystyle=\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}p_{j}\ \bm{x}_{j}\ -\ \sum_{j=l+1}^{N}p_{j}\ \bm{x}_{j}\right\|}}
=‖𝒎𝒙−∑j=1lpj1−γ​𝒙j+γ1−γ​∑j=1lpj​𝒙j−∑j=l+1Npj​𝒙j‖\displaystyle=\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\ +\ \frac{\gamma}{1-\gamma}\ \sum_{j=1}^{l}p_{j}\ \bm{x}_{j}\ -\ \sum_{j=l+1}^{N}p_{j}\ \bm{x}_{j}\right\|}}
⩽‖𝒎𝒙−∑j=1lpj1−γ​𝒙j‖+γ1−γ​‖∑j=1lpj​𝒙j‖+‖∑j=l+1Npj​𝒙j‖\displaystyle\leqslant\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\right\|}}\ +\ \frac{\gamma}{1-\gamma}\ {{\left\|\sum_{j=1}^{l}p_{j}\ \bm{x}_{j}\right\|}}\ +\ {{\left\|\sum_{j=l+1}^{N}p_{j}\ \bm{x}_{j}\right\|}}
⩽‖𝒎𝒙−∑j=1lpj1−γ​𝒙j‖+γ1−γ​∑j=1lpj​M+∑j=l+1Npj​M\displaystyle\leqslant\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\right\|}}\ +\ \frac{\gamma}{1-\gamma}\ \sum_{j=1}^{l}p_{j}\ M\ +\ \sum_{j=l+1}^{N}p_{j}\ M
⩽‖𝒎𝒙−∑j=1lpj1−γ​𝒙j‖+ 2​γ​M\displaystyle\leqslant\ {{\left\|\bm{m}_{\bm{x}}\ -\ \sum_{j=1}^{l}\frac{p_{j}}{1-\gamma}\ \bm{x}_{j}\right\|}}\ +\ 2\ \gamma\ M
⩽mmax+ 2​γ​M⩽mmax+ 2​ϵ​M,\displaystyle\leqslant\ m_{\max}\ +\ 2\ \gamma\ M\ \leqslant\ m_{\max}\ +\ 2\ \epsilon\ M\ ,
Δm\displaystyle\Delta_{m}\ =minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=𝒎𝒙T​𝒎𝒙−maxj,l<j⁡𝒎𝒙T​𝒙j.\displaystyle=\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \max_{j,l<j}\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ . (236)
Δm\displaystyle\Delta_{m}\ =minj,l<j⁡12​(‖𝒎𝒙‖2−‖𝒙j‖2+‖𝒎𝒙−𝒙j‖2)\displaystyle=\ \min_{j,l<j}\frac{1}{2}\ \left({{\left\|\bm{m}_{\bm{x}}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{j}\right\|}}^{2}\ +\ {{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right\|}}^{2}\right) (237)
=12​‖𝒎𝒙‖2−12​maxj,l<j⁡(‖𝒙j‖2−‖𝒎𝒙−𝒙j‖2).\displaystyle=\ \frac{1}{2}{{\left\|\bm{m}_{\bm{x}}\right\|}}^{2}\ -\ \frac{1}{2}\ \max_{j,l<j}\left({{\left\|\bm{x}_{j}\right\|}}^{2}\ -\ {{\left\|\bm{m}_{\bm{x}}\ -\ \bm{x}_{j}\right\|}}^{2}\right)\ .
Definition 4 (Sphere Sm\mathrm{S}_{m}).

The sphere Sm\mathrm{S}_{m} is defined as

Equation 238
Sm\displaystyle\mathrm{S}_{m}\ :={𝝃∣‖𝝃−𝒎𝒙‖⩽1β​mmax}.\displaystyle:=\ \left\{\bm{\xi}\mid{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ \leqslant\ \frac{1}{\beta\ m_{\max}}\right\}\ . (238)
Lemma A10.

With 𝛏\bm{\xi} given, if the assumptions

  1. A1:

    𝝃\bm{\xi} is inside sphere: 𝝃∈Sm\bm{\xi}\in\mathrm{S}_{m},

  2. A2:

    the center 𝒎𝒙\bm{m}_{\bm{x}} is well separated from other data 𝒙j\bm{x}_{j} with l<jl<j:

    Equation 239
    Δm\displaystyle\Delta_{m}\ ≥2​Mβ​mmax−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}),\displaystyle\geq\ \frac{2\ M}{\beta\ m_{\max}}\ -\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ , (239)
  3. A3:

    the distance mmaxm_{\max} of similar patterns to the center is sufficient small:

    Equation 240
    β​mmax2\displaystyle\beta\ m_{\max}^{2}\ ⩽ 1\displaystyle\leqslant\ 1 (240)

hold, then f⁡(𝛏)∈Smf(\bm{\xi})\in\mathrm{S}_{m}. Therefore, under conditions (A2) and (A3), ff is a mapping from Sm\mathrm{S}_{m} into Sm\mathrm{S}_{m}.

Equations 241–253
Δ~m\displaystyle\tilde{\Delta}_{m}\ =minj,l<j⁡(𝝃T​𝒎𝒙−𝝃T​𝒙j).\displaystyle=\ \min_{j,l<j}\left(\bm{\xi}^{T}\bm{m}_{\bm{x}}\ -\ \bm{\xi}^{T}\bm{x}_{j}\right)\ . (241)
|𝝃T​𝒙j−𝒎𝒙T​𝒙j|\displaystyle\left|\bm{\xi}^{T}\bm{x}_{j}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right| ⩽‖𝝃−𝒎𝒙‖​‖𝒙j‖⩽‖𝝃−𝒎𝒙‖​M.\displaystyle\leqslant\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ . (242)
Δ~m\displaystyle\tilde{\Delta}_{m}\ ≥minj,l<j⁡((𝒎𝒙T​𝒎𝒙−‖𝝃−𝒎𝒙‖​M)−(𝒎𝒙T​𝒙j+‖𝝃−𝒎𝒙‖​M))\displaystyle\geq\ \min_{j,l<j}\left(\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\ -\ \left(\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ +\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\right) (243)
=− 2​‖𝝃−𝒎𝒙‖​M+minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=Δm− 2​‖𝝃−𝒎𝒙‖​M\displaystyle=\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ +\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \Delta_{m}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M
≥Δm− 2​Mβ​mmax,\displaystyle\geq\ \Delta_{m}\ -\ 2\ \frac{M}{\beta\ m_{\max}}\ ,
∑i=1lpi\displaystyle\sum_{i=1}^{l}p_{i}\ ≥ 1−(N−l)​exp⁡(−β​Δ~m)= 1−ϵ~,\displaystyle\geq\ 1\ -\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ =\ 1\ -\ \tilde{\epsilon}\ , (244)
∑i=l+1Npi\displaystyle\sum_{i=l+1}^{N}p_{i}\ ⩽(N−l)​exp⁡(−β​Δ~m)=ϵ~.\displaystyle\leqslant\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ =\ \tilde{\epsilon}\ . (245)
‖𝒎𝒙−f⁡(𝝃)‖\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ f(\bm{\xi})\right\|}}\ ⩽mmax+ 2​ϵ~​M\displaystyle\leqslant\ m_{\max}\ +\ 2\ \tilde{\epsilon}\ M (246)
⩽mmax+ 2​(N−l)​exp⁡(−β​Δ~m)​M.\displaystyle\leqslant\ m_{\max}\ +\ 2\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ M\ .
⩽mmax+ 2​(N−l)​exp⁡(−β⁡(Δm− 2​Mβ​mmax))​M.\displaystyle\leqslant\ m_{\max}\ +\ 2\ (N-l)\ \exp(-\ \beta\ (\Delta_{m}\ -\ 2\ \frac{M}{\beta\ m_{\max}}))\ M\ .
‖𝒎𝒙−f⁡(𝝃)‖⩽mmax+ 2​(N−l)​exp⁡(−β⁡(Δm− 2​Mβ​mmax))​M\displaystyle{{\left\|\bm{m}_{\bm{x}}\ -\ f(\bm{\xi})\right\|}}\ \leqslant\ m_{\max}\ +\ 2\ (N-l)\ \exp\left(-\ \beta\ (\Delta_{m}\ -\ 2\ \frac{M}{\beta\ m_{\max}})\right)\ M (247)
⩽mmax+ 2​(N−l)​exp⁡(−β⁡(2​Mβ​mmax−\displaystyle\leqslant\ m_{\max}\ +\ 2\ (N-l)\ \exp\left(-\ \beta\ \left(\frac{2\ M}{\beta\ m_{\max}}\ -\right.\right.
1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M})− 2​Mβ​mmax))​M\displaystyle\left.\left.\frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ -\ 2\ \frac{M}{\beta\ m_{\max}}\right)\right)\ M
=mmax+ 2​(N−l)​1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}​M\displaystyle=\ m_{\max}\ +\ 2\ (N-l)\ \frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\ M
⩽mmax+1−β​mmax2β​mmax=1β​mmax,\displaystyle\leqslant\ m_{\max}\ +\ \frac{1\ -\ \beta\ m_{\max}^{2}}{\beta\ m_{\max}}\ =\ \frac{1}{\beta\ m_{\max}}\ ,
mmax=max1⩽i⩽l⁡‖𝒙i−𝒎𝒙‖\displaystyle m_{\max}=\max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}-\bm{m}_{\bm{x}}\right\|}} (248)
=max1⩽i⩽l‖𝒙i−1/l∑j=1l𝒙j‖\displaystyle=\max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}-1/l\sum_{j=1}^{l}\bm{x}_{j}\right\|}} (249)
=max1⩽i⩽l‖1/l∑j=1l(𝒙i−𝒙j)‖\displaystyle=\max_{1\leqslant i\leqslant l}{{\left\|1/l\sum_{j=1}^{l}(\bm{x}_{i}-\bm{x}_{j})\right\|}} (250)
⩽max1⩽i,j⩽l⁡‖𝒙i−𝒙j‖\displaystyle\leqslant\max_{1\leqslant i,j\leqslant l}{{\left\|\bm{x}_{i}-\bm{x}_{j}\right\|}} (251)
⩽max1⩽i⩽l⁡‖𝒙i‖+max1⩽j⩽l⁡‖𝒙i‖\displaystyle\leqslant\max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}\right\|}}+\max_{1\leqslant j\leqslant l}{{\left\|\bm{x}_{i}\right\|}} (252)
⩽2​M\displaystyle\leqslant 2M (253)
Lemma A11.

Assume that

  1. A1:
    Equation 254
    Δm\displaystyle\Delta_{m}\ ≥2​Mβ​mmax−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}),\displaystyle\geq\ \frac{2\ M}{\beta\ m_{\max}}\ -\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ , (254)

    and

  2. A2:
    Equation 255
    β​mmax2\displaystyle\beta\ m_{\max}^{2}\ ⩽ 1,\displaystyle\leqslant\ 1\ , (255)

then ff is a contraction mapping in Sm\mathrm{S}_{m}.

Equations 256–275
f⁡(𝝃)\displaystyle f(\bm{\xi})\ =f⁡(𝒎𝒙)+Jm​(𝝃−𝒎𝒙).\displaystyle=\ f(\bm{m}_{\bm{x}})\ +\ \mathrm{J}^{m}\ (\bm{\xi}\ -\ \bm{m}_{\bm{x}})\ . (256)
‖f⁡(𝝃)−f⁡(𝒎𝒙)‖\displaystyle{{\left\|f(\bm{\xi})\ -\ f(\bm{m}_{\bm{x}})\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒎𝒙‖.\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ . (257)
Δ~m\displaystyle\tilde{\Delta}_{m}\ =minj,l<j⁡(𝝃~T​𝒎𝒙−𝝃~T​𝒙j).\displaystyle=\ \min_{j,l<j}\left(\tilde{\bm{\xi}}^{T}\bm{m}_{\bm{x}}\ -\ \tilde{\bm{\xi}}^{T}\bm{x}_{j}\right)\ . (258)
ϵ~\displaystyle\tilde{\epsilon}\ =(N−l)​exp⁡(−β​Δ~m),\displaystyle=\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ , (259)
∑i=1lpi​(𝝃~)\displaystyle\sum_{i=1}^{l}p_{i}(\tilde{\bm{\xi}})\ ≥ 1−(N−l)​exp⁡(−β​Δ~m)= 1−ϵ~,\displaystyle\geq\ 1\ -\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ =\ 1\ -\ \tilde{\epsilon}\ , (260)
∑i=l+1Npi​(𝝃~)\displaystyle\sum_{i=l+1}^{N}p_{i}(\tilde{\bm{\xi}})\ ⩽(N−l)​exp⁡(−β​Δ~m)=ϵ~.\displaystyle\leqslant\ (N-l)\ \exp(-\ \beta\ \tilde{\Delta}_{m})\ =\ \tilde{\epsilon}\ . (261)
|𝝃~T​𝒙j−𝒎𝒙T​𝒙j|\displaystyle\left|\tilde{\bm{\xi}}^{T}\bm{x}_{j}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right| ⩽‖𝝃~−𝒎𝒙‖​‖𝒙j‖⩽‖𝝃~−𝒎𝒙‖​M.\displaystyle\leqslant\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ . (262)
Δ~m\displaystyle\tilde{\Delta}_{m}\ ≥minj,l<j⁡((𝒎𝒙T​𝒎𝒙−‖𝝃~−𝒎𝒙‖​M)−(𝒎𝒙T​𝒙j+‖𝝃~−𝒎𝒙‖​M))\displaystyle\geq\ \min_{j,l<j}\left(\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\ -\ \left(\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ +\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\right) (263)
=− 2​‖𝝃~−𝒎𝒙‖​M+minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=Δm− 2​‖𝝃~−𝒎𝒙‖​M\displaystyle=\ -\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ +\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \Delta_{m}\ -\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M
≥Δm− 2​‖𝝃−𝒎𝒙‖​M.\displaystyle\geq\ \Delta_{m}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ .
ϵ~\displaystyle\tilde{\epsilon}\ ⩽(N−l)​exp⁡(−β⁡(Δm− 2​‖𝝃−𝒎𝒙‖​M))\displaystyle\leqslant\ (N-l)\ \exp\left(-\ \beta\ \left(\Delta_{m}\ -\ 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\right) (264)
⩽(N−l)​exp⁡(−β⁡(Δm−2​Mβ​mmax)).\displaystyle\leqslant\ (N-l)\ \exp\left(-\ \beta\ \left(\Delta_{m}\ -\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ .
‖𝝃−𝒎𝒙‖⩽1β​mmax,\displaystyle{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ \leqslant\ \frac{1}{\beta\ m_{\max}}\ , (265)
2​‖𝝃−𝒎𝒙‖​M⩽2​Mβ​mmax.\displaystyle 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ \leqslant\ \frac{2\ M}{\beta\ m_{\max}}\ . (266)
Δ~m\displaystyle\tilde{\Delta}_{m}\ ≥minj,l<j⁡((𝒎𝒙T​𝒎𝒙+‖𝝃~−𝒎𝒙‖​M)−(𝒎𝒙T​𝒙j−‖𝝃~−𝒎𝒙‖​M))\displaystyle\geq\ \min_{j,l<j}\left(\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ +\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\ -\ \left(\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ -\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\right) (267)
= 2​‖𝝃~−𝒎𝒙‖​M+minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=Δm+ 2​‖𝝃~−𝒎𝒙‖​M\displaystyle=\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ +\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \Delta_{m}\ +\ 2\ {{\left\|\tilde{\bm{\xi}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M
≥Δm+ 2​‖𝝃−𝒎𝒙‖​M.\displaystyle\geq\ \Delta_{m}\ +\ 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ .
ϵ~\displaystyle\tilde{\epsilon}\ ≥(N−l)​exp⁡(−β⁡(Δm+2​Mβ​mmax)),\displaystyle\geq\ (N-l)\ \exp\left(-\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ , (268)
‖𝝃−𝒎𝒙‖⩽1β​mmax,\displaystyle{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ \leqslant\ \frac{1}{\beta\ m_{\max}}\ , (269)
2​‖𝝃−𝒎𝒙‖​M⩽2​Mβ​mmax.\displaystyle 2\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ \leqslant\ \frac{2\ M}{\beta\ m_{\max}}\ . (270)
‖J⁡(𝝃~)‖2\displaystyle{{\left\|\mathrm{J}(\tilde{\bm{\xi}})\right\|}}_{2}\ ⩽β⁡(mmax2+ϵ~​ 2​(2−ϵ~)​M2)\displaystyle\leqslant\ \beta\left(\ m_{\max}^{2}\ +\ \tilde{\epsilon}\ 2\ (2\ -\ \tilde{\epsilon})\ M^{2}\right) (271)
=β⁡(mmax2+ϵ~​4​M2− 2​ϵ~2​M2)\displaystyle=\ \beta\left(m_{\max}^{2}\ +\ \tilde{\epsilon}4\ M^{2}\ -\ 2\ \tilde{\epsilon}^{2}\ M^{2}\right)
⩽β⁡(mmax2+(N−l)​exp⁡(−β⁡(Δm−2​Mβ​mmax))​4​M2−\displaystyle\leqslant\ \beta\left(m_{\max}^{2}\ +\ (N-l)\ \exp\left(-\ \beta\ \left(\Delta_{m}\ -\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)4\ M^{2}\ -\right.
2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2).\displaystyle\left.2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\right)\ .
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽β⁡(mmax2+(N−l)​exp⁡(−β⁡(Δm−2​Mβ​mmax))​4​M2−\displaystyle\leqslant\ \beta\left(m_{\max}^{2}\ +\ (N-l)\ \exp\left(-\ \beta\ \left(\Delta_{m}\ -\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)4\ M^{2}\ -\right. (272)
2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2).\displaystyle\left.2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\right)\ .
Δm\displaystyle\Delta_{m}\ ≥2​Mβ​mmax−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}),\displaystyle\geq\ \frac{2\ M}{\beta\ m_{\max}}\ -\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ , (273)
Δm−2​Mβ​mmax\displaystyle\Delta_{m}\ -\ \frac{2\ M}{\beta\ m_{\max}}\ ≥−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}).\displaystyle\geq\ -\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ . (274)
‖Jm‖2⩽\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ \leqslant (275)
β⁡(mmax2+(N−l)​exp⁡(−β⁡(−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M})))\displaystyle\beta\left(m_{\max}^{2}\ +\ (N-l)\ \exp\left(-\ \beta\ \left(-\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\right)\right)\right.
4​M2− 2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2)\displaystyle\left.4\ M^{2}\ -\ 2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\right)
=β⁡(mmax2+(N−l)​exp⁡(ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}))\displaystyle=\ \beta\left(m_{\max}^{2}\ +\ (N-l)\ \exp\left(\ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\right)\right.
4​M2− 2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2)\displaystyle\left.4\ M^{2}\ -\ 2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\right)
=β⁡(mmax2+(N−l)​1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}​ 4​M2−\displaystyle=\ \beta\left(m_{\max}^{2}\ +\ (N-l)\ \frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\ 4\ M^{2}\ -\right.
2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2)\displaystyle\left.2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\right)
=β​mmax2+1−β​mmax2max⁡{mmax, 2​M}​ 2​M−\displaystyle=\ \beta m_{\max}^{2}\ +\ \frac{1\ -\ \beta\ m_{\max}^{2}}{\ \max\{m_{\max}\ ,\ 2\ M\}}\ 2\ M\ -
β​ 2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2\displaystyle\beta\ 2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}
⩽β​mmax2+ 1−β​mmax2−β​ 2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2\displaystyle\leqslant\ \beta m_{\max}^{2}\ +\ 1\ -\ \beta\ m_{\max}^{2}\ -\ \beta\ 2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}
= 1−β​ 2​(N−l)2​exp⁡(− 2​β​(Δm+2​Mβ​mmax))​M2<1.\displaystyle=\ 1\ -\ \beta\ 2\ (N-l)^{2}\ \exp\left(-\ 2\ \beta\ \left(\Delta_{m}\ +\ \frac{2\ M}{\beta\ m_{\max}}\right)\right)\ M^{2}\ <1\ .
Lemma A12.

Assume that

  1. A1:
    Equation 276
    Δm\displaystyle\Delta_{m}\ ≥2​Mβ​mmax−1β​ln⁡(1−β​mmax22​β​(N−l)​M​max⁡{mmax, 2​M}),\displaystyle\geq\ \frac{2\ M}{\beta\ m_{\max}}\ -\ \frac{1}{\beta}\ \ln\left(\frac{1\ -\ \beta\ m_{\max}^{2}}{2\ \beta\ (N-l)\ M\ \max\{m_{\max}\ ,\ 2\ M\}}\right)\ , (276)

    and

  2. A2:
    Equation 277
    β​mmax2\displaystyle\beta\ m_{\max}^{2}\ ⩽ 1,\displaystyle\leqslant\ 1\ , (277)

then ff has a fixed point in Sm\mathrm{S}_{m}.

Equations 278–296
M\displaystyle M\ :=maxi⁡‖𝒙i‖,\displaystyle:=\ \max_{i}{{\left\|\bm{x}_{i}\right\|}}\ , (278)
γ\displaystyle\gamma\ =∑i=l+1Npi⩽ϵ,\displaystyle=\ \sum_{i=l+1}^{N}p_{i}\ \leqslant\ \epsilon\ , (279)
1−γ\displaystyle 1-\gamma\ =∑i=1lpi≥ 1−ϵ,\displaystyle=\ \sum_{i=1}^{l}p_{i}\ \geq\ 1\ -\ \epsilon\ , (280)
p~i\displaystyle\tilde{p}_{i}\ :=pi1−γ⩽pi/(1−ϵ),\displaystyle:=\ \frac{p_{i}}{1-\gamma}\ \leqslant\ p_{i}/(1-\epsilon)\ , (281)
∑i=1lp~i\displaystyle\sum_{i=1}^{l}\tilde{p}_{i}\ = 1,\displaystyle=\ 1\ , (282)
𝒎𝒙\displaystyle\bm{m}_{\bm{x}}\ =1l​∑i=1l𝒙i,\displaystyle=\ \frac{1}{l}\ \sum_{i=1}^{l}\ \bm{x}_{i}\ , (283)
mmax\displaystyle m_{\max}\ =max1⩽i⩽l⁡‖𝒙i−𝒎𝒙‖.\displaystyle=\ \max_{1\leqslant i\leqslant l}{{\left\|\bm{x}_{i}\ -\ \bm{m}_{\bm{x}}\right\|}}\ . (284)
Varp~[𝒙1:l]\displaystyle\mathbf{\mathrm{Var}}_{\tilde{p}}[\bm{x}_{1:l}]\ =∑i=1lp~i​𝒙i​𝒙iT−(∑i=1lp~i​𝒙i)​(∑i=1lp~i​𝒙i)T\displaystyle=\ \sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\ \bm{x}_{i}^{T}\ -\ \left(\sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\right)\ \left(\sum_{i=1}^{l}\tilde{p}_{i}\ \bm{x}_{i}\right)^{T} (285)
=∑i=1lp~i​(𝒙i−∑i=1lp~i​𝒙i)​(𝒙i−∑i=1lp~i​𝒙i)T.\displaystyle=\ \sum_{i=1}^{l}\tilde{p}_{i}\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{l}\tilde{p}_{i}\bm{x}_{i}\right)\ \left(\bm{x}_{i}\ -\ \sum_{i=1}^{l}\tilde{p}_{i}\bm{x}_{i}\right)^{T}\ .
‖f⁡(𝝃)−𝒎𝒙∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{m}_{\bm{x}}^{*}\right\|}}\ =‖f⁡(𝝃)−f⁡(𝒎𝒙∗)‖⩽‖Jm‖2​‖𝝃−𝒎𝒙∗‖\displaystyle=\ {{\left\|f(\bm{\xi})\ -\ f(\bm{m}_{\bm{x}}^{*})\right\|}}\ \leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}^{*}\right\|}} (286)
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β⁡((1−γ)​mmax2+γ​ 2​(2−γ)​M2),\displaystyle\leqslant\ \beta\left((1-\gamma)\ m_{\max}^{2}\ +\ \gamma\ 2\ (2\ -\ \gamma)\ M^{2}\right)\ , (287)
‖J‖2\displaystyle{{\left\|\mathrm{J}\right\|}}_{2}\ ⩽β⁡(mmax2+ϵ​ 2​(2−ϵ)​M2).\displaystyle\leqslant\ \beta\left(\ m_{\max}^{2}\ +\ \epsilon\ 2\ (2\ -\ \epsilon)\ M^{2}\right)\ . (288)
βmmax2\displaystyle\beta\ \ m_{\max}^{2}\ < 1.\displaystyle<\ 1\ . (289)
Δm\displaystyle\Delta_{m}\ =minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=𝒎𝒙T​𝒎𝒙−maxj,l<j⁡𝒎𝒙T​𝒙j.\displaystyle=\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \max_{j,l<j}\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ . (290)
Δ~m\displaystyle\tilde{\Delta}_{m}\ =minj,l<j⁡(𝒙~T​𝒎𝒙−𝒙~T​𝒙j).\displaystyle=\ \min_{j,l<j}\left(\tilde{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \tilde{\bm{x}}^{T}\bm{x}_{j}\right)\ . (291)
|𝒙~T​𝒙j−𝒎𝒙T​𝒙j|\displaystyle\left|\tilde{\bm{x}}^{T}\bm{x}_{j}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right| ⩽‖𝒙~−𝒎𝒙‖​‖𝒙j‖⩽‖𝒙~−𝒎𝒙‖​M.\displaystyle\leqslant\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ {{\left\|\bm{x}_{j}\right\|}}\ \leqslant\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ . (292)
Δ~m\displaystyle\tilde{\Delta}_{m}\ ≥minj,l<j⁡((𝒎𝒙T​𝒎𝒙−‖𝒙~−𝒎𝒙‖​M)−(𝒎𝒙T​𝒙j+‖𝒙~−𝒎𝒙‖​M))\displaystyle\geq\ \min_{j,l<j}\left(\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\ -\ \left(\bm{m}_{\bm{x}}^{T}\bm{x}_{j}\ +\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\right)\right) (293)
=− 2​‖𝒙~−𝒎𝒙‖​M+minj,l<j⁡(𝒎𝒙T​𝒎𝒙−𝒎𝒙T​𝒙j)=Δm− 2​‖𝒙~−𝒎𝒙‖​M.\displaystyle=\ -\ 2\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ +\ \min_{j,l<j}\left(\bm{m}_{\bm{x}}^{T}\bm{m}_{\bm{x}}\ -\ \bm{m}_{\bm{x}}^{T}\bm{x}_{j}\right)\ =\ \Delta_{m}\ -\ 2\ {{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ M\ .
‖𝒙~−𝒎𝒙‖\displaystyle{{\left\|\tilde{\bm{x}}\ -\ \bm{m}_{\bm{x}}\right\|}}\ =‖λ​𝝃+(1−λ)​𝒎𝒙∗−𝒎𝒙‖\displaystyle=\ {{\left\|\lambda\bm{\xi}+(1-\lambda)\bm{m}_{\bm{x}}^{*}\ -\ \bm{m}_{\bm{x}}\right\|}} (294)
⩽λ⁡‖𝝃−𝒎𝒙‖+(1−λ)​‖𝒎𝒙∗−𝒎𝒙‖\displaystyle\leqslant\ \lambda\ {{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}}\ +\ (1-\lambda)\ {{\left\|\bm{m}_{\bm{x}}^{*}\ -\ \bm{m}_{\bm{x}}\right\|}}
⩽max⁡{‖𝝃−𝒎𝒙‖,‖𝒎𝒙∗−𝒎𝒙‖},\displaystyle\leqslant\ \max\{{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}},{{\left\|\bm{m}_{\bm{x}}^{*}\ -\ \bm{m}_{\bm{x}}\right\|}}\}\ ,
Δ~m\displaystyle\tilde{\Delta}_{m}\ ≥Δm− 2​max⁡{‖𝝃−𝒎𝒙‖,‖𝒎𝒙∗−𝒎𝒙‖}​M.\displaystyle\geq\ \Delta_{m}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}},{{\left\|\bm{m}_{\bm{x}}^{*}\ -\ \bm{m}_{\bm{x}}\right\|}}\}\ M\ . (295)
ϵ\displaystyle\epsilon\ =(N−l)​exp⁡(−β⁡(Δm− 2​max⁡{‖𝝃−𝒎𝒙‖,‖𝒎𝒙∗−𝒎𝒙‖}​M)).\displaystyle=\ (N-l)\exp(-\ \beta\ (\Delta_{m}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{m}_{\bm{x}}\right\|}},{{\left\|\bm{m}_{\bm{x}}^{*}\ -\ \bm{m}_{\bm{x}}\right\|}}\}\ M))\ . (296)

A.1.6 Properties of Fixed Points Near Stored Pattern

Capacity and retrieval quality trade off through separation. Packing more patterns closer together can reduce attraction margins, slow convergence and increase error. Storage here requires disjoint basins with unique attracting fixed points; it is more specific than merely counting stationary points.

A.1.6.1 Exponentially Many Patterns can be Stored.

The master inequality relates minimum separation to N, β and the largest norm M. For equal-radius patterns, separation becomes M²(1−cos α_min), allowing spherical-packing arguments. The placed-pattern claims in Theorems A3–A4 use Equation 315’s angular-coordinate counting rule as if it guaranteed pairwise angular spacing. It does not in general: A3’s d=4 example would require 64 vectors separated by at least π/2 in four dimensions, whereas at most eight can have pairwise nonpositive inner products. Those displayed claims and their proof remain unchanged, but the construction is not established as written.

The random-pattern result follows a separate route: a lower-tail bound on the minimum angle, estimates of its dimension-dependent constant, and a cosine bound. Theorem A5 solves the sufficient inequality using the principal Lambert W branch, giving a capacity lower bound √p·c^((d−1)/4) with failure probability p and radius K√(d−1), subject to its stated admissibility conditions. The cited examples use β=1,p=.001 and either K=3,d=20 or K=1,d=75. Corollary A1 replaces Lambert W with simpler bounds; Theorem A6 and Corollary A2 solve for dimension using the appropriate branch. Theorem A7 instead substitutes an expected minimum separation into the criterion; this is not the same as a high-probability guarantee that every random memory is retrievable.

Equations 297–299
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (297)
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡12​(‖𝒙i‖2−‖𝒙j‖2+‖𝒙i−𝒙j‖2)\displaystyle=\ \min_{j,j\not=i}\frac{1}{2}\ \left({{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{j}\right\|}}^{2}\ +\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right) (298)
=12​‖𝒙i‖2−12​maxj,j≠i⁡(‖𝒙j‖2−‖𝒙i−𝒙j‖2).\displaystyle=\ \frac{1}{2}{{\left\|\bm{x}_{i}\right\|}}^{2}\ -\ \frac{1}{2}\ \max_{j,j\not=i}\left({{\left\|\bm{x}_{j}\right\|}}^{2}\ -\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}^{2}\right)\ .
Si\displaystyle\mathrm{S}_{i}\ ={𝝃∣‖𝝃−𝒙i‖⩽1β​N​M}.\displaystyle=\ \left\{\bm{\xi}\mid{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}}\ \leqslant\ \frac{1}{\beta\ N\ M}\right\}\ . (299)
Definition 5 (Pattern Stored and Retrieved).

We assume that around every pattern 𝐱i\bm{x}_{i} a sphere Si\mathrm{S}_{i} is given. We say 𝐱i\bm{x}_{i} is stored if there is a single fixed point 𝐱i∗∈Si\bm{x}_{i}^{*}\in\mathrm{S}_{i} to which all points 𝛏∈Si\bm{\xi}\in\mathrm{S}_{i} converge, and Si∩Sj=∅\mathrm{S}_{i}\cap\mathrm{S}_{j}=\emptyset for i≠ji\not=j. We say 𝐱i\bm{x}_{i} is retrieved for a given ϵ\epsilon if iteration (update rule) Eq. (92) gives a point 𝐱~i\tilde{\bm{x}}_{i} that is at least ϵ\epsilon-close to the single fixed point 𝐱i∗∈Si\bm{x}_{i}^{*}\in\mathrm{S}_{i}. The retrieval error is ‖𝐱~i−𝐱i‖{{\left\|\tilde{\bm{x}}_{i}-\bm{x}_{i}\right\|}}.

Equations 300–311
Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​(N−1)​N​β​M2).\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ . (300)
2β​N+1β​ln⁡(2​N2​β​M2)>2β​N+1β​ln⁡(2​(N−1)​N​β​M2),\displaystyle\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ >\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ (N-1)\ N\ \beta\ M^{2}\right)\ , (301)
Δi\displaystyle\Delta_{i}\ ≥2β​N+1β​ln⁡(2​N2​β​M2),\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ , (302)
‖𝒙i−𝒙j‖\displaystyle{{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}\ ⩽2β​N​M.\displaystyle\leqslant\ \frac{2}{\beta\ N\ M}\ . (303)
Δi\displaystyle\Delta_{i}\ ⩽𝒙iT​(𝒙i−𝒙j)⩽‖𝒙i‖​‖𝒙i−𝒙j‖⩽M​2β​N​M=2β​N.\displaystyle\leqslant\ \bm{x}_{i}^{T}\left(\bm{x}_{i}\ -\ \bm{x}_{j}\right)\ \leqslant\ {{\left\|\bm{x}_{i}\right\|}}\ {{\left\|\bm{x}_{i}\ -\ \bm{x}_{j}\right\|}}\ \leqslant M\ \frac{2}{\beta\ N\ M}\ =\ \frac{2}{\beta\ N}\ . (304)
1\displaystyle 1\ < 2​(N−1)​N​β​M2.\displaystyle<\ 2\ (N-1)\ N\ \beta\ M^{2}\ . (305)
Δmin\displaystyle\Delta_{\min}\ =min1⩽i⩽N⁡Δi\displaystyle=\ \min_{1\leqslant i\leqslant N}\Delta_{i} (306)
Δmin\displaystyle\Delta_{\min}\ ≥2β​N+1β​ln⁡(2​N2​β​M2).\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ . (307)
∀i:‖𝒙i‖\displaystyle\forall_{i}:\ {{\left\|\bm{x}_{i}\right\|}}\ =M.\displaystyle=\ M\ . (308)
αmin\displaystyle\alpha_{\min}\ =min1⩽i<j⩽N⁡αi​j.\displaystyle=\ \min_{1\leqslant i<j\leqslant N}\alpha_{ij}\ . (309)
Δmin\displaystyle\Delta_{\min}\ =min1⩽i<j⩽N⁡M2​(1−cos⁡(αi​j))=M2​(1−cos⁡(αmin)),\displaystyle=\ \min_{1\leqslant i<j\leqslant N}M^{2}(1\ -\ \cos(\alpha_{ij}))\ =\ M^{2}(1\ -\ \cos(\alpha_{\min}))\ , (310)
M2​(1−cos⁡(αmin))\displaystyle M^{2}(1\ -\ \cos(\alpha_{\min}))\ ≥2β​N+1β​ln⁡(2​N2​β​M2).\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ . (311)
Theorem A3 (Storage Capacity (M=2): Placed Patterns).

We assume β=1\beta=1 and patterns on the sphere with radius MM. If M=2​d−1M=2\sqrt{d-1} and the dimension dd of the space is d≥4d\geq 4 or if M=1.7​d−1M=1.7\sqrt{d-1} and the dimension dd of the space is d≥50d\geq 50, then the number of patterns NN that can be stored (fixed point with attraction basin near pattern) is at least

Equation 312
N\displaystyle N\ = 22​(d−1).\displaystyle=\ 2^{2(d-1)}\ . (312)
Equations 313–319
M2​(1−cos⁡(αmin))\displaystyle M^{2}(1\ -\ \cos(\alpha_{\min}))\ ≥2β​N+1β​ln⁡(2​N2​β​M2).\displaystyle\geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ . (313)
∀i:minj,j≠i⁡αi​j=αmin,\displaystyle\forall_{i}:\ \min_{j,j\not=i}\alpha_{ij}\ =\ \alpha_{\min}\ , (314)
N\displaystyle N\ =(2​παmin)d−1\displaystyle=\ \left(\frac{2\pi}{\alpha_{\min}}\right)^{d-1} (315)
αmin\displaystyle\alpha_{\min}\ =2​πN1/(d−1).\displaystyle=\ \frac{2\pi}{N^{1/(d-1)}}\ . (316)
M2​(1−cos⁡(2​πN1/(d−1)))≥2β​N+1β​ln⁡(2​N2​β​M2).\displaystyle M^{2}\left(1\ -\ \cos\left(\frac{2\pi}{N^{1/(d-1)}}\right)\right)\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ . (317)
M2​(1−cos⁡(π2))≥2β​ 23​(d−1)+1β​ln⁡(2​β​M2)+1β​ 4​(d−1)​ln⁡2.\displaystyle M^{2}\left(1\ -\ \cos\left(\frac{\pi}{2}\right)\right)\ \geq\ \frac{2}{\beta\ 2^{3(d-1)}}\ +\ \frac{1}{\beta}\ \ln\left(2\ \beta\ M^{2}\right)\ +\ \frac{1}{\beta}\ 4\ (d-1)\ln 2\ . (318)
β​M2≥122​(d−1)−1+ln⁡(2​β​M2)+ 4​(d−1)​ln⁡2.\displaystyle\beta\ M^{2}\ \geq\ \frac{1}{2^{2(d-1)-1}}\ +\ \ln\left(2\ \beta\ M^{2}\right)\ +\ 4\ (d-1)\ln 2\ . (319)
Theorem A4 (Storage Capacity (M=5): Placed Patterns).

We assume β=1\beta=1 and patterns on the sphere with radius MM. If M=5​d−1M=5\sqrt{d-1} and the dimension dd of the space is d≥3d\geq 3 or if M=4​d−1M=4\sqrt{d-1} and the dimension dd of the space is d≥13d\geq 13, then the number of patterns NN that can be stored (fixed point with attraction basin near pattern) is at least

Equation 320
N\displaystyle N\ = 23​(d−1).\displaystyle=\ 2^{3(d-1)}\ . (320)
Equations 321–322
M2​(1−cos⁡(π4))≥2β​ 23​(d−1)+1β​ln⁡(2​β​M2)+1β​ 6​(d−1)​ln⁡2.\displaystyle M^{2}\left(1\ -\ \cos\left(\frac{\pi}{4}\right)\right)\ \geq\ \frac{2}{\beta\ 2^{3(d-1)}}\ +\ \frac{1}{\beta}\ \ln\left(2\ \beta\ M^{2}\right)\ +\ \frac{1}{\beta}\ 6\ (d-1)\ln 2\ . (321)
β​M2​(1−22)≥123​(d−1)−1+ln⁡(2​β​M2)+ 6​(d−1)​ln⁡2.\displaystyle\beta\ M^{2}\left(1\ -\ \frac{\sqrt{2}}{2}\right)\ \geq\ \frac{1}{2^{3(d-1)-1}}\ +\ \ln\left(2\ \beta\ M^{2}\right)\ +\ 6\ (d-1)\ln 2\ . (322)
Lemma A13 ((Brauchart et al., 2018)).

Let dd be the dimension of the pattern space,

Equation 323
κd\displaystyle\kappa_{d}\ :=1d​π​Γ⁡((d+1)/2)Γ⁡(d/2).\displaystyle:=\ \frac{1}{d\ \sqrt{\pi}}\ \frac{\Gamma((d+1)/2)}{\Gamma(d/2)}\ . (323)

and δ>0\delta>0 such that κd−12​δ(d−1)⩽1\frac{\kappa_{d-1}}{2}\delta^{(d-1)}\leqslant 1. Then

Equation 324
Pr⁡(N2d−1​αmin≥δ)\displaystyle\mathbf{\mathrm{Pr}}(N^{\frac{2}{d-1}}\alpha_{\min}\ \geq\ \delta)\ ≥ 1−κd−12​δd−1.\displaystyle\geq\ 1\ -\ \frac{\kappa_{d-1}}{2}\ \delta^{d-1}\ . (324)
Lemma A14.

For κd\kappa_{d} defined in Eq. (323) we have the following bounds for every d≥1d\geq 1:

Equation 325
1exp⁡(1/6)​e​π​d\displaystyle\frac{1}{\exp(1/6)\ \sqrt{e\ \pi\ d}}\ ⩽κd⩽exp⁡(1/12)2​π​d< 1.\displaystyle\leqslant\ \kappa_{d}\ \leqslant\ \frac{\exp(1/12)}{\sqrt{2\ \pi\ d}}\ <\ 1\ . (325)
Equations 326–328
1\displaystyle 1\ <Γ⁡(x)​(2​π)−12​x12−x​exp⁡(x)<exp⁡(112​x).\displaystyle<\ \Gamma(x)\ (2\ \pi)^{-\ \frac{1}{2}}x^{\frac{1}{2}\ -\ x}\exp(x)\ <\ \exp\left(\frac{1}{12\ x}\right)\ . (326)
κd=1d​π​Γ⁡((d+1)/2)Γ⁡(d/2)<1d​π​exp⁡(16​(d+1))​exp⁡(−d+12)​(d+12)d2exp⁡(−d2)​(d2)d2−12\displaystyle\kappa_{d}\ =\ \frac{1}{d\ \sqrt{\pi}}\ \frac{\Gamma((d+1)/2)}{\Gamma(d/2)}\ <\ \frac{1}{d\ \sqrt{\pi}}\ \frac{\exp\left(\frac{1}{6(d+1)}\right)\ \exp\left(-\ \frac{d+1}{2}\right)\ \left(\frac{d+1}{2}\right)^{\frac{d}{2}}}{\exp\left(-\ \frac{d}{2}\right)\ \left(\frac{d}{2}\right)^{\frac{d}{2}\ -\ \frac{1}{2}}} (327)
=1d​π​e​exp⁡(16​(d+1))​(1+1d)d2​d2⩽exp⁡(112)2​π​d.\displaystyle=\ \frac{1}{d\ \sqrt{\pi\ e}}\ \exp\left(\frac{1}{6(d+1)}\right)\ \left(1\ +\ \frac{1}{d}\right)^{\frac{d}{2}}\sqrt{\frac{d}{2}}\ \leqslant\ \frac{\exp\left(\frac{1}{12}\right)}{\sqrt{2\ \pi}\ \sqrt{d}}\ .
κd=1d​π​Γ⁡((d+1)/2)Γ⁡(d/2)>1d​π​exp⁡(−d+12)​(d+12)d2exp⁡(16​d)​exp⁡(−d2)​(d2)d2−12\displaystyle\kappa_{d}\ =\ \frac{1}{d\ \sqrt{\pi}}\ \frac{\Gamma((d+1)/2)}{\Gamma(d/2)}\ >\ \frac{1}{d\ \sqrt{\pi}}\ \frac{\exp\left(-\ \frac{d+1}{2}\right)\ \left(\frac{d+1}{2}\right)^{\frac{d}{2}}}{\exp\left(\frac{1}{6\ d}\right)\ \exp\left(-\frac{d}{2}\right)\ \left(\frac{d}{2}\right)^{\frac{d}{2}-\frac{1}{2}}} (328)
=1d​π​e​exp⁡(16​d)​(1+1d)d2​d2≥1exp⁡(16)​e​π​d,\displaystyle=\ \frac{1}{d\ \sqrt{\pi\ e}\ \exp\left(\frac{1}{6\ d}\right)}\ \left(1+\frac{1}{d}\right)^{\frac{d}{2}}\sqrt{\frac{d}{2}}\ \geq\ \frac{1}{\exp\left(\frac{1}{6}\right)\ \sqrt{e\ \pi\ d}}\ ,
Lemma A15.

For 0⩽x⩽π0\leqslant x\leqslant\pi the function cos\cos can be upper bounded by:

Equation 329
cos⁡(x)\displaystyle\cos(x)\ ⩽ 1−x25.\displaystyle\leqslant\ 1\ -\ \frac{x^{2}}{5}\ . (329)
Equations 330–332
cos⁡(x)\displaystyle\cos(x)\ =∏n=1∞(1−4​x2(2​n−1)2​π2).\displaystyle=\ \prod_{n=1}^{\infty}\left(1-\frac{4\ x^{2}}{(2n-1)^{2}\ \pi^{2}}\right)\ . (330)
1−4​x2(2​n−1)2​π2⩽ 1\displaystyle 1\ -\ \frac{4\ x^{2}}{(2n-1)^{2}\ \pi^{2}}\ \leqslant\ 1 (331)
cos⁡(x)\displaystyle\cos(x)\ ⩽∏n=12(1−4​x2(2​n−1)2​π2)=(1−4​x2π2)​(1−4​x29​π2)\displaystyle\leqslant\ \prod_{n=1}^{2}\left(1-\frac{4\ x^{2}}{(2n-1)^{2}\pi^{2}}\right)\ =\ \left(1\ -\ \frac{4\ x^{2}}{\pi^{2}}\right)\ \left(1\ -\ \frac{4\ x^{2}}{9\ \pi^{2}}\right) (332)
= 1−40​x29​π2+16​x49​π4⩽ 1−40​x29​π2+16​x29​π2\displaystyle=\ 1\ -\ \frac{40\ x^{2}}{9\ \pi^{2}}\ +\ \frac{16\ x^{4}}{9\ \pi^{4}}\ \leqslant\ 1\ -\ \frac{40\ x^{2}}{9\ \pi^{2}}\ +\ \frac{16\ x^{2}}{9\ \pi^{2}}
= 1−24​x29​π2⩽ 1−x25.\displaystyle=\ 1\ -\ \frac{24\ x^{2}}{9\ \pi^{2}}\ \leqslant\ 1\ -\ \frac{x^{2}}{5}\ .
Theorem A5 (Storage Capacity (Main): Random Patterns).

We assume a failure probability 0<p⩽10<p\leqslant 1 and randomly chosen patterns on the sphere with radius M:=K​d−1M:=K\sqrt{d-1}. We define

Equation 333
a\displaystyle a\ :=2d−1​(1+ln⁡(2​β​K2​p​(d−1))),b:=2​K2​β5,\displaystyle:=\ \frac{2}{d-1}\ (1\ +\ \ln(2\ \beta\ K^{2}\ p\ (d-1)))\ ,\quad b\ :=\ \frac{2\ K^{2}\ \beta}{5}\ ,
c\displaystyle c\ :=bW0​(exp⁡(a+ln⁡(b)),\displaystyle:=\ \frac{b}{W_{0}(\exp(a\ +\ \ln(b))}\ , (333)

where W0W_{0} is the upper branch of the Lambert WW function (Olver et al., 2010, (4.13)) and ensure

Equation 334
c\displaystyle c\ ≥(2p)4d−1.\displaystyle\geq\ \left(\frac{2}{\sqrt{p}}\right)^{\frac{4}{d-1}}\ . (334)

Then with probability 1−p1-p, the number of random patterns that can be stored is

Equation 335
N\displaystyle N\ ≥p​cd−14.\displaystyle\geq\ \sqrt{p}\ c^{\frac{d-1}{4}}\ . (335)

Therefore it is proven for c≥3.1546c\geq 3.1546 with β=1\beta=1, K=3K=3, d=20d=20 and p=0.001p=0.001 (a+ln⁡(b)>1.27a+\ln(b)>1.27) and proven for c≥1.3718c\geq 1.3718 with β=1\beta=1, K=1K=1, d=75d=75, and p=0.001p=0.001 (a+ln⁡(b)<−0.94a+\ln(b)<-0.94).

Equations 336–348
Pr⁡(M2​(1−cos⁡(αmin)))≥2β​N+1β​ln⁡(2​N2​β​M2))≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(M^{2}(1\ -\cos(\alpha_{\min})))\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)\ \geq\ 1\ -\ p\ . (336)
1−cos⁡(αmin)\displaystyle 1\ -\ \cos(\alpha_{\min})\ ≥15​αmin2.\displaystyle\geq\ \frac{1}{5}\ \alpha_{\min}^{2}\ . (337)
Pr⁡(M2​αm​i​n25≥2β​N+1β​ln⁡(2​N2​β​M2))≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(M^{2}\frac{\alpha_{min}^{2}}{5}\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)\ \geq\ 1\ -\ p\ . (338)
Pr⁡(N2d−1​αm​i​n≥5​N2d−1M​(2β​N+1β​ln⁡(2​N2​β​M2))12)≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(N^{\frac{2}{d-1}}\ \alpha_{min}\ \geq\ \frac{\sqrt{5}\ N^{\frac{2}{d-1}}}{M}\ \left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{1}{2}}\right)\ \geq\ 1\ -\ p\ . (339)
Pr⁡(N2d−1​αm​i​n≥5​N2d−1M​(2β​N+1β​ln⁡(2​N2​β​M2))12)\displaystyle\mathbf{\mathrm{Pr}}\left(N^{\frac{2}{d-1}}\ \alpha_{min}\ \geq\ \frac{\sqrt{5}\ N^{\frac{2}{d-1}}}{M}\ \left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{1}{2}}\right) (340)
≥ 1−κd−12​ 5d−12​N2​M−(d−1)​(2β​N+1β​ln⁡(2​N2​β​M2))d−12.\displaystyle\geq\ 1\ -\ \frac{\kappa_{d-1}}{2}\ 5^{\frac{d-1}{2}}\ N^{2}\ M^{-(d-1)}\left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{d-1}{2}}\ .
κd−12​ 5d−12​N2​M−(d−1)​(2β​N+1β​ln⁡(2​N2​β​M2))d−12−p⩽ 0.\displaystyle\frac{\kappa_{d-1}}{2}\ 5^{\frac{d-1}{2}}\ N^{2}\ M^{-(d-1)}\left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta M^{2}\right)\right)^{\frac{d-1}{2}}\ -\ p\ \leqslant\ 0\ . (341)
5d−12​p​cd−12​M−(d−1)​(1β+1β​ln⁡(2​p​cd−12​β​M2))d−12⩽p.\displaystyle 5^{\frac{d-1}{2}}\ p\ c^{\frac{d-1}{2}}\ M^{-(d-1)}\left(\frac{1}{\beta}\ +\ \frac{1}{\beta}\ \ln\left(2\ p\ c^{\frac{d-1}{2}}\ \beta M^{2}\right)\right)^{\frac{d-1}{2}}\leqslant\ p\ . (342)
5​cK2​(d−1)​(1β+1β​ln⁡(2​β​cd−12​p​K2​(d−1)))− 1⩽ 0.\displaystyle\frac{5\ c}{K^{2}\ (d-1)}\left(\frac{1}{\beta}\ +\ \frac{1}{\beta}\ \ln\left(2\ \beta\ c^{\frac{d-1}{2}}\ p\ K^{2}\ (d-1)\right)\right)\ -\ 1\ \leqslant\ 0\ . (343)
a​c+c​ln⁡(c)−b⩽ 0,\displaystyle a\ c\ +\ c\ \ln(c)\ -\ b\ \leqslant\ 0\ , (344)
a\displaystyle a\ :=2d−1​(1+ln⁡(2​β​K2​p​(d−1))),b:=2​K2​β5.\displaystyle:=\ \frac{2}{d-1}\ (1\ +\ \ln(2\ \beta\ K^{2}\ p\ (d-1)))\ ,\quad b\ :=\ \frac{2\ K^{2}\ \beta}{5}\ .
a​c^+c^​ln⁡(c^)−b= 0\displaystyle a\ \hat{c}\ +\ \hat{c}\ \ln(\hat{c})\ -\ b\ =\ 0 (345)
a​c^+c^​ln⁡(c^)−b= 0\displaystyle a\ \hat{c}\ +\ \hat{c}\ \ln(\hat{c})\ -\ b\ =\ 0 (346)
⇔\displaystyle\Leftrightarrow\ \ a+ln⁡(c^)=b/c^\displaystyle a\ +\ \ln(\hat{c})\ =\ b/\hat{c}
⇔\displaystyle\Leftrightarrow\ \ a+ln⁡(b)+ln⁡(c^/b)=b/c^\displaystyle a\ +\ \ln(b)\ +\ \ln(\hat{c}/b)\ =\ b/\hat{c}
⇔\displaystyle\Leftrightarrow\ \ b/c^+ln⁡(b/c^)=a+ln⁡(b)\displaystyle b/\hat{c}\ +\ \ln(b/\hat{c})\ =\ a\ +\ \ln(b)
⇔\displaystyle\Leftrightarrow\ \ b/c^​exp⁡(b/c^)=exp⁡(a+ln⁡(b))\displaystyle b/\hat{c}\ \exp(b/\hat{c})\ =\ \exp(a\ +\ \ln(b))
⇔\displaystyle\Leftrightarrow\ \ b/c^=W0​(exp⁡(a+ln⁡(b)))\displaystyle b/\hat{c}\ =\ W_{0}(\exp(a\ +\ \ln(b)))
⇔\displaystyle\Leftrightarrow\ \ c^=bW0​(exp⁡(a+ln⁡(b)),\displaystyle\hat{c}\ =\ \frac{b}{W_{0}(\exp(a\ +\ \ln(b))}\ ,
c^\displaystyle\hat{c}\ =bW0​(exp⁡(a+ln⁡(b)).\displaystyle=\ \frac{b}{W_{0}(\exp(a\ +\ \ln(b))}\ . (347)
N\displaystyle N\ ≥p​c^d−14.\displaystyle\geq\ \sqrt{p}\ \hat{c}^{\frac{d-1}{4}}\ . (348)
Corollary A1.

We assume a failure probability 0<p⩽10<p\leqslant 1 and randomly chosen patterns on the sphere with radius M=K​d−1M=K\sqrt{d-1}. We define

Equation
a\displaystyle a\ :=2d−1​(1+ln⁡(2​β​K2​p​(d−1))),b:=2​K2​β5.\displaystyle:=\ \frac{2}{d-1}\ (1\ +\ \ln(2\ \beta\ K^{2}\ p\ (d-1)))\ ,\quad b\ :=\ \frac{2\ K^{2}\ \beta}{5}\ .

Using the omega constant Ω≈0.56714329\Omega\approx 0.56714329 we set

Equation 349
c\displaystyle c\ ={b​ln⁡(Ω​exp⁡(a+ln⁡(b))+ 1Ω⁡(1+Ω))−1for ​a+ln⁡(b)⩽ 0,b​(a+ln⁡(b))−a+ln⁡(b)a+ln⁡(b)+ 1for ​a+ln⁡(b)> 0\displaystyle=\ \begin{cases}b\ \ln\left(\frac{\Omega\ \exp(a\ +\ \ln(b))\ +\ 1}{\Omega\ (1\ +\ \Omega)}\right)^{-1}&\text{for }\ a\ +\ \ln(b)\ \leqslant\ 0\ ,\\ b\ (a\ +\ \ln(b))^{-\frac{a\ +\ \ln(b)}{a\ +\ \ln(b)\ +\ 1}}&\text{for }\ a\ +\ \ln(b)\ >\ 0\end{cases} (349)

and ensure

Equation 350
c\displaystyle c\ ≥(2p)4d−1.\displaystyle\geq\ \left(\frac{2}{\sqrt{p}}\right)^{\frac{4}{d-1}}\ . (350)

Then with probability 1−p1-p, the number of random patterns that can be stored is

Equation 351
N\displaystyle N\ ≥p​cd−14.\displaystyle\geq\ \sqrt{p}\ c^{\frac{d-1}{4}}\ . (351)

Examples are c≥3.1444c\geq 3.1444 for β=1\beta=1, K=3K=3, d=20d=20 and p=0.001p=0.001 (a+ln⁡(b)>1.27a+\ln(b)>1.27) and c≥1.2585c\geq 1.2585 for β=1\beta=1 K=1K=1, d=75d=75, and p=0.001p=0.001 (a+ln⁡(b)<−0.94a+\ln(b)<-0.94).

Equations 352–358
W0​(exp⁡(u))\displaystyle W_{0}(\exp(u))\ ⩽ln⁡(exp⁡(u)+y1+ln⁡(y)).\displaystyle\leqslant\ \ln\left(\frac{\exp(u)\ +\ y}{1\ +\ \ln(y)}\right)\ . (352)
y\displaystyle y\ = 1/W0(1)= 1/Ω=expΩ=− 1/lnΩ≈ 1.76322,\displaystyle=\ 1/W_{0}(1)\ =\ 1/\Omega\ =\ \exp{\Omega}\ =\ -\ 1/\ln\Omega\ \approx\ 1.76322\ , (353)
Ω\displaystyle\Omega\ =(∫−∞∞d​t(et−t)2+π2)−1− 1≈ 0.56714329.\displaystyle=\ \left(\int_{-\infty}^{\infty}\frac{\mathrm{d}t}{\left(e^{t}\ -\ t\right)^{2}\ +\ \pi^{2}}\right)^{-1}\ -\ 1\ \approx\ 0.56714329\ . (354)
W0​(exp⁡(u))\displaystyle W_{0}(\exp(u))\ ⩽ln⁡(exp⁡(u)+ 1/Ω1+ln⁡(1/Ω))=ln⁡(Ω​exp⁡(u)+ 1Ω⁡(1+Ω)).\displaystyle\leqslant\ \ln\left(\frac{\exp(u)\ +\ 1/\Omega}{1\ +\ \ln(1/\Omega)}\right)\ =\ \ln\left(\frac{\Omega\ \exp(u)\ +\ 1}{\Omega(1\ +\ \Omega)}\right)\ . (355)
ln⁡(Ω​ 1+ 1Ω⁡(1+Ω))=ln⁡(1Ω)=−ln⁡(Ω)=Ω=W0​(1).\displaystyle\ln\left(\frac{\Omega\ 1\ +\ 1}{\Omega(1\ +\ \Omega)}\right)\ =\ \ln\left(\frac{1}{\Omega}\right)\ =\ -\ \ln\left(\Omega\right)\ =\ \Omega\ =\ W_{0}(1)\ . (356)
W0​(exp⁡(u))\displaystyle W_{0}(\exp(u))\ ⩽uu1+u.\displaystyle\leqslant\ u^{\frac{u}{1\ +\ u}}\ . (357)
W0​(exp⁡(a+ln⁡(b)))\displaystyle W_{0}(\exp(a\ +\ \ln(b)))\ ⩽{ln⁡(Ω​exp⁡(a+ln⁡(b))+ 1Ω⁡(1+Ω))−1for ​a+ln⁡(b)⩽ 0,(a+ln⁡(b))−a+ln⁡(b)a+ln⁡(b)+ 1for ​a+ln⁡(b)> 0\displaystyle\leqslant\ \begin{cases}\ln\left(\frac{\Omega\ \exp(a\ +\ \ln(b))\ +\ 1}{\Omega\ (1\ +\ \Omega)}\right)^{-1}&\text{for }\ a\ +\ \ln(b)\ \leqslant\ 0\ ,\\ (a\ +\ \ln(b))^{-\frac{a\ +\ \ln(b)}{a\ +\ \ln(b)\ +\ 1}}&\text{for }\ a\ +\ \ln(b)\ >\ 0\end{cases} (358)
Theorem A6 (Storage Capacity (d computed): Random Patterns).

We assume a failure probability 0<p⩽10<p\leqslant 1 and randomly chosen patterns on the sphere with radius M=K​d−1M=K\sqrt{d-1}. We define

Equation 359
a\displaystyle a\ :=ln⁡(c)2−K2​β5​c,b:= 1+ln⁡(2​p​β​K2),\displaystyle:=\ \frac{\ln(c)}{2}\ -\ \frac{K^{2}\ \beta}{5\ c}\ ,\quad b\ :=\ 1\ +\ \ln\left(2\ p\ \beta\ K^{2}\right)\ ,
d\displaystyle d\ ={1+1a​W​(a​exp⁡(−b))for ​a≠0,1+exp⁡(−b)for ​a=0,\displaystyle=\ \begin{cases}1\ +\ \frac{1}{a}\ W(a\ \exp(-b))&\text{for }a\not=0\ ,\\ 1\ +\ \exp(-b)&\text{for }a=0\ ,\end{cases} (359)

where WW is the Lambert WW function (Olver et al., 2010, (4.13)). For 0<a0<a the function WW is the upper branch W0W_{0} and for a<0a<0 we use the lower branch W−1W_{-1}. If we ensure that

Equation 360
c\displaystyle c\ ≥(2p)4d−1,−1e⩽a​exp⁡(−b),\displaystyle\geq\ \left(\frac{2}{\sqrt{p}}\right)^{\frac{4}{d-1}}\ ,\quad\ -\ \frac{1}{e}\ \leqslant\ a\ \exp(-b)\ , (360)

then with probability 1−p1-p, the number of random patterns that can be stored is

Equation 361
N\displaystyle N\ ≥p​cd−14.\displaystyle\geq\ \sqrt{p}\ c^{\frac{d-1}{4}}\ . (361)
Equations 362–376
Pr⁡(M2​(1−cos⁡(αmin)))≥2β​N+1β​ln⁡(2​N2​β​M2))≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(M^{2}(1\ -\cos(\alpha_{\min})))\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)\ \geq\ 1\ -\ p\ . (362)
1−cos⁡(αmin)\displaystyle 1\ -\ \cos(\alpha_{\min})\ ≥15​αmin2.\displaystyle\geq\ \frac{1}{5}\ \alpha_{\min}^{2}\ . (363)
Pr⁡(M2​αm​i​n25≥2β​N+1β​ln⁡(2​N2​β​M2))≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(M^{2}\frac{\alpha_{min}^{2}}{5}\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)\ \geq\ 1\ -\ p\ . (364)
Pr⁡(N2d−1​αm​i​n≥5​N2d−1M​(2β​N+1β​ln⁡(2​N2​β​M2))12)≥ 1−p.\displaystyle\mathbf{\mathrm{Pr}}\left(N^{\frac{2}{d-1}}\ \alpha_{min}\ \geq\ \frac{\sqrt{5}\ N^{\frac{2}{d-1}}}{M}\ \left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{1}{2}}\right)\ \geq\ 1\ -\ p\ . (365)
Pr⁡(N2d−1​αm​i​n≥5​N2d−1M​(2β​N+1β​ln⁡(2​N2​β​M2))12)\displaystyle\mathbf{\mathrm{Pr}}\left(N^{\frac{2}{d-1}}\ \alpha_{min}\ \geq\ \frac{\sqrt{5}\ N^{\frac{2}{d-1}}}{M}\ \left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{1}{2}}\right) (366)
≥ 1−κd−12​ 5d−12​N2​M−(d−1)​(2β​N+1β​ln⁡(2​N2​β​M2))d−12.\displaystyle\geq\ 1\ -\ \frac{\kappa_{d-1}}{2}\ 5^{\frac{d-1}{2}}\ N^{2}\ M^{-(d-1)}\left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\right)^{\frac{d-1}{2}}\ .
κd−12​ 5d−12​N2​M−(d−1)​(2β​N+1β​ln⁡(2​N2​β​M2))d−12−p⩽ 0.\displaystyle\frac{\kappa_{d-1}}{2}\ 5^{\frac{d-1}{2}}\ N^{2}\ M^{-(d-1)}\left(\frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta M^{2}\right)\right)^{\frac{d-1}{2}}\ -\ p\ \leqslant\ 0\ . (367)
5d−12​p​cd−12​M−(d−1)​(1β+1β​ln⁡(2​p​cd−12​β​M2))d−12⩽p.\displaystyle 5^{\frac{d-1}{2}}\ p\ c^{\frac{d-1}{2}}\ M^{-(d-1)}\left(\frac{1}{\beta}\ +\ \frac{1}{\beta}\ \ln\left(2\ p\ c^{\frac{d-1}{2}}\ \beta M^{2}\right)\right)^{\frac{d-1}{2}}\leqslant\ p\ . (368)
5​cK2​(d−1)​(1β+1β​ln⁡(2​β​cd−12​p​K2​(d−1)))− 1⩽ 0.\displaystyle\frac{5\ c}{K^{2}\ (d-1)}\left(\frac{1}{\beta}\ +\ \frac{1}{\beta}\ \ln\left(2\ \beta\ c^{\frac{d-1}{2}}\ p\ K^{2}\ (d-1)\right)\right)\ -\ 1\ \leqslant\ 0\ . (369)
1+ln⁡(2​p​β​cd−12​K2​(d−1))−(d−1)​K2​β5​c⩽ 0.\displaystyle 1\ +\ \ln\left(2\ p\ \beta\ c^{\frac{d-1}{2}}\ K^{2}\ (d-1)\right)\ -\ \frac{(d-1)\ K^{2}\ \beta}{5\ c}\ \leqslant\ 0\ . (370)
a\displaystyle a\ :=ln⁡(c)2−K2​β5​c,b:= 1+ln⁡(2​p​β​K2),\displaystyle:=\ \frac{\ln(c)}{2}\ -\ \frac{K^{2}\ \beta}{5\ c}\ ,\quad b\ :=\ 1\ +\ \ln\left(2\ p\ \beta\ K^{2}\right)\ ,
ln⁡(d−1)+a⁡(d−1)+b⩽ 0.\displaystyle\ln(d-1)\ +\ a\ (d-1)\ +\ b\ \leqslant\ 0\ . (372)
ln⁡(d^−1)+a⁡(d^−1)+b= 0.\displaystyle\ln(\hat{d}-1)\ +\ a\ (\hat{d}-1)\ +\ b\ =\ 0\ . (373)
ln⁡(d^−1)+a⁡(d^−1)+b= 0\displaystyle\ln(\hat{d}-1)\ +\ a\ (\hat{d}-1)\ +\ b\ =\ 0 (374)
⇔\displaystyle\Leftrightarrow\ \ a⁡(d^−1)+ln⁡(d^−1)=−b\displaystyle a\ (\hat{d}-1)\ +\ \ln(\hat{d}-1)\ =\ -\ b
⇔\displaystyle\Leftrightarrow\ \ (d^−1)​exp⁡(a⁡(d^−1))=exp⁡(−b)\displaystyle(\hat{d}-1)\exp(a\ (\hat{d}-1))\ =\ \exp(-b)
⇔\displaystyle\Leftrightarrow\ \ a⁡(d^−1)​exp⁡(a⁡(d^−1))=a​exp⁡(−b)\displaystyle a\ (\hat{d}-1)\exp(a\ (\hat{d}-1))\ =\ a\ \exp(-b)
⇔\displaystyle\Leftrightarrow\ \ a⁡(d^−1)=W⁡(a​exp⁡(−b))\displaystyle a\ (\hat{d}-1)\ =\ W(a\ \exp(-b))
⇔\displaystyle\Leftrightarrow\ \ d^− 1=1a​W​(a​exp⁡(−b))\displaystyle\hat{d}\ -\ 1\ =\ \frac{1}{a}\ W(a\ \exp(-b))
⇔\displaystyle\Leftrightarrow\ \ d^= 1+1a​W​(a​exp⁡(−b)),\displaystyle\hat{d}\ =\ 1\ +\ \frac{1}{a}\ W(a\ \exp(-b))\ ,
d^\displaystyle\hat{d}\ = 1+1a​W​(a​exp⁡(−b)).\displaystyle=\ 1\ +\ \frac{1}{a}\ W(a\exp(-b))\ . (375)
N\displaystyle N\ ≥p​c^d−14.\displaystyle\geq\ \sqrt{p}\ \hat{c}^{\frac{d-1}{4}}\ . (376)
Corollary A2.

We assume a failure probability 0<p⩽10<p\leqslant 1 and randomly chosen patterns on the sphere with radius M=K​d−1M=K\sqrt{d-1}. We define

Equation 377
a\displaystyle a\ :=ln⁡(c)2−K2​β5​c,b:= 1+ln⁡(2​p​β​K2),\displaystyle:=\ \frac{\ln(c)}{2}\ -\ \frac{K^{2}\ \beta}{5\ c}\ ,\quad b\ :=\ 1\ +\ \ln\left(2\ p\ \beta\ K^{2}\right)\ ,
d\displaystyle d\ = 1+1a​(−ln⁡(−a)+b),\displaystyle=\ 1\ +\ \frac{1}{a}\ \left(-\ \ln(-a)\ +\ b\right)\ , (377)

and ensure

Equation 378
c\displaystyle c\ ≥(2p)4d−1,−1e⩽a​exp⁡(−b),a< 0,\displaystyle\geq\ \left(\frac{2}{\sqrt{p}}\right)^{\frac{4}{d-1}}\ ,\quad\ -\ \frac{1}{e}\ \leqslant\ a\ \exp(-b)\ ,\quad\ a\ <\ 0\ , (378)

then with probability 1−p1-p, the number of random patterns that can be stored is

Equation 379
N\displaystyle N\ ≥p​cd−14.\displaystyle\geq\ \sqrt{p}\ c^{\frac{d-1}{4}}\ . (379)

Setting β=1\beta=1, K=3K=3, c=2c=2 and p=0.001p=0.001 yields d<24d<24.

Equations 380–382
d= 1+W−1​(a​exp⁡(−b))a= 1+W−1​(−exp⁡(−(−ln⁡(−a)+b−1)−1))a\displaystyle d\ =\ 1\ +\ \frac{W_{-1}(a\exp(-b))}{a}\ =\ 1\ +\ \frac{W_{-1}(-\exp\left(-(-\ln(-a)+b-1)-1\right))}{a} (380)
−ee−1​(u+1)\displaystyle-\ \frac{e}{e-1}\ (u+1)\ <W−1​(−exp⁡(−u−1))<−(u+1).\displaystyle<\ W_{-1}(-\ \exp(-u-1))\ <\ -\ (u+1)\ . (381)
d> 1+−ln⁡(−a)+ba.\displaystyle d\ >\ 1\ +\ \frac{-\ln(-a)+b}{a}\ . (382)
Lemma A16 (Proposition 3.6 in Brauchart et al. (2018)).

We have the following lower bound on the expectation of αmin\alpha_{\min}:

Equation 383
E⁡[N2d−1​αmin]\displaystyle\mathbf{\mathrm{E}}\left[N^{\frac{2}{d-1}}\ \alpha_{\min}\right]\ ≥(Γ⁡(d2)2​(d−1)​π​Γ​(d−12))−1d−1​Γ​(1+1d−1)​d−1d−1Γ⁡(2+1d−1):=Cd−1.\displaystyle\geq\ \left(\frac{\Gamma(\frac{d}{2})}{2(d-1)\ \sqrt{\pi}\ \Gamma(\frac{d-1}{2})}\right)^{-\frac{1}{d-1}}\Gamma(1+\frac{1}{d-1})\ \frac{d^{-\frac{1}{d-1}}}{\Gamma(2+\frac{1}{d-1})}\ :=\ C_{d-1}. (383)

The bound is valid for all N≥2N\geq 2 and d≥2d\geq 2.

Lemma A17.

The following estimate holds for d≥2d\geq 2:

Equation 384
Cd\displaystyle C_{d}\ ≥ 1−ln⁡(d+1)d.\displaystyle\geq\ 1\ -\ \frac{\ln(d+1)}{d}\ . (384)
Equations 385–386
Γ⁡(x+1)\displaystyle\Gamma(x+1)\ =x​Γ​(x).\displaystyle=\ x\ \Gamma(x)\ . (385)
Cd\displaystyle C_{d}\ ≥(2​d)1d​Γ​(1+1d)​(d+1)−1dΓ⁡(2+1d)=(2​d)1d​(d+1)−1d1−1d>(d+1)1d\displaystyle\geq\ (2\ \sqrt{d})^{\frac{1}{d}}\Gamma(1+\frac{1}{d})\ \frac{(d+1)^{-\ \frac{1}{d}}}{\Gamma(2+\frac{1}{d})}\ =\ (2\ \sqrt{d})^{\frac{1}{d}}\frac{(d+1)^{-\ \frac{1}{d}}}{1-\frac{1}{d}}\ >\ (d+1)^{\frac{1}{d}} (386)
=exp⁡(−1d​ln⁡(d+1))≥ 1−1d​ln⁡(d+1),\displaystyle=\ \exp(-\frac{1}{d}\ \ln(d+1))\ \geq\ 1\ -\ \frac{1}{d}\ \ln(d+1)\ ,
Theorem A7 (Storage Capacity (expected separation): Random Patterns).

We assume patterns on the sphere with radius M=K​d−1M=K\sqrt{d-1} that are randomly chosen. Then for all values c≥1c\geq 1 for which

Equation 387
15​(d−1)​K2​c−1​(1−ln⁡(d−1)(d−1))2\displaystyle\frac{1}{5}\ (d-1)\ K^{2}\ c^{-1}(1\ -\ \frac{\ln(d-1)}{(d-1)})^{2}\ ≥2β​cd−14+1β​ln⁡(2​cd−12​β​(d−1)​K2)\displaystyle\geq\ \frac{2}{\beta\ c^{\frac{d-1}{4}}}\ +\ \frac{1}{\beta}\ \ln\left(2\ c^{\frac{d-1}{2}}\ \beta\ (d-1)\ K^{2}\right) (387)

holds, the number of stored patterns for the expected minimal separation is at least

Equation 388
N\displaystyle N\ =cd−14.\displaystyle=\ c^{\frac{d-1}{4}}\ . (388)

The inequality Eq. (387) is e.g. fulfilled with β=1\beta=1, K=3K=3, c=2c=2 and d≥17d\geq 17.

Equations 389–392
E⁡[Δmin]\displaystyle\mathbf{\mathrm{E}}[\Delta_{\min}]\ =E[M2(1−cos(αmin)))]=M2(1−E[cos(αmin))]).\displaystyle=\ \mathbf{\mathrm{E}}[M^{2}(1\ -\ \cos(\alpha_{\min})))]\ =\ M^{2}(1\ -\ \mathbf{\mathrm{E}}[\cos(\alpha_{\min}))])\ . (389)
M2(1−E[cos(αmin))])≥2β​N+1βln(2N2βM2).\displaystyle M^{2}(1\ -\ \mathbf{\mathrm{E}}[\cos(\alpha_{\min}))])\ \geq\ \frac{2}{\beta\ N}\ +\ \frac{1}{\beta}\ \ln\left(2\ N^{2}\ \beta\ M^{2}\right)\ . (390)
1−E⁡[cos⁡(αmin)]\displaystyle 1\ -\ \mathbf{\mathrm{E}}[\cos(\alpha_{\min})]\ ≥15​E​[αmin2]≥15​E​[αmin]2.\displaystyle\geq\ \frac{1}{5}\ \mathbf{\mathrm{E}}\left[\alpha_{\min}^{2}\right]\ \geq\ \frac{1}{5}\ \mathbf{\mathrm{E}}[\alpha_{\min}]^{2}\ . (391)
E​[αmin]2\displaystyle\mathbf{\mathrm{E}}[\alpha_{\min}]^{2}\ ≥N−4d−1​E​[N2d−1​αmin]2≥N−4d−1​Cd−12≥N−4d−1​(1−ln⁡(d−1)(d−1))2,\displaystyle\geq\ N^{-\frac{4}{d-1}}\ \mathbf{\mathrm{E}}[N^{\frac{2}{d-1}}\ \alpha_{\min}]^{2}\ \geq\ N^{-\frac{4}{d-1}}\ C_{d-1}^{2}\ \geq\ N^{-\frac{4}{d-1}}\ (1-\frac{\ln(d-1)}{(d-1)})^{2}\ , (392)

A.1.6.2 Retrieval of Patterns with One Update and Small Retrieval Error.

Theorems A8–A9 distinguish error to the local fixed point from error to the original stored vector. Both decay exponentially in the separation margin after accounting for the query/fixed-point distances, β, M and competing-pattern count. The Gaussian-pair calculation illustrates how typical separation grows with dimension; it does not by itself bound the minimum over exponentially many competitors. One-step retrieval is therefore a local, margin-dependent statement rather than convergence from every query.

Theorem A8 (Pattern Retrieval with One Update).

With query 𝛏\bm{\xi}, after one update the distance of the new point f⁡(𝛏)f(\bm{\xi}) to the fixed point 𝐱i∗\bm{x}_{i}^{*} is exponentially small in the separation Δi\Delta_{i}. The precise bounds using the Jacobian J=∂f⁡(𝛏)∂𝛏\mathrm{J}=\frac{\partial f(\bm{\xi})}{\partial\bm{\xi}} and its value Jm\mathrm{J}^{m} in the mean value theorem are:

Equations 393–394
‖f⁡(𝝃)−𝒙i∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒙i∗‖,\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ , (393)
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (394)

For given ϵ\epsilon and sufficient large Δi\Delta_{i}, we have ‖f⁡(𝛏)−𝐱i∗‖<ϵ{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}<\epsilon, that is, retrieval with one update.

Equations 395–400
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (395)
‖f⁡(𝝃)−𝒙i∗‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽‖Jm‖2​‖𝝃−𝒙i∗‖.\displaystyle\leqslant\ {{\left\|\mathrm{J}^{m}\right\|}}_{2}\ {{\left\|\bm{\xi}\ -\ \bm{x}_{i}^{*}\right\|}}\ . (396)
Δi\displaystyle\Delta_{i}\ =minj,j≠i⁡(𝒙iT​𝒙i−𝒙iT​𝒙j)=𝒙iT​𝒙i−maxj,j≠i⁡𝒙iT​𝒙j.\displaystyle=\ \min_{j,j\not=i}\left(\bm{x}_{i}^{T}\bm{x}_{i}\ -\ \bm{x}_{i}^{T}\bm{x}_{j}\right)\ =\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \max_{j,j\not=i}\bm{x}_{i}^{T}\bm{x}_{j}\ . (397)
E⁡[𝒙T​𝒙−𝒙T​𝒚]\displaystyle\mathbf{\mathrm{E}}\left[\bm{x}^{T}\bm{x}\ -\ \bm{x}^{T}\bm{y}\right]\ =∑j=1dE⁡[xj2]+∑j=1dE⁡[xj]​∑j=1dE⁡[yj]=d.\displaystyle=\ \sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{2}\right]\ +\ \sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[x_{j}\right]\sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[y_{j}\right]\ =\ d\ . (398)
Var⁡[𝒙T​𝒙−𝒙T​𝒚]=E⁡[(𝒙T​𝒙−𝒙T​𝒚)2]−d2\displaystyle\mathbf{\mathrm{Var}}\left[\bm{x}^{T}\bm{x}\ -\ \bm{x}^{T}\bm{y}\right]\ =\ \mathbf{\mathrm{E}}\left[\left(\bm{x}^{T}\bm{x}\ -\ \bm{x}^{T}\bm{y}\right)^{2}\right]\ -\ d^{2} (399)
=∑j=1dE[xj4]+∑j=1,k=1,k≠jdE[xj2]E[xk2]− 2∑j=1dE[xj3]E[yj]−\displaystyle=\ \sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{4}\right]\ +\ \sum_{j=1,k=1,k\not=j}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{2}\right]\ \mathbf{\mathrm{E}}\left[x_{k}^{2}\right]\ \ -\ 2\ \sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{3}\right]\mathbf{\mathrm{E}}\left[y_{j}\right]\ -
2​∑j=1,k=1,k≠jdE⁡[xj2]​E​[xk]​E​[yk]+∑j=1dE⁡[xj2]​E​[yj2]+\displaystyle 2\ \sum_{j=1,k=1,k\not=j}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{2}\right]\mathbf{\mathrm{E}}\left[x_{k}\right]\mathbf{\mathrm{E}}\left[y_{k}\right]\ +\ \sum_{j=1}^{d}\mathbf{\mathrm{E}}\left[x_{j}^{2}\right]\ \mathbf{\mathrm{E}}\left[y_{j}^{2}\right]\ +
∑j=1,k=1,k≠jdE⁡[xj]​E​[yj]​E​[xk]​E​[yk]−d2\displaystyle\sum_{j=1,k=1,k\not=j}^{d}\mathbf{\mathrm{E}}\left[x_{j}\right]\mathbf{\mathrm{E}}\left[y_{j}\right]\mathbf{\mathrm{E}}\left[x_{k}\right]\mathbf{\mathrm{E}}\left[y_{k}\right]\ -\ d^{2}
=3​d+d⁡(d−1)+d−d2= 3​d.\displaystyle=3\ d\ +\ d\ (d-1)\ +\ d\ -\ d^{2}\ =\ 3\ d\ .
‖Jm‖2\displaystyle{{\left\|\mathrm{J}^{m}\right\|}}_{2}\ ⩽ 2​β​N​M2​(N−1)​exp⁡(−β⁡(d− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle\leqslant\ 2\ \beta\ N\ M^{2}\ (N-1)\exp(-\ \beta\ (d\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (400)
Theorem A9 (Exponentially Small Retrieval Error).

The retrieval error ‖f⁡(𝛏)−𝐱i‖{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}\right\|}} of pattern 𝐱i\bm{x}_{i} is bounded by

Equation 401
‖f⁡(𝝃)−𝒙i‖\displaystyle{{\left\|f(\bm{\xi})\ -\ \bm{x}_{i}\right\|}}\ ⩽ 2​(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M))​M\displaystyle\leqslant\ 2\ (N-1)\ \exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ M (401)

and for ‖𝐱i−𝐱i∗‖⩽12​β​M{{\left\|\bm{x}_{i}-\bm{x}_{i}^{*}\right\|}}\leqslant\frac{1}{2\ \beta\ M} together with ‖𝐱i−𝛏‖⩽12​β​M{{\left\|\bm{x}_{i}-\bm{\xi}\right\|}}\leqslant\frac{1}{2\ \beta\ M} by

Equation 402
‖𝒙i−𝒙i∗‖\displaystyle{{\left\|\bm{x}_{i}\ -\ \bm{x}_{i}^{*}\right\|}}\ ⩽ 2​e​(N−1)​M​exp⁡(−β​Δi).\displaystyle\leqslant\ 2\ e\ (N-1)\ M\ \exp(-\ \beta\ \Delta_{i})\ . (402)
Equations 403–405
‖𝒙i−f⁡(𝝃)‖\displaystyle{{\left\|\bm{x}_{i}\ -\ f(\bm{\xi})\right\|}}\ ⩽ 2​ϵ​M,\displaystyle\leqslant\ 2\ \epsilon\ M\ , (403)
ϵ\displaystyle\epsilon\ =(N−1)​exp⁡(−β⁡(Δi− 2​max⁡{‖𝝃−𝒙i‖,‖𝒙i∗−𝒙i‖}​M)).\displaystyle=\ (N-1)\exp(-\ \beta\ (\Delta_{i}\ -\ 2\ \max\{{{\left\|\bm{\xi}\ -\ \bm{x}_{i}\right\|}},{{\left\|\bm{x}_{i}^{*}\ -\ \bm{x}_{i}\right\|}}\}\ M))\ . (404)
ϵ\displaystyle\epsilon\ ⩽e⁡(N−1)​M​exp⁡(−β​Δi).\displaystyle\leqslant\ e\ (N-1)\ M\ \exp(-\ \beta\ \Delta_{i})\ . (405)

A.1.7 Learning Associations

Association can use raw patterns, one learned bilinear map, or separate query/key projections into a shared lower-dimensional space. The following derivatives explain how these retrieval operations participate in backpropagation.

A.1.7.1 Association of Raw Patterns – No Mapping in an Associative Space.

For raw queries and stored patterns, differentiating the update gives the familiar covariance Jacobian with respect to the query. The memory gradient additionally contains a direct weighted-output term because changing a stored pattern changes both its score and its retrieved value.

Equations 406–409
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑿​𝒑,\displaystyle=\ \bm{X}\ \bm{p}\ , (406)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\xi})\ . (407)
∂𝝃new∂𝝃\displaystyle\frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}}\ =β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T\displaystyle=\ \beta\ \bm{X}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{X}^{T} (408)
∂𝒂T​𝝃new∂𝑿\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{X}}\ =𝒂​𝒑T+β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​(𝝃T​𝒂).\displaystyle=\ \bm{a}\ \bm{p}^{T}\ +\ \beta\ \bm{X}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ (\bm{\xi}^{T}\bm{a})\ . (409)

A.1.7.2 Learning an Association Matrix – Only One Set is Mapped in an Associative Space.

One map can transform keys into query space, queries into key space, or define the bilinear association directly. The full recurrent update includes the transformed memory values; the simplified final-step version can absorb an outer map into the value projection. The displayed gradients distinguish these cases, adding the direct output contribution when differentiating the full map. Repeated retrieval must remain in the associative space.

Equations 410–420
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑾​𝒀​𝒑,\displaystyle=\ \bm{W}\ \bm{Y}\ \bm{p}\ , (410)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝒀T​𝑾T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\ \bm{Y}^{T}\bm{W}^{T}\bm{\xi})\ . (411)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑾​𝒀​𝒑=𝑾​𝒀​softmax​(β​𝒀T​𝑾T​𝝃)\displaystyle=\ \bm{W}\ \bm{Y}\ \bm{p}\ =\ \bm{W}\ \bm{Y}\ \mathrm{softmax}(\beta\ \bm{Y}^{T}\bm{W}^{T}\bm{\xi}) (412)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝒀​𝒑=𝒀​softmax​(β​𝒀T​𝑾T​𝝃)\displaystyle=\ \bm{Y}\ \bm{p}\ =\ \bm{Y}\ \mathrm{softmax}(\beta\ \bm{Y}^{T}\bm{W}^{T}\bm{\xi}) (413)
∂𝒂T​𝝃new∂𝑾\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}}\ =∂𝝃new∂𝑾​∂𝒂T​𝝃new∂𝝃new=∂𝝃new∂(𝑾T​𝝃)​∂(𝑾T​𝝃)∂𝑾​∂𝒂T​𝝃new∂𝝃new.\displaystyle=\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}^{T}\bm{\xi})}\ \frac{\partial(\bm{W}^{T}\bm{\xi})}{\partial\bm{W}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ . (414)
∂𝝃new∂(𝑾T​𝝃)\displaystyle\frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}^{T}\bm{\xi})}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T} (415)
∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =𝒂.\displaystyle=\ \bm{a}\ . (416)
∂(𝑾T​𝝃)∂𝑾​∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial(\bm{W}^{T}\bm{\xi})}{\partial\bm{W}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =∂(𝑾T​𝝃)∂𝑾​𝒂=𝝃T​𝒂​𝑰.\displaystyle=\ \frac{\partial(\bm{W}^{T}\bm{\xi})}{\partial\bm{W}}\ \bm{a}\ =\ \bm{\xi}^{T}\bm{a}\bm{I}\ . (417)
∂𝒂T​𝝃new∂𝑾\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​(𝝃T​𝒂)=J⁡(𝝃T​𝒂),\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}(\bm{\xi}^{T}\bm{a})\ =\ \mathrm{J}\ (\bm{\xi}^{T}\bm{a})\ , (418)
𝒂​𝒑T​𝒀T\displaystyle\bm{a}\ \bm{p}^{T}\bm{Y}^{T} (419)
∂𝒂T​𝝃new∂𝑾\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}}\ =𝒂​𝒑T​𝒀T+β​𝑾​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​(𝝃T​𝒂)\displaystyle=\ \bm{a}\ \bm{p}^{T}\bm{Y}^{T}\ +\ \beta\ \bm{W}\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}(\bm{\xi}^{T}\bm{a}) (420)
=𝒂​𝒑T​𝒀T+𝑾​J​(𝝃T​𝒂).\displaystyle=\ \bm{a}\ \bm{p}^{T}\bm{Y}^{T}\ +\ \bm{W}\ \mathrm{J}\ (\bm{\xi}^{T}\bm{a})\ .

A.1.7.3 Learning Two Association Mappings – Both Sets are Mapped in an Associative Space.

Separate key and query maps factor the association matrix through dimension dₖ. The appendix derives gradients for each map, including the additional direct term for keys in the full update. A simplified output may live in raw-key space, so applying it repeatedly without the stated mapping would change the dynamics. These formulas train the geometry; the earlier convergence results analyze retrieval after that geometry is fixed.

Equations 421–437
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑿​𝒑,\displaystyle=\ \bm{X}\ \bm{p}\ , (421)
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\xi})\ . (422)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝑾K​𝒀​𝒑=𝑾K​𝒀​softmax​(β​𝒀T​𝑾KT​𝑾Q​𝒓).\displaystyle=\ \bm{W}_{K}\ \bm{Y}\ \bm{p}\ =\ \bm{W}_{K}\ \bm{Y}\ \mathrm{softmax}(\beta\ \bm{Y}^{T}\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})\ . (423)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =𝒀​𝒑=𝒀​softmax​(β​𝒀T​𝑾KT​𝑾Q​𝒓).\displaystyle=\ \bm{Y}\ \bm{p}\ =\ \bm{Y}\ \mathrm{softmax}(\beta\ \bm{Y}^{T}\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})\ . (424)
∂𝒂T​𝝃new∂𝑾Q\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{Q}}\ =∂𝝃new∂𝑾Q​∂𝒂T​𝝃new∂𝝃new=∂𝝃new∂(𝑾Q​𝒓)​∂(𝑾Q​𝒓)∂𝑾Q​∂𝒂T​𝝃new∂𝝃new.\displaystyle=\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{Q}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}_{Q}\ \bm{r})}\ \frac{\partial(\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{Q}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ . (425)
∂𝝃new∂(𝑾Q​𝒓)\displaystyle\frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}_{Q}\ \bm{r})}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​𝑾KT\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}\bm{W}_{K}^{T} (426)
∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =𝒂.\displaystyle=\ \bm{a}\ . (427)
∂(𝑾Q​𝒓)∂𝑾Q​∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial(\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{Q}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =∂(𝑾Q​𝒓)∂𝑾Q​𝒂=𝒓T​𝒂​𝑰.\displaystyle=\ \frac{\partial(\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{Q}}\ \bm{a}\ =\ \bm{r}^{T}\bm{a}\ \bm{I}\ . (428)
∂𝒂T​𝝃new∂𝑾Q\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{Q}}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​𝑾KT​(𝒓T​𝒂)=J​𝑾KT​(𝒓T​𝒂),\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}\ \bm{W}_{K}^{T}(\bm{r}^{T}\bm{a})\ =\ \mathrm{J}\ \bm{W}_{K}^{T}(\bm{r}^{T}\bm{a})\ , (429)
∂𝒂T​𝝃new∂𝑾Q\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{Q}}\ =β​𝑾K​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​𝑾KT​(𝒓T​𝒂)=𝑾K​J​𝑾KT​(𝒓T​𝒂).\displaystyle=\ \beta\ \bm{W}_{K}\ \bm{Y}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}\ \bm{W}_{K}^{T}(\bm{r}^{T}\bm{a})\ =\ \bm{W}_{K}\ \mathrm{J}\ \bm{W}_{K}^{T}(\bm{r}^{T}\bm{a})\ . (430)
∂𝒂T​𝝃new∂𝑾K\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{K}}\ =∂𝝃new∂𝑾K​∂𝒂T​𝝃new∂𝝃new=∂𝝃new∂(𝑾KT​𝑾Q​𝒓)​∂(𝑾KT​𝑾Q​𝒓)∂𝑾K​∂𝒂T​𝝃new∂𝝃new.\displaystyle=\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{K}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =\ \frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})}\ \frac{\partial(\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{K}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ . (431)
∂𝝃new∂(𝑾KT​𝑾Q​𝒓)\displaystyle\frac{\partial\bm{\xi}^{\mathrm{new}}}{\partial(\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T} (432)
∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =𝒂.\displaystyle=\ \bm{a}\ . (433)
∂(𝑾KT​𝑾Q​𝒓)∂𝑾K​∂𝒂T​𝝃new∂𝝃new\displaystyle\frac{\partial(\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{K}}\ \frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{\xi}^{\mathrm{new}}}\ =∂(𝑾KT​𝑾Q​𝒓)∂𝑾K​𝒂=𝑾QT​𝒓T​𝒂​𝑰.\displaystyle=\ \frac{\partial(\bm{W}_{K}^{T}\bm{W}_{Q}\ \bm{r})}{\partial\bm{W}_{K}}\ \bm{a}\ =\ \bm{W}_{Q}^{T}\bm{r}^{T}\bm{a}\ \bm{I}\ . (434)
∂𝒂T​𝝃new∂𝑾K\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{K}}\ =β​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​(𝑾QT​𝒓T​𝒂)=J⁡(𝑾QT​𝒓T​𝒂),\displaystyle=\ \beta\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}\ (\bm{W}_{Q}^{T}\bm{r}^{T}\bm{a})\ =\ \mathrm{J}\ (\bm{W}_{Q}^{T}\bm{r}^{T}\bm{a})\ , (435)
𝒂​𝒑T​𝒀T\displaystyle\bm{a}\ \bm{p}^{T}\bm{Y}^{T} (436)
∂𝒂T​𝝃new∂𝑾K\displaystyle\frac{\partial\bm{a}^{T}\bm{\xi}^{\mathrm{new}}}{\partial\bm{W}_{K}}\ =𝒂​𝒑T​𝒀T+β​𝑾K​𝒀​(diag⁡(𝒑)−𝒑​𝒑T)​𝒀T​(𝑾QT​𝒓T​𝒂)\displaystyle=\ \bm{a}\ \bm{p}^{T}\bm{Y}^{T}\ +\ \beta\ \bm{W}_{K}\ \bm{Y}\ \left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ \bm{Y}^{T}(\bm{W}_{Q}^{T}\bm{r}^{T}\bm{a}) (437)
=𝒂​𝒑T​𝒀T+𝑾K​J​(𝑾QT​𝒓T​𝒂).\displaystyle=\ \bm{a}\ \bm{p}^{T}\bm{Y}^{T}\ +\ \bm{W}_{K}\ \mathrm{J}\ (\bm{W}_{Q}^{T}\bm{r}^{T}\bm{a})\ .

A.1.8 Infinite Many Patterns and Forgetting Patterns

Temporal use extends the current memory as observations arrive and can reduce the influence of older entries. These mechanisms alter the active memory or attention scores rather than proving unlimited storage with fixed computational cost.

A.1.8.1 Infinite Many Patterns.

At time t, retrieve only from the observed prefix Xₜ. Causal masking expresses the same restriction in attention notation. The “infinite” construction is a formal stream interpretation: each actual finite-time update uses the available prefix, and storage/computation can grow with t.

Equations 438–448
𝑿\displaystyle\bm{X}\ =(𝒙1,𝒙2,…,).\displaystyle=\ \left(\bm{x}_{1},\bm{x}_{2},\ldots,\right)\ . (438)
𝑿t\displaystyle\bm{X}_{t}\ =(𝒙1,𝒙2,…,𝒙t).\displaystyle=\ \left(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{t}\right)\ . (439)
Et\displaystyle\mathrm{E}_{t}\ =−lse⁡(β,𝑿tT​𝝃t)+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}_{t}^{T}\bm{\xi}_{t})\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2} (440)
=−β−1​ln⁡(∑i=1texp⁡(β​𝒙iT​𝝃t))+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2.\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{t}\exp(\beta\bm{x}_{i}^{T}\bm{\xi}_{t})\right)\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2}\ . (441)
𝝃tnew\displaystyle\bm{\xi}_{t}^{\mathrm{new}}\ =𝑿t​𝒑t=𝑿t​softmax​(β​𝑿tT​𝝃t),\displaystyle=\ \bm{X}_{t}\ \bm{p}_{t}\ =\ \bm{X}_{t}\ \mathrm{softmax}(\beta\ \bm{X}_{t}^{T}\bm{\xi}_{t})\ , (442)
𝒑t\displaystyle\bm{p}_{t}\ =softmax⁡(β​𝑿tT​𝝃t).\displaystyle=\ \mathrm{softmax}(\beta\ \bm{X}_{t}^{T}\bm{\xi}_{t})\ . (443)
𝑿t\displaystyle\bm{X}_{t}\ =(𝒙1,𝒙2,…,𝒙t,−α​𝝃t,−α​𝝃t,…),\displaystyle=\ \left(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{t},-\alpha\bm{\xi}_{t},-\alpha\bm{\xi}_{t},\ldots\right)\ , (444)
Et\displaystyle\mathrm{E}_{t}\ =−lse⁡(β,𝑿tT​𝝃t)+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}_{t}^{T}\bm{\xi}_{t})\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2} (445)
=−β−1​ln⁡(∑i=1texp⁡(β​𝒙iT​𝝃t)+∑i=t+1⌊α⌋exp⁡(−β​α​‖𝝃t‖2))+12​𝝃tT​𝝃t+\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{t}\exp(\beta\bm{x}_{i}^{T}\bm{\xi}_{t})\ +\ \sum_{i=t+1}^{\lfloor\alpha\rfloor}\exp(-\beta\alpha{{\left\|\bm{\xi}_{t}\right\|}}^{2})\right)\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ + (446)
β−1​ln⁡t+12​Mt2.\displaystyle~~~~~~~\beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2}\ .
Et\displaystyle\mathrm{E}_{t}\ =−lse⁡(β,𝑿tT​𝝃t)+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}_{t}^{T}\bm{\xi}_{t})\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2} (447)
=−β−1​ln⁡(∑i=1texp⁡(β​𝒙iT​𝝃t))+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2.\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{t}\exp(\beta\bm{x}_{i}^{T}\bm{\xi}_{t})\right)\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2}\ . (448)

A.1.8.2 Forgetting Patterns.

Subtract an age-dependent term from an old pattern’s logit to obtain exponentially decreasing temporal weight. The printed energy451 includes βγ(t−i), while452 uses γ(t−i), and updates453–454 omit that age term. These original formulas remain unchanged. An implementation must choose and consistently apply its intended time penalty in both energy and softmax; the printed sequence does not define that choice consistently.

Equations 449–454
𝑿\displaystyle\bm{X}\ =(𝒙1,𝒙2,…,).\displaystyle=\ \left(\bm{x}_{1},\bm{x}_{2},\ldots,\right)\ . (449)
𝑿t\displaystyle\bm{X}_{t}\ =(𝒙1,𝒙2,…,𝒙t).\displaystyle=\ \left(\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{t}\right)\ . (450)
Et\displaystyle\mathrm{E}_{t}\ =−lse⁡(β,𝑿tT​𝝃t−γ​(t−1,t−2,…,0)T)+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}_{t}^{T}\bm{\xi}_{t}\ -\ \gamma(t-1,t-2,\ldots,0)^{T})\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2} (451)
=−β−1​ln⁡(∑i=1Texp⁡(β​𝒙iT​𝝃t−γ⁡(t−i)))+12​𝝃tT​𝝃t+β−1​ln⁡t+12​Mt2.\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{T}\exp(\beta\bm{x}_{i}^{T}\bm{\xi}_{t}\ -\ \gamma(t-i))\right)\ +\ \frac{1}{2}\bm{\xi}_{t}^{T}\bm{\xi}_{t}\ +\ \beta^{-1}\ln t\ +\ \frac{1}{2}M_{t}^{2}\ . (452)
𝝃tnew\displaystyle\bm{\xi}_{t}^{\mathrm{new}}\ =𝑿t​𝒑t=𝑿t​softmax​(β​𝑿tT​𝝃t),\displaystyle=\ \bm{X}_{t}\ \bm{p}_{t}\ =\ \bm{X}_{t}\ \mathrm{softmax}(\beta\bm{X}_{t}^{T}\bm{\xi}_{t})\ , (453)
𝒑t\displaystyle\bm{p}_{t}\ =softmax⁡(β​𝑿tT​𝝃t).\displaystyle=\ \mathrm{softmax}(\beta\bm{X}_{t}^{T}\bm{\xi}_{t})\ . (454)

A.1.9 Number of Spurious States

A monotone transformation of the energy relates its minima to modes of a weighted isotropic Gaussian mixture. Such a mixture can have more modes than components, so a memory may possess additional stable states beyond the N intended pattern states. High storage capacity does not imply that all spurious states disappear.

Equations 455–458
E\displaystyle\mathrm{E}\ =−lse⁡(β,𝑿T​𝝃)+12​𝝃T​𝝃+β−1​ln⁡N+12​M2\displaystyle=\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}M^{2} (455)
=−β−1​ln⁡(∑i=1Nexp⁡(β​𝒙iT​𝝃))+β−1​ln⁡N+12​𝝃T​𝝃+12​M2.\displaystyle=\ -\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)\ +\ \beta^{-1}\ln N\ +\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}\ +\ \frac{1}{2}M^{2}\ . (456)
exp⁡(−E)\displaystyle\exp(-\mathrm{E})\ =exp⁡(lse⁡(β,𝑿T​𝝃))​exp⁡(−12​𝝃T​𝝃)​C\displaystyle=\ \exp(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi}))\ \exp(-\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi})\ C (457)
=(∑i=1Nexp⁡(β​𝒙iT​𝝃))β−1​exp⁡(−12​𝝃T​𝝃)​C\displaystyle=\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)^{\beta^{-1}}\ \exp(-\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi})\ C
=(∑i=1Nexp⁡(β​𝒙iT​𝝃))β−1​(exp⁡(−β​12​𝝃T​𝝃))β−1​C\displaystyle=\left(\sum_{i=1}^{N}\exp(\beta\bm{x}_{i}^{T}\bm{\xi})\right)^{\beta^{-1}}\ \left(\exp(-\ \beta\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi})\right)^{\beta^{-1}}\ C
=(∑i=1Nexp⁡(β⁡(𝒙iT​𝝃−12​𝝃T​𝝃)))β−1​C\displaystyle=\left(\sum_{i=1}^{N}\exp(\beta\ (\bm{x}_{i}^{T}\bm{\xi}\ -\ \frac{1}{2}\bm{\xi}^{T}\bm{\xi}))\right)^{\beta^{-1}}\ C
=(∑i=1Nexp⁡(12​β​𝒙iT​𝒙i−12​β​(𝝃−𝒙i)T​(𝝃−𝒙i)))β−1​C\displaystyle=\left(\sum_{i=1}^{N}\exp(\frac{1}{2}\ \beta\ \bm{x}_{i}^{T}\bm{x}_{i}\ -\ \frac{1}{2}\ \beta\ (\bm{\xi}\ -\ \bm{x}_{i})^{T}(\bm{\xi}\ -\ \bm{x}_{i}))\right)^{\beta^{-1}}\ C
=(∑i=1Nλ⁡(𝒙i,β)​G​(𝝃,𝒙i,β−1​𝑰))β−1​C,\displaystyle=\left(\sum_{i=1}^{N}\lambda(\bm{x}_{i},\beta)\ G(\bm{\xi};\bm{x}_{i},\beta^{-1}\ \bm{I})\right)^{\beta^{-1}}\ C\ ,
∑i=1Nλ⁡(𝒙i,β)​G​(𝝃,𝒙i,β−1​𝑰).\displaystyle\sum_{i=1}^{N}\lambda(\bm{x}_{i},\beta)\ G(\bm{\xi};\bm{x}_{i},\beta^{-1}\ \bm{I})\ . (458)

A.2 Properties of Softmax, Log-Sum-Exponential, Legendre Transform, Lambert W Function

Softmax is the gradient of log-sum-exp. Its Jacobian β(diag(p)−ppᵀ) is positive semidefinite, annihilates the all-ones vector and has spectral norm at most β/2; concentrated probabilities give tighter local bounds. These facts supply monotonicity, Lipschitz and cocoercivity statements. The section also develops convex-conjugate rules, the entropy/log-sum-exp duality, Lambert W identities used in capacity calculations, and an integral-Jacobian mean-value formula.

The basic entropy duality is standard, but Lemma A29’s formula/domain for log-sum-exp composed with a rectangular matrix is not generally valid as written. For X=[0,1], XXᵀ is nonsingular and log(1+e^ξ) has a finite conjugate on [0,1], while the proposed pseudoinverse-based simplex constraint does not recover that domain. The original lemma and proof remain for inspection; the earlier direct covariance and energy-descent arguments do not require this domain formula.

Definition A1 (Softmax).
Equations 459–460
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝒙)\displaystyle=\ \mathrm{softmax}(\beta\bm{x}) (459)
pi\displaystyle p_{i}\ =[softmax⁡(β​𝒙)]i=exp⁡(β​xi)∑kexp⁡(β​xk).\displaystyle=\ [\mathrm{softmax}(\beta\bm{x})]_{i}\ =\ \frac{\exp(\beta x_{i})}{\sum_{k}\exp(\beta x_{k})}\ . (460)
Definition A2 (Log-Sum-Exp Function).
Equation 461
lse⁡(β,𝒙)\displaystyle\mathrm{lse}(\beta,\bm{x})\ =β−1​ln⁡(∑i=1Nexp⁡(β​xi)).\displaystyle=\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta x_{i})\right)\ . (461)
Equations 462–463
βa\displaystyle\beta_{a}\ =βln⁡a,\displaystyle=\ \frac{\beta}{\ln a}\ , (462)
lse⁡(β,𝒙)\displaystyle\mathrm{lse}(\beta,\bm{x})\ =β−1​ln⁡(∑i=1Nexp⁡(β​xi))\displaystyle=\ \beta^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta\ x_{i})\right) (463)
=(βa​ln⁡a)−1​ln⁡(∑i=1Nexp⁡(βa​ln⁡a​xi))\displaystyle=\ \left(\beta_{a}\ \ln a\right)^{-1}\ln\left(\sum_{i=1}^{N}\exp(\beta_{a}\ \ln a\ x_{i})\right)
=(βa)−1​loga⁡(∑i=1Naβa​xi).\displaystyle=\ \left(\beta_{a}\right)^{-1}\log_{a}\left(\sum_{i=1}^{N}a^{\beta_{a}\ x_{i}}\right)\ .
Lemma A18.

The softmax is the gradient of the lse\mathrm{lse}:

Equation 464
softmax⁡(β​𝒙)\displaystyle\mathrm{softmax}(\beta\bm{x})\ =∇𝒙lse​(β,𝒙).\displaystyle=\ \nabla_{\bm{x}}\mathrm{lse}(\beta,\bm{x})\ . (464)
Lemma A19.

We define

Equation 465
L\displaystyle\mathrm{L}\ :=𝒛T​𝒙−β−1​∑i=1Nzi​ln⁡zi\displaystyle:=\ \bm{z}^{T}\bm{x}\ -\ \beta^{-1}\sum_{i=1}^{N}z_{i}\ln z_{i} (465)

with L≥𝐳T​𝐱\mathrm{L}\geq\bm{z}^{T}\bm{x}. The lse\mathrm{lse} is the maximum of L\mathrm{L} on the NN-dimensional simplex DD with D={𝐳∣∑izi=1,0⩽zi}D=\{\bm{z}\mid\sum_{i}z_{i}=1,0\leqslant z_{i}\}:

Equation 466
lse⁡(β,𝒙)\displaystyle\mathrm{lse}(\beta,\bm{x})\ =max𝒛∈D⁡𝒛T​𝒙−β−1​∑i=1Nzi​ln⁡zi.\displaystyle=\ \max_{\bm{z}\in D}\bm{z}^{T}\bm{x}\ -\ \beta^{-1}\sum_{i=1}^{N}z_{i}\ln z_{i}\ . (466)

The softmax 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is the argument of the maximum of L\mathrm{L} on the NN-dimensional simplex DD with D={𝐳∣∑izi=1,0⩽zi}D=\{\bm{z}\mid\sum_{i}z_{i}=1,0\leqslant z_{i}\}:

Equation 467
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝒙)=arg⁡max𝒛∈D​𝒛T​𝒙−β−1​∑i=1Nzi​ln⁡zi.\displaystyle=\ \mathrm{softmax}(\beta\bm{x})\ =\ \arg\max_{\bm{z}\in D}\bm{z}^{T}\bm{x}\ -\ \beta^{-1}\sum_{i=1}^{N}z_{i}\ln z_{i}\ . (467)
Lemma A20.

The Jacobian Js\mathrm{J}_{s} of the softmax 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is

Equation 468
Js\displaystyle\mathrm{J}_{s}\ =∂softmax⁡(β​𝒙)∂𝒙=β⁡(diag⁡(𝒑)−𝒑​𝒑T),\displaystyle=\ \frac{\partial\mathrm{softmax}(\beta\bm{x})}{\partial\bm{x}}\ =\ \beta\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\ , (468)

which gives the elements

Equation 469
[Js]i​j\displaystyle[\mathrm{J}_{s}]_{ij}\ ={β​pi​(1−pi)for​i=j−β​pi​pjfor​i≠j.\displaystyle=\ \begin{cases}\beta p_{i}(1-p_{i})&\text{for}\ i=j\\ -\beta p_{i}p_{j}&\text{for}\ i\not=j\end{cases}\ . (469)
Lemma A21.

The Jacobian Js\mathrm{J}_{s} of the softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) has a zero eigenvalue with eigenvector 𝟏\bm{1}.

Equation 470
[Js​𝟏]i\displaystyle[\mathrm{J}_{s}\bm{1}]_{i}\ =β⁡(pi​(1−pi)−∑j,j≠ipi​pj)=β​pi​(1−∑jpj)=0.\displaystyle=\ \beta\left(p_{i}(1-p_{i})\ -\ \sum_{j,j\not=i}p_{i}p_{j}\right)\ =\ \beta\ p_{i}(1\ -\ \sum_{j}p_{j})\ =0\ . (470)
Lemma A22.

The Jacobian Js\mathrm{J}_{s} of the softmax 𝐩=softmax⁡(β​𝛏)\bm{p}=\mathrm{softmax}(\beta\bm{\xi}) is symmetric and positive semi-definite.

Equation 471
𝒛T​(diag⁡(𝒑)−𝒑​𝒑T)​𝒛\displaystyle\bm{z}^{T}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{z}\ =∑ipi​zi2−(∑ipi​zi)2\displaystyle=\ \sum_{i}p_{i}z_{i}^{2}-\left(\sum_{i}p_{i}z_{i}\right)^{2} (471)
=(∑ipi​zi2)​(∑ipi)−(∑ipi​zi)2≥ 0.\displaystyle=\ \left(\sum_{i}p_{i}z_{i}^{2}\right)\ \left(\sum_{i}p_{i}\right)-\left(\sum_{i}p_{i}z_{i}\right)^{2}\ \geq\ 0\ .
Lemma A23.

The softmax softmax⁡(β​𝐱)\mathrm{softmax}(\beta\bm{x}) is monotone for β>0\beta>0, that is,

Equation 472
(softmax⁡(β​𝒙)−softmax⁡(β​𝒙′))T​(𝒙−𝒙′)\displaystyle\left(\mathrm{softmax}(\beta\bm{x})\ -\ \mathrm{softmax}(\beta\bm{x}^{\prime})\right)^{T}\left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ ≥ 0.\displaystyle\geq\ 0\ . (472)
Equations 473–475
softmax⁡(𝒙)−softmax⁡(𝒙′)\displaystyle\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\ =Jsm​(𝒙−𝒙′).\displaystyle=\ \mathrm{J}_{s}^{m}\ \left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ . (473)
(softmax⁡(𝒙)−softmax⁡(𝒙′))T​(𝒙−𝒙′)=(𝒙−𝒙′)T​Jsm​(𝒙−𝒙′)≥ 0,\displaystyle\left(\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\right)^{T}\left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ =\ \left(\bm{x}\ -\ \bm{x}^{\prime}\right)^{T}\mathrm{J}_{s}^{m}\ \left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ \geq\ 0\ , (474)
𝒙T​Jsm​𝒙\displaystyle\bm{x}^{T}\mathrm{J}_{s}^{m}\bm{x}\ =∫01𝒙T​Js​(λ​𝒙+(1−λ)​𝒙′)​𝒙​𝑑λ≥ 0\displaystyle=\ \int_{0}^{1}\bm{x}^{T}\mathrm{J}_{s}(\lambda\bm{x}\ +\ (1-\lambda)\bm{x}^{\prime})\ \bm{x}\ \mathrm{d}\lambda\ \geq\ 0 (475)
Lemma A24.

For a softmax 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) with m=maxi⁡pi​(1−pi)m=\max_{i}p_{i}(1-p_{i}), the spectral norm of the Jacobian Js\mathrm{J}_{s} of the softmax is bounded:

Equations 476–478
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ ⩽ 2​m​β,\displaystyle\leqslant\ 2\ m\ \beta\ , (476)
‖Js‖1\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{1}\ ⩽ 2​m​β,\displaystyle\leqslant\ 2\ m\ \beta\ , (477)
‖Js‖∞\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{\infty}\ ⩽ 2​m​β.\displaystyle\leqslant\ 2\ m\ \beta\ . (478)

In particular everywhere holds

Equation 479
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ ⩽12​β.\displaystyle\leqslant\ \frac{1}{2}\ \beta\ . (479)

If pmax=maxi⁡pi≥1−ϵ≥0.5p_{\max}=\max_{i}p_{i}\geq 1-\epsilon\geq 0.5, then for the spectral norm of the Jacobian holds

Equation 480
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ ⩽ 2ϵβ− 2ϵ2β< 2ϵβ.\displaystyle\leqslant\ 2\ \epsilon\ \beta\ \ -\ 2\ \epsilon^{2}\ \beta\ \ <\ 2\ \epsilon\ \beta\ . (480)
Equations 481–487
‖𝑨‖1\displaystyle{{\left\|\bm{A}\right\|}}_{1}\ =max⁡∑ij⁡|ai​j|\displaystyle=\ \max_{j}\sum_{i}{{\left|a_{ij}\right|}} (481)
‖𝑨‖∞\displaystyle{{\left\|\bm{A}\right\|}}_{\infty}\ =max⁡∑ji⁡|ai​j|.\displaystyle=\ \max_{i}\sum_{j}{{\left|a_{ij}\right|}}\ . (482)
∑j|ai​j|\displaystyle\sum_{j}{{\left|a_{ij}\right|}}\ =β⁡(pi​(1−pi)+∑j,j≠ipi​pj)=β​pi​(1− 2​pi+∑jpj)\displaystyle=\ \beta\ \left(p_{i}(1-p_{i})\ +\ \sum_{j,j\not=i}p_{i}p_{j}\right)\ =\ \beta\ p_{i}\ (1\ -\ 2p_{i}\ +\ \sum_{j}p_{j}) (483)
= 2​β​pi​(1−pi)⩽ 2​m​β,\displaystyle=\ 2\ \beta\ p_{i}\ (1-p_{i})\ \leqslant\ 2\ m\ \beta\ ,
∑i|ai​j|\displaystyle\sum_{i}{{\left|a_{ij}\right|}}\ =β⁡(pj​(1−pj)+∑i,i≠jpj​pi)=β​pj​(1− 2​pj+∑ipi)\displaystyle=\ \beta\ \left(p_{j}\ (1-p_{j})\ +\ \sum_{i,i\not=j}p_{j}p_{i}\right)\ =\ \beta\ p_{j}\ (1\ -\ 2p_{j}\ +\ \sum_{i}p_{i}) (484)
= 2​β​pj​(1−pj)⩽ 2​m​β.\displaystyle=\ 2\ \beta\ p_{j}\ (1-p_{j})\ \leqslant\ 2\ m\ \beta\ .
‖Js‖1\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{1}\ ⩽ 2​m​β,\displaystyle\leqslant\ 2\ m\ \beta\ , (485)
‖Js‖∞\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{\infty}\ ⩽ 2​m​β,\displaystyle\leqslant\ 2\ m\ \beta\ , (486)
‖Js‖2\displaystyle{{\left\|\mathrm{J}_{s}\right\|}}_{2}\ ⩽‖Js‖1​‖Js‖∞⩽ 2​m​β.\displaystyle\leqslant\ \sqrt{{{\left\|\mathrm{J}_{s}\right\|}}_{1}{{\left\|\mathrm{J}_{s}\right\|}}_{\infty}}\ \leqslant\ 2\ m\ \beta\ . (487)
Lemma A25.

The softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is (β/2)(\beta/2)-Lipschitz. The softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is (2​β​m)(2\beta m)-Lipschitz in a convex environment UU for which m=max𝐱∈U⁡maxi​pi​(1−pi)m=\max_{\bm{x}\in U}\max_{i}p_{i}(1-p_{i}). For pmax=min𝐱∈U⁡maxi​pi=1−ϵp_{\max}=\min_{\bm{x}\in U}\max_{i}p_{i}=1-\epsilon, the softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is (2​β​ϵ)(2\beta\epsilon)-Lipschitz. For β<2​m\beta<2m, the softmax 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is contractive in UU on which mm is defined.

Equations 488–492
softmax⁡(𝒙)−softmax⁡(𝒙′)\displaystyle\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\ =Jsm​(𝒙−𝒙′).\displaystyle=\ \mathrm{J}_{s}^{m}\ \left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ . (488)
‖Js​(𝒙~)‖2⩽ 2​m~​β,\displaystyle{{\left\|\mathrm{J}_{s}(\tilde{\bm{x}})\right\|}}_{2}\ \leqslant\ 2\ \tilde{m}\ \beta\ , (489)
‖Js​(𝒙~)‖2⩽ 2​m​β\displaystyle{{\left\|\mathrm{J}_{s}(\tilde{\bm{x}})\right\|}}_{2}\ \leqslant\ 2\ m\ \beta (490)
‖Jsm‖2⩽ 2​m​β.\displaystyle{{\left\|\mathrm{J}_{s}^{m}\right\|}}_{2}\ \leqslant\ 2\ m\ \beta\ . (491)
‖softmax⁡(𝒙)−softmax⁡(𝒙′)‖\displaystyle{{\left\|\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\right\|}}\ ⩽‖Jsm‖2​‖𝒙−𝒙′‖⩽ 2​m​β​‖𝒙−𝒙′‖.\displaystyle\leqslant\ {{\left\|\mathrm{J}_{s}^{m}\right\|}}_{2}\ {{\left\|\bm{x}\ -\ \bm{x}^{\prime}\right\|}}\ \leqslant\ 2\ m\ \beta\ {{\left\|\bm{x}\ -\ \bm{x}^{\prime}\right\|}}\ . (492)
Lemma A26.

For m=max𝐱∈U⁡maxi​pi​(1−pi)m=\max_{\bm{x}\in U}\max_{i}p_{i}(1-p_{i}), softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is 1/(2​m​β)1/(2m\beta)-cocoercive in UU, that is,

Equation 493
(softmax⁡(𝒙)−softmax⁡(𝒙′))T​(𝒙−𝒙′)\displaystyle\left(\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\right)^{T}\left(\bm{x}\ -\ \bm{x}^{\prime}\right)\ ≥12​m​β​‖softmax⁡(𝒙)−softmax⁡(𝒙′)‖.\displaystyle\geq\ \frac{1}{2\ m\ \beta}{{\left\|\mathrm{softmax}(\bm{x})\ -\ \mathrm{softmax}(\bm{x}^{\prime})\right\|}}. (493)

In particular the softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is (2/β)(2/\beta)-cocoercive everywhere. With pmax=min𝐱∈U⁡maxi​pi=1−ϵp_{\max}=\min_{\bm{x}\in U}\max_{i}p_{i}=1-\epsilon, the softmax function 𝐩=softmax⁡(β​𝐱)\bm{p}=\mathrm{softmax}(\beta\bm{x}) is 1/(2​β​ϵ)1/(2\beta\epsilon)-cocoercive in UU.

Definition A3 (Convex Conjugate).

The Convex Conjugate (Legendre-Fenchel transform) of a function ff from a Hilbert Space XX to [−∞,∞][-\infty,\infty] is f∗f^{*} which is defined as

Equation 494
f∗​(𝒙∗)\displaystyle f^{*}(\bm{x}^{*})\ =sup𝒙∈X(𝒙T​𝒙∗−f⁡(𝒙)),𝒙∗∈X\displaystyle=\ \sup_{\bm{x}\in X}(\bm{x}^{T}\bm{x}^{*}\ -\ f(\bm{x}))\ ,\quad\bm{x}^{*}\in X (494)
Definition A4 (Legendre Transform).

The Legendre transform of a convex function ff from a convex set X⊂ℝnX\subset\mathbb{R}^{n} to ℝ\mathbb{R} (f:X→ℝf:X\rightarrow\mathbb{R}) is f∗f^{*}, which is defined as

Equations 495–496
f∗​(𝒙∗)\displaystyle f^{*}(\bm{x}^{*})\ =sup𝒙∈X(𝒙T​𝒙∗−f⁡(𝒙)),𝒙∗∈X∗,\displaystyle=\ \sup_{\bm{x}\in X}(\bm{x}^{T}\bm{x}^{*}\ -\ f(\bm{x}))\ ,\quad\bm{x}^{*}\in X^{*}\ , (495)
X∗\displaystyle X^{*}\ ={𝒙∗∈ℝn∣sup𝒙∈X(𝒙T​𝒙∗−f⁡(𝒙))<∞}.\displaystyle=\ \left\{\bm{x}^{*}\in\mathbb{R}^{n}\mid\sup_{\bm{x}\in X}(\bm{x}^{T}\bm{x}^{*}\ -\ f(\bm{x}))<\infty\right\}\ . (496)
Definition A5 (Epi-Sum).

Let ff and gg be two functions from XX to (−∞,∞](-\infty,\infty], then the infimal convolution (or epi-sum) of ff and gg is

Equation 497
f​□​g:X→[−∞,∞],𝒙↦inf𝒚∈X(f⁡(𝒚)+g⁡(𝒙−𝒚))\displaystyle f\Box g:X\rightarrow[-\infty,\infty]\ ,\ \bm{x}\mapsto\inf_{\bm{y}\in X}\left(f(\bm{y})+g(\bm{x}-\bm{y})\right) (497)
Lemma A27.

Let ff and gg be functions from XX to (−∞,∞](-\infty,\infty]. Then the following hold:

  1. 1.

    Convex Conjugate of norm squared

    Equation 498
    (12‖.‖2)∗\displaystyle\left(\frac{1}{2}{{\left\|.\right\|}}^{2}\right)^{*}\ =12‖.‖2.\displaystyle=\ \frac{1}{2}{{\left\|.\right\|}}^{2}\ . (498)
  2. 2.

    Convex Conjugate of a function multiplied by scalar 0<α∈ℝ0<\alpha\in\mathbb{R}

    Equation 499
    (α​f)∗\displaystyle\left(\alpha\ f\right)^{*}\ =αf∗(./α).\displaystyle=\ \alpha\ f^{*}(./\alpha)\ . (499)
  3. 3.

    Convex Conjugate of the sum of a function and a scalar β∈ℝ\beta\in\mathbb{R}

    Equation 500
    (f+β)∗\displaystyle\left(f\ +\ \beta\right)^{*}\ =f∗−β.\displaystyle=\ f^{*}\ -\ \beta\ . (500)
  4. 4.

    Convex Conjugate of affine transformation of the arguments. Let 𝑨\bm{A} be a non-singular matrix and 𝒃\bm{b} a vector

    Equation 501
    (f⁡(𝑨​𝒙+𝒃))∗\displaystyle\left(f\left(\bm{A}\bm{x}\ +\ \bm{b}\right)\right)^{*}\ =f∗​(𝑨−T​𝒙∗)−𝒃T​𝑨−T​𝒙∗.\displaystyle=\ f^{*}\left(\bm{A}^{-T}\bm{x}^{*}\right)\ -\ \bm{b}^{T}\bm{A}^{-T}\bm{x}^{*}\ . (501)
  5. 5.

    Convex Conjugate of epi-sums

    Equation 502
    (f​□​g)∗\displaystyle\left(f\Box g\right)^{*}\ =f∗+g∗.\displaystyle=\ f^{*}+g^{*}\ . (502)
Equations
α​f∗​(𝒙∗α)=α​sup𝒙∈X(𝒙T​𝒙∗α−f⁡(𝒙))=sup𝒙∈X(𝒙T​𝒙∗−α​f​(𝒙))=(α​f)∗​(𝒙∗)\displaystyle\alpha f^{*}\left(\frac{\bm{x}^{*}}{\alpha}\right)=\alpha\sup_{\bm{x}\in X}\left(\bm{x}^{T}\frac{\bm{x}^{*}}{\alpha}\ -\ f(\bm{x})\right)=\sup_{\bm{x}\in X}(\bm{x}^{T}\bm{x}^{*}-\alpha f(\bm{x}))=(\alpha f)^{*}(\bm{x}^{*})
(f⁡(𝑨​𝒙+𝒃))∗​(𝒙∗)\displaystyle\left(f\left(\bm{A}\bm{x}+\bm{b}\right)\right)^{*}(\bm{x}^{*}) =sup𝒙∈X(𝒙T​𝒙∗−f⁡(𝑨​𝒙+𝒃))\displaystyle=\sup_{\bm{x}\in X}\left(\bm{x}^{T}\bm{x}^{*}-f\left(\bm{A}\bm{x}+\bm{b}\right)\right)
=sup𝒙∈X((𝑨​𝒙+𝒃)T​𝑨−T​𝒙∗−f⁡(𝑨​𝒙+𝒃))−𝒃T​𝑨−T​𝒙∗\displaystyle=\sup_{\bm{x}\in X}\left(\left(\bm{A}\bm{x}+\bm{b}\right)^{T}\bm{A}^{-T}\bm{x}^{*}-f\left(\bm{A}\bm{x}+\bm{b}\right)\right)-\bm{b}^{T}\bm{A}^{-T}\bm{x}^{*}
=sup𝒚∈X(𝒚T​𝑨−T​𝒙∗−f⁡(𝒚))−𝒃T​𝑨−T​𝒙∗\displaystyle=\sup_{\bm{y}\in X}\left(\bm{y}^{T}\bm{A}^{-T}\bm{x}^{*}-f\left(\bm{y}\right)\right)-\bm{b}^{T}\bm{A}^{-T}\bm{x}^{*}
=f∗​(𝑨−T​𝒙∗)−𝒃T​𝑨−T​𝒙∗\displaystyle=f^{*}\left(\bm{A}^{-T}\bm{x}^{*}\right)-\bm{b}^{T}\bm{A}^{-T}\bm{x}^{*}
(f​□​g)∗​(𝒙∗)\displaystyle\left(f\Box g\right)^{*}(\bm{x}^{*}) =sup𝒙∈X(𝒙T​𝒙∗−inf𝒚∈X(f⁡(𝒚)−g⁡(𝒙−𝒚)))\displaystyle=\sup_{\bm{x}\in X}\left(\bm{x}^{T}\bm{x}^{*}-\inf_{\bm{y}\in X}\left(f(\bm{y})-g(\bm{x}-\bm{y})\right)\right)
=sup𝒙,𝒚∈X(𝒙T​𝒙∗−f⁡(𝒚)−g⁡(𝒙−𝒚))\displaystyle=\sup_{\bm{x},\bm{y}\in X}\left(\bm{x}^{T}\bm{x}^{*}-f(\bm{y})-g(\bm{x}-\bm{y})\right)
=sup𝒙,𝒚∈X((𝒚T​𝒙∗−f⁡(𝒚))+((𝒙−𝒚)T​𝒙∗−g⁡(𝒙−𝒚)))\displaystyle=\sup_{\bm{x},\bm{y}\in X}\left(\left(\bm{y}^{T}\bm{x}^{*}-f(\bm{y})\right)+\left(\left(\bm{x}-\bm{y}\right)^{T}\bm{x}^{*}-g(\bm{x}-\bm{y})\right)\right)
=f∗​(𝒙∗)+g∗​(𝒙∗)\displaystyle=f^{*}(\bm{x}^{*})+g^{*}(\bm{x}^{*})
Lemma A28.

The Legendre transform of the lse\mathrm{lse} is the negative entropy function, restricted to the probability simplex and vice versa. For the log-sum exponential

Equation 503
f⁡(𝒙)\displaystyle f(\bm{x})\ =ln⁡(∑i=1nexp⁡(xi)),\displaystyle=\ \ln\left(\sum_{i=1}^{n}\exp(x_{i})\right)\ , (503)

the Legendre transform is the negative entropy function, restricted to the probability simplex:

Equation 504
f∗​(𝒙∗)\displaystyle f^{*}(\bm{x}^{*})\ ={∑i=1nxi∗​ln⁡(xi∗) for ​ 0⩽xi∗​ and ​∑i=1nxi∗=1∞ otherwise .\displaystyle=\ \begin{cases}\sum_{i=1}^{n}x_{i}^{*}\ln(x^{*}_{i})&\text{ for }\ 0\leqslant x^{*}_{i}\ \text{ and }\ \sum_{i=1}^{n}x^{*}_{i}=1\\ \infty&\text{ otherwise }\end{cases}\ . (504)

For the negative entropy function, restricted to the probability simplex:

Equation 505
f⁡(𝒙)\displaystyle f(\bm{x})\ ={∑i=1nxi​ln⁡(xi) for ​ 0⩽xi​ and ​∑i=1nxi=1∞ otherwise .\displaystyle=\ \begin{cases}\sum_{i=1}^{n}x_{i}\ln(x_{i})&\text{ for }\ 0\leqslant x_{i}\ \text{ and }\ \sum_{i=1}^{n}x_{i}=1\\ \infty&\text{ otherwise }\end{cases}\ . (505)

the Legendre transform is the log-sum exponential

Equation 506
f∗​(𝒙∗)\displaystyle f^{*}(\bm{x}^{*})\ =ln⁡(∑i=1nexp⁡(xi∗)),\displaystyle=\ \ln\left(\sum_{i=1}^{n}\exp(x_{i}^{*})\right)\ , (506)
Lemma A29.

Let 𝐗​𝐗T\bm{X}\bm{X}^{T} be non-singular and XX a Hilbert space. We define

Equation 507
X∗={𝒂∣0⩽𝑿T(𝑿𝑿T)−1𝒂, 1T𝑿T(𝑿𝑿T)−1𝒂= 1}.\displaystyle X^{*}\ =\ \left\{\bm{a}\mid 0\ \leqslant\ \bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{a}\ ,\ \ \bm{1}^{T}\bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{a}\ =\ 1\right\}\ . (507)

and

Equation 508
Xv={𝒂∣𝒂=𝑿T𝝃,𝝃∈X}.\displaystyle X^{v}\ =\ \left\{\bm{a}\mid\bm{a}=\bm{X}^{T}\bm{\xi}\ ,\ \ \bm{\xi}\in X\right\}\ . (508)

The Legendre transform of lse⁡(β,𝐗T​𝛏)\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi}) with 𝛏∈X\bm{\xi}\in X is

Equation 509
(lse⁡(β,𝑿T​𝝃))∗​(𝝃∗)\displaystyle\left(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\right)^{*}(\bm{\xi}^{*})\ =(lse⁡(β,𝒗))∗​(𝑿T​(𝑿​𝑿T)−1​𝝃∗),\displaystyle=\ \left(\mathrm{lse}(\beta,\bm{v})\right)^{*}\left(\bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{\xi}^{*}\right)\ , (509)

with 𝛏∗∈X∗\bm{\xi}^{*}\in X^{*} and 𝐯∈Xv\bm{v}\in X^{v}. The domain of (lse⁡(β,𝐗T​𝛏))∗\left(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\right)^{*} is X∗X^{*}.

Furthermore we have

Equation 510
(lse⁡(β,𝑿T​𝝃))∗⁣∗\displaystyle\left(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\right)^{**}\ =lse⁡(β,𝑿T​𝝃).\displaystyle=\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ . (510)
Equations 511–514
(lse⁡(β,𝑿T​𝝃))∗​(𝝃∗)=sup𝝃∈X𝝃T​𝝃∗−lse⁡(β,𝑿T​𝝃)\displaystyle\left(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\right)^{*}(\bm{\xi}^{*})\ =\ \sup_{\bm{\xi}\in X}\bm{\xi}^{T}\bm{\xi}^{*}\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi}) (511)
=sup𝝃∈X(𝑿T​𝝃)T​𝑿T​(𝑿​𝑿T)−1​𝝃∗−lse⁡(β,𝑿T​𝝃)\displaystyle=\ \sup_{\bm{\xi}\in X}\left(\bm{X}^{T}\bm{\xi}\right)^{T}\bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{\xi}^{*}\ -\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})
=sup𝒗∈Xv𝒗T​𝑿T​(𝑿​𝑿T)−1​𝝃∗−lse⁡(β,𝒗)\displaystyle=\ \sup_{\bm{v}\in X^{v}}\bm{v}^{T}\bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{\xi}^{*}\ -\ \mathrm{lse}(\beta,\bm{v})
=sup𝒗∈Xv𝒗T​𝒗∗−lse⁡(β,𝒗)\displaystyle=\ \sup_{\bm{v}\in X^{v}}\bm{v}^{T}\bm{v}^{*}\ -\ \mathrm{lse}(\beta,\bm{v})
=(lse⁡(β,𝒗))∗​(𝒗∗)=(lse⁡(β,𝒗))∗​(𝑿T​(𝑿​𝑿T)−1​𝝃∗),\displaystyle=\ \left(\mathrm{lse}(\beta,\bm{v})\right)^{*}(\bm{v}^{*})\ =\ \left(\mathrm{lse}(\beta,\bm{v})\right)^{*}\left(\bm{X}^{T}\left(\bm{X}\bm{X}^{T}\right)^{-1}\bm{\xi}^{*}\right)\ ,
𝒑\displaystyle\bm{p}\ =softmax⁡(β​𝑿T​𝝃),\displaystyle=\ \mathrm{softmax}(\beta\bm{X}^{T}\bm{\xi})\ , (512)
∂2lse⁡(β,𝑿T​𝝃)∂𝝃2\displaystyle\frac{\partial^{2}\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})}{\partial\bm{\xi}^{2}}\ =β​𝑿​(diag⁡(𝒑)−𝒑​𝒑T)​𝑿T\displaystyle=\ \beta\ \bm{X}\left(\mathrm{diag}(\bm{p})-\bm{p}\bm{p}^{T}\right)\bm{X}^{T} (513)
(lse⁡(β,𝑿T​𝝃))∗⁣∗\displaystyle\left(\mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\right)^{**}\ =lse⁡(β,𝑿T​𝝃).\displaystyle=\ \mathrm{lse}(\beta,\bm{X}^{T}\bm{\xi})\ . (514)
Definition A6 (Lambert Function).

The Lambert WW function (Olver et al., 2010, (4.13)) is the inverse function of

Equation 515
f⁡(y)\displaystyle f(y)\ =y​ey.\displaystyle=\ ye^{y}\ . (515)

The Lambert WW function has an upper branch W0W_{0} for −1⩽y-1\leqslant y and a lower branch W−1W_{-1} for y⩽−1y\leqslant-1. We use WW if a formula holds for both branches. We have

Equation 516
W⁡(x)\displaystyle W(x)\ =y⇒y​ey=x.\displaystyle=\ y\ \Rightarrow ye^{y}\ =\ x\ . (516)
Lemma A30.

Identities for the Lambert WW function are

Equations 517–529
W⁡(x)​eW⁡(x)\displaystyle W(x)\ e^{W(x)}\ =x,\displaystyle=\ x\ , (517)
W⁡(x​ex)\displaystyle W(xe^{x})\ =x,\displaystyle=\ x\ , (518)
eW⁡(x)\displaystyle e^{W(x)}\ =xW⁡(x),\displaystyle=\ \frac{x}{W(x)}\ , (519)
e−W⁡(x)\displaystyle e^{-W(x)}\ =W⁡(x)x,\displaystyle=\ \frac{W(x)}{x}\ , (520)
en​W​(x)\displaystyle e^{nW(x)}\ =(xW⁡(x))n,\displaystyle=\ \left(\frac{x}{W(x)}\right)^{n}\ , (521)
W0​(x​ln⁡x)\displaystyle W_{0}\left(x\ \ln x\right)\ =ln⁡xfor ​x≥1e,\displaystyle=\ \ln x\quad\text{for }\ x\ \geq\ \frac{1}{e}\ , (522)
W−1​(x​ln⁡x)\displaystyle W_{-1}\left(x\ \ln x\right)\ =ln⁡xfor ​x⩽1e,\displaystyle=\ \ln x\quad\text{for }\ x\ \leqslant\ \frac{1}{e}\ , (523)
W⁡(x)\displaystyle W(x)\ =ln⁡xW⁡(x)for ​x≥−1e,\displaystyle=\ \ln\frac{x}{W(x)}\quad\text{for }\ x\ \geq\ -\ \frac{1}{e}\ , (524)
W⁡(n​xnW​(x)n−1)\displaystyle W\left(\frac{n\ x^{n}}{W\left(x\right)^{n-1}}\right)\ =nW(x)for n,x> 0,\displaystyle=\ n\ W(x)\quad\text{for }\ n,x\ >\ 0\ , (525)
W⁡(x)+W⁡(y)\displaystyle W(x)\ +\ W(y)\ =W(xy(1W⁡(x)+1W⁡(y)))for x,y> 0,\displaystyle=\ W\left(x\ y\ \left(\frac{1}{W(x)}\ +\ \frac{1}{W(y)}\right)\right)\quad\text{for }\ x,y\ >\ 0\ , (526)
W0​(−ln⁡xx)\displaystyle W_{0}\left(-\ \frac{\ln x}{x}\right)\ =−ln⁡xfor ​ 0<x⩽e,\displaystyle=\ -\ \ln x\quad\text{for }\ 0\ <\ x\ \leqslant\ e\ , (527)
W−1​(−ln⁡xx)\displaystyle W_{-1}\left(-\ \frac{\ln x}{x}\right)\ =−ln⁡xfor ​x>e,\displaystyle=\ -\ \ln x\quad\text{for }\ x\ >\ e\ , (528)
e−W⁡(−ln⁡x)\displaystyle e^{-\ W(-\ \ln x)}\ =W⁡(−ln⁡x)−ln⁡xfor ​x≠ 1.\displaystyle=\ \frac{W(-\ \ln x)}{-\ \ln x}\quad\text{for }\ x\ \neq\ 1\ . (529)
Lemma A31.
Equations 530–538
W⁡(0)\displaystyle W(0)\ = 0,\displaystyle=\ 0\ , (530)
W⁡(e)\displaystyle W(e)\ = 1,\displaystyle=\ 1\ , (531)
W⁡(−1e)\displaystyle W\left(-\frac{1}{e}\right)\ =−1,\displaystyle=\ -1\ , (532)
W⁡(e1+e)\displaystyle W\left(e^{1+e}\right)\ =e,\displaystyle=\ e\ , (533)
W⁡(2​ln⁡2)\displaystyle W\left(2\ln 2\right)\ =ln⁡2,\displaystyle=\ \ln 2\ , (534)
W⁡(1)\displaystyle W(1)\ =Ω,\displaystyle=\ \Omega\ , (535)
W⁡(1)\displaystyle W(1)\ =e−W⁡(1)=ln⁡(1W⁡(1))=−ln⁡W⁡(1),\displaystyle=\ e^{-W(1)}\ =\ \ln\left(\frac{1}{W(1)}\right)\ =\ -\ \ln W(1)\ , (536)
W⁡(−π2)\displaystyle W\left(-\frac{\pi}{2}\right)\ =i​π2,\displaystyle=\ \frac{i\pi}{2}\ , (537)
W⁡(−1)\displaystyle W(-1)\ ≈−0.31813+1.33723​i,\displaystyle\approx\ -0.31813+1.33723i\ , (538)

where the Omega constant Ω\Omega is

Equation 539
Ω\displaystyle\Omega\ =(∫−∞∞d​t(et−t)2+π2)−1− 1≈ 0.56714329.\displaystyle=\ \left(\int_{-\infty}^{\infty}\frac{\mathrm{d}t}{\left(e^{t}\ -\ t\right)^{2}\ +\ \pi^{2}}\right)^{-1}\ -\ 1\ \approx\ 0.56714329\ . (539)
Lemma A32 (Mean Value Theorem).

Let U⊂ℝnU\subset\mathbb{R}^{n} be open, f:U→ℝmf:U\to\mathbb{R}^{m} continuously differentiable, and 𝐱∈U\bm{x}\in U as well as 𝐡∈ℝn\bm{h}\in\mathbb{R}^{n} vectors such that the line segment 𝐱+t​𝐡\bm{x}+t\bm{h} for 0⩽t⩽10\leqslant t\leqslant 1 is in UU. Then the following holds:

Equation 540
f⁡(𝒙+𝒉)−f⁡(𝒙)\displaystyle f(\bm{x}\ +\ \bm{h})\ -\ f(\bm{x})\ =(∫01J⁡(𝒙+t​𝒉)​𝑑t)​𝒉,\displaystyle=\ \left(\int_{0}^{1}J(\bm{x}\ +\ t\ \bm{h})\ \mathrm{d}t\right)\ \bm{h}\ , (540)

where JJ is the Jacobian of ff and the integral of the matrix is component-wise.

Equations 541–542
gi​(t)\displaystyle g_{i}(t)\ =fi​(𝒙+t​𝒉),\displaystyle=\ f_{i}(\bm{x}\ +\ t\ \bm{h})\ , (541)
fi​(𝒙+𝒉)−fi​(𝒙)=gi​(1)−gi​(0)=∫01g′​(t)​𝑑t\displaystyle f_{i}(\bm{x}\ +\ \bm{h})\ -\ f_{i}(\bm{x})\ =\ g_{i}(1)\ -\ g_{i}(0)\ =\ \int_{0}^{1}g^{\prime}(t)\ \mathrm{d}t (542)
∫01(∑j=1n∂fi∂xj​(𝒙+t​𝒉)​hj)​𝑑t=∑j=1n(∫01∂fi∂xj​(𝒙+t​𝒉)​𝑑t)​hj.\displaystyle\int_{0}^{1}\left(\sum_{j=1}^{n}\frac{\partial f_{i}}{\partial x_{j}}(\bm{x}\ +\ t\ \bm{h})\ h_{j}\right)\ \mathrm{d}t\ =\ \sum_{j=1}^{n}\left(\int_{0}^{1}\frac{\partial f_{i}}{\partial x_{j}}(\bm{x}\ +\ t\ \bm{h})\ \mathrm{d}t\right)\ h_{j}\ .

A.3 Modern Hopfield Networks: Binary States (Krotov and Hopfield)

The appendix relates the proposed continuous network to earlier binary modern Hopfield networks. Their interaction functions, coordinatewise dynamics and probabilistic capacity results are distinct from the continuous update analyzed above.

A.3.1 Modern Hopfield Networks: Introduction

The background separates external memory architectures from the development of high-capacity binary associative memories.

A.3.1.1 Additional Memory and Attention for Neural Networks.

Neural Turing machines explicitly read and write an external memory; memory networks provide input, generalization, output and response components. End-to-end memory networks replace a discrete memory selection with differentiable softmax. The review traces connections to recurrent associative memories, key/value attention, transformers and BERT. Similar mechanisms do not imply identical energies, learning rules or retrieval guarantees.

A.3.1.2 Modern Hopfield networks: Overview.

Capacity depends on the update rule, pattern distribution, required stability and allowable error. The appendix contrasts classical binary capacities with polynomial interactions and exponential interactions. Exponentially many energy minima are not the same as exponentially many reliably retrievable intended memories. The reported earlier exponential-capacity result uses random binary patterns and a probability statement tied to the permitted corruption radius.

A.3.2 Energy and Update Rule for Binary Modern Hopfield Networks

For binary states, an interaction function is applied to each pattern–state inner product and summed into the energy. A coordinate update compares the energies of setting that coordinate to +1 or −1. Theorem A10 quotes the exponential-interaction result: random-pattern storage rate must stay below a corruption-dependent large-deviation threshold, and one update of each component recovers the target with probability tending to one as dimension grows. The proof is delegated to Demircigil et al. The subsequent mean-value argument expresses the coordinate decision through a softmax-weighted pattern sum, motivating the continuous construction without equating the two dynamics.

Equations 543–545
𝑿\displaystyle\bm{X}\ =(𝒙1,…,𝒙N).\displaystyle=\ \left(\bm{x}_{1},\ldots,\bm{x}_{N}\right)\ . (543)
E\displaystyle\mathrm{E}\ =−∑i=1NF(𝝃T𝒙i)\displaystyle=\ -\ \sum_{i=1}^{N}F\left(\bm{\xi}^{T}\bm{x}_{i}\right) (544)
Tj​(𝝃)\displaystyle T_{j}(\bm{\xi})\ :=sgn[∑i=1N(F⁡(xi​j+∑l≠jxi​l​ξl)−F⁡(−xi​j+∑l≠jxi​l​ξl))]\displaystyle:=\ \mathop{\mathrm{sgn}\,}\Bigl[\sum\limits_{i=1}^{N}\bigl(F\bigl(x_{ij}\ +\ \sum\limits_{l\neq j}x_{il}\ \xi_{l}\bigr)\ -\ F\bigl(-\ x_{ij}\ +\ \sum\limits_{l\neq j}x_{il}\ \xi_{l}\bigr)\bigr)\Bigr] (545)
Theorem A10 (Storage Capacity for Binary Modern Hopfield Nets (Demircigil et al. 2017)).

Consider the generalized Hopfield model with the dynamics described in Eq. (545) and interaction function FF given by F⁡(x)=exF(x)=e^{x}. For a fixed 0<α<ln⁡(2)/20<\alpha<\ln(2)/2 let N=exp⁡(α​d)+1N=\exp\left(\alpha d\right)+1 and let 𝐱1,…,𝐱N\bm{x}_{1},\ldots,\bm{x}_{N} be NN patterns chosen uniformly at random from {−1,+1}d\{-1,+1\}^{d}. Moreover fix ϱ∈[0,1/2)\varrho\in[0,1/2). For any ii and any 𝐱~i\widetilde{\bm{x}}_{i} taken uniformly at random from the Hamming sphere with radius ϱ​d\varrho d centered in 𝐱i\bm{x}_{i}, 𝒮⁡(𝐱i,ϱ​d)\mathcal{S}(\bm{x}_{i},\varrho d), where ϱ​d\varrho d is assumed to be an integer, it holds that

Equation
Pr(∃i∃j:Tj(𝒙~i)≠xi​j)→ 0,\displaystyle\mathbf{\mathrm{Pr}}\left(\exists i\;\exists j:\ T_{j}\left(\widetilde{\bm{x}}_{i}\right)\ \neq\ x_{ij}\right)\ \rightarrow\ 0\ ,

if α\alpha is chosen in dependence of ϱ\varrho such that

Equation
α<I⁡(1−2​ϱ)2\displaystyle\alpha\ <\ \frac{I(1-2\varrho)}{2}

with

Equation
I:a↦12​((1+a)​ln⁡(1+a)+(1−a)​ln⁡(1−a)).\displaystyle I:\ a\ \mapsto\ \frac{1}{2}\left((1+a)\ln(1+a)\ +\ (1-a)\ln(1-a)\right)\ .
Equations 546–547
Pr(∃i∃j:Tj(𝒙~i)≠xi​j)→ 0\displaystyle\mathbf{\mathrm{Pr}}\left(\exists i\;\exists j:\ T_{j}\left(\widetilde{\bm{x}}_{i}\right)\ \neq\ x_{ij}\right)\ \rightarrow\ 0
Pr(∃i∃j:Tj(𝒙i)≠xi​j)→ 0\displaystyle\mathbf{\mathrm{Pr}}\left(\exists i\;\exists j:\ T_{j}\left(\bm{x}_{i}\right)\ \neq\ x_{ij}\right)\ \rightarrow\ 0
Tj​(𝝃)\displaystyle T_{j}(\bm{\xi})\ =sgn[−E⁡(ξj=1)+E⁡(ξj=−1)]=sgn[exp⁡(lse⁡(ξj=1))−exp⁡(lse⁡(ξj=−1))]\displaystyle=\ \mathop{\mathrm{sgn}\,}\Bigl[-\ \mathrm{E}(\xi_{j}=1)\ +\ \mathrm{E}(\xi_{j}=-1)\Bigr]\ =\ \mathop{\mathrm{sgn}\,}\Bigl[\exp(\mathrm{lse}(\xi_{j}=1))\ -\ \exp(\mathrm{lse}(\xi_{j}=-1))\Bigr]
=sgn[−(2​𝒆j)T​∇𝝃E​(ξj=v)]=sgn[exp⁡(lse⁡(ξj=v))​(2​𝒆j)T​lse⁡(ξj=v)∂𝝃]\displaystyle=\ \mathop{\mathrm{sgn}\,}\Bigl[-\ (2\bm{e}_{j})^{T}\nabla_{\bm{\xi}}\mathrm{E}(\xi_{j}=v)\Bigr]\ =\ \mathop{\mathrm{sgn}\,}\Bigl[\exp(\mathrm{lse}(\xi_{j}=v))\ (2\bm{e}_{j})^{T}\frac{\mathrm{lse}(\xi_{j}=v)}{\partial\bm{\xi}}\Bigr] (546)
=sgn[exp⁡(lse⁡(ξj=1))​(2​𝒆j)T​𝑿​softmax​(𝑿T​𝝃​(ξj=v))]\displaystyle=\ \mathop{\mathrm{sgn}\,}\Bigl[\exp(\mathrm{lse}(\xi_{j}=1))\ (2\bm{e}_{j})^{T}\bm{X}\mathrm{softmax}(\bm{X}^{T}\bm{\xi}(\xi_{j}=v))\Bigr]
=sgn[[𝑿​softmax​(𝑿T​𝝃​(ξj=v))]j]=sgn[[𝑿​𝒑​(ξj=v)]j],\displaystyle=\ \mathop{\mathrm{sgn}\,}\Bigl[[\bm{X}\mathrm{softmax}(\bm{X}^{T}\bm{\xi}(\xi_{j}=v))]_{j}\Bigr]\ =\ \mathop{\mathrm{sgn}\,}\Bigl[[\bm{X}\bm{p}(\xi_{j}=v)]_{j}\Bigr]\ ,
𝒑\displaystyle\bm{p}\ =softmax⁡(𝑿T​𝝃).\displaystyle=\ \mathrm{softmax}(\bm{X}^{T}\bm{\xi})\ . (547)

A.4 Hopfield Update Rule is Attention of The Transformer

Map queries and keys into the same associative space, apply one Hopfield update with β=1/√dₖ, and project the result into value space. Equation548 becomes the usual scaled dot-product attention formula. Here the values factor through the projected keys as K Wᵥ; the correspondence identifies this retrieval computation, not every residual, normalization or feedforward component of a transformer. Repeated energy-descent guarantees concern fixed memories and the specified associative state update.

Equation 548
softmax⁡(1/dk​𝑸​𝑲T)​𝑽=softmax⁡(β​𝑹​𝑾𝑸​𝑾𝑲T​𝒀T)​𝒀​𝑾𝑲​𝑾𝑽.\displaystyle\mathrm{softmax}\left(1/\sqrt{d_{k}}\ \bm{Q}\ \bm{K}^{T}\right)\ \bm{V}\ =\mathrm{softmax}\left(\beta\ \bm{R}\bm{W}_{\bm{Q}}\ \bm{W}_{\bm{K}}^{T}\bm{Y}^{T}\right)\ \bm{Y}\bm{W}_{\bm{K}}\bm{W}_{\bm{V}}\ . (548)
Complete original Figure A.2: We generalized
the energy
of binary modern Hopfield networks for allowing continuous states
while keeping fast convergence and
storage capacity properties.
We defined for the new energy also a new update
rule that minimizes the energy.
The new update rule is the attention mechanism of the transformer.
Formulae are modified to express softmax \mathrm{softmax} as row vector as for transformers.
" = = "-sign means "keeps the properties".
Figure A.2: We generalized the energy of binary modern Hopfield networks for allowing continuous states while keeping fast convergence and storage capacity properties. We defined for the new energy also a new update rule that minimizes the energy. The new update rule is the attention mechanism of the transformer. Formulae are modified to express softmax\mathrm{softmax} as row vector as for transformers. "=="-sign means "keeps the properties".

A.5 Experiments

Additional experiments describe attention-head behavior, massive and conventional multiple instance learning, small tabular datasets, and molecular classification. Their datasets, validation procedures and metrics differ.

A.5.1 Experiment 1: Attention in Transformers described by Hopfield dynamics

The attention experiments classify heads by how many patterns account for90% of their probability mass, track that statistic through training, and test input-independent Gaussian averaging replacements.

A.5.1.1 Analysis of operating modes of the heads of a pre-trained BERT model.

For pretrained bert-base-cased, Figure A.3 displays a distribution of the smallest count k accumulating90% of attention mass for every head. Heads are sorted within each layer, so columns do not track a fixed head identity across layers. Large-averaging heads are common in lower layers; small metastable states often appear in middle layers; medium states occur toward the top. These are descriptive classes based on attention concentration, not a direct measurement proving convergence to a unique energy minimum.

Complete original Figure A.3: Analysis of operating modes of the heads of a pre-trained BERT model.
For each head in each layer, the distribution of the minimal number k k of patterns required to sum up the softmax \mathrm{softmax} values to 0.90 0.90 is displayed as a violin plot in a panel. k k indicates the size of a metastable state.
The bold number in the center of each panel gives the median k ¯ \bar{k} of the distribution.
The heads in each layer are sorted according to k ¯ \bar{k} .
Attention heads belong to the class they mainly operate in. Class (IV) in blue: Small metastable state or fixed point close to a single pattern, which
is abundant in the middle layers (6, 7, and 8). Class (II) in orange: Large metastable state, which is
prominent in middle layers (3, 4, and 5). Class (I) in red: Very large metastable state or global fixed point,
which is predominant in the first layer.
These heads can potentially be replaced by averaging operations. Class (III) in green: Medium metastable state,
which is frequently observed in higher layers.
We hypothesize that these heads are used to collect information required to
perform the respective task.
These heads should be the main target to improve transformer and BERT models.
Figure A.3: Analysis of operating modes of the heads of a pre-trained BERT model. For each head in each layer, the distribution of the minimal number kk of patterns required to sum up the softmax\mathrm{softmax} values to 0.900.90 is displayed as a violin plot in a panel. kk indicates the size of a metastable state. The bold number in the center of each panel gives the median k¯\bar{k} of the distribution. The heads in each layer are sorted according to k¯\bar{k}. Attention heads belong to the class they mainly operate in. Class (IV) in blue: Small metastable state or fixed point close to a single pattern, which is abundant in the middle layers (6, 7, and 8). Class (II) in orange: Large metastable state, which is prominent in middle layers (3, 4, and 5). Class (I) in red: Very large metastable state or global fixed point, which is predominant in the first layer. These heads can potentially be replaced by averaging operations. Class (III) in green: Medium metastable state, which is frequently observed in higher layers. We hypothesize that these heads are used to collect information required to perform the respective task. These heads should be the main target to improve transformer and BERT models.

A.5.1.2 Experimental Setup.

The training study uses a reduced BERT-small model:12layers,4heads, hidden size256, embedding size128, sequence length128 and batch size128. It trains1.45million updates, reported as roughly four days on one V100. The data approximate the original Wikipedia/BookCorpus protocol, but use an uncased BookCorpus snapshot because of access restrictions. These details limit direct comparisons with full BERT training.

A.5.1.3 Hopfield Operating Classes of Transformer and BERT Models.

Figure A.4 tracks the distribution of90%-mass counts during BERT-small training and groups the final counts into four classes. Thresholds are divided by four to account for the reduction from512 to128tokens. Some heads commit early to small metastable-state behavior. This rescaled classification is a heuristic description of attention concentration.

A.5.1.4 Learning Dynamics of Transformer and BERT Models.

During the first20,000steps, several middle-layer heads change toward concentrated attention around9,000–10,000steps, near a second loss decrease. Figure A.5 also tracks the Frobenius norm of the softmax Jacobian. The reported later stability of these heads’ attention does not on its own prove that the mode transition caused the loss change or that every parameter gradient vanishes: the Jacobian is one factor in the full derivative.

Complete original Figure A.4: Left : Ridge plots of the distribution of counts k k over time for BERT-small Right : Violin plot of counts k k after 1 , 450000 1,450000 steps, divided into the four classes from the main paper. The thresholds were adapted to the shorter sequence length.
Figure A.4: Left: Ridge plots of the distribution of counts kk over time for BERT-small Right: Violin plot of counts kk after 1,4500001,450000 steps, divided into the four classes from the main paper. The thresholds were adapted to the shorter sequence length.
Complete original Figure A.5: (a) : change of count density during training is depicted for the first 20,000 20,000 steps. (b) : the corresponding distribution of the Frobenius norm of the Jacobian of the softmax \mathrm{softmax} function
is depicted.
The gradients with respect to the weights are determined by the Jacobian J \mathrm{J} defined in Eq. ( 59 )
as can be seen in Eq. ( 418 ), Eq. ( 429 ), and Eq. ( 435 ).
Figure A.5: (a): change of count density during training is depicted for the first 20,00020,000 steps. (b): the corresponding distribution of the Frobenius norm of the Jacobian of the softmax\mathrm{softmax} function is depicted. The gradients with respect to the weights are determined by the Jacobian J\mathrm{J} defined in Eq. (59) as can be seen in Eq. (418), Eq. (429), and Eq. (435).

A.5.1.5 Attention Heads Replaced by Gaussian Averaging Layers.

Gaussian averaging heads replace query-dependent scores with normalized discrete Gaussian kernels over token positions. Each position has a learned location and scale. Initialization samples locations in[−1,1] and scales in[.75,1.25], or centers each kernel at its corresponding support point; Figure A.6 shows the latter option. This uses2N location/scale parameters per head instead of the2dₖdᵧ query/key projection parameters being compared. The printed BERT-base arithmetic gives95.5×, although the displayed dimensions yield96×. This count excludes other attention components and trades input-dependent expressiveness for a smaller positional mechanism; it is not a whole-model parameter or runtime ratio.

Equations
sj=(j−1)−0.5​(N−1)0.5​(N−1).\displaystyle s_{j}=\frac{(j-1)-0.5~(N-1)}{0.5~(N-1)}.
[A]i,j=1zi​exp⁡{−12​(sj−μiσi)2},\displaystyle[A]_{i,j}=\frac{1}{z_{i}}\exp\left\{-\frac{1}{2}\big(\frac{s_{j}-\mu_{i}}{\sigma_{i}}\big)^{2}\right\},
zi=∑j=1Nexp⁡{−12​(sj−μiσi)2}.\displaystyle z_{i}=\sum_{j=1}^{N}\exp\left\{-\frac{1}{2}\big(\frac{s_{j}-\mu_{i}}{\sigma_{i}}\big)^{2}\right\}.
Complete original Figure A.6: Attentions of a Gaussian averaging head at initialization for sequence length N = 128 N=128 . Every line depicts one Gaussian kernel. Here, the location parameters are initialized with the value of the supporting points μ i = s i \mu_{i}=s_{i} .
Figure A.6: Attentions of a Gaussian averaging head at initialization for sequence length N=128N=128. Every line depicts one Gaussian kernel. Here, the location parameters are initialized with the value of the supporting points μi=si\mu_{i}=s_{i}.

A.5.2 Experiment 2: Multiple Instance Learning Datasets.

Multiple instance learning predicts a bag-level label from many instances. The experiments distinguish large immune-sequence repertoires from smaller image and histopathology bags.

A.5.2.1 Immune Repertoire Classification.

DeepRC maps immune amino-acid sequences through a position-aware one-dimensional convolutional network and uses learned static queries to pool class-indicative instances. The tests include simulated, LSTM-generated and real repertoires with implanted motifs, plus a real CMV-status dataset; several categories contain about300,000sequences per repertoire. The complete table retains every motif frequency and comparison. On real CMV, DeepRC reports AUC .832±.022 versus .825±.022 for the MinMax SVM, with errors across five folds. The difference is small relative to those deviations and is not presented here as a significance test. Implanted-signal results and biological generalization answer different questions; the capacity theorem alone does not guarantee either empirical result.

Real-world Real-world data with implanted signals LSTM-generated data Simulated
CMV OM 1% OM 0.1% MM 1% MM 0.1% 10% 1% 0.5% 0.1% 0.05% avg.
DeepRC 0.832 ±\pm 0.022 1.00 ±\pm 0.00 0.98±\pm 0.01 1.00±\pm 0.00 0.94±\pm0.01 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 0.846±\pm 0.223
SVM (MM) 0.825 ±\pm 0.022 1.00 ±\pm 0.00 0.58±\pm 0.02 1.00±\pm 0.00 0.53±\pm0.02 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 0.99±\pm 0.01 0.827±\pm 0.210
SVM (J) 0.546 ±\pm 0.021 0.99 ±\pm 0.00 0.53±\pm 0.02 1.00±\pm 0.00 0.57±\pm0.02 0.98±\pm 0.04 1.00±\pm 0.00 1.00±\pm 0.00 0.90±\pm 0.04 0.77±\pm 0.07 0.550±\pm 0.080
KNN (MM) 0.679 ±\pm 0.076 0.74 ±\pm 0.24 0.49±\pm 0.03 0.67±\pm 0.18 0.50±\pm0.02 0.70±\pm 0.27 0.72±\pm 0.26 0.73±\pm 0.26 0.54±\pm 0.16 0.52±\pm 0.15 0.634±\pm 0.129
KNN (J) 0.534 ±\pm 0.039 0.65 ±\pm 0.16 0.48±\pm 0.03 0.70±\pm 0.20 0.51±\pm0.03 0.70±\pm 0.29 0.61±\pm 0.24 0.52±\pm 0.16 0.55±\pm 0.19 0.54±\pm 0.19 0.501±\pm 0.007
Log. regr. 0.607 ±\pm 0.058 1.00 ±\pm 0.00 0.54±\pm 0.04 0.99±\pm 0.00 0.51±\pm0.04 1.00±\pm 0.00 1.00±\pm 0.00 0.93±\pm 0.15 0.60±\pm 0.19 0.43±\pm 0.16 0.826±\pm 0.211
Burden test 0.699 ±\pm 0.041 1.00 ±\pm 0.00 0.64±\pm 0.05 1.00±\pm 0.00 0.89±\pm0.02 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 0.79±\pm 0.28 0.549±\pm 0.074
Log. MIL (KMER) 0.582 ±\pm 0.065 0.54 ±\pm 0.07 0.51±\pm 0.03 0.99±\pm 0.00 0.62±\pm0.15 1.00±\pm 0.00 0.72±\pm 0.11 0.64±\pm 0.14 0.57±\pm 0.15 0.53±\pm 0.13 0.665±\pm 0.224
Log. MIL (TCRβ) 0.515 ±\pm 0.073 0.50 ±\pm 0.03 0.50±\pm 0.02 0.99±\pm 0.00 0.78±\pm0.03 0.54±\pm 0.09 0.57±\pm 0.16 0.47±\pm 0.09 0.51±\pm 0.07 0.50±\pm 0.12 0.501±\pm 0.016
Known motif b. – 1.00 ±\pm 0.00 0.70±\pm 0.03 0.99±\pm 0.00 0.62±\pm0.04 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 1.00±\pm 0.00 0.890±\pm 0.168
Known motif c. – 0.92 ±\pm 0.00 0.56±\pm 0.03 0.65±\pm 0.03 0.52±\pm0.03 1.00±\pm 0.00 1.00±\pm 0.00 0.99±\pm 0.01 0.72±\pm 0.09 0.63±\pm 0.09 0.738±\pm 0.202
Table A.1: Results immune repertoire classification across all datasets. Results are given in terms of AUC of the competing methods on all datasets. The reported errors are standard deviations across 55 cross-validation (CV) folds (except for the column “Simulated”). Real-world CMV: Average performance over 55 CV folds on the cytomegalovirus (CMV) dataset Emerson et al. (2017). Real-world data with implanted signals: Average performance over 55 CV folds for each of the four datasets. A signal was implanted with a frequency (=wittness rate) of 1%1\% or 0.1%0.1\%. Either a single motif (“OM”) or multiple motifs (“MM”) were implanted. LSTM-generated data: Average performance over 55 CV folds for each of the 55 datasets. In each dataset, a signal was implanted with a frequency of 10%10\%, 1%1\%, 0.5%0.5\%, 0.1%0.1\%, and 0.05%0.05\%, respectively. Simulated: Here we report the mean over 18 simulated datasets with implanted signals and varying difficulties. The error reported is the standard deviation of the AUC values across the 18 datasets.

A.5.2.2 Multiple Instance Learning Benchmark Datasets.

Elephant, Fox and Tiger each have200image bags, represented by region descriptors; UCSB has58histopathology bags and2,002patch instances. Embedded instances enter HopfieldPooling with a trainable static query, followed by a classifier. A manual validation-set search varies widths, head counts, associative dimensions, β and regularization. Train160epochs with AdamW and exponential learning-rate decay; report average AUC from five repetitions of nested ten-fold cross-validation. Bag dropout addresses the severe small-data overfitting risk. Table A.2 retains the full search space.

parameter values
learning rates {10−3\{10^{-3}, 10−5}10^{-5}\}
learning rate decay (γ) {0.98,0.96,0.94}\{0.98,0.96,0.94\}
embedding layers {1,2,3}\{1,2,3\}
layer widths {32,64,256,1024,2048}\{32,64,256,1024,2048\}
number of heads {8,12,16,32}\{8,12,16,32\}
head dimensions {16,32,64}\{16,32,64\}
scaling factors {0.1,1.0,10.0}\{0.1,1.0,10.0\}
hidden dimensions {32,64,128}\{32,64,128\}
bag dropout {0.0,0.75}\{0.0,0.75\}
Table A.2: Hyperparameter search-space of a manual hyperparameter selection on the respective validation sets of the Elephant, Fox, Tiger and UCSB breast cancer datasets.

A.5.3 Experiment 3: Classification on Small UCI Benchmark Datasets

A learned associative memory supplies prototypes for classification on75small UCI datasets. The protocol uses held-out validation and test sets, rather than the multiple-instance cross-validation scheme.

A.5.3.1 Motivation.

An optional self-normalizing embedding feeds a Hopfield layer with learned stored patterns; its retrieved output feeds the classifier. The motivation is to learn class-indicative prototypes for datasets with fewer than1,000samples, where standard neural networks often struggle. The number of patterns and associative dimension are hyperparameters rather than fixed theoretical optima.

A.5.3.2 Methods compared.

The comparison spans17families, including SVMs, forests, boosting, regression and discriminant methods, bagging, nearest neighbors, trees, ensembles, Bayesian methods and multiple neural-network variants. These categories encompass the25methods used in the final rank comparison; a family label is not a separate uniform implementation.

A.5.3.3 Experimental design and implementation details.

For each dataset, search the architecture and hyperparameters in Table A.3. Train up to100epochs with cross entropy, batch size4, and SGD without momentum, weight decay or dropout. Early stopping selects the model using validation accuracy averaged over16consecutive epochs; the selected model is evaluated on a separate test set. Embedding depth is0/1/7, width32/128/1024, and the number of stored patterns is one or eight times the class count. Validation selection and historical baseline scores should be considered when interpreting aggregate ranks.

parameter values
learning rates {0.05}\{0.05\}
embedding layers {0,1,7}\{0,1,7\}
hidden units {32,128,1024}\{32,128,1024\}
heads {1,32}\{1,32\}
β\beta {1.0,0.1,0.001}\{1.0,0.1,0.001\}
# stored patterns {1,8}⋅n​_​c​l​a​s​s​e​s\{1,8\}\cdot n\_classes
Table A.3: Hyperparameter search-space for grid-search on small UCI benchmark datasets. All models were trained for 100100 epochs using stochastic gradient descent with early stopping based on the validation set accuracy and a minibatch size of 44 samples. The number of stored patterns is depending on the number of target classes of the individual tasks.

A.5.3.4 Results.

Rank each of25methods by test accuracy within each dataset, then average those ranks across the75datasets. The baseline scores come from the cited earlier study. A rank difference is not an average percentage-point accuracy gain, and this aggregate does not establish superiority for every dataset.

A.5.4 Experiment 4: Drug Design Benchmark Datasets

The molecular experiments retrieve information from a reference set of training compounds and labels. This is a different use of the layer from a learned static prototype matrix.

A.5.4.1 Experimental design and implementation details.

Evaluate HIV, BACE, BBBP and SIDER from MoleculeNet. The training inputs act as stored keys, labels as retrieved values, and the current compound as query. Separate validation sets select β, head count, normalization, bias, dropout and other choices in Table A.4. The source describes selecting the highest-validation-AUC model on five random splits; the result table describes50random splits, so that discrepancy is retained rather than silently resolved. Applicable learned models train four epochs with Adam and batch size1. Test labels must remain outside the stored training reference set.

parameter values
beta {0.0001,0.001,0.01,0.1,0.2,0.3}\{0.0001,0.001,0.01,0.1,0.2,0.3\}
learning rates {0.0002}\{0.0002\}
heads {1,32,128,512}\{1,32,128,512\}
dropout {0.0,0.1,0.2}\{0.0,0.1,0.2\}
state-pattern bias {0.0,−0.1,−0.125,0.15,−0.2}\{0.0,-0.1,-0.125,0.15,-0.2\}
association-activation {None, LeakyReLU }
state- and stored-pattern static {False, True}
normalize state- and stored-pattern {False, True}
normalize association projection {False, True}
learnable stored-pattern {False, True}
Table A.4: Hyperparameter search-space for grid-search on HIV, BACE, BBBP and SIDER. All models were trained if applicable for 44 epochs using Adam and a batch size of 11 sample.

A.5.4.2 Results.

Table A.5 compares against SVM, XGBoost, forests, dense neural networks and several graph models, with historical baseline results taken from Jiang et al. The Hopfield layer reports ROC-AUC .902±.023 on BACE and .672±.019 on SIDER, the best table entries for those datasets. It does not win every dataset. The comparison combines reported baselines with the stated validation protocol, and its split-count discrepancy prevents treating every cell as a fully controlled rerun.

Table A.5: Results on drug design benchmark datasets. Predictive performance (ROCAUC) on test set as reported by Jiang et al. (2020) for 50 random splits
Model HIV BACE BBBP SIDER
SVM 0.822±0.0200.822\pm 0.020 0.893±0.0200.893\pm 0.020 0.919±0.0280.919\pm 0.028 0.630±0.0210.630\pm 0.021
XGBoost 0.816±0.0200.816\pm 0.020 0.889±0.0210.889\pm 0.021 0.926±0.026\mathbf{0.926\pm 0.026} 0.642±0.0200.642\pm 0.020
RF 0.820±0.0160.820\pm 0.016 0.890±0.0220.890\pm 0.022 0.927±0.025\mathbf{0.927\pm 0.025} 0.646±0.0220.646\pm 0.022
GCN 0.834±0.025\mathbf{0.834\pm 0.025} 0.898±0.0190.898\pm 0.019 0.903±0.0270.903\pm 0.027 0.634±0.0260.634\pm 0.026
GAT 0.826±0.0300.826\pm 0.030 0.886±0.0230.886\pm 0.023 0.898±0.0330.898\pm 0.033 0.627±0.0240.627\pm 0.024
DNN 0.797±0.0180.797\pm 0.018 0.890±0.0240.890\pm 0.024 0.898±0.0330.898\pm 0.033 0.627±0.0240.627\pm 0.024
MPNN 0.811±0.0310.811\pm 0.031 0.838±0.0270.838\pm 0.027 0.879±0.0370.879\pm 0.037 0.598±0.0310.598\pm 0.031
Attentive FP 0.822±0.0260.822\pm 0.026 0.876±0.0230.876\pm 0.023 0.887±0.0320.887\pm 0.032 0.623±0.0260.623\pm 0.026
Hopfield (ours) 0.815±0.0230.815\pm 0.023 0.902±0.023\mathbf{0.902\pm 0.023} 0.910±0.0260.910\pm 0.026 0.672±0.019\mathbf{0.672\pm 0.019}

A.6 PyTorch Implementation of Hopfield Layers

The authors provide a PyTorch implementation at ml-jku/hopfield-layers. The appendix explains the source-version interface and its relation to the mathematical update; it is not a claim that every present-day release retains exactly the same API.

A.6.1 Introduction

Hopfield layers can associate sets, pool a variable-size bag, or retrieve from prototypes, training examples or a reference memory. The appendix relates the raw queries R and stored patterns Y to associative queries Q and keys K, then projects retrieved states into output values. Permutation-invariant pooling requires an appropriate query/output arrangement; sequence-order tasks may additionally need positional information.

A.6.2 Functionality

The interface exposes six main controls. Query/key projections associate two possibly different input sets. Multiple updates iterate in associative space before the final value projection, with a maximum count or state-change tolerance. β controls concentration together with pattern geometry and affects gradients. Associative dimension affects separation and representational capacity, with larger dimensions also increasing computation. Static patterns are independent of the current input but can be learned through backpropagation. Optional normalization changes pattern norms and centers, which influence averaging and metastable behavior. Figure A.7 preserves the complete flow, including the separate key and value inputs. Theorems for fixed patterns give intuition for these choices but do not prescribe a universally optimal β, dimension or normalization during training.

Equations 549–555
𝑸\displaystyle\bm{Q}\ =𝚵T=𝑹​𝑾Q,\displaystyle=\ \bm{\Xi}^{T}\ =\ \bm{R}\ \bm{W}_{Q}\ , (549)
𝑲\displaystyle\bm{K}\ =𝑿T=𝒀​𝑾K.\displaystyle=\ \bm{X}^{T}\ =\ \bm{Y}\ \bm{W}_{K}\ . (550)
𝝃new\displaystyle\bm{\xi}^{\mathrm{new}}\ =f⁡(𝝃)=𝑿​softmax​(β​𝑿T​𝝃),\displaystyle=\ f(\bm{\xi})\ =\ \bm{X}\ \mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\xi})\ , (551)
𝚵new\displaystyle\bm{\Xi}^{\mathrm{new}}\ =f⁡(𝚵)=𝑿​softmax​(β​𝑿T​𝚵),\displaystyle=\ f(\bm{\Xi})\ =\ \bm{X}\ \mathrm{softmax}(\beta\ \bm{X}^{T}\bm{\Xi})\ , (552)
(𝑸new)T\displaystyle\left(\bm{Q}^{\mathrm{new}}\right)^{T}\ =𝑲T​softmax​(β​𝑲​𝑸T).\displaystyle=\ \bm{K}^{T}\mathrm{softmax}(\beta\ \bm{K}\ \bm{Q}^{T})\ . (553)
𝑸new\displaystyle\bm{Q}^{\mathrm{new}}\ =softmax⁡(1/dk​𝑸​𝑲T)​𝑲,\displaystyle=\ \mathrm{softmax}(1/\sqrt{d_{k}}\ \bm{Q}\ \bm{K}^{T})\ \bm{K}\ , (554)
𝒁\displaystyle\bm{Z}\ =softmax⁡(1/dk​𝑸​𝑲T)​𝑽.\displaystyle=\ \mathrm{softmax}(1/\sqrt{d_{k}}\ \bm{Q}\ \bm{K}^{T})\ \bm{V}\ . (555)
Complete original Figure A.7: A flowchart of the Hopfield layer.
First, the raw state (query) patterns 𝑹 \bm{R} and the raw stored (key) patterns 𝒀 \bm{Y} are optionally normalized (with layer normalization),
projected and optionally normalized (with layer normalization) again.
The default setting is a layer normalization of the input
patterns, and no layer normalization of the projected patterns.
The raw stored patterns 𝒀 \bm{Y} can in principle be also two different input tensors.
Optionally, multiple updates take place in the
projected space of 𝑸 \bm{Q} and 𝑲 \bm{K} .
This update rule is obtained e.g. from the full update
Eq. ( 423 ) or the simplified update
Eq. ( 424 )
in the appendix.
Figure A.7: A flowchart of the Hopfield layer. First, the raw state (query) patterns 𝑹\bm{R} and the raw stored (key) patterns 𝒀\bm{Y} are optionally normalized (with layer normalization), projected and optionally normalized (with layer normalization) again. The default setting is a layer normalization of the input patterns, and no layer normalization of the projected patterns. The raw stored patterns 𝒀\bm{Y} can in principle be also two different input tensors. Optionally, multiple updates take place in the projected space of 𝑸\bm{Q} and 𝑲\bm{K}. This update rule is obtained e.g. from the full update Eq. (423) or the simplified update Eq. (424) in the appendix.

A.6.3 Usage

The module can implement pooling, set association and attention, while encoder/decoder wrappers recover the corresponding PyTorch transformer-style behavior with the extra functionalities disabled. It can also replace recurrent components in some designs, but matching the information flow, sequence order, masks, tensor shapes and recurrent semantics requires an architecture-specific choice. A substitution is an interface capability rather than a guarantee of identical learned behavior.

Concept demonstration

Open image ↗