Binary pattern retrieval with Kuramoto-type oscillators via a least orthogonal lift of three patterns

Given a set of standard binary patterns and a defective pattern, the pattern retrieval task is to ﬁnd the closest pattern to the defective one among these standard patterns. The Hebbian network of Kuramoto oscillators with second-order coupling provides a dynamical model for this task, and the mutual orthogonality in memorised patterns enables us to distinguish these memorised patterns from most others in terms of stability. For the sake of error-free retrieval for general problems lacking orthogonality, a uniﬁed approach was proposed which transforms the problem into a series of subproblems with orthogonality using the orthogonal lift for two patterns. In this work, we propose the least orthogonal lift for three patterns, which evidently reduces the time of solving subproblems and even the dimensions of subproblems. Furthermore, we provide an estimate for the critical strength for stability/instability of binary patterns, which is convenient in practical use. Simulation results are presented to illustrate the eﬀectiveness of the proposed approach.


Introduction
The famous Hopfield model of associative memory [13] provides basic ideas for neural computing and has attracted a lot of interest.Neurone in the network takes discrete value (e.g., −1 or 1) to reflect a binary pattern, and a set of patterns is stored such that when a new pattern is presented, the network responds by producing a stored pattern that most closely resembles the new pattern.This is the basic mechanism for the binary pattern recognition using an associative-memory network, and the memories are reflected by dynamically stable attractors [1,2,9,10].One advantage of this type of model is that it can be implemented using variety of small devices, such as phase-locked loop circuits [9].
As a paradigm for synchronisation of coupled oscillators, Kuramoto model [14] was proposed in the 1970s and has attracted much attention owing to its significance in mathematical physics and various applications in different fields such as neuroscience and engineering [3][4][5]7].Recent findings suggest that the system of coupled Kuramoto-type oscillators with Hebbian learning rule provides a simple model to implement the associative memory for error-free binary pattern retrieval [6,11,12,16,19,20].The purpose of this paper is to continue the earlier study in refs.[16,19] regarding the following system for binary pattern retrieval problems: where ϕ i is a function of time and denotes the phase of i-th oscillator, N is the size of the network and ε > 0 is the strength of the second-order Fourier term.The coupling matrix C = [C ij ], reflecting the so-called Hebbian rule [8], is set as C ij := M k=1 ξ k i ξ k j , where {ξ 1 , ξ 2 , . . ., ξ M } ⊆ {−1, 1} N is the set of so-called memorised binary patterns stored in the network.To the knowledge of the authors, this model was proposed in ref. [18] by Nishikawa, Lai, and Hoppensteadt.Notice that ξ k and −ξ k can be regarded as the same pattern; in fact, the coupling coefficient C ij does not change if ξ k is replaced by −ξ k .
For any binary pattern ξ ∈ {−1, 1} N , there is a unique (up to constant translation) phase-locked state ϕ * (ξ ) = (ϕ * 1 (ξ ), ϕ * 2 (ξ ), . . ., ϕ * N (ξ )) corresponding to the pattern ξ , which is characterised by where i, j = 1, 2, . . ., N and i = j.The coefficient ε can be regarded as an adjustable parameter that controls the stability/instability of equilibria and leads to rich dynamical properties.The symmetry of the matrix C guarantees that (1.1) can be written as a gradient system φ = −∇f (ϕ) with ϕ = (ϕ 1 , ϕ 2 , . . ., ϕ N ) and potential For any initial data, the solution of the network (1.1) will evolve towards an equilibrium which locally minimises the potential [15].Let ξ def be a defective copy of one of the memorised patterns and ϕ(0) be an initial data reflecting ξ def , the solution of (1.1) will converge to a phase-locked state corresponding to a memorised pattern that most closely resembles the defective pattern.For notational convenience, we denote this process or the problem by PR(ξ 1 , ξ 2 , . . ., ξ M ;ξ def ).The overlap measures the closeness of the solution ϕ(t) to the pattern ξ .Due to the global phase shift invariance in (1.1), m(ξ ) is invariant under global rotations.Let {η 1 , η 2 , . . ., η M } be a given set of standard binary patterns, and η def be a defective copy of one of the standard patterns, our task is to recognise the standard pattern closest to the defective one.For the systems solving the above problems, a principal issue concerns the stability/instability of equilibria corresponding to binary patterns.The stability of standard patterns {ξ k } M k=1 and instability of others {−1, 1} N \ {ξ k } M k=1 are desired, so that the standard patterns can be produced in the evolution rather than others.When we simply set the memorised patterns identical to the standard ones, the stability was studied by linearisation in refs.[17,18] where the stability/instability is conditional to the eigenspectrum of Jacobian.For system (1.1), the mutual orthogonality in memorised patterns was incorporated [16,19], and the authors proposed a concept that a binary pattern η is called ε-independently (asymptotically) stable if ϕ * (η) is (asymptotically) stable for any ε > 0. Then the benefits are at last two-fold.First, easy conditions rather than the eigenvalues for stability/instability are presented.Second, the memorised patterns are ε-independently stable, which enables us to distinguish the memorised patterns and most others.
However, it is highly desired to solve problems with standard patterns lacking the orthogonality in practical situations since the orthogonality usually does not hold.For this aim, a unified approach and the corresponding algorithm are proposed in ref. [20], which combines the strategy of subgrouping and orthogonal lifting of two general patterns, where the strategy of orthogonal lifting enables us to exploit the advantages of mutual orthogonality, see Subsection 2.2.A natural question then arises: Can we use the orthogonal lifting for subgroups containing more than two patterns so that the times of retrieval processes can be reduced evidently?
An example in ref. [20,Subsection III.C] shows that the natural extension of the orthogonal lift for three patterns, which quadruples the dimension of patterns, typically fails to identify the appropriate memorised patterns.In this paper, we propose a least orthogonal lift for a group of three patterns, which enables us to transform the general problem into subproblems with three mutually orthogonal memorised patterns in lowest possible dimension.Moreover, we investigate the critical strength of the second-order Fourier term for any set of three mutually orthogonal memorised patterns, which gives a criterion for the set-up of the parameter making the memorised patterns stable and all other patterns unstable.This strategy guarantees error-free retrieval as well, and it reduces the number of subproblems for solving the pattern retrieval problem in comparison with [20].Simulations are presented to illustrate the approach.
Organisation of paper.In Section 2, we review the Kuramoto-type system and present some existing results.In Section 3, we consider the model with three memorised binary patterns, present the least orthogonal lift and estimate the range of the parameter for achieving error-free retrieval.In Section 4, an approach for general standard patterns and simulations are presented.Section 5 is devoted to a brief conclusion.
[16], it is shown that the equality always holds for M = 2 and M = 3.That is, if ξ 1 , ξ 2 and ξ 3 are orthogonal to each other, then The equality can fail for M ≥ 4, see [16, Example 4.1].

An approach to general problems
Let N 1 ∈ N and {ξ 1 , ξ 2 , . . ., ξ M } ⊆ {−1, 1} N 1 be a set of general standard patterns that can be orthogonal or nonorthogonal to each other.Let ξ def ∈ [−1, 1] N 1 be a defective pattern.Here, ξ def can be a non-binary pattern, which represents a grey-scale image.In ref. [20], an approach was developed by subgrouping the standard patterns in pairs which converts the problem to a series of subproblems.In each subproblem, an orthogonal lift is employed to obtain an associated problem with orthogonal memorised patterns.Here, an (orthogonal) lift of a pattern set The procedure for solving the pattern retrieval task is proposed as follows, referred to as procedure A here.
(A1) Subgroup the standard patterns so that each one contains two patterns.A subgroup containing only one pattern is allowed when M is odd.Then we have M 2 subgroups, where x denotes the smallest integer not less than x.
In the above procedure, we need to build subsystems with dimension 2N 1 and carry out the pattern retrieval processes for M − 1 times.In fact, one standard pattern is excluded in each process and therefore the total times should be M − 1.

The theory for three memorised patterns
In this section, we consider the case of three memorised binary patterns.We explore the least orthogonal lift and the critical strength of the second-order term.

Least orthogonal lift
Let ξ 1 , ξ 2 and ξ 3 be three memorised binary patterns in {−1, 1} N 1 with N 1 ∈ N. We first introduce some notation.Denote Suppose Then we have the following relations: ) Our aim is to find an orthogonal lift of {ξ 1 , ξ 2 , ξ 3 }.
Necessity.Let { ξ 1 , ξ 2 , ξ 3 } be a lift of {ξ 1 , ξ 2 , ξ 3 }.Following the notation in (3.1) and (3.2), it is an orthogonal lift if and only if (3.12)This is a system of non-homogeneous linear equations.Owing to Cramer's rule, since the coefficient determinant the system of Equations (3.12) has a unique solution where .
https://doi.org/10.1017/S0956792524000238Published online by Cambridge University Press It is easy to see that where we used (3.5)-(3.7) in the last equality.Similarly, we derive So we have Therefore, (3.8) and (3.9) are obtained.
Remark 1.In the above theorem, the relation (3.8) describes the dimension, while (3.9) gives the construction, of an orthogonal lift, for a given set of three patterns.For computational simplicity, it is reasonable to seek an orthogonal lift with the minimal dimension.For a set of three patterns, an orthogonal lift with the minimal dimension is called a least orthogonal lift of the set.
Following Theorem 3.1, we now present the least orthogonal lift.

set of three memorised binary patterns. Then the dimension of the least orthogonal lift is
Remark 2. The lift { ξ 1 , ξ 2 , ξ 3 } given in (3.10), with N 2 in Theorem 3.2, is a least orthogonal lift of {ξ 1 , ξ 2 , ξ 3 }.To construct this lift, we set and  illustrates this point.Let {ξ 1 , ξ 2 , ξ 3 } be the three memorised patterns with 44 rows and 22 columns (N = 968) indicated as in Figure 1a-1c.By calculating the eigenvalues, we see that none of ξ 1 , ξ 2 and ξ 3 is ε-independently stable; however, the binary pattern ξ in Figure 1d is ε-independently stable.This means that ξ 1 , ξ 2 and ξ 3 cannot be distinguished from other patterns by selecting ε.

Set-up of the parameter ε
In this subsection, we consider the criterion for selecting the parameter ε to make the orthogonal memorised patterns stable and other binary patterns unstable, for system (3.1) with three mutually orthogonal memorised patterns.
Case 1: and it can be seen that From (3.2), we obtain For convenience, let's say l = 1, i.e., Then by which yields Then Note that .
So we obtain .
Note that the stability of η and −η is the same.In this case, we can consider −η and find that Then the desired result can be obtained through Case 2.

Remark 4.
Comparing to the bound given in Equation (2.2), the lower bound of critical strength, 1  6 , is independent of the patterns ξ 1 , ξ 2 , ξ 3 and η, and independent of the dimension N. Remark 5. We acknowledge that the bound 1  6 is still conservative.According to Lemma 2.1, the critical strength which distinguishes orthogonal memorised patterns {ξ 1 , ξ 2 , ξ 3 } from all other binary patterns is given by Here, we consider the mutually orthogonal memorised patterns {ξ 1 , ξ 2 , ξ 3 } shown in Figure 2 and examine the stability of binary patterns by computing the eigenvalues of Jacobians.
For given ε, we count the number of binary patterns at which the Jacobian matrix has (N − 1) negative eigenvalues (0 is an eigenvalue due to global phase shift invariance).Table 1 shows, as the strength ε increases, the number of (asymptotically) stable binary patterns is gradually increased.It's easy to see that

An approach for general standard patterns and simulations
In this section, we propose an approach for the pattern retrieval problem with general standard binary patterns that can be orthogonal or nonorthogonal to each other.Consider a set of standard binary patterns

The case of general standard patterns: M ≥ 4
Combining the procedure A and B, we use an idea of subgrouping to solve a general pattern retrieval problem.Let {ξ 1 , ξ 2 , . . ., ξ M } be a set of standard patterns with M ≥ 4 and let ξ def be a defective pattern.The procedure is as follows, referred as procedure C: (C1) Subgroup the standard patterns so that each subgroup contains three patterns.A subgroup containing one or two patterns is allowed.We have M 3 subgroups.(C2) For each subgroup containing three patterns, we apply procedure B to retrieve one from the subgroup for the defective pattern ξ def .For the subgroup with two patterns, we apply (A2) in procedure A to retrieve one from the subgroup.For the subgroup containing only one Step 1. Subgroup the standard patterns so that each one contains at most three patterns.Output M 3 subgroups.
Step 2. For each subgroup containing three patterns {ξ k , ξ l , ξ s }, we apply procedure B to retrieve a pattern from the subgroup.For the subgroup containing two patterns {ξ k , ξ l } (if exists), we apply (A2) in procedure A to retrieve a pattern from the subgroup.For the subgroup containing only one pattern (if exists), output this pattern.
Step 3. Collect the outputs for all subgroups obtained in Step 2, and take the collection as a new set of standard patterns.If the set contains more than one pattern, we redo Step 1-Step 2 with the new standard pattern set; if the set contains only one pattern, the process stops and this pattern is the final output.
In each retrieval process, two patterns are excluded.Therefore, the total time of the retrieval processes required in the task is M−1

2
, where M is the number of standard patterns.The algorithm realising procedure C is given in Algorithm 1.
In this simulation, we terminate the programme in each subproblem if one of the overlaps is greater than 0.95.We explore the CPU time consumed in Algorithm 1, and it is 47.5318 seconds.In contrast, it takes 106.8977 seconds for the procedure A in ref. [20].This suggests that Algorithm 1 can save time.
In Figure 7, we consider a different way of subgrouping the number symbols {1, 2, . . ., 9, 0} and it shows that the defective symbol 6 is successfully recognised again.

Conclusion
In this paper, we investigated the Hebbian network of Kuramoto oscillators with applications in binary pattern retrieval.We focused on the system with three memorised patterns and developed a theory of least orthogonal lift, which converts the problems with three general memorised patterns to problems with mutually orthogonal memorised patterns.Meanwhile, we provided an estimate for the critical strength of second-order coupling, which tells us how to set-up the parameter ε such that the memorised patterns are stable while other binary patterns are all unstable in each subproblem.In this sense, the error-free retrieval can be guaranteed.This estimate for the critical strength is independent of the memorised patterns and dimensions, therefore, it is extremely convenient for applications.Combining this theory  with the idea of subgrouping, we designed an approach and an algorithm for solving the problems with general standard patterns.

Figure 6 .
Figure 6.The retrieval processes for the grey-scale defective symbol 6 in Figure 3b.Here, Figure 6b-6f illustrate the evolution of overlaps in the processes shown in Figure 6a.

Figure 7 .
Figure 7.The retrieval processes for the gray-scale defective symbol 6 in Figure 3b.

Table 1 .
The number of stable binary patterns for different values of ε.Here, the six stable patterns for small ε are ±ξ 1 , ±ξ 2 and ±ξ 3 .