\documentstyle[12pt]{article}
\renewcommand{\baselinestretch}{1.5}
\setlength{\oddsidemargin}{.0in}
\setlength{\textwidth}{6.5in}
\setlength{\topmargin}{-.25in}
\setlength{\textheight}{8.5in}
\newtheorem{thm}{Theorem}
\newtheorem{defn}{Definition}
\newtheorem{example}{Example}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{algorithm}{Algorithm}
\newcommand{\markex}{\nopagebreak\begin{center}\rule{5in}{.02in}\end{center}}
\newcommand{\mb}{\mbox}
\newcommand{\bm}{\boldmath}
\def\b#1{\mb{\bm$#1$}}
\newfont{\Bb}{msbm10 scaled\magstep1}
\begin{document}
\section{Brain-State-in-a-Box (BSB) Models}
\subsection{Associative Memories}
In this section, we analyze a class of neural network models that can be used to construct the so called associative memories. We distinguish between two types of association: autoassociation and heteroassociation. In autoassociation, after storing a set of patterns by a neural network, where patterns are usually represented as vectors, a distorted (noisy) pattern of a stored pattern is subsequently presented to the network. The task of the neural network is to retrieve (recall) the original stored pattern from the noisy pattern. In heteroassociation an arbitrary set of input patterns is paired with another arbitrary set of output patterns. Operation of an associative memory is characterized by two stages: storage phase, where patterns are being stored by the neural network, and recall phase, where memorized patterns are being retrieved in response to a noisy pattern presented to the network. Figure~\ref{fig:patt_assoc} illustrates the recall phase. In this figure, the input pattern represents a noisy version of stored pattern $\b x_j$. The pattern associator produces the output pattern $\b y$ in response to the input pattern $\b x$. For perfect recall in an autoassociative memory, we should have $\b y=\b x_j$. When $\b y \ne \b x_j$, the associative memory is said to have made a recall error (error in recall).
\begin{figure}
\setlength{\unitlength}{.1in}
\begin{center}
\begin{picture}(60,20)
\thicklines
\put(11,9){\vector(1,0){11}}
\put(11,10){Input pattern}
\put(16,7){$\b x$}
\put(22,5){\framebox(10,8){\shortstack{Pattern\\\\associator}}}
\put(32,9){\vector(1,0){12}}
\put(33,10){Output pattern}
\put(37,7){$\b y$}
\end{picture}
\end{center}
\caption{Operation of pattern associator.}
\label{fig:patt_assoc}
\end{figure}
A neural network that can be used as an associative memory is the Brain-State-in-a-Box~(BSB) neural model, which was proposed by Anderson, Silverstein, Ritz, and Jones in 1977~\cite{anderson77}. It can recognize a pattern from its given noisy version. For this reason the BSB neural model is often referred to as an associative memory. The BSB neural model is a discrete dynamical system whose dynamics are described by the difference equation,
\begin{equation}
\b x(k+1)=\b g\left(\b x(k)+\alpha \b W\b x(k)\right)
\label{eq:classic_bsb}
\end{equation}
subject to the initial condition
\[
\b x(0)=\b x_0,
\]
where $\b x(k) \in \mb{\Bb R}^n$ is the state of the BSB network at time $k$, $\alpha>0$ is the step size, and $\b W \in \mb{\Bb R}^{n \times n}$ is a symmetric weight matrix determined during the training procedure. The function $\b g:\mb{\Bb R}^n \to \mb{\Bb R}^n$ is a vector valued function whose i-th component $(\b g)_i$ is defined as:
\[
(\b g(\b x))_i=\left\{ \begin{array}{lll}
1 & \mbox{if}& \quad x_i \ge 1 \\
x_i & \mbox{if} & \quad -1<x_i<1 \\
-1 & \mbox{if}& \quad x_i \le -1.
\end{array}
\right.
\]
The above nonlinear activation function is sometimes referred to as the linear saturating nonlinearity. A plot of such a  function is shown in Figure~\ref{fig:lin_sat}.
\begin{figure}
\vspace{3in}
\hskip1.5in\special{ps:c:/mybook/sat_non.ps y=3in}
\caption{A plot of linear saturating nonlinearity used in the BSB model as an activation function.}
\label{fig:lin_sat}
\end{figure}
The BSB model gets its name from the fact that the state of the network trajectories are constrained to be in the hypercube $H_n=\left[\begin{array}{cc}
-1 & 1
\end{array}\right]^n$. In Figure~\ref{fig:bsb_tr2}, we show a state plane portrait of the BSB model~(\ref{eq:classic_bsb}) with the weight matrix
\begin{equation}
\b W=\left[\begin{array}{cc}
7 & -3\\
-3 & 10
\end{array}\right]
\label{eq:bsbweighton}
\end{equation}
and the step size is $\alpha=0.1$. The state space of this two-dimensional neural net is the square $H_2=\left[\begin{array}{cc}
-1 & 1
\end{array}\right]^2$.
\begin{figure}
\vspace{3in}
\hskip1.75in\special{eps:c:/mybook/bsb_tr2.eps y=3in}
\caption{Trajectories of the BSB model with the weight matrix $\b W$ given by~(\ref{eq:bsbweighton}).}
\label{fig:bsb_tr2}
\end{figure}
Each vertex of the square has its basin of attraction. In this example all four vertices are asymptotically stable equilibrium states of the BSB neural net. In Figure~\ref{fig:bsb_tr1}, we show a state plane portrait of the BSB model~(\ref{eq:classic_bsb}) with the weight matrix
\begin{equation}
\b W=\left[\begin{array}{cc}
5 & -3\\
-3 & 2
\end{array}\right]
\label{eq:bsbweighttwo}
\end{equation}
and the step size $\alpha=0.1$. In this example only two vertices are asymptotically stable equilibrium states of the BSB neural net.
\begin{figure}
\vspace{3in}
\hskip1.75in\special{eps:c:/mybook/bsb_tr1.eps y=3in}
\caption{Trajectories of the BSB model with the weight matrix $\b W$ given by~(\ref{eq:bsbweighttwo}).}
\label{fig:bsb_tr1}
\end{figure}
When the BSB model is used as an associative memory, asymptotically stable equilibrium points of the model represent stored patterns. Noisy patterns are represented by points in the basins of attraction of the stable equilibrium points. Thus, suppose we are given a noisy, or imperfect, pattern represented by an initial condition $\b x_0$ that is located in a basin of attraction of a stable equilibrium point corresponding to the correct pattern. Then, the corresponding trajectory of the BSB model evolves in time to this stable equilibrium point representing the correct stored pattern. Hence, in this case, the process of association and information retrieval is modeled by the BSB model---a nonlinear discrete dynamical system. An associative memory is able to recall all the information stored in the memory when it is excited with a sufficient large portion of that memory's information. This is in contrast to the address addressable memory (AAM) used in digital computers. In this type of memory, a desired set of information is recalled only when the entire correct address of the information is given. For this reason, we can view associative memories as content addressable memories (CAM).

In summary, realization of associative memories with neural networks requires storage of a set of given patterns as asymptotically stable equilibrium points of the network. The points in basins of attraction about each stable equilibrium point correspond to noisy versions of the stored patterns. When a noisy version of a stored pattern is sufficiently close to it, the network trajectory should converge to the equilibrium point corresponding to the correct pattern. Neural network associative memories should possess the following useful properties:
\begin{enumerate}
\item \label{aemem:one} Each given pattern to be stored should be stored as an asymptotically stable equilibrium point of the network;
\item \label{aemem:two}The number of asymptotically stable equilibrium points of the network that do not correspond to the given stored patterns (that is, spurious states) is minimal;
\item \label{aemem:three} A nonsymmetric interconnection structure, which eases difficulties in implementing associative memory using neural networks;
\item \label{aemem:four} The ability to control the extent of the basin of attraction about the equilibrium point corresponding to each stored pattern;
\item \label{aemem:five} Learning capability, that is, the ability to add patterns to be stored as asymptotically stable equilibrium points to an existing set of stored patterns without affecting the existing equilibria;
\item \label{aemem:six} Forgetting capability, that is, the ability to delete specified equilibria from a given set of stored equilibria without affecting the rest of the equilibria;
\item \label{aemem:seven} A high storage and retrieval efficiency, that is, the ability to efficiently store and retrieve a large number, compared to the order $n$ of the network, of patterns.
\end{enumerate}
\subsection{Analysis of BSB Models}
The BSB neural net has a symmetric weight matrix, which is not a desirable property of a neural associative memory. Hui and \.{Z}ak~\cite{hui92bsb} modified the BSB model to allow for nonsymmetric weight matrix as well as to better control the extent of basins of attraction. A modified form of the BSB neural model, which we refer to as the generalized Brain-State-in-a-Box~(gBSB) neural model, is well suited for implementing associative memories. Its dynamics are described by the equation
\begin{equation}
\b x(k+1)=\b g((\b I_n+\alpha \b W)\b x(k)+\alpha \b b),
\label{eq:gbsbmo}
\end{equation}
where $\b I_n$ is the $n \times n$ identity matrix, $\b b \in \mb{\Bb R}^n$, and the weight matrix $\b W\in \mb{\Bb R}^{n \times n}$ need not be symmetric as in the BSB model. Thus, the original BSB model can be viewed as a special case of the gBSB model when the interconnection matrix $\b W$ is symmetric and $\b b=\b 0$. We add that the analysis of the BSB model behavior on the boundary regions of the hypercube $H_n$ can be reduced to the study of the reduced order gBSB type models rather than that of the BSB type.

We now describe an algorithm for computing the weight matrix $\b W$ and the vector $\b b$ that result in the given patterns being stored as asymptotically stable vertices of the hypercube $H_n$. In further discussion, we use the following notation and definitions. Let
\[
\b L(\b x)=\left(\b I_n+\alpha \b W\right)\b x+\alpha \b b,
\]
and
\[
\b T(\b x)=\b g(\b L(\b x)).
\]
\begin{defn}
A point $\b x \in H_n$ is an equilibrium point of $\b x(k+1)=\b T(\b x(k))$ if $\b x=\b T(\b x)$.
\end{defn}
Note that an equilibrium point is a constant solution to the system $\b x(k+1)=\b T(\b x(k))$.

Let $(\b L(\b x))_i$ be the $i$-th component of $\b L(\b x)$ and $v_i^{(j)}$ be the $i$-the component of the vector $\b v^{(j)}$. Let $\b v=\left[\begin{array}{ccc}
v_1 & \cdots & v_n
\end{array}\right]^T \in \{-1, 1\}^n$. Then, obviously $\b v$ is a vertex of $H_n$ because for $i=1,2,\ldots, n$, we have $v_i=\pm 1$. For a vertex $\b v$ of $H_n$ to be an equilibrium point of the gBSB, we have to have
\[
v_i=\left( \b g\left( \b L(\b v)\right)\right)_i\quad i=1,2,\ldots, n,
\]
that is, if $v_i=1$, then we should have
\begin{equation}
\left( \b L(\b v)\right)_i \ge 1,
\label{eq:eqpforvone}
\end{equation}
and for $v_1=-1$, we should have
\begin{equation}
\left( \b L(\b v)\right)_i \le -1.
\label{eq:eqpforvmin}
\end{equation}
Combining~(\ref{eq:eqpforvone}) and (\ref{eq:eqpforvmin}), we obtain the following necessary and sufficient conditions for the vertex $\b v$ to be an equilibrium point.
\begin{thm}
\label{thm:nascforeq}
A vertex $\b v$ of the hypercube $H_n$ is an equilibrium point of the gBSB neural model if and only if
\[
(\b L(\b v))_iv_i \ge 1, \quad i=1,2, \ldots ,n.
\]
\end{thm}
Equivalently, a vertex $\b v$ of the hypercube $H_n$ is not an equilibrium point of the gBSB neural model if and only if
\[
(\b L(\b v))_iv_i < 1, \quad \mbox{ for some } \quad i=1,2, \ldots ,n.
\]
\begin{example}\rm
We verify that the vertex $\b v^{(1)}=\left[\begin{array}{cc}
1 & 1
\end{array}\right]^T$ of $H_2$ is not an equilibrium point of the BSB model with the weight matrix given by (\ref{eq:bsbweighttwo}) and the step size $\alpha=0.1$. Note that for the BSB model,
\[
\b L(\b x)=(\b I_n+\alpha \b W)\b x.
\]
We first compute $\b L\left(\b v^{(1)} \right)$ to get
\[
\b L\left( \b v^{(1)}\right) = \left(\b I_2+\alpha\b W\right)\b v^{(1)}
= \left[\begin{array}{c}
1.2\\
-0.1
\end{array}\right].
\]
Note that because
\[
\left(\b L\left(\b v^{(1)}\right)\right)_2v_2^{(1)}=-0.1<0,
\]
the vertex $\b v^{(1)}$ is not an equilibrium point of the above BSB model.

On the other hand, the vertex $\b v^{(2)}=\left[\begin{array}{cc}
1 & -1
\end{array}\right]^T$ is an equilibrium point of this BSB model because
\[
\left(\b L\left(\b v^{(2)}\right)\right)_1v_1^{(2)}=1.8\ge 0\quad\mb{and}\quad \left(\b L\left(\b v^{(2)}\right)\right)_2v_2^{(2)}=0.5\ge 0,
\]
that is, the necessary and sufficient conditions of Theorem~\ref{thm:nascforeq} are satisfied.
\end{example}\markex
In our analysis we use stability definitions in the sense of Lyapunov, which we now recall.
\begin{defn}
An equilibrium point $\b x^*$ of $\b x(k+1)=\b T(\b x(k))$ is stable if for every $\varepsilon > 0$ there is a $\delta=\delta(\varepsilon) > 0$ such that if $\|\b x(0)-\b x^*\|< \delta $ then  $\|\b x(k)-\b x^*\|< \varepsilon $ for all $k \ge 0$,
where $\| \cdot\|$ may be any $p$-norm of a vector, for example the Euclidean norm $\|\cdot\|_2$.
\end{defn}
The above definition says that if an equilibrium is stable then if a trajectory starts sufficiently close to the equilibrium, it stays close to the equilibrium for all future times. If the equilibrium is not stable, then it is called unstable.
\begin{defn}
The equilibrium solution $\b x^*$ is asymptotically stable if it is stable and if there exists an $\eta > 0$ such that if $\left\Arrowvert\b x(0)-\b x^* \right\Arrowvert < \eta$ then $\lim_{k \rightarrow \infty}\b x(k) = \b x^*$.
\end{defn}
The following theorem gives us a sufficient condition for a vertex of $H_n$ to be an asymptotically stable equilibrium point.
\begin{thm}
\label{thm_asymstab}
Let $\b v$ be a vertex of the hypercube $H_n$. If
\[
(\b L(\b v))_i v_i > 1, \quad i=1,2, \ldots ,n,
\]
then $\b v$ is an asymptotically stable equilibrium point of the gBSB neural model.
\end{thm}
{\bf Proof}\quad Because $(\b L(\b x))_i x_i$ is a continuous function, there exists a neighborhood $N(\b v)$ about $\b v$ such that
\[
(\b L(\b x))_i x_i > 1\quad\mb{for all}\quad i=1,2,\ldots, n\quad\mb{and}\quad \b x\in N(\b v).
\]
Therefore, for any $\b x\in N(\b v)$, we have
\[
\b g(\b L(\b x))=\b v,
\]
which means that $\b v$ is indeed asymptotically stable equilibrium point.\\
\hspace*{5.5in} $\Box$

In our further discussion, we use a special class of matrices that are characterized in the following definition.
\begin{defn}
A matrix $\mbox{\boldmath$ D$} \in \mb{\Bb R}^{n \times n}$ is said to be row diagonal dominant if
\[
d_{ii} \ge \sum_{{j=1}\atop{j \ne i}}^{n}|d_{ij}|, \quad i=1,2,\ldots ,n.
\]
It is strongly row diagonal dominant if
\[
d_{ii} > \sum_{{j=1}\atop{j \ne i}}^{n}|d_{ij}|, \quad i=1,2,\ldots ,n.
\]
\end{defn}
Now, suppose we are given a matrix $\mbox{\boldmath$ V$} \in \mb{\Bb R}^{n \times r}$. Its pseudo-inverse, or the Moore-Penrose inverse, is defined as follows.
\begin{defn}
Given $\b V \in \mb{\Bb R}^{n \times r}$, a matrix $\b V^{\dagger} \in \mb{\Bb R}^{r \times n}$ is called a pseudo-inverse of $\mbox{\boldmath$ V$}$ if
\begin{enumerate}
\item $\b V\b V^{\dagger}\b V=\b V$;
\item $\b V^{\dagger}\b V \b V^{\dagger}=\b V^{\dagger} $;
\item $(\b V\b V^{\dagger})^T=\b V\b V^{\dagger}$;
\item $(\b V^{\dagger}\b V)^T=\b V^{\dagger}\b V$.
\end{enumerate}
\end{defn}
The pseudo-inverse has the following useful properties:
\begin{itemize}
\item $(\b V^T)^{\dagger} =(\b V^{\dagger})^T$;
\item $(\b V^{\dagger})^{\dagger}=\b V$.
\end{itemize}
%For further properties of pseudo-inverses, we refer the reader to Campbell %and \newline Meyer~(1991) and Boullion and Odell~(1971).

\subsection{Synthesis of Neural Associative Memory}

We now present an expression for constructing the weight matrix $\b W$ ensuring that the given patterns are stored as asymptotically stable equilibrium points of the hypercube $H_n$. Let $d_{ij}$ be the $ij$-th element of the matrix $\b D$, $b_i$ be the $i$-th element of the vector $\b b$, and $v_i^{(j)}$ be the $i$-the component of the vector $\b v^{(j)}$. Recall that we denote the $i$-th component of a vector $\b L(\b v)$ as $(\b L(\b v))_i$.
\begin{thm}
\label{thm_lindep}
Let $\b V=\left[\b v^{(1)},\b v^{(2)},\ldots ,\b v^{(r)}\right] \in \{ -1,1 \} ^{n \times r}$ be the matrix of patterns that we wish to have stored as asymptotically stable points of the gBSB net. Let $\b B =\left[\b b,\b b, \ldots ,\b b\right] \in \mb{\Bb R}^{n \times r}$ be the matrix  consisting of the column vector $\b b$ repeated $r$ times, $\b \Lambda \in \mb{\Bb R}^{n \times n}$, ${\beta}_j=\sum_{i=1}^{r}\left(\b V^{\dagger}\b v^{(j)}\right)_{i}$, and $\beta=-\max _j|\beta _j|$, where $j=1,2,\ldots,r$. Suppose
\[
d_{ii} > \sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|+|b_i(1- \beta)| \qquad i=1,2, \ldots ,n.
\]
If in the gBSB model its weight matrix
\begin{equation}
\b W=(\b D\b V-\b B)\b V^{\dagger}+\b \Lambda\left(\b I_n-\b V\b V^{\dagger}\right),
\label{eq:matwingbsb}
\end{equation}
then all the patterns $\b v^{(j)}$, $j=1,2, \ldots ,r$, will be stored as asymptotically stable equilibrium points of the gBSB model.
\end{thm}
{\bf Proof}\quad Using the first property of the More-Penrose inverse of $\b V$, we obtain
\begin{eqnarray*}
\b W\b V&=& (\b D\b V-\b B)\b V^{\dagger}\b V+\b \Lambda\left(\b I_n-\b V\b V^{\dagger}\right)\b V\\
&=&\b D\b V-\b B\b V^{\dagger}\b V\\
&=& \b D\b V-\left[\begin{array}{cccc}
\b B\b V^{\dagger}\b v^{(1)} & \b B\b V^{\dagger}\b v^{(2)}& \cdots & \b B\b V^{\dagger}\b v^{(r)}
\end{array}\right]\\
&=& \b D\b V-\left[\begin{array}{cccc}
\b b\beta_1 & \b b\beta_2 & \cdots & \b b\beta_r
\end{array}\right].
\end{eqnarray*}
The $j$-th column of the above is
\[
\b W\b v^{(j)}=\b D\b v^{(j)}-\b b\beta _j \qquad j=1,2, \ldots ,r.
\]
We now show that a sufficient condition for a pattern $\b v^{(j)}$ to be an asymptotically stable equilibrium of $\b x(k+1)=\b T(\b x(k))$ is satisfied. Performing simple manipulations and taking into account the assumptions we obtain for $j=1,2, \ldots ,r$:
\begin{eqnarray*}
\left(\b L\left(\b v^{(j)}\right)\right)_iv_i^{(j)}&=&\left((\b I_n+\alpha\b W)\b v^{(j)}+\alpha\b b\right)_iv_i^{(j)} \\
&=& \left(\b v^{(j)} + \alpha \b D\b v^{(j)}-\alpha \beta_j\b b + \alpha \b b\right)_i v_i^{(j)}\\
&=&1+\alpha \left(d_{ii}+\sum_{{k=1}\atop{k \ne i}}^{n} d_{ik}v_k^{(j)}v_i^{(j)}+b_i(1-\beta_j)v_i^{(j)}\right) \\
&\ge& 1+{\alpha}\left(d_{ii}-\left[\sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|+\left\arrowvert b_i\left(1-\left(-\max_j|\beta_j|\right)\right)\right\arrowvert \right]\right)\\
&= &1+{\alpha}\left(d_{ii}-\left[\sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|+|b_i(1-\beta)|\right]\right)\\
&>&1,
\end{eqnarray*}
where the last inequality follows from the assumption on the matrix $\b D$. Hence, the given patterns are asymptotically stable equilibrium points of the gBSB model.\\
\hspace*{5.5in} $\Box$


If the patterns are linearly independent and $r<n$, then $\mbox{rank}\b V =r$ and therefore $\b V^{\dagger}\b V=\b I_r$. In this case $\beta _j=1$ for all $j$ and
\[
\b W\b v^{(j)}=\b D\b v^{(j)}-\b b\qquad j=1,2, \ldots ,r,
\]
Thus in this case, a sufficient condition for the patterns $\b v^{(j)}$, $j=1,2, \ldots ,r$, to be stored as asymptotically stable equilibrium points of the gBSB model is that for $i=1,2,\ldots ,n$,
\[
d_{ii} > \sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|.
\]
The above condition indeed guarantees that all the desired patterns are stored as asymptotically stable equilibria. However, if $d_{ii}$'s are too large, then we may end up storing many other vertices of $H_n$ in addition to the given patterns to be stored. To avoid this scenario, we impose the constraints of the form
\[
d_{ii} < \sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|+|b_i|, \qquad i=1,2,\ldots ,n,
\]
on the matrix $\b D$ while synthesizing the weight matrix $\b W$.

We now discuss the role of the different terms in the expression~(\ref{eq:matwingbsb}) on the weight matrix in storing the desired patterns and minimizing the number of spurious states. First, the term $(\b D\b V-\b B)\b V^{\dagger}$ is responsible for the given patterns to be stored as asymptotically stable equilibrium points of the gBSB model. This is because for a given pattern, say $\b v^{(j)}$, we have $\b \Lambda (\b I_n-\b V\b V^{\dagger})\b v^{(j)}=\b 0$, where $\b 0$ is a vector of zeros. The role of the term  $\b \Lambda (\b I_n-\b V\b V^{\dagger})$ is to destabilize patterns that are not in $\mb{Span} \left[\b v^{(1)},\b v^{(2)}, \ldots ,\b v^{(r)}\right]$. Specifically, if a pattern $\b y$ is orthogonal to $\mb{Span}\left[\b v^{(1)},\b v^{(2)}, \ldots ,\b v^{(r)}\right]$, then the term $\b \Lambda (\b I_n-\b V\b V^{\dagger})$ will not only destabilize the vertex of $H_n$ corresponding to the pattern $\b y$, if it is an equilibrium point of the gBSB model, it will make this point a non-equilibrium point of the gBSB model. To see this, consider the expression
\[
\b W\b y=(\b D\b V-\b B)\b V^{\dagger}\b y+\b \Lambda (\b I_n-\b V\b V^{\dagger})\b y.
\]
Because 
\[
\b y \perp \mb{Span} \left[\b v^{(1)},\b v^{(2)}, \ldots ,\b v^{(r)}\right],
\]
we have
\begin{eqnarray*}
\b W\b y&=&\b \Lambda (\b I_n-\b V\b V^{\dagger })\b y \\
&=&\b \Lambda \b y.
\end{eqnarray*}
Therefore,
\begin{eqnarray*}
(\b L(\b y))_iy_i&=&((\b I_n+\alpha \b W)\b y+\alpha \b b)_iy_i \\
&=&(\b y+\alpha (\b \Lambda \b y+\b b))_iy_i \\
&=&1+\alpha ({\lambda }_{ii}+\sum_{{j=1}\atop{j \ne i}}^{n}{\lambda }_{ij}y_jy_i+b_iy_i).
\end{eqnarray*}
Hence, if we choose $\b \Lambda$ such that for some $i=1,2,\ldots ,n$,
\[
{\lambda }_{ii}<-\sum_{{j=1}\atop{j \ne i}}^{n}|{\lambda }_{ij}|-|b_i|,
\]
then
\[
(\b L(\b y))_iy_i<1 \quad\mb{for some}\quad i=1,2,\ldots ,n.
\]
Therefore, by Theorem~\ref{thm:nascforeq}, the point $\b y$ cannot be an equilibrium point. The inclusion of the matrix $\b B$ in the synthesis formula for the weight matrix $\b W$ was to prevent an automatic storage of the negatives of the given patterns as well as to somehow help to direct the gBSB model trajectories towards the given patterns. For this reason, we select $\b b$ to be in $\mb{Span}\{ \b V\}$. We recommend making $\b b$ a weighted sum of $\b v^{(i)}$, $i=1,2, \ldots ,r$, with positive weight coefficients. The above results can be used as guides for selecting the components of the matrices $\b D$, $\b \Lambda$, the vector $\b b$, and consequently the matrix $\b B$. We summarize the above considerations in the form of an algorithm for synthesizing the weight matrix $\b W$.
\begin{center}
{\bf The Weight Matrix Construction Algorithm}
\end{center}
Given: $\b v^{(j)}$, $j=1,2, \ldots ,r$.
\begin{itemize}
\item Form $\b B=\left[\b b,\b b,\ldots ,\b b\right]$, where
\[
\b b=\sum_{j=1}^{r}\epsilon _j\b v^{(j)},\quad \epsilon_j >0,\quad j=1,2,\ldots , r.
\]
\item Form $\b D$ such that
\begin{eqnarray*}
d_{ii} &>& \sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|,\quad\mbox{and}\\
d_{ii} &<& \sum_{{k=1}\atop{k \ne i}}^{n}|d_{ik}|+|b_i|, \qquad i=1,2,\ldots ,n.
\end{eqnarray*}
\item Form $\b \Lambda$ such that for $i=1,2, \ldots, n$,
\[
{\lambda }_{ii}<-\sum_{{j=1}\atop{j \ne i}}^{n}|{\lambda }_{ij}|-|b_i|.
\]
\item Compute
\[
\b W=(\b D\b V-\b B)\b V^{\dagger}+\b \Lambda(\b I_n-\b V\b V^{\dagger}).
\]
\item Apply the condition $(\b L(\b v))_iv_i \ge 1$ to all vertices of $H_n$ to identify spurious equilibrium patterns.
\end{itemize}
\markex
When the set of patterns to be stored is not linearly independent, we of course can still use the above algorithm to compute the weight matrix $\b W$. However, the second constraint on the matrix $\b D$ may lead to a violation of the condition on $\b D$ in Theorem~\ref{thm_lindep}. In such a case we cannot guarantee that all the given patterns will be stored, nor we will have any control on the number of spurious states. We may then relax this condition. If attempts in synthesizing associative memory for a set of linearly dependent patterns result in unacceptable designs, we may try a different approach. We can start by picking a maximal set of patterns that are linearly independent from the given set of patterns. Let us assume that are $s$ patterns to be stored and $r$ of them form a maximal linearly independent subset. Suppose, for notational simplicity, that the patterns $\b v^{(q)}$, $q=1, \ldots ,r$, form a linearly independent set. We then apply the above algorithm to generate the weight matrix $\b W$ based on the linearly independent patterns $\b v^{(q)}$, $q=1, \ldots ,r$. After that, we modify the obtained weight matrix and the vector $\b b$ to ensure that the remaining patterns $\b v^{(j)}$, $j=r+1,r+2, \ldots ,s$ will also be stored. We will focus our attention on ensuring that the pattern $\b v^{(r+1)}$ is stored as an asymptotically stable point. The remaining patterns $\b v^{(j)}$, $j=r+2,r+3, \ldots ,s$, can be stored using the same procedure. Because
\[
\b v^{(r+1)}\in \mb{Span}\left\{\b v^{(1)},\b v^{(2)},\ldots ,\b v^{(r)}\right\},
\]
that is, $\b v^{(r+1)}$ is a linear combination of $\b v^{(i)}$, $i=1,2,\ldots, r$, it can be represented as
\[
\b v^{(r+1)}=\sum_{q=1}^{r}\gamma _q\b v^{(q)}=\b V \b \gamma,
\]
where $\b \gamma =\left[\begin{array}{ccc}
\gamma_1  & \cdots & \gamma_r
\end{array}\right]^T \in \mb{\Bb R}^n$. Then, using~(\ref{eq:matwingbsb}), we obtain 
\begin{eqnarray*}
\b W\b v^{(r+1)}&=&\b D\b V\b \gamma -\b B\b \gamma\\
&=&\b D\b v^{(r+1)}-\left( \sum_{q=1}^{r}\gamma _q \right)\b b.
\end{eqnarray*}
Recall that $\b L(\b v)=(\b I_n+\alpha \b W)\b v+\alpha \b b$. Hence,
\begin{eqnarray*}
\b L\left(\b v^{(r+1)}\right)&=&\b v^{(r+1)}+\alpha(\b W\b v^{(r+1)}+\b b)\\
&=&\b v^{(r+1)}+\alpha\left(\b D\b v^{(r+1)}-\left[\sum_{q=1}^{r}\gamma_q-1\right]\b b\right).
\end{eqnarray*}
Recall now that the conditions for the asymptotic stability of $\b v^{(r+1)}$ are
\[
\left(\b L\left(\b v^{(r+1)}\right)\right)_iv^{(r+1)}_i>1 \qquad i=1,2,\ldots ,n,
\]
or equivalently, since $\alpha >0$,
\[
\left(\b D\b v^{(r+1)}-\left[\sum_{q=1}^{r}\gamma_q-1\right]\b b\right)_iv^{(r+1)}_i>0,\qquad i=1,2,\ldots ,n.
\]
Observe that because of the assumptions on the matrix $\b D$, we have
\[
\mb{sign}\left(\b D\b v^{(r+1)}\right)_i=\mb{sign} v^{(r+1)}_i.
\]
With the above in mind, we obtain stronger sufficient conditions for $\b v^{(r+1)}$ to be asymptotically stable. They are:
\[
\mb{sign}\left[\left(1-\sum_{q=1}^{r}\gamma _q\right)\b b\right]_i=\mb{sign}\left(v^{(r+1)}_i\right) \qquad i=1,2,\ldots ,n.
\]
We can represent the above conditions as
\[
\mb{sign}\,b_i=\mb{sign}\left(v^{(r+1)}_i\right)\,\mb{sign}\left(1-\sum_{q=1}^{r}\gamma_q\right)\qquad i=1,2,\ldots ,n.
\]
Thus, if we adjust the components of $\b b$ so that the above conditions are satisfied, then we are guaranteed that $\b v^{(r+1)}$ is stored as an asymptotically stable equilibrium point. We may try to further adjust the components of $\b b$, if possible, to store the remaining patterns. Then, using the same matrices $\b D$, $\b \Lambda$, and $\b V$ used to compute the original weight matrix $\b W$, we can choose an appropriate $\mb{\bm$b$}$ to generate a new weight matrix $\b W$.

If $\b b$ cannot be adjusted in the above way to accommodate all the patterns, then we may have to use the following procedure. First, recall the stability conditions that the patterns $\b v^{(r+1)},\ldots ,\b v^{(s)}$ should satisfy,
\[
\left(\b D\b v^{(k)}-\left[\sum_{j=q}^{r}\gamma_q-1\right]\b b\right)_i v^{(k)}_i>0,\qquad i=1,2,\ldots ,n, \quad k=r+1,r+2,\ldots ,s.
\]
Using the requirement for the matrix $\mb{\bm$D$}$ to be diagonally dominant, we arrive at stronger conditions of the form
\[
\left(d_{ii}-\sum_{{j=1}\atop{j \ne i}}^{n}|d_{ij}|\right)+\left(1-\sum_{j=q}^{r}\gamma _{q}\right)b_i v^{(k)}_i > 0,\quad i=1,\ldots ,n,\quad
k=r+1,\ldots ,s.
\]
Thus, we may use the above conditions and try to decrease the magnitude of the $b_i$'s until the conditions are met for all $k=r+1,\ldots ,s$. However, we are not guaranteed to always satisfy the above conditions. If the conditions are satisfied, then using the same matrices $\mb{\bm$D$}$, $\mb{\bm$\Lambda$}$, and $\mb{\bm$V$}$ used to compute the original weight matrix $\mb{\bm$W$}$, we choose an appropriate $\mb{\bm$b$}$ to generate a new weight matrix $\mb{\bm$W$}$.
\begin{example}
\label{ex_synthes}
\rm
We synthesize a gBSB based associative memory that is supposed to store the following three patterns:
\[
\mb{\bm$v$}^{(1)}=\left[\begin{array}{c}
-1\\
-1\\
1\\
1\\
1
\end{array}\right],\qquad \mb{\bm$v$}^{(2)}=\left[\begin{array}{c}
1\\
1\\
-1\\
-1\\
-1
\end{array}\right],\quad \mb{ and }\quad \mb{\bm$v$}^{(3)}=\left[\begin{array}{c}
1\\
-1\\
1\\
-1\\
1
\end{array}\right].
\]
We first choose matrices $\mb{\bm$D$}$ and $\mb{\bm$\Lambda$}$. Let
\[
\mb{\bm$D$}=\left[\begin{array}{ccccc}
1 & 0 & 0 & 0 & -0.5\\
0 & 1 & 0 & 0 & 0\\
0 & -0.2 & 1 & 0 & 0\\
0 & 0 & 0 & 1 & 0\\
0 & 0 & 0 & 0 & 1
\end{array}\right],\qquad \mb{\bm$\Lambda$}=-2\mb{\bm$I$}_5.
\]
Set $\mb{\bm$V$}=\left[\begin{array}{ccc}\mb{\bm$v$}^{(1)} & \mb{\bm$v$}^{(2)} & \mb{\bm$v$}^{(3)} \end{array}\right]$ and $\mb{\bm$B$}=\left[\begin{array}{ccc}\mb{\bm$b$} & \mb{\bm$b$} & \mb{\bm$b$} \end{array}\right]$, where
\[ 
\mb{\bm$b$}=\left[\begin{array}{ccccc} 0.6 & -0.5 & 0.7 & -0.4 & 0.8 \end{array}\right]^T.
\]
Note that $\mb{rank}\mb{\bm$V$}=2$ and the step size $\alpha =0.9$. Using the above we compute the weight matrix $\mb{\bm$W$}$ to get
\[
\mb{\bm$W$}=\left[\begin{array}{ccccc}
-0.6500 & 0.2667 & -0.2667 & -1.3500 & -0.2667\\
0.1250 & -1.0833 & -0.9167 & -0.1250 & -0.9167\\
-0.1750 & -0.9500 & -1.0500 & 0.1750 & 0.9500\\
-1.4000 & -0.0667 & 0.0667 & -0.6000 & 0.0667\\
-0.2000 & -0.8667 & 0.8667 & 0.2000 & -1.1333
\end{array}\right].
\]
We then check which vertices of $H_n$ satisfy the sufficiency condition of Theorem~\ref{thm_asymstab} for asymptotic stability. In other words, we wish to identify the spurious states. In this example only the given patterns satisfy the condition of Theorem~\ref{thm_asymstab}.
\end{example}\markex
\subsection{Learning}
A pattern is learned by a memory if its noisy or an incomplete version presented to the memory is mapped back to this pattern. In our discussion of the proposed iterative learning, we will use the following theorem which allows one to iteratively compute the pseudo-inverse of a given matrix. The theorem can be found in Campbell and Meyer~\cite[page~54]{campbell91}. 
\begin{thm}
\label{thm_pseuditer}
Let $\mbox{\boldmath$ U$}\in \mb{\Bb R}^{n \times (r+1)}$ be partitioned by slicing off its last column, so that
\[
\mbox{\boldmath$ U$}=\left[\mbox{\boldmath$V$} \vdots \mbox{\boldmath$y$}\right]. 
\]
Let
\begin{eqnarray*}
\mbox{\boldmath$ k$}&=&\mbox{\boldmath$ V$}^{\dagger}\mbox{\boldmath$y$},\\
\mbox{\boldmath$ u$}&=&\left(\mbox{\boldmath$ I$}_n-\mbox{\boldmath$ VV$}^{\dagger}\right)\mbox{\boldmath$ y$},\\
\mbox{\boldmath$ z$}&=&\left\{ \begin{array}{cc}
\mbox{\boldmath$ u$}^{\dagger} & \mbox{ if } \mbox{\boldmath$ u \ne 0$}\\
{\textstyle \frac{\mbox{\boldmath$ k$}^T\mbox{\boldmath$ V$}^{\dagger}}{1+\mbox{\boldmath$ k$}^T\mbox{\boldmath$ k$}}} & \mbox{ if } \mbox{\boldmath$ u=0$}.
\end{array}
\right.
\end{eqnarray*}
Then,
\[
\mbox{\boldmath$ U$}^{\dagger}=\left[ \begin{array}{c}
\mbox{\boldmath$ V$}^{\dagger}-\mbox{\boldmath$ kz$}\\
\mbox{\boldmath$ z$}
\end{array}
\right].
\]
\end{thm}

Learning a new pattern requires updating the weight matrix $\mb{\bm$W$}$. This in turn involves augmenting the matrices $\mb{\bm$V$}$ and $\mb{\bm$B$}$, and recomputing $\mb{\bm$V$}^{\dagger}$. Augmenting  $\mb{\bm$V$}$ and $\mb{\bm$B$}$ is straightforward, and updating $\mb{\bm$V$}^{\dagger}$ can be performed recursively using the above theorem. The above procedure guarantees that the new pattern will be stored as an asymptotically stable equilibrium point if it is linearly independent of the learned patterns. If, however, the pattern to be learned is a linear combination of the stored patterns, then we do not update the matrix $\mb{\bm$V$}$ but rather modify the vector $\mb{\bm$b$}$ as described at the end of the previous subsection. After that we update the weight matrix $\mb{\bm$W$}$. We now illustrate the iterative learning process on a numerical example.
\begin{example}
\label{ex_learn}
\rm
Consider Example~\ref{ex_synthes}. Suppose that we wish to add 
\[
\mb{\bm$v$}^{(4)}=\left[\begin{array}{ccccc}
-1 &-1 & -1 &-1 & 1
\end{array}\right]^T
\]
to the previously designed memory. We augment the pattern matrix to include $\mb{\bm$v$}^{(4)}$ in it. Thus, we get $\mb{\bm$V$}=\left[\begin{array}{cccc}
\mb{\bm$v$}^{(1)} & \mb{\bm$v$}^{(2)} & \mb{\bm$v$}^{(3)} & \mb{\bm$v$}^{(4)}
\end{array}\right]$. We also augment the matrix $\mb{\bm$B$}$ by adding extra $\mb{\bm$b$}$ column to it. We then update $\mb{\bm$V$}^{\dagger}$ using Theorem~\ref{thm_pseuditer} and recompute the weight matrix $\mb{\bm$W$}$ to obtain
\[
\mb{\bm$W$}=\left[\begin{array}{ccccc}
0.1714 & 0.8143 & 0.8286 & -0.5286 & -0.8143\\
0.4643 & -0.8571 & -0.4643 & 0.2143 & -1.1429\\
0.7786 & -0.3143 & 0.2214 & 1.1286 & 0.3143\\
-0.8286 & 0.3143 & 0.8286 & -0.0286 & -0.3143\\
-0.4857 & -1.0571 & 0.4857 & -0.0857 & -0.9429
\end{array}\right].
\]
Next, we check which vertices of $H_n$ satisfy the condition of Theorem~\ref{thm_asymstab} for asymptotic stability. We found that only the given four patterns satisfy the condition of Theorem~\ref{thm_asymstab}.
\end{example}\markex 
\subsection{Forgetting}
A pattern, previously stored, is forgotten or deleted from the memory if a stimulus that is a perturbed version of that pattern, when presented to the memory is not mapped back to this pattern. We now discuss the situation when we would like to update the weight matrix in such a way that the specified patterns are removed from memory. We refer to the process of removing patterns from the associative memory as forgetting. Our discussion uses the following theorem which can be considered as the inverse of Theorem~\ref{thm_pseuditer}. A general version of the theorem appears in Boullion and Odell~\cite[page~14]{boullion71}.
\begin{thm}
\label{thm_pseudinv}
Let $\mbox{\boldmath$ U$}\in \mb{\Bb R}^{n \times (r+1)}$ be partitioned by slicing off its last column, so that
\[
\mbox{\boldmath$U$}=\left[\mbox{\boldmath$V$} \vdots \mbox{\boldmath$y$}\right], 
\]
and let its pseudoinverse $\mbox{\boldmath$U$}^{\dagger} \in \mb{\Bb R}^{(r+1) \times n}$ be partitioned by slicing off its last row, so that
\[
\mbox{\boldmath$U$}^{\dagger}=\left[\begin{array}{c}
\mbox{\boldmath$S$}\\
\mbox{\boldmath$z$}
\end{array}\right].
\]
Then,
\[
\mbox{\boldmath$V$}^{\dagger}=\left\{\begin{array}{cc}
\mbox{\boldmath$S$}+\mbox{\boldmath$S$}\frac{\mbox{\boldmath$yz$}}{1-\mbox{\boldmath$zy$}} & \mbox{if}\quad \mbox{\boldmath$zy$}\ne 1\\
\mbox{\boldmath$S$}-\mbox{\boldmath$S$}\mbox{\boldmath$z$}^{\dagger}\mbox{\boldmath$z$} & \mbox{if}\quad \mbox{\boldmath$zy$}=1.
\end{array}\right.
\]
\end{thm}
{\bf Proof}\quad Referring to Theorem~\ref{thm_pseuditer}, we first show that $\mbox{\boldmath$u$}=\mbox{\boldmath$0$}$ if and only if $\mbox{\boldmath$zy$}\ne 1$, or equivalently $\mbox{\boldmath$u$}\ne \mbox{\boldmath$0$}$ if and only if $\mbox{\boldmath$zy$}= 1$. Indeed, if $\mbox{\boldmath$u$}=\mbox{\boldmath$0$}$, then
\[
\mbox{\boldmath$z$}=\frac{\mbox{\boldmath$k$}^T\mbox{\boldmath$V$}^{\dagger}}{1+\mbox{\boldmath$ k$}^T\mbox{\boldmath$ k$}}.
\]
Hence,
\[
\mbox{\boldmath$zy$}=\frac{\mbox{\boldmath$k$}^T\mbox{\boldmath$V$}^{\dagger}\mbox{\boldmath$y$}}{1+\mbox{\boldmath$ k$}^T\mbox{\boldmath$ k$}}=\frac{\mbox{\boldmath$ k$}^T\mbox{\boldmath$ k$}}{1+\mbox{\boldmath$ k$}^T\mbox{\boldmath$ k$}}<1.
\]
On the other hand, if $\mbox{\boldmath$u$}\ne\mbox{\boldmath$0$}$, then
\[
\mbox{\boldmath$z$}=\mbox{\boldmath$u$}^{\dagger}=\left\{(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger})\mbox{\boldmath$y$}\right\}^{\dagger}.
\]
Taking into account the fact that $\mbox{\boldmath$u$}\ne \mbox{\boldmath$0$}$ and that $\mbox{\boldmath$u$}\in \mb{\Bb R}^n$, its pseudoinverse is
\[
\mbox{\boldmath$u$}^{\dagger}=(\mbox{\boldmath$u$}^T\mbox{\boldmath$u$})^{-1}\mbox{\boldmath$u$}^T=\left(\mbox{\boldmath$y$}^T\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger}\right)\mbox{\boldmath$y$}\right)^{-1}\mbox{\boldmath$y$}^T\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger}\right),
\]
since $\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger}\right)$ is an orthogonal projector. Hence,
\[
\mbox{\boldmath$zy$}=\mbox{\boldmath$u$}^{\dagger}\mbox{\boldmath$y$}=1.
\]
We now prove the expressions for $\mbox{\boldmath$V$}^{\dagger}$. We start with the case when $\mbox{\boldmath$zy$}\ne 1$. It follows from Theorem~\ref{thm_pseuditer} that
\begin{equation}
\mbox{\boldmath$S$}=\mbox{\boldmath$V$}^{\dagger}-\mbox{\boldmath$V$}^{\dagger}\mbox{\boldmath$yz$}=\mbox{\boldmath$V$}^{\dagger}\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$yz$}\right).
\label{eq:afapptfo}
\end{equation}
Since $\mbox{\boldmath$zy$}\ne 1$, the matrix $\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$yz$}\right)$ is invertible. Using the matrix inversion formula~(\ref{eq:specialsmw}), from Chapter~\ref{chp:m_review}, gives
\begin{equation}
\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$yz$}\right)^{-1}=\mbox{\boldmath$I$}_n+\frac{\mbox{\boldmath$yz$}}{1-\b z \b y}.
\label{eq:simpappofminvf}
\end{equation}
Applying formula~(\ref{eq:simpappofminvf}) to (\ref{eq:afapptfo}) gives
\[
\mbox{\boldmath$V$}^{\dagger}=\mbox{\boldmath$S$}+\mbox{\boldmath$S$}\frac{\mbox{\boldmath$yz$}}{1-\mbox{\boldmath$zy$}}.
\]

We now prove the expression for $\mbox{\boldmath$V$}^{\dagger}$ for the case when $\mbox{\boldmath$zy$}=1$. From Theorem~\ref{thm_pseuditer} we have
\begin{equation}
\mbox{\boldmath$S$}=\mbox{\boldmath$V$}^{\dagger}-\mbox{\boldmath$kz$},
\label{eq:exfos}
\end{equation}
and hence, because $\b z\ne \b 0$,
\begin{equation}
\mbox{\boldmath$S$}\mbox{\boldmath$z$}^{\dagger}=\mbox{\boldmath$V$}^{\dagger}\mbox{\boldmath$z$}^{\dagger}-\mbox{\boldmath$k$}.
\label{eq:szpsinv}
\end{equation}
But in this case, $\mbox{\boldmath$u$}\ne \mbox{\boldmath$0$}$, and therefore
\[
\mbox{\boldmath$z$}^{\dagger}=\left(\mbox{\boldmath$u$}^{\dagger}\right)^{\dagger}=\mbox{\boldmath$u$}=\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger}\right)\mbox{\boldmath$y$}.
\]
Substituting the above expression for $\mbox{\boldmath$z$}^{\dagger}$ into~(\ref{eq:szpsinv}), we get
\begin{eqnarray*}
\mbox{\boldmath$S$}\mbox{\boldmath$z$}^{\dagger}&=&\mbox{\boldmath$V$}^{\dagger}\left(\mbox{\boldmath$I$}_n-\mbox{\boldmath$VV$}^{\dagger}\right)\mbox{\boldmath$y$}-\mbox{\boldmath$k$}\\
&=&\left(\mbox{\boldmath$V$}^{\dagger}-\mbox{\boldmath$V$}^{\dagger}\right)\mbox{\boldmath$y$}-\mbox{\boldmath$k$}\\
&=&-\mbox{\boldmath$k$}.
\end{eqnarray*}
We now substitute the above into~(\ref{eq:exfos}) to obtain
\[
\mbox{\boldmath$V$}^{\dagger}=\mbox{\boldmath$S$}-\mbox{\boldmath$S$}\mbox{\boldmath$z$}^{\dagger}\mbox{\boldmath$z$},
\]
which completes the proof of the theorem.\\
\hspace*{5.5in} $\Box$

We can delete a pattern from the memory by first removing it from $\mb{\bm$V$}$, then updating $\mb{\bm$V$}^{\dagger}$ using Theorem~\ref{thm_pseudinv}, removing one column from $\mb{\bm$B$}$, and recomputing the weight matrix $\mb{\bm$W$}$. We illustrate the iterative forgetting process on the following numerical example.
\begin{example}
\label{ex_forget}
\rm
We consider  Example~\ref{ex_learn}. Suppose that we wish to delete
\[
\mb{\bm$v$}^{(3)}=\left[\begin{array}{ccccc}
1 &-1 &1 &-1 & 1
\end{array}\right]^T
\]
from the previously designed memory. We can use Theorem~\ref{thm_pseudinv} to update $\mb{\bm$V$}^{\dagger}$, that is to find the pseudoinverse of $\mb{\bm$V$}=\left[\begin{array}{ccc}
\mb{\bm$v$}^{(1)} & \mb{\bm$v$}^{(2)} & \mb{\bm$v$}^{(4)}
\end{array}\right]$. We also update the matrix $\mb{\bm$B$}$ by deleting from it one column. We leave the remaining data unchanged. We recompute the weight matrix $\mb{\bm$W$}$ to obtain
\[
\mb{\bm$W$}=\left[\begin{array}{ccccc}
-0.7333 & 1.2667 & 0.1500 & 0.1500 & -1.2667\\
0.9167 & -1.0833 & -0.1250 & -0.1250 & -0.9167\\
0.0500 & 0.0500 & -0.3250 & 1.6750 & -0.0500\\
-0.0667 & -0.0667 & 1.4000 & -0.6000 & 0.0667\\
-0.8667 & -0.8667 & 0.2000 & 0.2000 & -1.1333
\end{array}\right].
\]
We then check which vertices of $H_n$ satisfy the condition of Theorem~\ref{thm_asymstab} for asymptotic stability. We found that only $\mb{\bm$v$}^{(1)}$, $\mb{\bm$v$}^{(2)}$, and $\mb{\bm$v$}^{(4)}$,  satisfy the condition of Theorem~\ref{thm_asymstab} as desired.
\end{example}\markex




The dynamics of the BSB model and its variants were further analyzed by Greenberg~\cite{greenberg88}, Golden~\cite{golden86,golden93}, Grossberg\cite{grossberg88}, Hui and \.{Z}ak~\cite{hui92bsb}, Hakl~\cite{hakl92}, Hui, Lillo, and \.{Z}ak~\cite{hui93b}, Schultz~\cite{schultz93}, Lillo et al.~\cite{lillo94}, Anderson~\cite{anderson95}, Perfetti~\cite{perfetti95}, Hassoun~\cite{hassoun95}, \.{Z}ak, Lillo, and Hui~\cite{zak96}, Chan and \.{Z}ak~\cite{chan97,chan98}, Park, Cho, and Park~\cite{parkcho99}. A continuous counterpart of the BSB model referred to as the linear system operating in a saturated mode was analyzed in~Li, Michel, and Porod~\cite{jhli89}.


One of the first techniques used to design neural network associative memories is the {\em outer product method}. This method was first used by Hopfield in~\cite{hopfield82} and in~\cite{hopfield84}. The outer product method gives a network with a symmetric interconnection structure that does not necessarily store the desired patterns as equilibrium points of the network. The network designed using this method can efficiently store up to $0.15n$ arbitrary patterns and has learning capabilities.

The {\em projection learning rule\/} proposed by Personnaz, Guyon, and Dreyfus in~\cite{personnaz85} and in~\cite{personnaz86} gives a network with a symmetric interconnection structure that always stores the desired patterns as equilibrium points although not necessarily asymptotically stable. The network designed with this method can effectively store up to $0.5n$ desired patterns and has learning capabilities.

The {\em eigenstructure method\/} used by Li et al.~\cite{jhli89} yields a neural network associative memory capable of storing any given pattern as an asymptotically stable equilibrium point. The network designed with this method is capable of effectively storing a number of patterns that may exceed the order of the network. However, the resulting network will have a symmetric interconnection structure. Furthermore, there are no provisions for learning for networks designed by the eigenstructure method.

In a series of papers, Michel, Farrell, and Porod~\cite{michel89}, Michel and Farrell~\cite{michel90a}, Michel, Farrell, and Sun~\cite{michel90b}, and Yen and Michel~\cite{yen91} modified the eigenstructure method. The modified method can be used to design networks capable of storing on order of $0.5n$ patterns as asymptotically stable equilibrium points. Networks designed by this method need not have a symmetric interconnecting structures. Furthermore, learning is also incorporated into the method. Networks designed by the previous methods use energy functions techniques. A benefit of using energy functions is that the resulting networks are globally stable, that is, the networks' trajectories always tend to some equilibrium points. On the other hand, if the modified method yields a nonsymmetric interconnecting structure, the resulting network is not guaranteed to be globally stable.

\bibliography{brain,root2,mybook}
\bibliographystyle{plain}
\end{document}