\documentstyle[12pt]{article}
\renewcommand{\baselinestretch}{1.5}
\setlength{\topmargin}{-.25in}
\setlength{\textheight}{8.5in}
\setlength{\oddsidemargin}{0in}
\setlength{\textwidth}{6.5in}
\newtheorem{thm}{Theorem}
\newtheorem{defn}{Definition}
\newtheorem{example}{Example}
\newtheorem{lemma}{Lemma}
\newtheorem{proposition}{Proposition}
\newtheorem{algorithm}{Algorithm}
\newcommand{\mb}{\mbox}
\newcommand{\bm}{\boldmath}
\def\b#1{\mb{\bm$#1$}}
\newfont{\Bb}{msbm10 scaled\magstep1}
\begin{document}

\section{Analog-to-Digital Converter}
In this section, we show how the Hopfield neural network can be used to solve some engineering problems. Specifically, we discuss the construction of an analog-to-digital~(A/D) converter using the Hopfield neural network. The idea was put forward by Tank and Hopfield~\cite{tank86}. In A/D conversion, an analog value is transformed into its corresponding binary representation. The conversion, we discuss, consists of two steps. The first step is to round the given analog value to the nearest integer. The second step is to convert the integer into its binary equivalent. For example, 3.4 is first rounded to 3, then 3 is converted into its binary equivalent ``11'' if we have a 2-bit converter, ``011'' if we have a 3-bit converter, or ``0011'' if we have a 4-bit converter.

The first step of rounding a number to its nearest integer by a neural network can be organized as solving an optimization problem by the neural network. The second step of converting the integer representation  from integer to binary can be accomplished using a neural network with a suitable nonlinear activation function mapping real numbers into the interval $[0, 1]$. Therefore, an $N$-bit A/D converter can be viewed as a device that assigns to a given analog input a corresponding vertex of the hypercube $[0, 1]^N$.

Since A/D conversion is viewed as an optimization problem, we need to formulate a suitable cost, or penalty, function that we will minimize. In fact finding a suitable cost function will be the first step in constructing a neural network based A/D converter.

To better understand the idea behind solving A/D conversion problems using neural networks, we first consider constructing a 2-bit A/D converter. Let $V_1$ be the most significant bit~(MSB), $V_0$ be the least significant bit~(LSB) and $x$ an analog value to be converted. In the actual neural network, $V_1$ and $V_0$ are the outputs of the two neurons in the neural network based A/D converter. We first consider a very simple cost function,
\begin{equation}
E=\frac{1}{2}\left( V_1 2^1+V_0 2^0-x\right)^2.
\label{eq:quacost}
\end{equation}
A plot of the above function, for an analog input of $x=1.5$, is shown in Figure~\ref{fig:ad_conv}(a). Note that $E$ is defined over the two-dimensional hypercube $[0,1]^2$.
\begin{figure}
\vspace{4.5in}
\hskip.5in \special{eps:c:/brain/ad_conv.eps x=5.1in}
\caption{Constructing the cost function for a 2-bit A/D converter; plots of the cost function and its components.}
\label{fig:ad_conv}
\end{figure}
Observe that minimizing~(\ref{eq:quacost}) corresponds to choosing $V_1$ and $V_0$ in a least-squares sense. However, constructing a circuit minimizing the cost function~(\ref{eq:quacost}) would not yield good results since $V_0$ and $V_1$ would take value from the interval $[0,1]$ rather than the set $\{0, 1\}$. To overcome the problem, we add to~(\ref{eq:quacost}) two terms whose purpose is to penalize for non-integer values of $V_0$ and $V_1$. The modified cost function is
\begin{equation}
E=\frac{1}{2}\left( V_1 2^1+V_0 2^0-x\right)^2+aV_1(V_1-1)+bV_0(V_0-1),
\label{eq:mod_cost}
\end{equation}
where $a$ and $b$ are weighting coefficients. Tank and Hopfield~\cite{tank86} suggest selecting $a$ and $b$ so that the quadratic terms $V_0^2$ and $V_1^2$ are cancelled out. In particular, if we set $a=-2$ and $b=-0.5$, then~(\ref{eq:mod_cost}) becomes
\begin{equation}
E=\frac{1}{2}x^2+2V_0V_1-2V_1x-V_0x+2V_1+\frac{1}{2}V_0.
\label{eq:final_cost}
\end{equation}
The term used to cancel the quadratic terms is plotted in Figure~\ref{fig:ad_conv}(b). The final cost function given by~(\ref{eq:final_cost}) is plotted in Figure~\ref{fig:ad_conv}(c). When an analog input $x$ is converted to its binary equivalent, the cost function~(\ref{eq:final_cost}) is minimized. One can interpret this process as follows. A given analog input $x$ defines a corresponding cost function $E$. We can view $E$ as a landscape. Now a ball placed somewhere on this landscape will move towards the lowest point of this landscape, which should correspond to one of the vertices of the hypercube $[0,1]^2$.

Our next task is to construct differential equations that model a neural network implementing an A/D converter. The parameters of the state equations of the network will be chosen so that the time derivative of $E$ evaluated on the solutions of the equations describing the network is negative. We consider a general case of an $N$-bit A/D converter. After this, we will derive state equations for a 2-bit A/D converter. Note that
\[
\frac{dE}{dt}=\frac{\partial E}{\partial V_0}\dot{V}_0+\frac{\partial E}{\partial V_1}\dot{V}_1+\cdots + \frac{\partial E}{\partial V_{N-1}}\dot{V}_{N-1},
\]
where $V_i=g_i(u_i)$. Hence, similarly as in~(\ref{eq:deocoefu}), it is enough to set
\begin{equation}
\frac{\partial E}{\partial V_i}=-C_i \dot{u}_i=-C_i \frac{d g_i^{-1}(V_i)}{d V_i}\dot{V}_i,\quad i=0,1,\ldots,N-1
\label{eq:parewrtvi}
\end{equation}
to obtain
\begin{equation}
\frac{d E}{dt}=-\sum_{i=0}^{N-1} C_i \frac{d g_i^{-1}(V_i)}{d V_i}\dot{V}_i^2\le 0.
\label{eq:dedtinad}
\end{equation}
Thus, if the state equations of an $N$-bit A/D converter satisfy~(\ref{eq:parewrtvi}), then $dE/dt\le 0$ on the trajectories of the converter. Using~(\ref{eq:parewrtvi}), we can write the state equations that model the 2-bit A/D converter consisting of two neurons. Let $V_0=g_0(u_0)$ and $V_1=g_1(u_1)$. Then, we have
\begin{equation}
\left.\begin{array}{ccl}
C_0\dot{u}_0 &=& -\frac{\partial E}{\partial V_0}=-\left( 2V_1-x+0.5\right)=-(2g_1(u_1)-x+0.5)\\
C_1\dot{u}_1 &=& -\frac{\partial E}{\partial V_1}=-2\left( V_0-x+1\right)=-2(g_0(u_0)-x+1)
\end{array}\right\}
\label{eq:twobitadcoe}
\end{equation}
For simplicity, we assume that $C_0=C_1=1$ and that $g_0=g_1=g$. Note that since $C$ is present in both state equations, assigning $C_0=C_1=1$ does not distort the dynamic behavior of the network; it only changes the time scale of both equations by the same factor. Observing that for $i=1,2$,
\[
\dot{u}_i=\frac{d g_i^{-1}(V_i)}{d V_i}\dot{V}_i,
\]
we represent~(\ref{eq:twobitadcoe}) as
\begin{equation}
\left.\begin{array}{ccl}
\dot{V}_0 &=& -\left( 2V_1-x+0.5\right)\left( \frac{d g^{-1}(V_0)}{d V_0}\right)^{-1}\\
\dot{V}_1 &=& -2\left( V_0-x+1\right)\left(\frac{d g^{-1}(V_1)}{d V_1}\right)^{-1}
\end{array}\right\}
\label{eq:stsptwobitad}
\end{equation}
We display a plot of the cost function $E$ for the analog value $x=1.5$ in Figure~(\ref{fig:adphase})(a) and a phase portrait of the 2-bit A/D converter in Figure~(\ref{fig:adphase})(b). We used the activation function
\[
g(V_i)=\frac{1}{1+e^{-4V_i}}.
\]
Thus,
\[
\frac{d g^{-1}(V_i)}{d V_i}=\frac{1}{4 V_i(1-V_i)}.
\]
\begin{figure}
\vspace{4.5in}
\hskip.75in\special{eps:c:/brain/adphase.eps x=5.1in}
\caption{Plot of the cost function and a phase portrait of the 2-bit A/D converter for the analog value $x=1.5$.}
\label{fig:adphase}
\end{figure}
Note that for $x=1.5$, the cost function has two minimizers in the hypercube $[0,1]^2$ that are separated by a ``ridge''. Initial conditions on each side of this ridge yield a different final value of the state vector $[V_0, V_1]$. Depending on the initial conditions of the state vector of the A/D converter, the analog value $x=1.5$ will either be converted to the binary ``01'' or ``10'', which are both valid solutions for $x=1.5$. Thus, initial conditions influence the dynamic behavior of the A/D converter. Also, the  activation nonlinearity influences the dynamical behavior of the A/D converter via the multiplicative term $\left( \frac{d g^{-1}(V_i)}{d V_i}\right)^{-1}$ present in the state equations~(\ref{eq:stsptwobitad}). One can see that the multiplicative factor influences the direction in which the converter state trajectory evolves.

Using the above technique, we can easily construct higher order A/D converters. For example, to construct a 3-bit A/D converter, we begin with the cost function
\[
E=\frac{1}{2}\left( V_2 2^2+V_1 2^1 +V_0 2^0-x\right)^2+aV_2(V_2-1)+bV_1(V_1-1)+cV_0(V_0-1).
\]
To cancel the quadratic terms, select $a=-8$, $b=-2$, $c=-1/2$. The state equations of the converter are obtained from the equations
\[
\dot{V}_i=-\frac{\partial E}{\partial V_i}\left( \frac{d g_i^{-1}(V_i)}{d V_i}\right)^{-1}.
\]

\end{document}