1 Analog-to-Digital Converter

In this section, we show how the Hopfield neural network can be used to solve some engineering problems. Specifically, we discuss the construction of an analog-to-digital (A/D) converter using the Hopfield neural network. The idea was put forward by Tank and Hopfield [?]. In A/D conversion, and analog value is transformed into its corresponding binary representation. The conversion, we discuss, consists of two steps. The first step is to round the given analog value to the nearest integer. The second step is to convert the integer into its binary equivalent. For example, 3.4 is first rounded to 3, then 3 is converted into its binary equivalent “11” if we have a 2-bit converter, “011” if we have a 3-bit converter, or “0011” if we have a 4-bit converter.

The first step of rounding a number to its nearest integer by a neural network can be organized as solving an optimization problem by the neural network. The second step of converting the integer representation from integer to binary can be accomplished using a neural network with a suitable nonlinear activation function mapping real numbers into the interval [0, 1]. Therefore, an N-bit A/D converter can be viewed as a device that assigns to a given analog input a corresponding vertex of the hypercube [0, 1]^{N}.

Since A/D conversion is viewed an optimization problem, we need to formulate a suitable cost, or penalty, function that we will minimize. In fact finding a suitable cost function will be the first step in constructing a neural network based A/D converter.

To better understand the idea behind solving A/D conversion problems using neural networks, we first consider constructing a 2-bit A/D converter. Let V_{1} be the most significant bit (MSB), V_{0} be the least significant bit (LSB) and x an analog value to be converted. In the actual neural network, V_{1} and V_{0} are the outputs of the two neurons in the neural network based A/D converter. We first consider a very simple cost function,

E = \frac{1}{2} (V_{1}2^{1} + V_{0}2^{0} − x)^{2} .  (1)

A plot of the above function, for an analog input of x = 1.5, shown in Figure 1(a). Note that E is defined over the two-dimensional hypercube [0, 1]^{2}. Observe that minimizing (1)

Figure 1: Constructing the cost function for a 2-bit A/D converter; plots of the cost function and its components. corresponds to choosing V_{1} and V_{0} in a least-squares sense. However, constructing a circuit minimizing the cost function (1) would not yield good results since V_{0} and V_{1} would take value from the interval [0, 1] rather than the set {0, 1}. To overcome the problem, we add to (1) two terms whose purpose is to penalize for non-integer values of V_{0} and V_{1}. The modified cost function is

E = \frac{1}{2} (V_{1}2^{1} + V_{0}2^{0} − x)^{2} + aV_{1}(V_{1} − 1) + bV_{0}(V_{0} − 1),  (2)

where a and b are weighting coefficients. Tank and Hopfield suggest selecting a and b so that the quadratic terms V_{0}^{2} and V_{1}^{2} are cancelled out. In particular, if we set a = −2 and b = −0.5, then (2) becomes

E = \frac{1}{2}x^{2} + 2V_{0}V_{1} − 2V_{1}x − V_{0}x + 2V_{1} + \frac{1}{2}V_{0}.  (3)

The term used to cancelled the quadratic terms is plotted in Figure 1(b). The final cost function given by (3) is plotted in Figure 1(c).When an analog input x is converted to its binary equivalent, the cost function (3) is minimized. One can interpret this process as follows. A given analog input x defines a corresponding cost function E. We can view E as a landscape. Now a ball placed somewhere on this landscape will move towards the lowest point of this landscape, which should correspond to one of the vertices of the hypercube [0, 1]^{2}.

Our next task is to construct differential equations that model a neural network based A/D converter. The parameters of the state equations of the network will be chosen so that the time derivative of E evaluated on the solutions of the equations describing the network is negative. We consider a general case of an N-bit A/D converter. After this, we will derive state equations for a 2-bit A/D converter. Note that

\frac{dE}{dt} = \frac{∂E}{∂V_{0}} V̇_{0} + \frac{∂E}{∂V_{1}} V̇_{1} + ⋯ + \frac{∂E}{∂V_{N−1}} V̇_{N−1}.

Hence, similarly as in (??), it is enough to set

\frac{∂E}{∂V_{i}} = −C_{i}u̇_{i} = −C_{i} \frac{dg_{i}^{−1}(V_{i})}{dV_{i}} V̇_{i}, i = 0, 1, …, N − 1  (4)

to obtain

\frac{dE}{dt} = − ∑_{i=1}^{N−1} C_{i} \frac{dg_{i}^{−1}(V_{i})}{dV_{i}} V̇_{i}^{2} ≤ 0.  (5)

Thus, if the state equations of an N-bit A/D converter satisfy (4), then dE/dt ≤ 0 of the trajectories of the converter. Using (4), we can write the state equations that model the 2-bit A/D converter consisting of two neurons. Let V_{0} = g_{0}(u_{0}) and V_{1} = g_{1}(u_{1}). Then, we have

\begin{matrix}C_{0}u̇_{0} & = & −\frac{∂E}{∂V_{0}} = − (2V_{1} − x + 0.5) = −(2g(u_{1}) − x + 0.5) \\ C_{1}u̇_{1} & = & −\frac{∂E}{∂V_{1}} = −2 (V_{0} − x + 1) = −2(g(u_{0}) − x + 1) \end{matrix} \}  (6)

For simplicity, we assume that C_{0} = C_{1} = 1 and that g_{0} = g_{1} = g. Note that since C is present in both state equations, assigning C_{0} = C_{1} = 1 does not distort the dynamic behavior of the network; it only changes the time scale of both equations by the same factor. Observing that for i = 1, 2,

u̇_{i} = \frac{dg_{i}^{−1}(V_{i})}{dV_{i}} V̇_{i},

we represent (6) as

\begin{matrix}V̇_{0} & = & − (2V_{1} − x + 0.5) (\frac{dg^{−1}(V_{0})}{dV_{0}})^{−1} \\ V̇_{1} & = & −2 (V_{0} − x + 1) (\frac{dg^{−1}(V_{1})}{dV_{1}})^{−1} \end{matrix} \}  (7)

In Figure 2, we display the plot of the cost function E for the analog value x = 1.5 in Figure (2)(a) and a phase portrait of the 2-bit A/D converter in Figure (2)(b). We used the activation function

g(V_{i}) = \frac{1}{1 + e^{−4V_{i}}} .

Thus,

\frac{dg^{−1}(V_{i})}{dV_{i}} = \frac{1}{4V_{i}(1 − V_{i})}.

Note that for x = 1.5, the cost function has two minimizer over the hypercube [0, 1]^{2} that are separated by a “ridge”. Each side of this ridge corresponds to a different final value of the state vector [V_{0}, V_{1}]. Depending on the initial conditions of the state vector of the A/D converter, the analog value x = 1.5 will either be converted to the binary “01” or “10”, which are both valid solutions for x = 1.5. Thus, initial conditions influence the dynamic behavior of the A/D converter. Also, the activation nonlinearity influences the dynamical Figure 2: Plot of the cost function and a phase portrait of the 2-bit A/D converter for the analog value x = 1.5. behavior of the A/D converter via the multiplicative term (\frac{dg^{−1}(V_{i})}{dV_{i}})^{−1} present in the state equations (7). One observe that the multiplicative factor influences the direction in which the converter state trajectory evolves.

Using the above technique, we can easily construct higher order A/D converters. For example, to construct a 3-bit A/D converter, we begin with the cost function

E = \frac{1}{2} (V_{2}2^{2} + V_{1}2^{1} + V_{0}2^{0} − x)^{2} + aV_{2}(V_{2} − 1) + bV_{1}(V_{1} − 1) + cV_{0}(V_{0} − 1).

To cancel the quadratic terms, select a = −8, b = −2, c = −1/2. The state equations of the converter are obtained from the equations

V̇_{i} = −\frac{∂E}{∂V_{i}} (\frac{dg_{i}^{−1}(V_{i})}{dV_{i}})^{−1} .