Neural Network TLU Training

Steve Lamberson, Jon Nehrbass, and Mike Skillen  

ECE 595C   Funwork #1  

September 25, 2004

Table of Contents

  1. Example 1 - Problem 4.3 from Neural Network Design by Hagan, Demuth, and Beale (PWS Publishing Company, 1999)

  2. Example 2 - Funwork #1 example.
  3. Example 3 - We actually take credit for this one.
  1. trainTLU - single TLU perceptron
  2. TLUnet - network of TLUs utilizing calls to the single TLU perceptron
  3. plotClass - a function used to plot points in different classes with different symbols

  

Introduction

A Threshold Logic Unit (TLU) models a biological neuron, and returns a single value based on a series of inputs.  A simple model of a TLU is provided below.

 

As is shown, the output of the TLU is y = sign([w,x(i)]-t), given a series of n weights (w1,w2, … wn), threshold t, and input vector x(i).

Depending on the output (either +1 or -1), the TLU will categorize the inputs into one of two classes.  Therefore, with the use of a one-neuron perceptron, two classes can be obtained; similarly, an S-neuron perceptron can categorize 2S classes.  A simple illustration of a 2-neuron perceptron categorizing 4 classes is shown below.  

 back to top

Example Problems  

Example 1 - Problem 4.3 from Neural Network Design by Hagan, Demuth, and Beale (PWS Publishing Company, 1999)

 

back to top

Example 2 - Funwork #1 Example.

 

back to top

Example 3 - We actually take credit for this one.

back to top

Algorithms

trainTLU - single TLU perceptron

% trainTLU Trains a Threshold Logic Unit.
%
% [w,t] = trainTLU(x,d) sorts the points given in x into two classes
% as prescribed by d. trainTLU outputs the weight vector (w) and the
% threshold value (t). trainTLU will not be reliable if the input
% points (x) and corresponding separation preferences (d) describe
% are not linearly separable.
%
% trainTLU is derived from the algorithm called the 'Simplified TLU
% Learning Algorithm' on page 475 of the text 'Systems and Control'
% by Stanislaw H. Zak. (Oxford University Press, 2003).
%
% Input:
%     x == n x p matrix, where p is the number of training points and
%     n is the dimensions of each training point

%
%     d == a row vector of lenth p whose values are -1 or 1 and describe
%     the desired output of the TLU.
%
% Output:
%     w == a n x 1 vector, called the weight vector, that describes
%     the normal to a hyperplane separating x into classes -1 and 1

%
%     t == the threshold value for the TLU
%
% Examples:
%     Given:
%         x = [ 2 0 -1 -1 1 3;
%                 2 3 1 -1 -1 0];
%         d = [-1 1 1 1 -1 -1];
%
%     [w,t] = trainTLU(x,d) will separate the training pairs described in x
%     into two classes. Points 1, 5, and 6 will belong two one class
%     (from the -1 in the corresponding elements of d), and points 2, 3,
%     and 4 will belong to another class. The vector w will describe the
%     vector normal to the line (and through the origin) that separates
%     these classes, and t is the associated threshold. The equation to
%     plot the separating line is w(1)*x + w(2)*y = t.
%
%
% Created for:
%     Funwork 1
%     ECE 595C: Introduction to Biologically Inspired Engineering
%     Module 2: Neural Networks
%     Fall 2004
%     Prof. Stanislaw H. Zak
%     Purdue University
%     West Lafayette Campus
%
% Created by:
%     Steve Lamberson
%         lamberss@ecn.purdue.edu
%     Jon Nehrbass
%         jnehrbas@purdue.edu
%     Mike Skillen
%         skillen@purdue.edu
%
% Created on:
%     September 24, 2004
%
% Last Modified on:
%     September 25, 2004


function [w,t] = trainTLU(x,d)

% find dimensions of problem
[n,p] = size(x);

% initialize variables
w = zeros(n+1,1);
ws = rand(n+1,1);
ws = ws/sqrt(sum(ws.^2));
L = 0;
delta = 0;
s = 0*d;
r = 0*d;

% Augment x to contain d
x(size(x,1)+1,:) = -ones(1,p);

for i = 1:p

    % Alter x to be negative of x if d = -1
    x(:,i) = x(:,i)*d(i);

    % find the magnitude of each training pair
    s(i) = sqrt(sum(x(:,i).^2));

    % find the dot product of each training pair with ws
    r(i) = ws'*x(:,i);
end

% find the training pair with the largest magnitude
L = max(s);

% find the minimum dot product
delta = 0.95*min(r);

% determine the upper bound of needed weight vector changes
K = (L/delta)^2;

% Train TLU
k = 0;
while (k < K)
    error = zeros(1,p);
    for i = 1:p
        if (w'*x(:,i) <= 0)
            w = w + x(:,i);
            k = k + 1;
            error(i) = 1;
        end
    end

    % exit loop if weight vector accurately predicts
    % all training pairs
    if sum(sum(error)) == 0
        break;
    end
end

% fix output parameters
t = w(n+1);
w = w(1:n);

 

back to top

TLUnet - network of TLUs utilizing calls to trainTLU

% TLUnet Trains a network composed of TLUs
%
% [W,T] = TLUnet(x,d) sorts the training pairs x into the classes
% prescribed by d. TLUnet returns W (the weight matrix) and
% T (the threshold vector) for the network. If the training pairs
% are two dimensional, TLUnet also plots the classification lines
% described by the W and T to the current figure (or a new figure
% if none are open). The plot is generated on top of the current
% figure (the 'hold on' command is used).
%
% Input:
%     x == n x p matrix, where p is the number of training points and
%     n is the dimensions of each training point

%
%     d == N x p matrix in which each row contains only -1 or 1 elements
%     (the perceptrons will separated the training pairs that are
%     associated with a -1 from the rest). N is the number of perceptrons
%     needed (or desired) to solve the problem.
%
% Output:
%     W == n x N matrix, whos rows are the weight vector describing
%     the normal to a hyperplane which separates the classes 
%     according to the output of the appropriate perceptron.

%
%     T == N x 1 vector, whos elements are the threshold values calculated
%     the corresponding perceptron.
%
% Examples:
%     Given:
%         x = [ 2 0 -1 -1 1 3;
%                 2 3 1 -1 -1 0];
%         d = [-1 1 1 1 -1 -1;
%                  1 1 -1 -1 -1 1;
%                -1 -1 -1 1 1 1];
%
%     [W,T] = TLUnet(x,d) will separate the training pairs described in x,
%     using 3 perceptrons, into 6 different classes. Perceptron 1 will
%     find the line separating points 1, 5, and 6 from points 2, 3, and
%     4. Perceptron 2 will find the line separating points 3, 4, and 5
%     from points 1, 2, and 6. Finally, perceptron 3 will separate points
%     1, 2, and 3 from points 4, 5, and 6. Since the problem is two-
%     dimensional, TLUnet also plots the separating lines on the current
%     figure. W will be a 2 by 3 matrix, where each row is the weight
%     vector of a single trained unit. T will be a vector with length of 3,
%     whose elements are the thresholds of the trained units.
%
% Created for:
%     Funwork 1
%     ECE 595C: Introduction to Biologically Inspired Engineering
%     Module 2: Neural Networks
%     Fall 2004
%     Prof. Stanislaw H. Zak
%     Purdue University
%     West Lafayette Campus
%
% Created by:
%     Steve Lamberson
%         lamberss@ecn.purdue.edu
%     Jon Nehrbass
%         jnehrbas@purdue.edu
%     Mike Skillen
%         skillen@purdue.edu
%
% Created on:
%     September 24, 2004
%
% Last Modified on:
%     September 25, 2004


function [W,T] = TLUnet(x,d)

% determine size of problem
[n,p] = size(x);
c = size(d,1);

% initialize output
T = zeros(c,1);
W = zeros(c,n);

% initialize variables needed for plotting
if n == 2
    pmin = min(min(x));
    pmax = max(max(x));
    X = zeros(1,100);
    Y = zeros(c,100);
end


for i = 1:c
    % train perceptron
    [w,t] = trainTLU(x,d(i,:));

    % assign solution to output variables
    W(i,:) = w';
    T(i) = t;

    % generate plot
    if n == 2
        if w(2) == 0
            Y(i,:) = linspace(pmin,pmax,100);
            X = (t-w(2)*Y(i,:))/w(1);
        else
            X = linspace(pmin,pmax,100);
            Y(i,:) = (t-w(1)*X)/w(2);
        end
        hold on;
        plot(X,Y(i,:));
    end
end

back to top

 

plotClass - a function used to plot points in different classes with different symbols

 

% plotClass Plots (x,y) pairs based on class
%
% hndl = plotClass(x,d) plots the (x,y) pairs described in
% 'x' according to their class described by 'd'. The function
% returns the plot handle in 'hndl'. The plot will also have
% the number of the point close to the symbol representing
% the point.
%
% Inputs:
%     x == 2 x p matrix, with row one containing x coordinates
%     and row two containing y coordinates for p points
% i    n cartesian space.

%
%     d == a vector of dimension p, whose elements state which 
%     class the corresponding point described in x belongs to
%
% Outputs:
%     hndl == the handle of the figure generated
%
% Examples:
%     Given:
%         x = [0 1 2; 0 1 2];
%         d = [1 2 2];
%
%     hndl = plotClass(x,d) will plot the point (0,0) using one
%     symbol (a circle), and will plot (1,1) and (2,2) using
%     a different symbol (a square). The point (0,0) will have
%     the number '1' by it, the point (1,1) will have '2' by it,
%     etc.
%
% Notes:
%     Only works if the number of classes <= 12. 
%
% Created for:
%     Funwork 1
%     ECE 595C: Introduction to Biologically Inspired Engineering
%     Module 2: Neural Networks
%     Fall 2004
%     Prof. Stanislaw H. Zak
%     Purdue University
%     West Lafayette Campus
%
% Created by:
%     Steve Lamberson
%         lamberss@ecn.purdue.edu
%     Jon Nehrbass
%         jnehrbas@purdue.edu
%     Mike Skillen
%         skillen@purdue.edu
%
% Created on:
%     September 24, 2004
%
% Last Modified on:
%     September 25, 2004


function hndl = plotClass(x,d)

% sort class vector
[m,k] = sort(d);
p = length(d);

% sort x vector to correspond with sorted class vector
for i = 1:p
    xn(:,i) = x(:,k(i));
end

% generate graph, using different point styles for different classes
g = -1;
h = 0;
R = ['ob';'sb';'db';'vb';'^b';'<b';'>b';'*b';'xb';'+b';'pb';'hb'];
hold on;
for i = 1:p
    if m(i) > g
        g = m(i);
        h = h + 1;
        r = R(h,:);
    end
    hndl = plot(xn(1,i),xn(2,i),r);
    text(xn(1,i)+0.05,xn(2,i),sprintf('%d',i));
end

 

back to top