| |
- builtins.object
-
- ComputationalGraphPrimer
- Exp
class ComputationalGraphPrimer(builtins.object) |
|
ComputationalGraphPrimer(*args, **kwargs)
|
|
Methods defined here:
- __init__(self, *args, **kwargs)
- Initialize self. See help(type(self)) for accurate signature.
- backprop_and_update_params_multi_neuron_model(self, y_error, class_labels)
- First note that loop index variable 'back_layer_index' starts with the index of
the last layer. For the 3-layer example shown for 'forward', back_layer_index
starts with a value of 2, its next value is 1, and that's it.
Stochastic Gradient Gradient calls for the backpropagated loss to be averaged over
the samples in a batch. To explain how this averaging is carried out by the
backprop function, consider the last node on the example shown in the forward()
function above. Standing at the node, we look at the 'input' values stored in the
variable "input_vals". Assuming a batch size of 8, this will be list of
lists. Each of the inner lists will have two values for the two nodes in the
hidden layer. And there will be 8 of these for the 8 elements of the batch. We average
these values 'input vals' and store those in the variable "input_vals_avg". Next we
must carry out the same batch-based averaging for the partial derivatives stored in the
variable "deriv_sigmoid".
Pay attention to the variable 'vars_in_layer'. These stores the node variables in
the current layer during backpropagation. Since back_layer_index starts with a
value of 2, the variable 'vars_in_layer' will have just the single node for the
example shown for forward(). With respect to what is stored in vars_in_layer', the
variables stored in 'input_vars_to_layer' correspond to the input layer with
respect to the current layer.
- backprop_and_update_params_one_neuron_model(self, y_error, vals_for_input_vars, deriv_sigmoid)
- As should be evident from the syntax used in the following call to backprop function,
self.backprop_and_update_params_one_neuron_model( y_error_avg, data_tuple_avg, deriv_sigmoid_avg)
^^^ ^^^ ^^^
the values fed to the backprop function for its three arguments are averaged over the training
samples in the batch. This in keeping with the spirit of SGD that calls for averaging the
information retained in the forward propagation over the samples in a batch.
- calculate_loss(self, predicted_val, true_val)
- ######################################################################################################
###################################### Utility Functions ############################################
- display_network1(self)
- display_network2(self)
- Provides a fancier display of the network graph
- eval_expression(self, exp, vals_for_vars, vals_for_learnable_params, ind_vars=None)
- forward_prop_multi_neuron_model(self, data_tuples_in_batch)
- During forward propagation, we push each batch of the input data through the
network. In order to explain the logic of forward, consider the following network
layout in 4 nodes in the input layer, 2 nodes in the hidden layer, and 1 node in
the output layer.
input
x x = node
x x| | = sigmoid activation
x|
x x|
x
layer_0 layer_1 layer_2
In the code shown below, the expressions to evaluate for computing the
pre-activation values at a node are stored at the layer in which the nodes reside.
That is, the dictionary look-up "self.layer_exp_objects[layer_index]" returns the
Expression objects for which the left-side dependent variable is in the layer
pointed to layer_index. So the example shown above, "self.layer_exp_objects[1]"
will return two Expression objects, one for each of the two nodes in the second
layer of the network (that is, layer indexed 1).
The pre-activation values obtained by evaluating the expressions at each node are
then subject to Sigmoid activation, followed by the calculation of the partial
derivative of the output of the Sigmoid function with respect to its input.
In the forward, the values calculated for the nodes in each layer are stored in
the dictionary
self.forw_prop_vals_at_layers[ layer_index ]
and the gradients values calculated at the same nodes in the dictionary:
self.gradient_vals_for_layers[ layer_index ]
- forward_prop_one_neuron_model(self, data_tuples_in_batch)
- As the one-neuron model is characterized by a single expression, the main job of this function is
to evaluate that expression for each data tuple in the incoming batch. The resulting output is
fed into the sigmoid activation function and the partial derivative of the sigmoid with respect
to its input calculated.
- forward_propagate_one_input_sample_with_partial_deriv_calc(self, sample_index, input_vals_for_ind_vars)
- If you want to look at how the information flows in the DAG when you don't have to worry about
estimating the partial derivatives, see the method gen_gt_dataset(). As you will notice in the
implementation code for that method, there is nothing much to pushing the input values through
the nodes and the arcs of a computational graph if we are not concerned about estimating the
partial derivatives.
On the other hand, if you want to see how one might also estimate the partial derivatives as
during the forward flow of information in a computational graph, the forward_propagate...()
presented here is the method to examine. We first split the expression that the node
variable depends on into its constituent parts on the basis of '+' and '-' operators and
subsequently, for each part, we estimate the partial of the node variable with respect
to the variables and the learnable parameters in that part.
- gen_gt_dataset(self, vals_for_learnable_params={})
- This method illustrates that it is trivial to forward-propagate the information through
the computational graph if you are not concerned about estimating the partial derivatives
at the same time. This method is used to generate 'dataset_size' number of input/output
values for the computational graph for given values for the learnable parameters.
- gen_gt_dataset_with_activations(self, vals_for_learnable_params={})
- This method illustrates that it is trivial to forward-propagate the information through
the computational graph if you are not concerned about estimating the partial derivatives
at the same time. This method is used to generate 'dataset_size' number of input/output
values for the computational graph for given values for the learnable parameters.
- gen_training_data(self)
- parse_expressions(self)
- This method creates a DAG from a set of expressions that involve variables and learnable
parameters. The expressions are based on the assumption that a symbolic name that starts
with the letter 'x' is a variable, with all other symbolic names being learnable parameters.
The computational graph is represented by two dictionaries, 'depends_on' and 'leads_to'.
To illustrate the meaning of the dictionaries, something like "depends_on['xz']" would be
set to a list of all other variables whose outgoing arcs end in the node 'xz'. So
something like "depends_on['xz']" is best read as "node 'xz' depends on ...." where the
dots stand for the array of nodes that is the value of "depends_on['xz']". On the other
hand, the 'leads_to' dictionary has the opposite meaning. That is, something like
"leads_to['xz']" is set to the array of nodes at the ends of all the arcs that emanate
from 'xz'.
- parse_multi_layer_expressions(self)
- This method is a modification of the previous expression parser and meant specifically
for the case when a given set of expressions are supposed to define a multi-layer neural
network. The naming conventions for the variables, which designate the nodes in the layers
of the network, and the learnable parameters remain the same as in the previous function.
- plot_loss(self)
- run_training_loop_multi_neuron_model(self)
- ### Introduced in 1.0.5
######################################################################################################
######################################## multi neuron model ##########################################
- run_training_loop_one_neuron_model(self)
- ### Introduced in 1.0.5
######################################################################################################
######################################### one neuron model ###########################################
- run_training_with_torchnn(self, option)
- The value of the parameter 'option' must be either 'one_neuron' or 'multi_neuron'.
For either option, the number of input nodes is specified by the expressions specified in the
constructor of the class ComputationalGraphPrimer.
When the option value is 'one_neuron', we use the OneNeuronNet for the learning network and
when the option is 'multi_neuron' we use the MultiNeuronNet.
Assuming that the number of input nodes specified by the expressions is 4, the MultiNeuronNet
class creates the following network layout in which we have 2 nodes in the hidden layer and
one node for the final output:
input
x x = node
x x| | = ReLU activation
x|
x x|
x
layer_0 layer_1 layer_2
- train_on_all_data(self)
- The purpose of this method is to call forward_propagate_one_input_sample_with_partial_deriv_calc()
repeatedly on all input/output ground-truth training data pairs generated by the method
gen_gt_dataset(). The call to the forward_propagate...() method returns the predicted value
at the output nodes from the supplied values at the input nodes. The "train_on_all_data()"
method calculates the error associated with the predicted value. The call to
forward_propagate...() also returns the partial derivatives estimated by using the finite
difference method in the computational graph. Using the partial derivatives, the
"train_on_all_data()" backpropagates the loss to the interior nodes in the computational graph
and updates the values for the learnable parameters.
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
Data and other attributes defined here:
- AutogradCustomization = <class 'ComputationalGraphPrimer.ComputationalGraphPrimer.AutogradCustomization'>
- This class illustrates how you can add additional functionality of Autograd by
following the instructions posted at
https://pytorch.org/docs/stable/notes/extending.html
|
class Exp(builtins.object) |
|
Exp(exp, body, dependent_var, right_vars, right_params)
|
|
Methods defined here:
- __init__(self, exp, body, dependent_var, right_vars, right_params)
- Initialize self. See help(type(self)) for accurate signature.
Data descriptors defined here:
- __dict__
- dictionary for instance variables (if defined)
- __weakref__
- list of weak references to the object (if defined)
| |