BME 69500DL

(cross-listed as ECE 695)

Deep Learning

by

Avi Kak   and   Charles Bouman


Spring 2021

PREREQUISITES FOR THIS CLASS

In order to do well in this class, you must be proficient in programming with Python. Beyond that, all that is required for you to enroll in this class is that you be a graduate student in engineering, computer science, quantitative psychology, mathematics, etc.



Video Lectures: link
Piazza: link


Week 1   Tuesday,  Jan 19: Course Intro: [Slides]

  Thursday, Jan 21: (Bouman) [slides] Intro to ML: What is machine learning?; single layer
  neural networks; the MSE loss function

Week 2   Tuesday,  Jan 26: (Kak) [Slides] Python OO for DL    [updated: Jan 26, 2021]

  Thursday, Jan 28: (Bouman) [Slides] Intro to ML: Gradient descent optimization;
Week 3   Tuesday,  Feb 2: (Kak) [slides] Torchvision and Random Tensors    [updated: May 5, 2021]

  Thursday, Feb 4: (Bouman) [slides] Intro to ML: Tensors; GD for single layer NNs


  (Some of the Torchvision related material during this week will be illustrated with the functionality built into the
  RegionProposalGenerator module that you can access by clicking here.)
Week 4   Tuesday,  Feb 9:  (Kak) [Slides] Autograd: For Automatic Differentiation and for
  Auto-Construction of Computational Graphs    [updated: March 9, 2021]

  Thursday, Feb 11:  (Bouman) [Slides] Intro to ML: Optimization of deep functions; GD
  on acyclic graph structures; general loss functions


  (Several of the key ideas used for automatic differentiation in Autograd will be explained with the help of the
  ComputationalGraphPrimer module that you can access by clicking here.)
Week 5   Tuesday,   Feb 16: (Kak) [slides] A First Introduction to Torch.nn for Designing Deep Networks
  and to DLStudio for Experimenting with Them    [updated: Feb 23, 2021]

  Thursday, Feb 18:: (Bouman) [slides] Intro to NNs: Convolutional NNs; adjoint gradient for CNNs


  (We will talk about torch.nn with the help of the new DLStudio module that you can access by clicking here.)
Week 6   Tuesday,  Feb 23: (Kak) [slides] Demystifying the Convolutions in PyTorch    [updated: March 4, 2021]

  Thursday, Feb 25: (Bouman) [slides] Intro to ML: Probability and estimation; Frequentist
  and Bayesian estimation; the bias variance tradeoff
Week 7   Tuesday,  March 2: (Kak) [slides] Using Skip Connections and Batch Normalization to Mitigate
  the Problem of Vanishing Gradients in Deep Networks.    [updated: March 9, 2021]

  Thursday, March 4: (Bouman) [Slides] Training and Generalization

  (The material related to skip connections will be explained with the help an in-class demo based on the inner class SkipConnections
  of the DLStudio module that you can access by clicking here.)
Week 8   Tuesday,  March 9: (Kak) [Slides] Object Detection and Localization with Deep Networks
  [updated: March 24, 2021]

  Thursday, March 11: Mid-Term Test 1 ( exams )


  (The material related to object detection and localization will be explained with the help an in-class demo based on the inner class
  DetectAndLocalize of the DLStudio module that you can access by clicking here.)
Week 9   Tuesday,  March 16:   (Kak) [slides] Multi-Instance Object Detection -- Anchor Boxes and Region
  Proposals    [updated: April 9, 2021]

  Thursday, March 18:   (Bouman) [slides] Intro to Optimization: stochastic gradient descent;
  momenturn; ADAM optimization

  (You must solve the problem of multi-instance object detection and localization when an image is allowed to contain multiple objects
  of interest. In such cases, the input/output relationship for a neural network is made complicated by the presences of multiple bounding
  boxes and multiple class labels in the same image. This problem has been solved with the help of region proposals and anchor boxes.
  The goal of this lecture is to introduce you to these concepts. My code for explaining these ideas is in Version 2.0.1 of my
  RegionProposalGenerator module that you can access by clicking here.)
Week 11   Tuesday,  March 23: (Kak) [slides] Encoder-Decoder Architectures for Semantic Segmentation of
  Images    [updated: April 12, 2021]

  Thursday, March 25: (Bouman) [slides] Widely Used DL Techniques: Vanishing gradients; Batch
  normalization;


  (The material related to semantic segmentation is based on the nUnet network which is my implementation of the Unet. You will find the
  code for mUnet in my DLStudio module that you can access by clicking here.)
Week 12   Tuesday,  March 30: (Kak) [Slides] Generative Adversarial Networks for Data Modeling
  [updated: May 1, 2021]

  Thursday, April 1: (Bouman) [slides] Recurrent Neural Networks: LSTM; GRU;


  (The lecture by Kak on Adversarial Learning for data modeling will be explained with the help of demos based on the code in
  DLStudio's AdversarialNetworks class that you can access by clicking here.)
Week 13   Tuesday,  April 6: (Kak) [Slides] Recurrent Neural Networks for Text Classification     [updated: May 13, 2021]

  Thursday, April 8: (Bouman) [slides] Unsupervised Training: Autoencoders; Self and Zero-shot
  training


  (The material related to text classification is based on the TEXTnet, TEXTnetOrder2, and GRUnet networks of the DLStudio
   that you can access by clicking here.)
Week 14   Tuesday,  April 13: (Kak) [Slides] Word Embeddings and Sequence-to-Sequence Learning
  [updated: April 21, 2021]

  Thursday, April 15: (Bouman) [slides] Adversarial Learning: Nash Equilibrium; Zero-sum games;
  and GANs;
Week 15   Tuesday,  April 20: (Bouman) Generative Adversarial Networks (GAN): GANs; Conditional GANs;
   Wasserstein distance

  Thursday, April 22: Mid-Term Test 2 ( exams )
Week 16   Tuesday,  April 27: (Kak) [Slides] Reinforcement Learning with Discrete and Continuous State
  Spaces

  Thursday, April 29: (Bouman) Reinforcement learning


Links to documentation pages you will be visiting frequently:

- DLStudio    [updated: April 12, 2021]
- ComputationalGraphPrimer    [updated: February 22, 2021]
- RegionProposalGenerator    [updated: April 7, 2021]
- Master documentation page for PyTorch
- A direct link to the torch.nn module
- Torchvision Datasets
- Master documentation page for Torchvision
- A direct link to Torchvision Transforms
- Master documentation page for Torchtext
- A useful summary of many of the most basic operations on PyTorch Tensors
- The homepage for CIFAR-10 and CIFAR-100 image datasets

Recommended Books:

- https://www.manning.com/books/grokking-deep-learning
- Deep Learning
- Neural Networks and Deep Learning
- Data Science from Scratch
- Python Machine Learning
- http://machinelearningmastery.com/deep-learning-courses/
- Pytorch tutorials

Recommended Supplementary Course Material:

- CMU deep learning
- Stanford Class by FeiFei and Karpathy. For videos
- Udacity Deep Learning nano-degree

Valid HTML 4.01 Transitional Valid CSS!