ECE695: Inference and Learning in Generative Models

Lecture Hours: 3 

Credits: 3

Area of Specialization: Computer Engineering

Catalog Description: Generative models are a powerful alternative to discriminative models that, when properly specified, estimate their parameters more efficiently and can generate samples from the distribution of their input data, but also can be used (like discriminative models) to infer features or labels from their inputs. However, the generative and inferential faculties typically come at each other's expense. This course will cover five different attempts at finessing this trade-off, and the resulting learning algorithms: exact inference in directed graphical models (EM algorithm); sampling-based methods; deterministic approximate inference (variational EM); RBM-like architectures (contrastive-divergence learning); and generative adversarial networks (adversarial training).

Required Text(s): None

Recommended Reference: Bishop, Christopher M. Pattern Recognition and Machine Learning. New York: Springer, 2006. Print. Information Science and Statistics. ISBN: 9780387310732

Lecture Outline:

  1. Preliminaries
  2. Supervised Learning of Generative Models
  3. Exact Inference in Directed Graphical Models
  4. Sampling-Based Inference in Directed Graphical Models
  5. Variational Inference
  6. The Exponential-Family Harmonium
  7. Generative Adversarial Networks
  8. Autoregressive models
  9. Current State of the Art

Assessments: Approximately 10 homework assignments, 2 exams, 1 project

Prerequisites: Linear algebra, calculus, and basic probability theory