An Introduction to Modern
Statistical Learning

Contents

• Preface
• 1 Introduction
• I Representation and Inference
  • 2 Directed Generative models
    • 2.1 Static models
      • 2.1.1 Mixture models and the GMM
      • 2.1.2 Jointly Gaussian models and factor analyzers
      • 2.1.3 Sparse coding
    • 2.2 Dynamical models
      • 2.2.1 The hidden Markov Model and the $\alpha$-$\gamma$ algorithm
      • 2.2.2 State-space models, the Kalman filter, and the RTS smoother
  • 3 Undirected Generative Models
    • 3.1 The exponential-family harmonium
    • 3.2 Deep belief networks
    • 3.3 The Helmholtz machine
    • 3.4 Recurrent EFHs
      • 3.4.1 The recurrent temporal RBM
      • 3.4.2 The recurrent EFH
    • 3.5 General inference algorithms
• II Learning
  • 4 A Mathematical Framework for Learning
    • 4.1 Learning as optimization
    • 4.2 Minimizing relative entropy and maximizing likelihood
  • 5 Learning Discriminative Models
    • 5.1 Supervised Learning
      • 5.1.1 Linear regression
      • 5.1.2 Generalized linear models
      • 5.1.3 Artificial neural networks
    • 5.2 Unsupervised learning
      • 5.2.1 “InfoMax” in deterministic, invertible models
  • 6 Learning Generative Models with Latent Variables
    • 6.1 Introduction
    • 6.2 Latent-variable density estimation
    • 6.3 Expectation-Maximization
      • 6.3.1 Derivation of EM
      • 6.3.2 Information-theoretic perspectives on EM
  • 7 Learning Invertible Generative Models
    • 7.1 The Gaussian mixture model and $K$-means
      • 7.1.1 $K$-means
    • 7.2 The hidden Markov model
    • 7.3 Factor analysis and principal-components analysis
      • 7.3.1 Principal-components analysis
    • 7.4 Linear-Gaussian state-space models
  • 8 Learning Non-Invertible Generative Models
    • 8.1 Gaussian recognition distributions and sparse coding
      • 8.1.1 Independent sources and Gaussian emissions: sparse coding and other models
      • 8.1.2 Independent-components analysis
    • 8.2 Variational Autoencoders
      • 8.2.1 Monte Carlo gradient estimators
      • 8.2.2 Examples
    • 8.3 Diffusion Models
      • 8.3.1 Gaussian diffusion models
    • 8.4 Variational inference
  • 9 Learning with Reparameterizations
    • 9.0.1 A duality between generative and discriminative learning
    • 9.1 InfoMax ICA, revisited
    • 9.2 Nonlinear independent-component estimation
  • 10 Learning Energy-Based Models
    • 10.1 Noise-Contrastive Objectives
      • 10.1.1 Noise-Contrastive Estimation
      • 10.1.2 InfoNCE
      • 10.1.3 “Local” NCE
    • 10.2 EFH-like models
      • 10.2.1 Learning with exponential-family harmoniums
      • 10.2.2 The method of contrastive divergence
      • 10.2.3 Learning with deep belief networks
      • 10.2.4 Dynamical models: the TRBM, RTRBM, and rEFH
  • A Bonus Material
    • A.1 Leave-one-out cross validation for linear regression in one step
  • B Mathematical Appendix
    • B.1 Matrix Calculus
      • B.1.1 Derivatives with respect to vectors
      • B.1.2 Derivatives with respect to matrices
      • B.1.3 More useful identities
    • B.2 Probability and Statistics
    • B.3 Matrix Identities
  • C A Review of Probabilistic Graphical Models
  • Bibliography