Chapter 2 Exponential Families

So-called exponential families are among the most common probability distributions: Gaussian, Poisson, multinomial (including the special cases of Bernoulli, binomial, and categorical), gamma, Laplace, etc. They have several attractive properties, which we explore in this chapter. In modern practice, perhaps the most conspicuous is that they have closed-form relative entropies, which makes them good candidates for statistical models. Consequently, exponential families provide many of the loss functions for neural networks.

Here we begin with a different attractive property, namely that they are in some sense the least restrictive distributions; that is, given some set of constraints, they are the most “agnostic” distributions. In fact, we derive exponential families from this perspective, and then go on to prove other useful properties.