5.2 Unsupervised learning

To motivate the objective, consider the deterministic input-output relationship

Here the “final” output $\bm{\check{z}}$ is computed by passing each element of the vector $\bm{\check{x}}$ through some sort of invertible, element-wise “squashing” function (cf. a layer of a neural network, Eq. 5.17), i.e. a monotonic function from the reals to some bounded interval on the real line:

We will justify the need for such functions below. Now, when there are fewer outputs than inputs, $\dim\mathopen{}\mathclose{{}\left(\bm{\check{z}}}\right)<\dim\mathopen{}\mathclose{{}\left(\bm{y}}\right)$, changing $\bm{\theta}$ so as to maximize mutual information between inputs and outputs will yield a function $\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\bm{\theta})$ that recodes its inputs more efficiently. However, we begin with a seemingly more näive, but mathematically tractable, alternative: Let the number of outputs equal the number inputs, $\dim\mathopen{}\mathclose{{}\left(\bm{\check{z}}}\right)=\dim\mathopen{}\mathclose{{}\left(\bm{y}}\right)$, and furthermore let $\bm{d}$ be invertible (in its first argument), so that $\bm{y}=\bm{d}^{-1}(\bm{\psi}^{-1}(\bm{\check{z}}),\bm{\theta})$. This may seem somewhat perverse, since invertible transformations are automatically information preserving. However, recall that mutual information nevertheless depends on $\bm{\theta}$:

That is, although the conditional entropy is zero for all values of $\bm{\theta}$ (under the assumption that $\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\bm{\theta})$ doesn’t lose its invertibility), the output entropy still depends on $\bm{\theta}$. In fine, for invertible $\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\bm{\theta})$, maximizing mutual information amounts to maximizing output entropy.⁸⁸ 8 If there were noise in the map from ${\bm{Y}}$ to ${\bm{\check{Z}}}$, this would add an extra term to the gradient, and the goal of maximizing output entropy would have to be balanced against noise-proofing the transmission. Since a vector random variable ${\bm{\check{Z}}}$ is maximally entropic only if its components are statistically independent, this suggests that the “InfoMax” criterion can underwrite a form of independent-components analysis^††margin: independent-components analysis . That is, maximizing input-output mutual information or output entropy will “unmix” the inputs ${\bm{Y}}$ into their independent components.

Now we can say why the squashing function is necessary. If the input-output map were unbounded, the output entropy could be driven arbitrarily large. For concreteness, and without loss of generality, we can let $\psi_{k}$ be an increasing function, with its range the interval $[0,1]$. Now let us try to re-express the objective in Eq. 5.27. More precisely, we consider its negation, $-\text{H}{\mathopen{}\mathclose{{}\left[{\bm{\check{Z}}};\bm{\theta}}\right]}$, for consistency with the standard objectives of this book, which are losses. First we note that, although we have not specified a distribution for ${\bm{\check{Z}}}$, ${\check{p}_{{\bm{\check{Z}}}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\check{z}}}\right)}$ is inherited directly from the data distribution ${p_{{\bm{Y}}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}}\right)}$ via the deterministic relationship in Eq. 5.25. In particular, since the relationship is (by assumption) invertible, the two distributions are related by the standard change-of-variables formula, which we apply on the third line:

The last line follows because the squashing functions are assumed to act element-wise, so the determinant is just the product of the diagonal elements. Evidently, maximizing mutual information through an invertible transformation is identical to minimizing our standard loss, the relative entropy between data and model distributions, with the latter equal to the Jacobian determinant of that transformation:

The assimilation to density estimation can be completed by reinterpreting Eq. 5.25 as defining (via its inverse) the emission density of a generative model [3]. We defer this reinterpretion until Chapter 6, when we take up learning in generative models in earnest.