9.2 Nonlinear independent-component estimation

We now expand our view to nonlinear, albeit still invertible, transformations [6, 40, 7, 24]. In particular, consider a “generative function” $\bm{c}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}},\bm{\theta})$ that consists of a series of invertible transformations. Once again, to emphasize that it is the inverse of a recognition or discriminative function, $\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\bm{\theta})$, we write $\bm{c}$ as $\bm{d}^{-1}$:

This change of variables is called a flow^††margin: flow [40]. Let us still assume a factorial prior, Eq. 9.2, and furthermore that it does not depend on any parameters. Since the transformations are (by assumption) invertible, the change-of-variables formula still applies. Therefore, Eq. 9.3 still holds, but the Jacobian determinant of composed functions becomes the product of the individual Jacobian determinants:

with Jacobians given by

(For the sake of writing derivatives, we have named the argument of the $l^{\text{th}}$ function $\bm{x}_{l}$. This makes $\bm{x}_{1}=\bm{y}$.) The functions induced by multiplying the initial distribution (the Jacobian determinant $\mathbf{J}_{0}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}})$ at left) by, in turn, the determinants of each of the ${L}$ Jacobians $\mathbf{J}_{l}$ at right, are “automatically” normalized and positive, and consequently valid probability distributions. This sequence is accordingly called a normalizing flow^††margin: normalizing flow [40].

Since the generative function $\bm{c}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}},\bm{\theta})=\bm{d}^{-1}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}},\bm{\theta})$ is invertible, we can certainly compute the arguments to the Jacobians. However, to keep the problem tractable, we also need to be able to compute the Jacobian determinants efficiently. Generically, this computation is cubic in the dimension of the data. This is intolerable, so we will generally limit the expressiveness of each $\bm{c}_{l}$ to achieve something more practical.

Perhaps the most obvious limitation is to require that the transformations be “volume preserving”; that is, to require that Jacobian determinants are always unity [6] This can be achieved, for example, by splitting a data vector into two parts, and requiring (1) that the flow at a particular step $l$ of only one of the parts may depend on the other (this ensures that the Jacobian is upper triangular); and (2) that the flows of both parts depend on their previous values only through an identity transformation (this ensures that the two diagonal blocks of the Jacobian are identity matrices). In equations,

… [[multiple layers of this]]

Now our loss is, as usual, the relative entropy. With the “recognition functions” $\bm{d}_{l}$ of Eq. 9.6 and the corresponding model density of Eq. 9.7, the relative entropy becomes

(For concision, the Jacobians are written as a function directly of ${\bm{Y}}$.)