9.2 Nonlinear independent-component estimation

We now expand our view to nonlinear, albeit still invertible, transformations [6, 40, 7, 24]. In particular, consider a “generative function” $\bm{c}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb%}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}%},\bm{\theta})$ that consists of a series of invertible transformations. Once again, to emphasize that it is the inverse of a recognition or discriminative function, $\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb%}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\bm{%\theta})$ , we write $\bm{c}$ as $\bm{d}^{-1}$ :

\bm{\hat{y}}=\bm{c}(\bm{\hat{x}},\bm{\theta})=\bm{d}^{-1}(\bm{\hat{x}},\bm{%\theta})=\bm{d}_{1}^{-1}\circ\bm{d}_{2}^{-1}\cdots\bm{d}_{{L}-1}^{-1}\circ\bm{%d}_{{L}}^{-1}(\bm{\hat{x}},\bm{\theta}).

(9.6)

This change of variables is called a flow^†^†margin: flow [40]. Let us still assume a factorial prior, Eq. 9.2, and furthermore that it does not depend on any parameters. Since the transformations are (by assumption) invertible, the change-of-variables formula still applies. Therefore, Eq. 9.3 still holds, but the Jacobian determinant of composed functions becomes the product of the individual Jacobian determinants:

{\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}%\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5%}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{};\bm{\theta}}\right)}=\mathopen{}%\mathclose{{}\left\lvert\frac{\partial{\bm{\psi}}}{\partial{\bm{\hat{x}}}^{%\text{T}}}\mathopen{}\mathclose{{}\left(\bm{d}(\leavevmode\color[rgb]{.5,.5,.5%}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.%5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta});\bm{\theta}}\right)}%\right\rvert\mathopen{}\mathclose{{}\left\lvert\frac{\partial{\bm{d}}}{%\partial{\bm{y}}^{\text{T}}}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[%named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta})}\right\rvert\\=\prod_{l=0}^{{L}}\mathopen{}\mathclose{{}\left\lvert\mathbf{J}_{l}(%\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{%.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}})%}\right\rvert,

(9.7)

with Jacobians given by

\displaystyle\mathbf{J}_{0}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named%]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{y}})=\frac{\partial{\bm{\psi}}}{\partial{%\bm{\hat{x}}}^{\text{T}}}\mathopen{}\mathclose{{}\left(\bm{d}_{{L}}\circ\cdots%\circ\bm{d}_{1}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{%pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta})}\right),

\displaystyle\mathbf{J}_{l}=\frac{\partial{\bm{d}_{l}}}{\partial{\bm{x}_{l}}^{%\text{T}}}\mathopen{}\mathclose{{}\left(\bm{d}_{l-1}\circ\cdots\circ\bm{d}_{1}%(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{%.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},%\bm{\theta})}\right).

(For the sake of writing derivatives, we have named the argument of the $l^{\text{th}}$ function $\bm{x}_{l}$ . This makes $\bm{x}_{1}=\bm{y}$ .) The functions induced by multiplying the initial distribution (the Jacobian determinant $\mathbf{J}_{0}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{%pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{y}})$ at left) by, in turn, the determinants of each of the ${L}$ Jacobians $\mathbf{J}_{l}$ at right, are “automatically” normalized and positive, and consequently valid probability distributions. This sequence is accordingly called a normalizing flow^†^†margin: normalizing flow [40].

Since the generative function $\bm{c}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb%}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}%},\bm{\theta})=\bm{d}^{-1}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]%{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{x}},\bm{\theta})$ is invertible, we can certainly compute the arguments to the Jacobians. However, to keep the problem tractable, we also need to be able to compute the Jacobian determinants efficiently. Generically, this computation is cubic in the dimension of the data. This is intolerable, so we will generally limit the expressiveness of each $\bm{c}_{l}$ to achieve something more practical.

Perhaps the most obvious limitation is to require that the transformations be “volume preserving”; that is, to require that Jacobian determinants are always unity [6] This can be achieved, for example, by splitting a data vector into two parts, and requiring (1) that the flow at a particular step $l$ of only one of the parts may depend on the other (this ensures that the Jacobian is upper triangular); and (2) that the flows of both parts depend on their previous values only through an identity transformation (this ensures that the two diagonal blocks of the Jacobian are identity matrices). In equations,

\begin{bmatrix}\bm{x}^{a}_{l+1}\\\bm{x}^{b}_{l+1}\end{bmatrix}=\begin{bmatrix}\bm{x}^{a}_{l}\\\bm{x}^{b}_{l}+\bm{m}(\bm{x}^{a}_{l},\bm{\theta})\end{bmatrix}\implies\frac{%\partial{\bm{d}_{l}}}{\partial{\bm{x}_{l}}^{\text{T}}}=\begin{bmatrix}\mathbf{%I}&\mathbf{0}\\\frac{\partial{\bm{m}}}{\partial{\bm{x}^{a}_{l}}^{\text{T}}}&\mathbf{I}\end{%bmatrix}\implies\mathopen{}\mathclose{{}\left\lvert\frac{\partial{\bm{d}_{l}}}%{\partial{\bm{x}_{l}}^{\text{T}}}}\right\rvert=1.

… [[multiple layers of this]]

Now our loss is, as usual, the relative entropy. With the “recognition functions” $\bm{d}_{l}$ of Eq. 9.6 and the corresponding model density of Eq. 9.7, the relative entropy becomes

\operatorname*{\text{D}_{\text{KL}}}\mathopen{}\mathclose{{}\left\{{p\mathopen%{}\mathclose{{}\left({\bm{Y}}}\right)}\middle\|\prod_{l=0}^{{L}}\mathopen{}%\mathclose{{}\left\lvert\mathbf{J}_{l}({\bm{Y}})}\right\rvert}\right\}=\mathbb%{E}_{{\bm{Y}}{}}{\mathopen{}\mathclose{{}\left[\log{p\mathopen{}\mathclose{{}%\left({\bm{Y}}}\right)}-\sum_{k=1}^{{K}}\log\mathopen{}\mathclose{{}\left(%\frac{\partial{\psi_{k}}}{\partial{\hat{x}_{k}}}({\bm{Y}},\bm{\theta})}\right)%-\sum_{l=1}^{{L}}\log\mathopen{}\mathclose{{}\left\lvert\frac{\partial{\bm{d}_%{l}}}{\partial{\bm{x}_{l}}^{\text{T}}}({\bm{Y}},\bm{\theta})}\right\rvert}%\right]}.

(9.8)

(For concision, the Jacobians are written as a function directly of ${\bm{Y}}$ .)

The discriminative dual.

The model defined by Eq. 9.7, along with the loss in Eq. 9.8, has been called “nonlinear independent component analysis” (NICE) [6]. To see if the name is apposite, we employ our discriminative/generative duality, reinterpreting the minimization of the relative-entropy in Eq. 9.8 as a maximization of mutual information between the data, ${\bm{Y}}$ , and a random variable ${\bm{\hat{Z}}}$ defined by the (reverse) flow in Eq. 9.6, $\bm{d}$ , followed by (elementwise) transformation by the CDF of the prior. Can this still be thought of as an “unmixing” operation, as in InfoMax ICA? The question is acute particularly in the case where the prior is chosen to be normal, since (as we have just seen) ICA reduces to whitening in such circumstances.

In this case, the generative marginal given by the normalizing flow, Eq. 9.7, becomes

{\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}%\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5%}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{};\bm{\theta}}\right)}=\mathcal{N}%\mathopen{}\mathclose{{}\left(\bm{d}(\leavevmode\color[rgb]{.5,.5,.5}%\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5%}\pgfsys@color@gray@fill{.5}\bm{\hat{y}};\bm{\theta});\bm{0},\>\mathbf{I}}%\right)\mathopen{}\mathclose{{}\left\lvert\frac{\partial{\bm{d}}}{\partial{\bm%{y}}^{\text{T}}}(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{%pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}%\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta})}\right\rvert.

Despite the appearance of a normal distribution in this expression, this marginal distribution is certainly not normal—even though the generative prior, ${\hat{p}_{{\bm{\hat{X}}}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{%.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}%\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}}{};\bm{%\theta}}\right)}$ , is. So fitting this ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}%\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5%}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{};\bm{\theta}}\right)}$ to the data will not in general merely fit their second-order statistics.

[[Connection to HMC]]