6.2 Latent-variable density estimation

Starting with the gradient of a generic marginal cross entropy, we move the derivative into the expection, “anti-marginalize” to restore the latent variables, and then rearrange terms:

where in the final line we have combined the data marginal and the model posterior into a single hybrid joint distribution^††margin: hybrid joint distribution , ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}$. This looks promising. It almost says that the gradient of the marginal (or “incomplete”) cross entropy is the same as the gradient of a joint (or “complete”) cross entropy. But the derivative cannot pass outside the expectation, because the hybrid joint distribution depends, like the model distribution, on the parameters $\bm{\theta}$.

To see the implications of this dependence on the parameters, we return to our workhorse example, the Gaussian mixture model. Inserting Eq. 6.2 into the final line of Eq. 6.3 shows that

The gradient with respect to (e.g.) $\bm{\mu}_{k}$ is therefore

This formula looks elegant only if we forget that $\bm{\mu}_{k}$ is on both sides. The expectations under the model posterior, ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}$, involve $\bm{\mu}_{k}$—indeed, they involve all of the parameters (recall Eq. 2.6)! This is reminiscent of the problem with the direct optimization of the marginal cross entropy, Eq. 6.1, so it seems perhaps we have made no progress.