Chapter 8 Learning Non-Invertible Generative Models

To widen our view of target models for EM, let us drop the very restrictive assumption that the generative model can be inverted, i.e. that ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{x}}{}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{};\bm{\theta}}\right)}$ can be computed. It will therefore no longer be possible to tighten the bound completely in the E step. At this price, however, we have bought the ability to work with much more complicated generative models. The form of the recognition distribution will now be fixed, not by the generative posterior, but by our choice.