4.2 The exponential-family harmonium

The joint distribution for undirected graphical models is the (normalized) product over all maximal cliques of the graph, which in this case are all the (hidden, visible) pairs. To make any progress beyond this, we impose an additional constraint, namely that the clique energies are bilinear in (some functions of) ${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{x}}$ and ${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{y}}$. I will write this bilinear function in a particular form that stays completely general but facilitates the interpretations we make next:

There are a few important things to note about this formulation. First, it clearly contains redundancies, at least under the assumption that the component functions are arbitrary. For example, the second and third terms are redundant with the final two. (Note that we are assuming $h$ and $g$ to be strictly positive.) The weight $w_{km}$ likewise adds no expressive power. However, below we remove these redundancies by committing to a particular form for the functions, but letting the parameters $w_{km}$, $d_{k}$, and $c_{m}$ be free (learnable).

Second, the bilinear form greatly restricts the set of energies expressible by the model. However, the EFH need not be restricted this way: The functions ${T}_{k}$ and ${U}_{m}$ could be replaced with vector-valued functions, and the first term of the energy with their dot product, in which case the set of energies is essentially unlimited (for long enough vectors). In fact, an obvious EFH can be written in this form, as we will see below. But since we will anyway derive a more generic model in the next section, we do not consider this generalization here.

Now, the marginal distribution over (e.g.) the visible units is

(For discrete ${\bm{\hat{X}}}$, replace the integral with a sum.) To simplify this expression, let us introduce the definitions

under which the marginal becomes

It is now possible to compute the posterior as the ratio of the joint (Eq. 4.1) to this marginal (Eq. 4.3):

Thus, the posterior takes the form of an exponential family. Now we see that our symbol choices for the functions introduced in the definitions Eq. 4.2 were felicitous, since they are precisely the natural parameters and the log-partition function. Notice that this is not merely a case of ${A}(\cdot)$ showing up in the right place in Eq. 4.4; according to our definition in Eq. 4.2, ${A}(\cdot)$ is precisely the log-partition function for this posterior distribution.

A similar progression yields the emission distribution. In particular, if we define

then we obtain

and

The foregoing derivation may look like mere algebraic legerdemain; it is important to see what we have and haven’t accomplished. The conditional distributions, Eqs. 4.4 and 4.7, still contain (log) normalizers which have not been computed explicitly. For generic functions ${T}_{k}({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}x_{k}})$, there is no guarantee that the integral in Eq. 4.2 (or a sum over discrete configurations) is tractable. On the other hand, our notatation suggests, what is true, that for simple functions (e.g., low-order polynomials), the integral (or sum) is not only tractable but known, corresponding to the partition function of some exponential-family distribution or other.

Let us consider the special case in which the random variables are supported on the set $\{0,1\}$. Up to reparameterization, this is the standard Bernoulli distribution. Under the standard parameterization, the sufficient statistics are each the identity function, ${T}_{k}({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}x_{k}})={\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}x_{k}}$, ${U}_{m}({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}y_{m}})={\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}y_{m}}$; and the base measures are unity, $h({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}x_{k}})=g({\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}y_{m}})=1$. Therefore the joint distribution is

Note that there are many exponential families with identity sufficient statistics and therefore with joint energy functions similar to the RBM’s in Eq. 4.8. However, they have different base measures, and therefore their joint energies will not be identical to the energy in Eq. 4.8.

In the previous section, we assumed a certain graphical model (Fig. 4.2), with a restriction on the form of the clique potentials, and derived as a consequence that the posterior and emission are exponential-family distributions. We now rederive EFHs from the other direction, starting with exponential-family emission and posterior, and working backwards to derive the joint distribution. We therefore define the emission and posterior to be

(Cf. Eqs. 4.4 and 4.7.) Note that these need not be the same exponential family; indeed, the several elements of (e.g.) ${\bm{\hat{X}}}$ need not even belong to the same one.

Now, the ratio of the conditionals is also the ratio of the marginals,

Therefore, it must factor entirely into pieces that refer to at most one of ${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{x}}$ or ${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{y}}$. The ratio contains only such factors, but the exponential term does not. This term will factor correctly if and only if

for some functions $\mu$ and $\nu$. It is easily verified that to satisfy Eq. 4.9, it is sufficient (although not strictly necessary) for the natural parameters to take the form

with a shared, albeit transposed, linear transformation $\mathbf{{W}}$. In this case, the marginal distributions become (up to the proportionality constants)

and the conditional distributions become

Multiplying a conditional by the appropriate marginal yields the joint distribution:

The resulting joint distribution (Eq. 4.11) certainly looks very similar to our previous starting point, Eq. 4.1, and it clearly has the same parameter gradient.²² 2 We take up the learning problem in Section 12.2. However, there is a subtle difference. We have not assumed the latent variables to be independent conditioned on the observations, or vice versa. That is, we have not assumed the graph of Fig. 4.2. This provides a significant advantage.

Intuitively, good latent variables should be marginally independent, otherwise there is something (their dependence) left to explain; and conditionally dependent, because they should complete to explain the observations. For example, on a page of (handwritten) algebra, the symbols $2$ and $z$ should be (perhaps) independent—finding a $z$ doesn’t change my expectation of finding a $2$. However, for any particular handwritten character on the page, $2$ and $z$ provide highly dependent (indeed, mutually exclusive) explanations. The traditional EFH cannot model such dependencies, because the posterior is conditionally independent, but the generalized EFH can—e.g., letting the latent vector be a one-hot representation of a categorical random variable, like character ID.