12.1 Direct minimization

Samples generated with Langevin dynamics can be used to approximate the expectation under the model. For simple EBMs, this works. However, for any sufficiently complex model, it has been observed that the burn-in of the Markov chain requires an intolerable number of Langevin steps—on the order of tens of thousands [51, 50]! Somewhat remarkably, training such EBMs under only 100 or so steps of Langevin still yields a good generator of (e.g.) images—so long as, at test time, a similar number of steps of Langevin is used. If more steps are taken, the model generates images closer to its own preferences—which turn out to be highly saturated (bad) images. Although these models are in some sense EBMs, they do not assign the EBM probability (Eq. 12.1) to data, nor are they trained according to the proper training rule.

To circumvent in a single move both the long burn-in time and the high variance of the energy-gradient estimator, Hinton proposed an alternative that appears, at first glance, to be very crude [27]: initializing the Markov chain at data, and then taking only a small number of steps. This procedure has a few clear merits. Late in training, when the model distribution resembles the data distribution, data initialization could conceivably shorten the burn-in period, since the initial sample is likely already to be in the “thick” part of the model distribution from which we hope to draw samples (in contrast to an arbitrary initial vector). Early in training, data samples are unlikely to have high probability under the model. But even here data initialization offers something, so long as the same data are used in both averages in Eq. 12.2. In particular, it lowers the variance of our estimate of the loss gradient:

If only a few ($n$) steps are taken down the Markov chain, the resulting sample ${\bm{\hat{Y}}}^{n}_{m}$ will be correlated with its initialization—to wit, ${\bm{Y}}^{0}_{m}$. The gradients evaluated at these samples will likewise covary, and therefore the covariance term above will be positive, reducing the overall variance of the estimator [1]. Under standard MCMC, in contrast, the covariance term vanishes.

To emphasize that the “model” and data samples are from the same Markov chain, I have superscripted them both with the respective number of steps down that chain ($n$ and 0). For the same reason, we should also nest the model-dependent average inside the data average in the proposed gradient estimator:

Likewise, I will also use $p^{0}$, $p^{n}$, and $p^{\infty}$ for the data, $n$-step, and model distributions (respectively). (Recall that after infinite steps down the chain, samples will be drawn from the model distribution.)

The heuristic arguments for Eq. 12.3 are compelling³³ 3 At least, in conjunction with the empirical success that CD enjoyed in its first decade, approximately 2000–2010., but it would be useful to justify it more rigorously. To that end, we work backwards: what loss function might yield this gradient? Hinton proposed [25] a “contrastive divergence,” i.e., the difference or contrast between the original relative entropy and a new one:

We note two important facts about this objective:

• •
  ​

  The contrastive divergence is non-negative: The $n$-step distribution $p^{n}$ is “trapped” between $p^{0}$ and $p^{\infty}$. That is, MCMC sampling brings a distribution closer to $p^{\infty}$, which will consequently diverge less from “later” than “earlier” distributions. So the second term is always less then the first, and their difference never negative.

• •
  ​

  The contrastive divergence is zero at precisely the same points as the first term—i.e., the standard relative-entropy loss (with the gradient given by Eq. 12.2). By the previous argument, Eq. 12.4 can be zero only when the divergences are equal, i.e. when $p^{n}\approx p^{0}$. But this implies that $p^{0}$ is not altered by the first $n$ steps of sampling; that is, it is an equilibrium distribution of the Markov chain. Hence, $p^{0}\approx p^{\infty}$, at which point the standard relative entropy also vanishes.⁴⁴ 4 We might take this as a specific instance of a more general procedure: if the gradient of a function is difficult, replace it with a different function with the same minimum but an easier gradient.

Now we would like to show that the gradient of Eq. 12.4 is equal to the right-hand side of Eq. 12.3. We have already worked out the gradient of the first relative entropy, Eq. 12.2, but we break the computation into steps to reuse portions for the derivative of the second divergence. For brevity, we suppress the arguments:

On the third line we have substituted for the fourth term by exploiting the analogy with the second term (which we worked out in Eq. 12.2 above). In fine, the gradient of the contrastive divergence, Eq. 12.4, is equal to the desired update, plus one extra term.⁵⁵ 5 Note that, although I haven’t made it explicit, the sample average under ${\bm{\hat{Y}}}^{n}$ is implicitly nested within the sample average under the data ${\bm{Y}}^{0}$.

Is this close enough? Hinton concluded from extensive simulation that the third term is small in practice, and that in any case, including it rarely alters the gradient by more than 90 degrees (i.e., point in the wrong direction) [25].⁶⁶ 6 A recent attempt to model the third term approximately actually initializes its Markov chains at noise, rather than data [13], and is therefore inconclusive. Empirically, the method worked well in its era (c. 2000–2010). Furthermore, the relevance of this term can be decreased by increasing $n$, since for large enough $n$, the objective approaches the standard relative entropy—although this must be balanced against the introducton of estimator variance, discussed above.