10.1 Noise-Contrastive Objectives

The cross-entropy loss is the negative log of the posterior distribution (Eq. 10.2), averaged under the data (Eq. 10.1):

This is evidently a discriminative problem, but with a twist. The canonical generative approach to binary classification is to model the generative distribution ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x};\bm{\theta}}\right)}{\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x};\bm{\theta}}\right)}$ (like NCE); acquire the parameters by minimizing the joint cross entropy $\text{H}_{p\hat{p}}{\mathopen{}\mathclose{{}\left[{X},{\bm{Y}}}\right]}$ (unlike NCE); and then invert to ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}};\bm{\theta}}\right)}$ with Bayes’ rule. For Gaussian mixtures, this is known as linear/quadratic discriminant analysis (depending on whether the covariance is the same/different across classes). The canonical discriminative approach to binary classification is to model ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}};\bm{\theta}}\right)}$ directly (unlike NCE); and then minimize the cross entropy $\text{H}_{p\hat{p}}{\mathopen{}\mathclose{{}\left[{X}|{\bm{Y}}}\right]}$ (like NCE). This is logistic regression. NCE mixes both methods: it models the generative distribution ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x};\bm{\theta}}\right)}{\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\hat{x};\bm{\theta}}\right)}$, but first inverts with Bayes rule, and finally minimizes the discriminative cross entropy $\text{H}_{p\hat{p}}{\mathopen{}\mathclose{{}\left[{X}|{\bm{Y}}}\right]}$. In the classic case of the mixture of two Gaussians/binary classification, this would amount to learning the two (mean, covariance) pairs by minimizing the posterior cross entropy—as opposed to learning these parameters by minimizing the joint cross entropy (generative), or learning a separating hyperplane by minimizing the posterior cross entropy (discriminative).

[[Nice properties of the estimator….]]

Suppose that the observation $\bm{y}$ is not a single sample but a collection of ${K}$ “examples,” $(\bm{y}^{1},\ldots,\bm{y}^{{K}}$), precisely one of which is not noise. Then the goal is not to determine whether or not the sample is noise, but rather to determine which of the examples is noise. This means that rather than use the model-noise energy difference (Eq. 10.3) directly to assign the example to the positive or negative class, as in NCE, we will compare ${K}$ energy differences to each other (with the softmax function).

In this setup, the latent variable is categorical (conceived as a one-hot vector ${\bm{\hat{X}}}$) rather than Bernoulli, and the data distribution is:

Again we have set the prior uniform, since we have no reason to make any one of the elements more or less likely to be noise than any other. We emphasize that this is not a mixture model: a single sample contains ${K}$ “examples”: one positive, and ${K}-1$ negative.

The generative model takes the same form, with the model distribution taking the place of the data marginal. Writing it in terms of energies, we obtain

Again we ignore the fact that the emission is unnormalized and simply compute a (normalized) posterior distribution with Bayes’ rule

that is, the $i^{\text{th}}$ output of the softmax function. Eq. 10.6 is evidently a kind of generalization of Eq. 10.2.¹¹ 1 However, note that the multi-example version of NCE does not quite reduce to the single-example case even when ${K}=2$. Eq. 10.2 can indeed be re-written with a softmax as in Eq. 10.6, with the first argument equal to $h\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{i},\bm{\theta}}\right)$ and the second equal to 0. The latter reflects our indifferent prior, which provides no additional information. In the two-example version of the generalization under discussion, on the other hand, the second argument encodes the relative probability of the second example being data or noise. In short, deciding which of two samples is “real” is easier than deciding whether or not a single sample is. Putting this together with the data distribution, we can write the conditional cross entropy as

In the final line, we are selecting only that output of the softmax function that corresponds to the actual positive sample (whose index will of course differ from trial to trial).

We can shed light on the role played by the negative examples by considering them separately from the positive example in the posterior probability of a positive example:

Now notice that the negative-sample terms sum approximately to a constant:

The approximate equality becomes more exact as the number of negative examples increases. (And technically, the final equality requires the model and noise distributions to have the same support.) Eq. 10.8 says that, if we had in hand an expression for the normalizer, we could do without the negative samples altogether—they drop out of the loss function. Indeed, the loss now becomes

where the final line follows for large ${K}$.²² 2 The authors of the original paper [51] interpret this approximation as a lower bound when the model distribution matches the data distribution. Presumably the idea is that, for a very good model ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}};\bm{\theta}}\right)}$, the noise-to-model ratio will usually be less than one when evaluated on positive examples. Therefore, the neglected $+1$ will dominate the neglected $-1$. It would take more work to prove this. This makes sense: the whole point of using negative examples was to force unnormalized models to learn the correct normalization. Since we want to use models for which computing $Z(\bm{\theta})$ is intractable, we will not use Eq. 10.9 as our objective—but we will use it below to prove that optimizing the multi-example NCE loss (Eq. 10.7) increases mutual information in a certain setting.

We have assumed up to this point that the source of our “noise” samples is also an evaluatable expression for the probability of samples. What if we have only samples from the noise distribution? Can we still learn a model of the positive data?

One obvious solution is to learn a model for the negative as well as the positive samples; for example, to build a parameterized model for the noise energy, $E_{\text{n}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{-},\bm{\theta}}\right)$, and use it in the generative model. But if we wanted to get a normalized version of the model energy, $E\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{+},\bm{\theta}}\right)$, we would have to be able to get or to know the normalizer for this noise energy, $E_{\text{n}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{-},\bm{\theta}}\right)$, which is troubling. However, as noted at the outset, getting a probability model for the data, normalized or unnormalized, is not the goal of InfoNCE. So instead we will directly model the energy difference, i.e. the left-hand rather than right-hand side of Eq. 10.3. Rather than asking for the probabilities of an example $\bm{y}^{k}$ under the two models (positive and negative), we are asking for its relative probability.

One subtlety with modeling $h\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta}}\right)$ directly is that we are still at liberty to interpret this as fitting $E\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{+},\bm{\theta}}\right)$ only, that is to say, not fitting the noise energy, $E_{\text{n}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}^{-}}\right)$. In other words, we can attribute any error in $h\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\bm{\theta}}\right)$ to an error in $E\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}^{+},\bm{\theta}}\right)$ rather than $E_{\text{n}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}^{-}}\right)$. Consequently, the denominator in Eq. 10.8 can still be interpreted as ${p_{\text{n}}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}^{-}}\right)}$, and the equation still goes through. We will use it below.

In the third departure from the original NCE, the InfoNCE method proposes to learn to model a conditional distribution ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}\middle|\bm{z}}\right)}$, given some auxiliary variable $\bm{z}$. More importantly, we use the data marginal, ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}}\right)}$, as the noise distribution. Thus, ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}}\right)}$ has switched roles, from data to “noise,” or more felicitously from the source of positive to negative examples. The intuition behind this choice of distributions is that a model that can distinguish them must be able to extract information about ${\bm{Y}}$ from ${\bm{Z}}$.

This can be made precise in the language of information theory. However, the information we aim to increase is not precisely a mutual information, neither between ${\bm{Z}}$ and ${\bm{\hat{Y}}}$ nor anything else, because it depends on two different distributions: the model, ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{z};\bm{\theta}}\right)}$, and the data, $p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y},\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{z}}\right)$. The standard mutual information can of course be written as

The information quantity we are interested in retains the marginal entropy over ${\bm{Y}}$, since the model has no effect on it (see previous section), but replaces the conditional entropy with the conditional cross entropy:

We might accordingly call this (for want of something better) the “cross mutual information.” Intuitively, it is the portion of the (actual) entropy of ${\bm{Y}}$ that is explained by ${\bm{Z}}$ under the model ${\hat{p}\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}}{}\middle|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{z};\bm{\theta}}\right)}$.

Now, Gibbs’s inequality tells us that $\text{H}_{p\hat{p}}{\mathopen{}\mathclose{{}\left[{\bm{Y}}|{\bm{Z}}}\right]}\geq\text{H}_{p}{\mathopen{}\mathclose{{}\left[{\bm{Y}}|{\bm{Z}}}\right]}$, so consequently $\mathcal{I}_{p\hat{p}}\mathopen{}\mathclose{{}\left({\bm{Y}};{\bm{Z}}}\right)\leq\mathcal{I}\mathopen{}\mathclose{{}\left({\bm{Y}};{\bm{Z}}}\right)$: the cross mutual information is never greater than the actual mutual information. Equality is reached when the model matches the true data conditional. Although this is also the point at which the posterior cross entropy in Eq. 10.7 reaches its minimum, this is not quite the same as saying that improving the latter increases the cross mutual information. Still, it is intuitive, since we expect training to oblige the model to make increasing use of ${\bm{Z}}$ in order to distinguish the conditional data from the marginal data. And indeed, we can show this. The cross mutual information of Eq. 10.10 between ${\bm{Y}}$ and ${\bm{Z}}$ can be written more explicitly in terms of log probabilities, and then related to the (approximate) loss function in Eq. 10.9:

The final (approximate) equality follows because of the (somewhat subtle) fact that the expectation includes only positive samples, to wit, samples in which $\bm{z}$ is correctly paired with $\bm{y}$, and therefore Eq. 10.9 applies.

Eq. 10.11 tells us that decreasing the posterior cross entropy (on the right-hand side of Eq. 10.11) increases, at least approximately, the cross mutual information (on the left). The larger ${K}$, the less approximate the final equality (see Eq. 10.8). (And although this also increases the $\log{K}$ term in Eq. 10.11 and therefore the discrepancy between the cross mutual information and the cross entropy, it does not increase the discrepancy between their gradients.) In sum, minimizing the NCE loss in Eq. 10.7, with $h\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{z}},\bm{\theta}}\right)$ defined to be the difference between the conditional and marginal energies, maximizes the information extracted from ${\bm{Z}}$ by the function that assigns energies to ${\bm{Y}}$, $E\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{z}},\bm{\theta}}\right)$.

But now notice that there is nothing mathematical to distinguish the roles played by ${\bm{Y}}$ and ${\bm{Z}}$. In terms of the data, they are either paired (positive examples) or unpaired (negative examples), so they play symmetrical roles. In terms of the model, they enter the loss only through the generic function $h\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{y}},\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{\hat{z}},\bm{\theta}}\right)$, which is learned and has no pre-specified role for its first and second arguments. So we can equally interpret descent of the InfoNCE loss as learning to extract useful information from ${\bm{Y}}$ about ${\bm{Z}}$ rather than the other way around. Indeed, perhaps the most felicitous interpretation, which emphasizes this symmetry, is that the training scheme asks the model to distinguish between the joint distribution ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{z},\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}}\right)}$ and the product of the marginals, ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{z}}\right)}{p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}{}}\right)}$.

There are many possibilities, but one nice application of InfoNCE is to time-series data, and in particular with learning to extract useful information from the “auxiliary variable” $\mathopen{}\mathclose{{}\left({\bm{Y}}_{1},\ldots,{\bm{Y}}_{t}{}}\right)$, about the variable of interest, ${\bm{Y}}_{t+s}{}$ (for some positive integer $s$). That is, we want to learn how to “summarize” sequences of random variables so as best to predict their future state. (For example, for linear dynamical systems, the optimal summary is a weighted sum of past states, with weights decaying exponentially into the past.) Thus the positive and negative (“noise”) distributions are, respectively, $p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}_{t+s}{}|\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}_{1},\ldots,\bm{y}_{t}{}}\right)$ and ${p\mathopen{}\mathclose{{}\left(\leavevmode\color[rgb]{.5,.5,.5}\definecolor[named]{pgfstrokecolor}{rgb}{.5,.5,.5}\pgfsys@color@gray@stroke{.5}\pgfsys@color@gray@fill{.5}\bm{y}_{t+s}}\right)}$.

As lately discussed, the authors model the difference between the conditional and unconditional energies, rather than the energies themselves. In particular, they let this model have the form

where $f_{\text{s}}$ is a static “encoder” ANN and $f_{\text{RNN}}$ is an RNN. In order to decrease the posterior cross entropy (Eq. 10.7), the encoder and the RNN must extract representations from the data history (on the one hand) and a future sample (on the other) that expose the shared information between them to a bilinear form. The parameters $\bm{\theta}$ and $\mathbf{W}$ are all learned by stochastic gradient descent of Eq. 10.7.