B.2 Probability and Statistics

• •
  ​

  Have: $p_{xyo}(x,y,o)$

• •
  ​

  Define: ${Z}\mathrel{\vbox{\hbox{.}\hbox{.} }}=a{X}+b{Y}$

• •
  ​

  Want: $p_{zo}$

Let us use the change-of-variables formula to eliminate ${Y}$ in favor of ${Z}$. We will therefore need (1) to express ${Y}$ as a function of the other variables,

and (2) the derivative of this function with respect to its dependence on the variable we are introducing, ${Z}$:

Then by the change-of-variables formula,

For the simple case of $a=b=1$ (“coordinate transformation”), this reduces to

Thus to get $p_{zo}$, one substitutes $({Z}-{X})$ for ${Y}$ in the original joint and integrates out ${X}$.

The score is defined as the gradient of the log-likelihood (with respect to the parameters, $\bm{\theta}$), $\frac{\mathrm{d}{}}{\mathrm{d}{\bm{\theta}}}\log{\hat{p}\mathopen{}\mathclose{{}\left(\bm{y}{};\bm{\theta}}\right)}$. The mean of the score is zero:

The variance of the score is known as the Fisher information. Because its mean is zero, it is also the expected square of the score.

This turns out to take a simple form. For a (vector) random variable ${\bm{Y}}$ and “parameters” $\bm{\theta}$ (that may themselves be random variables):

the Fisher information is:

where in the last line we have used the fact that the derivatives of the log-normalizer are the cumulants of the sufficient statistics (${\bm{T}}{}$) under the distribution. A perhaps more interesting equivalent can be derived by noting that:

Therefore, using the shorthand $\bm{\mu}\mathrel{\vbox{\hbox{.}\hbox{.} }}=\mathbb{E}_{{\bm{Y}}{}|\bm{\theta}}{\mathopen{}\mathclose{{}\left[{\bm{T}}{}({\bm{Y}})\middle|\bm{\theta}{}}\right]}$, we can write

[[[table]]]