A.1 The Calculus of Variations

Since our object is now the function that optimizes the functional $J$, rather than the point that optimizes the function $\mathcal{L}$, we need a more sophisticated method. The basic approach is to transform this optimization into the simple one over scalars from introductory calculus. In particular, consider small displacements to the optimal trajectory $\bm{x}^{*}$ by another (nicely behaved) trajectory $\bm{\eta}(t)$; i.e. consider trajectories $\bm{x}^{*}(t)+\epsilon\bm{\eta}(t)$. Thus, we have expressed a family of trajectories near the optimum in terms of a scalar parameter $\epsilon$, which allows us to optimize $J$ with the standard method of single-variable calculus. The additional wrinkle is that the optimization must hold for any choice of displacement trajectories $\bm{\eta}(t)$. However, we do still require that the displaced trajectory satisfy the boundary conditions, in this case that $\bm{x}^{*}(0)+\epsilon\bm{\eta}(0)=\bm{0}$, which implies that $\bm{\eta}(0)=0$.^††margin: Precisely why is this necessary?

Accordingly, we set the derivative equal to zero and then proceed with some lemmas from multivariable calculus:

The second term vanishes under the assumption that the displaced trajectory satisfy the boundary condition. The remaining terms involve inner products with the displacement trajectory $\bm{\eta}(t)$. Since the equation must hold for any choice of displacements trajectories, the other vectors in these inner products must be zero. Making this explicit (and restoring the suppressed arguments of functions), we obtain the Euler-Lagrange equations^††margin: Euler-Lagrange equations ,

and the transversality condition^††margin: transversality condition ,

In use of the ELEs, the partial derivative $\partial{\mathcal{L}}/\partial{{\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{\dot{x}}}}$ will play a special role. Therefore we endow it with a name, the conjugate variable^††margin: conjugate variable $\bm{p}$, and re-write Eqs. A.1 and A.2 in terms of it:

The first two equations have a nice symmetry. Note well that the first is a definition and holds for all trajectories ${\color[rgb]{.75,0,.25}\definecolor[named]{pgfstrokecolor}{rgb}{.75,0,.25}\bm{x}}(t)$, whereas the second and third hold only for optimal trajectories $\bm{x}^{*}(t)$.

Let the Lagrangian and terminal cost be parameterized with some time-invariant parameters $\bm{\theta}$, which we wish to optimize. Then we require the total derivative of the accumlated loss $J$,

(Arguments have been suppressed on the second line for brevity.) This is certainly complicated. However, at an optimal trajectory, $\bm{x}^{*}$, the total derivative simplifies considerably. In particular, at an optimal trajectory, the Euler-Lagrange equations (Eq. A.1) and transversality condition (Eq. A.2) hold. Substituting these into the gradient, we find

To reach the third line, we integrated by parts. To reach the last line, we observed that the initial states $\bm{x}(0)$ are fixed, so $\mathrm{d}{\bm{x}}/\mathrm{d}{\bm{\theta}}$ must be 0. Crucially, the final line is the first line, with all total derivatives replaced by partial derivatives. Thus, along optimal trajectories, only the direct effects of parameters on terminal and running costs contribute to the total-cost gradient.