\chapter{Pontryagin's Minimum Principle}

In this chapter, we provide a derivation of the Minimum Principle of Pontryagin for the minimum time regulator problem. We analyzed this problem briefly in the previous chapter. A generalization to a broader class of problems will be discussed after we familiarize ourselves with basics of calculus of variations. In this chapter we analyze only time-invariant dynamic system models. Specifically, we consider dynamic systems modeled by
\[
\dot{\b x}=\b f(\b x,\b u),\quad \b x(t_0)=\b x_0.
\]
The control vector $\b u$ is assumed to belong to the set of admissible controllers denoted by $U$. In particular, the components $u_i$ of the control vector $u$ are assumed to satisfy the following amplitude constraints
\[
\alpha_i\le u_i\le \beta_i, \quad i=1,2,\ldots,m,
\]
where $\alpha_i$, $\beta_i$ are fixed real numbers. Recall from the previous chapter that our goal is to determine an admissible control law that steers system's trajectory from a given initial state $\b x(t_0)=\b x_0$ to a specified final state $\b x_f$ in the shortest possible time. This is equivalent to minimizing the performance index
\[
J=\int_{t_0}^{t_f}dt= t_f-t_0.
\]
We call such a controller time optimal. We assume that for any $\b x \ne \b x_f$ there exists an admissible time optimal control transferring the system from $\b x$ to $\b x_f$ in minimum time. We denote by
\[
J^*(\b x(t))
\]
the time of the optimal transfer from $\b x(t)$ to the specified $\b x_f$. Note that in our discussion now $J^*(\b x(t))$ is time-invariant. It follows from our previous discussion of this problem that the optimal controller must satisfy the equation
\[
\min_{\b u\in U}H(\b x,\b u,\b p)=0,
\]
where $H(\b x,\b u,\b p)=1+\b p^T\b f(\b x,\b u)$, and $\b p=\left[\begin{array}{cccc}
\frac{\partial J^*}{\partial x_1}&\frac{\partial J^*}{\partial x_2}&\cdots&\frac{\partial J^*}{\partial x_n}
\end{array}\right]^T$. In other words, for an optimal control $\b u^*$ and the corresponding optimal trajectory $\b x^*$ we have $H(\b x^*,\b u^*,\b p^*)=0$. Otherwise $H(\b x,\b u,\b p)\ge 0$.

To proceed further we need one more assumption in addition to the assumptions we made while discussing the HJB equation. We assume that $J^*(\b x(t))$ has continuous second partial derivatives $\frac{\partial ^2 J^*(\b x)}{\partial _i \partial _j}$, that is, the functions $p_i$ have continuous first derivatives $\frac{\partial p_i}{\partial x_j}$, and the components of $\b f$ have continuous first derivatives $\frac{\partial f_i(\b x,\b u)}{\partial x_j}$. This implies that $\frac{\partial \b p}{\partial \b x}=\left( \frac{\partial \b p}{\partial \b x}\right)^T$. We then compute
\begin{eqnarray*}
\frac{\partial H(\b x,\b u,\b p)}{\partial \b x} &=& \frac{\partial}{\partial \b x}\left(1+\b p^T \b f(\b x,\b u)\right)\\
&=& \b f^T\frac{\partial \b p}{\partial \b x}+\b p^T\frac{\partial \b f}{\partial \b x}.
\end{eqnarray*}
From our previous discussion it follows that the Hamiltonian function attains its minimum on the optimal trajectory, that is, $H(\b x^*,\b u^*,\b p^*)=0$. Taking this into account, we conclude that for the optimal process
\[
\frac{\partial H}{\partial \b x}=\b 0 \quad \mb{for} \quad (\b x,\b u)=(\b x^*,\b u^*).
\]
Hence, for the optimal process $(\b x^*,\b u^*)$, we have
\begin{equation}
\b f^T\frac{\partial \b p}{\partial \b x}+\b p^T\frac{\partial \b f}{\partial \b x}=\b 0.
\label{eq:derhwx}
\end{equation}
We now differentiate $\b p$ with respect to time to get
\begin{equation}
\dot{\b p} = \frac{\partial \b p}{\partial \b x}\dot{\b x}= \frac{\partial \b p}{\partial \b x}\b f.
\label{eq:pdotwt}
\end{equation}
Combining~(\ref{eq:derhwx}) and~(\ref{eq:pdotwt}), we obtain
\[
\dot{\b p}^*=-\left(\frac{\partial \b f}{\partial \b x}\right)^T\b p^*.
\]
since by assumption $\frac{\partial \b p}{\partial \b x}=\left( \frac{\partial \b p}{\partial \b x}\right)^T$. Observe now that $\b p^*$ is the solution to the differential equation $\dot{\b p}^*=-\left(\frac{\partial \b f}{\partial \b x}\right)^T\b p^*$, and hence $\b p^*=\b p^*(t)$. Therefore $\frac{\partial \b p}{\partial \b x}=\b 0$, and we have
\[
\frac{\partial H}{\partial \b x}=\b p^{*T}\frac{\partial \b f}{\partial \b x}.
\]
Hence
\[
\dot{\b p}^*=-\left( \frac{\partial H}{\partial \b x}\right)^T
\]
We summarize our development for the minimum time regulator problem in the following theorem that we can view as Pontryagin's Minimum Principle for the minimum time regulator problem.
\begin{thm}
Necessary conditions for $\b u$ to minimize
\[
J=\int_{t_0}^{t_f}dt
\]
subject to
\[
\dot{\b x}=\b f(\b x,\b u),\quad \b x(t_0)=\b x_0,\quad \b x(t_f)=\b x_f,
\]
are
\[
\dot{\b p}^{*T}=-\left( \frac{\partial H}{\partial \b x}\right),
\]
where $H(\b x,\b u,\b p)=1+\b p^T\b f(\b x,\b u)$, and
\[
H(\b x^*,\b u^*)=\min_{\b u\in U}H(\b x,\b u).
\]
\end{thm}
The equation $\dot{\b p}^{*T}=-\left( \frac{\partial H}{\partial \b x}\right)$ is called in the literature the {\em adjoint\/} or {\em costate\/} equation.

We now illustrate the above theorem on the following well-known example.
\begin{example}\rm
We consider a controlled object modeled by
\begin{eqnarray*}
\left[\begin{array}{c}
\dot{x}_1\\
\dot{x}_2
\end{array}\right]&=&\left[\begin{array}{cc}
0 & 1\\
0 & 0
\end{array}\right]+\left[\begin{array}{c}
{x}_1\\
{x}_2
\end{array}\right]+\left[\begin{array}{c}
0\\
1
\end{array}\right]u\\
&=& \b A\b x + \b bu
\end{eqnarray*}
with the performance index
\[
J=\int_{0}^{t_f}dt.
\]
The control is required to satisfy
\[
|u(t)|\le 1
\]
for all $t\in[0,t_f]$. This constraint means that the control must have magnitude no greater than $1$. Our objective is to find an admissible control that minimizes $J(u)$ and transfers the system from some initial state $\b x_0$ to the origin.

We begin by finding the Hamiltonian function for the problem. We have
\begin{eqnarray*}
H &=& 1+\b p^T(\b A\b x + \b bu)\\
&=& 1 + \left[\begin{array}{cc}
p_1 & p_2
\end{array}\right]\left(\left[\begin{array}{cc}
0 & 1\\
0 & 0
\end{array}\right]\left[\begin{array}{c}
{x}_1\\
{x}_2
\end{array}\right]+\left[\begin{array}{c}
0\\
1
\end{array}\right]u \right)\\
&=& 1 + p_1x_2 + p_2 u.
\end{eqnarray*}
The costate equations are
\[
\left[\begin{array}{c}
\dot{p}_1\\
\dot{p}_2
\end{array}\right] = -\left[\begin{array}{c}
\frac{\partial H}{\partial x_1}\\
\frac{\partial H}{\partial x_2}
\end{array}\right]= \left[\begin{array}{c}
0\\
-p_1
\end{array}\right].
\]
Solving the costate equations yields
\begin{eqnarray*}
p_1 &=& d_1,\\
p_2 &=& -d_1t+d_2
\end{eqnarray*}
where $d_1$ and $d_2$ are integration constants. We now find an admissible control minimizing the Hamiltonian. We have
\begin{eqnarray*}
\arg_{u}\min H &=& \arg_u \min (1 + p_1x_2+p_2u)\\
&=& \arg_u \min (p_2u)\\
&=& \left\{\begin{array}{ccc}
u(t)=1 & \mb{if}& p_2<0\\
u(t)=-1 & \mb{if}& p_2>0
\end{array}\right..
\end{eqnarray*}
Hence,
\[
u^*(t) = -\mb{sign}(p_2^*)= -\mb{sign}(-d_1t+d_2),
\]
where
\[
\mb{sign}(z)=\left\{\begin{array}{ccc}
1 & \mb{if}& z>0\\
-1 & \mb{if}& z<0
\end{array}\right..
\]
Thus, every time optimal controller is piecewise constant taking the values $1$ or $-1$. This controller has at most two intervals of constancy since the argument of the sign function is a linear function $-d_1t+d_2$ that changes its sign at most once. This type of control is called a {\em bang-bang control\/} since it switches back and forth between its extreme values. We now sketch a phase portrait for the problem. Recall that a phase portrait is a family of system's trajectories in two dimensional state space. A two dimensional state space is a state plane. We know that the optimal control for the problem takes only two values: $1$ or $-1$. We first consider the case when $u=1$. With this control, the system equations are:
\begin{eqnarray*}
\dot{x}_1 &=& x_2\\
\dot{x}_2 &=& 1.
\end{eqnarray*}
Solving the second equation yields
\[
x_2=t+c_1,
\]
where $c_1$ is an integration constant. We substitute $x_2=t+c_1$ into $\dot{x}_1=x_2$ to get
\[
\dot{x}_1=t+c_1.
\]
Solving the above equation, we obtain
\begin{eqnarray*}
x_1 &=& \int(t+c_1)dt\\
&=& \frac{1}{2}(t+c_1)^2+c_2\\
&=& \frac{1}{2}x^2_2+c_2,
\end{eqnarray*}
where $c_2$ is an integration constant. In the state plane the above equation represents a family of parabolas open toward the positive $x_1$ axis. The representative point moves upward along these parabolas since $\dot{x}_2=1$, that is, $\dot{x}_2>0$. Typical system trajectories for this case are shown in Figure~\ref{fig:bbu1}.
\begin{figure}
\vspace{4in}
\hskip.2in\special{bmp:c:/mybook/bbol1.bmp x=5.33in y=4in}
\caption{Trajectories of the system when {\em u}=1.}
\label{fig:bbu1}
\end{figure} 
Similarly, when $u=-1$, we have
\[
\dot{x}_2=-1.
\]
Solving the above differential equation, yields
\[
x_2=-t+\tilde{c}_1,
\]
where $\tilde{c}_1$ is an integration constant. Substituting $x_2=-t+\tilde{c}_1$ into $\dot{x}_1=x_2$, and then solving the resulting equation yields
\[
x_1=-\frac{1}{2}x^2_2+\tilde{c}_2,
\]
where $\tilde{c}_2$ is an integration constant. In the state plane, this equation represents a family of parabolas open toward the negative $x_1$ axis. The representative point moves downward along these parabolas since $\dot{x}_2=-1$, that is, $\dot{x}<0$. Typical system trajectories for this case are shown in Figure~\ref{fig:bbu2}.
\begin{figure}
\vspace{4in}
\hskip.2in\special{bmp:c:/mybook/bbol2.bmp x=5.33in y=4in}
\caption{Trajectories of the system when {\em u}=-1.}
\label{fig:bbu2}
\end{figure} 
Note that only one parabola from each family passes through the specified terminal point $\b x=\left[\begin{array}{cc}
0& 0
\end{array}\right]^T$ in the state plane. Segments of the two parabolas through the origin form the switching curve. The equation of the switching curve is
\[
x_1=-\frac{1}{2}x^2_2\, \mb{sign}(x_2).
\]
This means that if an initial state is above the switching curve, then $u=-1$ is used until the switching curve is reached. Then $u=1$ is used to reach the origin. For an initial state below the switching curve, the control $u=1$ is first used to reach the switching curve, and then the control is switched to $u=-1$. The above control action can be described as:
\[
u=\left\{\begin{array}{ccc}
1&\mb{if}& v<0\\
-\mb{sign}(x_2)&\mb{if}&v=0\\
-1&\mb{if}& v>0,
\end{array}\right.
\]
where $v=v(x_1,x_2)=x_1+\frac{1}{2}x^2_2\mb{sign}(x_2)=0$ is the equation describing the switching curve. We can use the above equation to synthesize a closed-loop system such that starting at an arbitrary initial state in the state plane, the representative point will always be moving in an optimal fashion towards the origin. Once the origin is reached, the representative point will stay there. A block diagram of the closed-loop time optimal system is shown in Figure~\ref{fig:bbcloop}.
\begin{figure}
\vspace{4.5in}
\hskip.2in\special{bmp:c:/mybook/bbcloop.bmp x=6in y=4.5in}
\caption{Time optimal closed-loop system with state feedback.}
\label{fig:bbcloop}
\end{figure}
A phase portrait of the closed-loop time optimal system is given in Figure~\ref{fig:bbclp}.
\begin{figure}
\vspace{4in}
\hskip.2in\special{bmp:c:/mybook/bbclp.bmp x=5.33in y=4in}
\caption{A phase-plane portrait of the time optimal closed-loop system.}
\label{fig:bbclp}
\end{figure}
\end{example}
\markex
We are now ready to discuss a general minimum time regulator problem for a class of multi-input dynamic systems modeled by
\[
\dot{\b x}=\b A\b x+\b B\b u.
\]
The goal is transfer the system from some initial state to the origin in minimum time, subject to
\[
|u_i|\le k_i,\quad k_i>0,\quad i=1,2,\ldots,m.
\]
We begin our analysis by forming the associated Hamiltonian. We have
\begin{eqnarray*}
H &=& 1+\b p^T(\b A\b x+\b B\b u)\\
&=& 1+\b p^T\b A\b x+\sum_{i=1}^m\left(\b p^T\b b_i\right)u_i,
\end{eqnarray*}
where $\b b_i$ is the $i$-th column of $\b B$. Let
\[
s_i(t)=\left(\b p^*(t)\right)^T\b b_i.
\]
Then, applying Pontryagin's Minimum Principle we obtain the necessary condition for optimal control of the form
\[
u^{*}_i(t)=-k_i\mb{sign}(s_i(t)),\quad i=1,2,\ldots,m.
\]
Thus, the optimal control for the $i$-th control channel has the bang-bang form $u^*_i=\pm k_i$. Note that $s_i(t)=\left(\b p^*(t)\right)^T\b b_i=0$ is the switching curve for the $i$-th control channel. We compute the costate vector from the costate equation
\begin{eqnarray*}
\dot{\b p}^{*T} &=& -\frac{\partial H}{\partial \b x}\\
&=& -\frac{\partial}{\partial \b x}\left(\b p^{*T}\b A\b x\right)\\
&=& -\b p^{*T}\b A,
\end{eqnarray*}
or
\[
\dot{\b p}^* = -\b A^T\b p^*.
\]
The solution to the costate equation is
\[
\b p^*(t)=e^{-\b A^Tt}\b p^*(0).
\]
Thus, the $i$-th switching curve can be represented as
\[
s_i(t) = \b p^{*T}(0)e^{-\b At}\b b_i= 0.
\]
Observe that if $s_i(t)=0$ on some time interval, then $u^*_i(t)$ is indefinite on this interval. We now show that under some mild assumptions there is no time interval on which $s_i(t)=0$, that is, there is no time interval on which $u^*_i(t)$ is indefinite. First, we assume that $\b b_i\ne \b 0$. Next, we note that for the problem we consider we have $\min_{\b u\in U}H(\b x,\b u)=0$, that is, $1+\b p^{*T}(t)(\b A\b x^*+\b B\b u^*)=0$ for all $t$ for the optimal process. Thus, the optimal costate vector $\b p^*(t)$ cannot be zero for any value of $t$. Finally, if $s_i=\b p^{*T}\b b_i$ is zero on some time interval, then this implies that
\[
\dot{s}_i = \dot{\b p}^{*T}\b b_i= -\b p^{*T}\b A\b b_i=0.
\]
Similarly, higher derivatives of $s_i(t)=\b p^{*T}\b b_i$ will also be zero on the time interval on which $s_i(t)=\b p^{*T}\b b_i$ is zero, that is, $s^{(j)}_i(t)=(-1)^j\b p^{*T}\b A^j\b b_i=0$, $j=1,2,\ldots$. We represent the above relations as
\[
\b p^{*T}\left[\begin{array}{ccccc}
\b b_i& \b A\b b_i&\cdots&\b A^{n-1}\b b_i&\cdots
\end{array}\right]=\b 0.
\]
If the system is completely controllable by the i-th input acting alone, then
\[
\mb{rank}\left[\begin{array}{ccccc}
\b b_i& \b A\b b_i&\cdots&\b A^{n-1}\b b_i&\cdots
\end{array}\right]=n,
\]
and we would have to have $\b p^*=\b p^*(t)=\b 0$. However, we already ruled out this case and thus $s_i(t)=\b p^{*T}\b b_i$ cannot be zero on any time interval. In conclusion, if the system is controllable by each input acting alone, then there is no time interval on which optimal control vector is indefinite.

\section{Exercises}
\begin{exercise}\rm
Show
\end{exercise}