ECE 69500 - Stochastic Control

Course Details

Lecture Hours: 3 Credits: 3

Areas of Specialization:

  • Automatic Control

Normally Offered:

Each Fall

Campus/Online:

On-campus only

Requisites:

ECE 38200; ECE 60200 and ECE 60000 (concurrent)

Requisites by Topic:

Control theory (introductory level), probability (introductory level), access to MATLAB.

Catalog Description:

This course aims to provide a theoretical overview of stochastic optimal control. Topics to be covered include Markov Decision Processes (MDPs), Linear-Quadratic-Gaussian (LQG) controls, Bayesian filtering, tabular reinforcement learning methods, variational methods, and path integral controls. This course also introduces an elementary stochastic calculus necessary to describe stochastic optimal control problems in continuous time. Rather than providing in-depth discussions on individual topics, this course aims to draw theoretical connections among topics traditionally studied separately in different disciplines.

Required Text(s):

None.

Recommended Text(s):

  1. Markov Decision Processes: Discrete Stochastic Dynamic Programming. , M. L. Puterman , John Wiley & Sons , 2014 , ISBN No. 978-1118625873
  2. Neuro-dynamic programming , D. P. Bertsekas and J. N. Tsitsiklis , Athena Scientific , 1996 , ISBN No. 978-1886529106
  3. Partially Observed Markov Decision Processes - Filtering, Learning and Controlled Sensing. , 2nd Edition , V. Krishnamurthy , Cambridge University Press , ISBN No. 978-1009449434
  4. Reinforcement Learning: An Introduction , R. S. Sutton and A. G. Barto , A Bradford Book , 2018 , ISBN No. 978-0262039246
  5. Stochastic Optimal Control - The Discrete Time Case , D. P. Bertsekas and S. E. Shreve , Athena Scientific , 1996 , ISBN No. 978-1886529038
  6. Stochastic differential equations , B. Oksendal , Springer Berlin Heidelberg , 2003 , ISBN No. 978-3540047582

Lecture Outline:

Week Week
1 Introduction
2 Classification of Stochastic Optimal Control Problems
3 Control of Fully Observable Systems (MDP, Discrete-Time Stochastic LQR)
4 Bayesian Filtering (HMM filtering, Kalman filtering)
5 Control of Partially Observable Systems (POMDP, Discrete-time LQG Control)
6 Reinforcement Learning: Model-Based Methods
7 Reinforcement Learning: Simulator-Driven and Data-Driven Methods
8 Introduction to Stochastic Calculus (Brownian Motion, Stochastic Differential Equation, Ito Formula)
9 Continuous-Time Stochastic Optimal Control Problems (Stochastic Hamilton-Jacobi-Bellman Equation)
10 Solving Hamilton-Jacobi-Bellman Equations
11 Variational Formula
12 Linearly Solvable MDPs
13 Path Integral Control: Derivation
14 Path Integral Control: Applications
15 Additional topics and project presentations

Assessment Method:

Participation, homework, project (4/2026)