Optimization Methods for Systems and Control - ECE58000
This course provides an introduction to various methods of obtaining the extremum (minimum or maximum) of a non-dynamical system and the use of these methods in real-life applications. Computational methods for nonlinear optimization; unconstrained optimization. Constrained optimization; linear programming; simplex method for solving linear programs; Lagrange's conditions, the Karush-Kuhn-Tucker (KKT) conditions, Least squares, Convex optimization, Global optimization methods: Genetic algorithms and Particle swarm optimization (PSO) method.
Credit Hours: 3
Instructor(s): Stanislaw Zak
Phone: (765) 494-6443
Email: zak@purdue.edu
Fall 2020 SyllabusLearning Objective:
Use various methods to compute minimum or maximum of nonlinear functions of many variables Solve linear programming problems Find minimum or maximum of nonlinear functions of many variables using population-based methods Apply various optimization methods learned in the course to real-life design problems
Topics Covered:| Week | Modules |
|---|---|
| 1 |
1. Welcome, Introduction, and Motivation 2. Differentiation of functions of many variables 3. Taylor series expansion and feasible directions |
| 2 |
4. First-order necessary condition (FONC) 5. Necessary and sufficient conditions for a point to be a minimizer 6. Quadratic forms |
| 3 |
7. Quadratic forms and second-order sufficient condition (SOSC) 8. Proof of the second-order sufficient condition (SOSC) 9. Golden section search and Fibonacci method |
| 4 |
10. The Fibonacci line search method and Newton's methods 11. Introduction to the method of steepest descent (SD) 12. Performance analysis of the steepest descent (SD) gradient method |
| 5 |
13. Further analysis of the steepest descent (SD) gradient method 14. Newton's method to minimize functions of many variables 15. The asymptomatic symbols and the conjugate directions (CD) method |
| 6 |
16. Linear independence of conjugate directions 17. Implementation of the Conjugate Gradient (CG) algorithm 18. Review for Midterm 1 |
| 7 |
19. An introduction to quasi-Newton methods 20. The rank one correction of Davidon-Fletcher-Powell methods 21. The Broyden-Fletcher-Goldfarb-Shanno algo |
| 8 |
22. The least-squares method development and its applications 23. The least squares and the recursive least-squares (RLS) methods 24. Underdetermined systems of linear equations |
| 9 |
25. Solving a general system of linear equations 26. The pseudo-inverse and population based optimization methods 27. Particle Swarm Optimization and Canonical Genetic Algorithms |
| 10 |
28. The canonical genetic algorithm (GA) development 29. The canonical genetic algorithm implementation and analysis 30. The canonical genetic algorithm MATLAB implementation |
| 11 |
31. The schema theorem and the real-number genetic algorithms 32. Real-number genetic algorithms and linear programming 33. Fundamental theorem of linear programming |
| 12 |
34. Fundamental theorem of linear programming and convex sets 35. Basic feasible solutions and extreme points of the feasible set |
| 13 |
36. Simplex method for solving linear programming problems 37. The two-phase simplex method and and introduction to duality 38. An introduction to nonlinear optimization subject to constraints |
| 14 |
39. Nonlinear optimization subject to equality constraints 40. The Lagrange multiplier theorem 41. Nonlinear optimization subject to inequality constraints |
| 15 |
42. Convex functions and convex optimization problem |
Prerequisites:
MA 511 or equivalent (first graduate course in linear algebra) Linear algebra, calculus of several variables. In particular: matrix manipulation, linear spaces, quadratic forms, tangent planes. Elements of multivariable calculus, in particular, differentiation of real-valued functions of n variables, gradients, and the chain rule.
Applied / Theory:
30 / 70
Homework:
Approximately 5 homework assignments
Projects:
None.
Exams:
Two midterm exams; one final exam.
Textbooks:
An Introduction to Optimization by E. K. P. Chong and S. H. Zak (4th edition), published by Wiley & Sons, Inc., New York 2013. Full text available online to Purdue students through Purdue Libraries.
Computer Requirements:
ProEd minimum computer requirements. Access to MATLAB or the Student Edition of MATLAB (registered students can access via Purdue GoRemote--http://goremote.ics.purdue.edu).
Other Requirements:
MATLAB