Optimization Methods for Systems and Control

This course provides an introduction to various methods of obtaining the extremum (minimum or maximum) of a non-dynamical system and the use of these methods in real-life applications. Computational methods for nonlinear optimization; unconstrained optimization. Constrained optimization; linear programming; simplex method for solving linear programs; Lagrange's conditions, the Karush-Kuhn-Tucker (KKT) conditions, Least squares, Convex optimization, Global optimization methods: Genetic algorithms and Particle swarm optimization (PSO) method.

ECE58000

Credit Hours:

3

Learning Objective:

  • Use various methods to compute minimum or maximum of nonlinear functions of many variables
  • Solve linear programming problems
  • Find minimum or maximum of nonlinear functions of many variables using population-based methods
  • Apply various optimization methods learned in the course to real-life design problems

Description:

This course provides an introduction to various methods of obtaining the extremum (minimum or maximum) of a non-dynamical system and the use of these methods in real-life applications. Computational methods for nonlinear optimization; unconstrained optimization. Constrained optimization; linear programming; simplex method for solving linear programs; Lagrange's conditions, the Karush-Kuhn-Tucker (KKT) conditions, Least squares, Convex optimization, Global optimization methods: Genetic algorithms and Particle swarm optimization (PSO) method.

Fall 2020 Syllabus

Topics Covered:

Week Modules
1

1. Welcome, Introduction, and Motivation

2. Differentiation of functions of many variables

3. Taylor series expansion and feasible directions

2

4. First-order necessary condition (FONC)

5. Necessary and sufficient conditions for a point to be a minimizer

6. Quadratic forms

3

7. Quadratic forms and second-order sufficient condition (SOSC)

8. Proof of the second-order sufficient condition (SOSC)

9. Golden section search and Fibonacci method

4

10. The Fibonacci line search method and Newton's methods

11. Introduction to the method of steepest descent (SD)

12. Performance analysis of the steepest descent (SD) gradient method

5

13. Further analysis of the steepest descent (SD) gradient method

14. Newton's method to minimize functions of many variables

15. The asymptomatic symbols and the conjugate directions (CD) method

6

16. Linear independence of conjugate directions

17. Implementation of the Conjugate Gradient (CG) algorithm

18. Review for Midterm 1

7

19. An introduction to quasi-Newton methods

20. The rank one correction of Davidon-Fletcher-Powell methods

21. The Broyden-Fletcher-Goldfarb-Shanno algo

8

22. The least-squares method development and its applications

23. The least squares and the recursive least-squares (RLS) methods

24. Underdetermined systems of linear equations

9

25. Solving a general system of linear equations

26. The pseudo-inverse and population based optimization methods

27. Particle Swarm Optimization and Canonical Genetic Algorithms

10

28. The canonical genetic algorithm (GA) development

29. The canonical genetic algorithm implementation and analysis

30. The canonical genetic algorithm MATLAB implementation

11

31. The schema theorem and the real-number genetic algorithms

32. Real-number genetic algorithms and linear programming

33. Fundamental theorem of linear programming

12

34. Fundamental theorem of linear programming and convex sets

35. Basic feasible solutions and extreme points of the feasible set

13

36. Simplex method for solving linear programming problems

37. The two-phase simplex method and and introduction to duality

38. An introduction to nonlinear optimization subject to constraints

14

39. Nonlinear optimization subject to equality constraints

40. The Lagrange multiplier theorem

41. Nonlinear optimization subject to inequality constraints

15

42. Convex functions and convex optimization problem

Prerequisites:

  • MA 511 or equivalent (first graduate course in linear algebra)
  • Linear algebra, calculus of several variables. In particular: matrix manipulation, linear spaces, quadratic forms, tangent planes. Elements of multivariable calculus, in particular, differentiation of real-valued functions of n variables, gradients, and the chain rule.

Applied / Theory:

30 / 70

Homework:

Approximately 5 homework assignments

Projects:

None.

Exams:

Two midterm exams; one final exam.

Textbooks:

An Introduction to Optimization by E. K. P. Chong and S. H. Zak (4th edition), published by Wiley & Sons, Inc., New York 2013. Full text available online to Purdue students through Purdue Libraries.

Computer Requirements:

ProEd minimum computer requirements. Access to MATLAB or the Student Edition of MATLAB (registered students can access via Purdue GoRemote--http://goremote.ics.purdue.edu).

Other Requirements:

MATLAB

ProEd Minimum Requirements:

view