ECE 50024 - Machine Learning

Course Details

Lecture Hours: 3 Credits: 3

Counts as:

  • EE Elective
  • CMPE Special Content Elective

Normally Offered:

Each Spring

Campus/Online:

On-campus and online

Requisites:

ECE 20875, ECE 26400, ECE 30200, MA 26500

Requisites by Topic:

Programming (Python & C), probability, optimization, linear algebra

Catalog Description:

The goal of this class is to help students gain a deeper understanding of the mathematical intuition and connection behind a variety of machine learning methods rather than programming per se. The four clusters of topics that will be covered in this course are: 1) Mathematical Preliminaries. Matrices, vectors, Lp norm, geometry of the norms, symmetry, positive definiteness, eigen-decomposition. Unconstrained optimization, gradient descent, convex functions, Lagrange multipliers, linear least squares. Probability space, random variables, joint distributions, multi-dimensional Gaussians. 2) Linear Classifiers. Linear discriminant analysis, separating hyperplane, multi-class classification, Bayesian decision rule, geometry of Bayesian decision rule, linear regression, logistic regression, perceptron algorithms, support vector machines, nonlinear transformations. 3) Learning Theory. Bias and variance, training and testing, generalization, PAC framework, Hoeffding inequality, VC dimension. 4) Robustness. Adversarial attack, targeted and untargeted attack, minimum distance attack, maximum loss attack, regularization-based attack. Perturbation through noises. Robustness of SVM. These topics could help you understand the principles and limitations of machine learning methods, which can be generalized to various popular tools nowadays not covered in this class. If you are taking this course as your first class in machine learning, it could lay a solid mathematical foundation for you as you journey on in this field. If you already have machine learning backgrounds, the class could possibly provide you additional understanding of machine learning from a mathematical perspective.

Required Text(s):

None.

Recommended Text(s):

  1. Introduction to Probability for Data Science , First Edition , Chan, Stanley , Draft
  2. Learning from Data , 1st Edition , Abu-Mostafa, Yaser , ALM , 2012 , ISBN No. 1600490069
  3. Pattern Recognition and Machine Learning , Bishop, Chris , Springer , 2011 , ISBN No. 978-0387310732

Learning Outcomes:

A student who successfully fulfills the course requirements will have demonstrated an ability to:
  1. classify data using statistical learning methods, and an understanding of the limitations of the methods.. [1]
  2. estimate model parameters using regression methods.. [1]
  3. apply optimization algorithms to achieve the statistical learning tasks.. [1]
  4. evaluate results generated by different machine learning algorithms, and make interpretations.. [1,6]
  5. apply machine learning algorithms to solve complex engineering problems.. [1,6,7]

Lecture Outline:

Week Lecture Topics
1 Course overview and mathematics review; linear regression
2 Examples of linear regression; ridge and LASSO regression
3 Optimization
4 Linear separability
5 Bayesian classifier; classification error and ROC curves
6 Parameter estimation; logistic regression
7 Logistic regression; kernel trick
8 Intro to neural networks; convolutional structures and back propagation
9 Convolutional structures; recurrent networks and transformers
10 Attention and transformers; adversarial attacks
11 Probability inequality; Is learning feasible?
12 Probability-approximately correct; generalization bound
13 Growth function and VC dimension; sample and model complexity
14 Bias and variance; overfitting
15 Regularization and validation; validation and conclusion

Assessment Method:

The grade for this course will be determined by homework, quizzes, and a final project. (11/22)