ECE 595 / STAT 598: Machine Learning

Spring 2021
Jan 19, 2021 - May 1, 2021

Announcement:

  • Nov 23, 2020. Welcome to this course website. Please kindly note that a lot of details are still being determined. Please check back often for the latest update.

Lectures

Instructor: Professor Stanley Chan
TuTh: 4:30pm-5:45pm Eastern Time
Room: Zoom, meeting link available at Brightspace

Course Description

The course is structured in four parts.

  • Mathematical preliminaries. Matrices, vectors, Lp norm, geometry of the norms, symmetry, positive definiteness, eigen-decomposition. Unconstrained optimization, graident descent, convex functions, Lagrange multipliers, linear least squares. Probability space, random variables, joint distributions, multi-dimensional Gaussians.

  • Linear Classifiers. Linear discriminant analysis, separating hyperplane, multi-class classification, Bayesian decision rule, geometry of Bayesian decision rule, linear regression, logistic regression, perceptron algorithms, support vector machines, nonlinear transformations.

  • Learning Theory. Bias and variance, training and testing, generalization, PAC framework, Hoeffding inequality, VC dimension.

  • Robustness. Adversarial attack, targeted and untargeted attack, minimum distance attack, maximum loss attack, regularization-based attack. Perturbation through noies. Robustness of SVM.

Pre-requisites

We expect audience to have good knowledge in the following three subjects:

To help you determine if you have adequate pre-requisites, we encourage you to try homework 0:

Homework 0: (PDF) 230KB

If you can finish all the problems without hacking, you have met the pre-requisites.

Grades

All students will be graded by the following rubric. Graduate students and undergraduate students will be graded on two different curves.

  • Homework (20%).

  • Midterm 1 (20%). Feb 4, 2021. (This is tentative. I mainly want to use this exam to help you calibrate the course. Will decide when the semester starts.)

  • Midterm 2 (20%). Mar 18, 2021. (This is tentative. I may not do this exam. Will decide when the semester starts.)

  • Final Project (40%). Due Apr 30, 2021

These weights are approximate. We reserve the right to change them later.

Textbook and References

There is no official textbook for this course. Please refer to the lecture note section of the website for our lecture materials.

A few good reference books for this course are:

Programming

We will be primarily using Python. As such, we expect audience to have elementary programming skills, e.g., writing a hello world program. More information and resources on how to use Python can be found in the programming section of this website.

Besides Python, we use optimization packages to solve optimization problems. Of particular importance is CVX.

FAQ

  • Am I ready to take the course?

    • There is no official pre-requisite of the course (e.g., taking a prior course), although we expect students to have good background in linear algebra, optimization and probability.

    • Historically, undergraduate students and non-ECE PhD students have found this course difficult.

    • Pleaes check out the information about pre-requisite to see if you are ready for the course.

  • What is the difference between ECE 595 and other machine learning courses on campus?

    • We focus on general principles of learning.

    • Our goal is to provide an in-depth discussion of the subject, rather than superficially glancing through different topics

    • We put significant emphasis on understanding the mathematics behind the algorithms

    • We have plenty of hands-on programming exercises

  • What will I learn after taking ECE 595?

    • You will know what a linear model is, such as Bayesian decision rule, perceptron algorithm, logistic regressoin, support vector machine, etc.

    • You will know how to understand a linear classifier from a geometric perspective.

    • You will know how to attack a classifier.

    • You will know the how much a machine learning algorithm can do, and what a machine learning algorithm cannot do.

    • You will know how to implement machine learning algorithms using Python and CVX.

  • Can I audit the class?

    • No. Sorry, we do not allow audit in Spring 2021.

  • Where can I get help for programming problems?

    • Please check out our teaching staff information and one of the TA's office hours.