ECE 595 / STAT 598: Machine Learning

Lectures

Professor Stanley Chan
TuTh: 4:30pm-5:45pm Eastern Time
Room: Zoom, meeting link available at Brightspace

For Spring 2021, I will do synchronous Zoom presentation for our lectures. This applies to both on-campus or off-campus students. Please do not go to Stewart Center 183.

To encourage classroom participation, at this point I am not planning to record the live lectures. I will more or less follow my last year's schedule, and so you can watch the recorded videos if you cannot join the live lecture. If I teach anything new, I will record a new video and post it in Brightspace.

Course Description

This is a graduate-level machine learning course at Purdue ECE. When I created the course in 2018, my vision is to do something different from what is already available on the internet (and other institutions). Perhaps needless to say, there is a tremendous demand for machine learning nowadays. However, if we go around the internet, it is indeed quite shocking to see the quality variation from one source to another. Many so-called data science courses on online platforms are more or less bubbles with no real content. Students who have taken these courses can only speak the jargon but cannot do any actual work. Great courses in some of our peer institutions are sometimes tailored to computer science Ph.D. students. For non-computer science students, the content may appear abstract. So I am seeing a need for engineering students, like ours at Purdue ECE. We need to develop a rigorous machine learning course, focus on the big pictures, and do something relevant to engineering students.

So here comes our Purdue ECE595ML. It is a uniquely designed machine learning course for Purdue graduate students. The target audience group is engineering students with sufficient exposure to engineering problems, and have enough mathematical background to appreciate the depth. Engineering students are different from computer science students, in the sense that they are more interested in physics and intuitions than programming per se. To educate these students, big pictures and connections across different topics are more important than effcieint programming skills. Purdue ECE graduate students are also very solid in mathematics (Well, at least this is what I demand our Communication, Networking, Signal and Image Processing students). So, to feed their needs, we need to be rigorous. We need to explain the concepts to a point that they can carry over to their research/industry problems.

Given the variety of materials in the literature, I have distilled a set of four fundamental machine learning topics that I believe every engineering student should be fluent at.

• Mathematical preliminaries. Matrices, vectors, Lp norm, geometry of the norms, symmetry, positive definiteness, eigen-decomposition. Unconstrained optimization, graident descent, convex functions, Lagrange multipliers, linear least squares. Probability space, random variables, joint distributions, multi-dimensional Gaussians.

• Linear Classifiers. Linear discriminant analysis, separating hyperplane, multi-class classification, Bayesian decision rule, geometry of Bayesian decision rule, linear regression, logistic regression, perceptron algorithms, support vector machines, nonlinear transformations.

• Learning Theory. Bias and variance, training and testing, generalization, PAC framework, Hoeffding inequality, VC dimension.

• Robustness. Adversarial attack, targeted and untargeted attack, minimum distance attack, maximum loss attack, regularization-based attack. Perturbation through noies. Robustness of SVM.

I hope that after you have taken this course, you will become a different person than someone who took an online bubble course. I hope that you will be able to confidently speak about the principles behind the machine learning tools. And more importantly, I hope that you will be able to understand the limitations of machine learning tools. Since the first offering in 2018, the course has received enormous positive feedback. Students generally find the course a very different learning experience than other machine learning courses they have taken.

Remark for Spring 2021: I may modify some of the lectures, by removing a few and adding some new ones. Stay in tune on this.

Pre-requisites

I understand the huge demand for taking a machine learning course nowadays. However, ECE595ML may not be suitable for everyone. If you are not our Communication, Networking, Signal, and Image Processing (CNSIP) graduate students, you may want to check the pre-requisites below to see if you are ready to take the course. Historically speaking, students without proper mathematics background (e.g., students from a non-engineering school) have found this course difficult. If you are not ready, I would recommend you consider taking a few pre-requisite courses before attempting ECE595ML.

• Linear Algebra

  • MIT 18.06 Textbook: http://math.mit.edu/~gs/linearalgebra/

  • Stanford ENGR 108 Textbook: https://web.stanford.edu/~boyd/vmls/

  • Purdue MA 511 Linear Algebra

• Optimization

  • MIT 6.079

  • Stanford EE 364A Textbook: https://stanford.edu/~boyd/cvxbook/

  • Purdue ECE 647 Convex and Stochastic Optimization and Applications

• Probability

  • MIT 6.012 Textbook: https://www.amazon.com/Introduction-Probability-2nd-Dimitri-Bertsekas/dp/188652923X

  • Purdue ECE 302 Textbook: https://engineering.purdue.edu/ChanGroup/eBook.html

To help you determine if you have adequate pre-requisites, we encourage you to try homework 0:

Homework 0: (PDF) 230KB

If you can finish all the problems without hacking (you know what I mean by hacking), you are fine.

Grades

All students will be graded by the rubrics listed below. Everyone (PhD, MS, online MS, undergrad) will be graded on the same curve. If you choose Pass-No Pass, you still need to do everything. If you are above the cut off, you will pass. There is no audit for Spring 2021 due to Zoom security.

• Homework (10%). There are six homeworks. I will drop the worst one. Each homework will have 2 points: If you complete the homework, you get 2. If you are partially done with the homework, you get 1. If you do not submit, you get 0. I do not accept late homework.

• Quiz (40%). There will be six quizzes. I will drop the worst one. Each quiz is 30 minutes long. Each quiz will carry 8%. The quizzes are conducted right after the due date of the homework. You will be given a 24-hour window to complete the 30-min quiz, completely online. During the quiz, I will ask you lecture questions. I will also ask you homework questions. For example, if in the homework I ask you to plot a figure, I may ask you to change a parameter and re-plot the figure. Quizzes will be open-book, open-note, open-computer. However, with only 30 minutes, you probably will not have time to read anything besides answering the questions. So, please do the homework.

• Final Project (50%). Due Apr 30. I will give you a list of papers and topics you can choose from. After you have picked a topic, I will ask you to complete several tasks. The tasks will involve extending the paper, or proposing something new. I will grade your report according to the general standard in ICML, NeurISP, CVPR, and ICCV. Think of that we are reviewers for your papers.
  
  
  

Textbook and References

There is no official textbook for this course. Please refer to the lecture note section of the website for our lecture materials.

A few good reference books for this course are:

• Introduction to Probability for Data Science, by Stanley Chan, draft. 1st edition. 2020.

• Pattern Classification, by Duda, Hart and Stork, Wiley-Interscience; 2 edition, 2000.

• Learning from Data, by Abu-Mostafa, Magdon-Ismail and Lin, AMLBook, 2012.

• Elements of Statistical Learning, by Hastie, Tibshirani and Friedman, Springer, 2 edition, 2009.

• Pattern Recognition and Machine Learning, by Bishop, Springer, 2006.
  
  
  

Programming

We will be primarily using Python. As such, I expect you to have elementary programming skills, e.g., writing a hello world program. More information and resources on how to use Python can be found in the programming section of this website. I found Google Colab a fairly easy-to-use platform for Python programming. You can check this out.

Besides Python, we use optimization packages to solve optimization problems. Of particular importance is CVX.

FAQ

• Am I ready to take the course?

  • There is no official pre-requisite of the course (e.g., taking a prior course), although we expect students to have good background in linear algebra, optimization and probability.

  • Historically, undergraduate students and non-ECE PhD students have found this course difficult.

  • Pleaes check out the information about pre-requisite to see if you are ready for the course.

• What is the difference between ECE 595 and other machine learning courses on campus?

  • We focus on general principles of learning.

  • Our goal is to provide an in-depth discussion of the subject, rather than superficially glancing through different topics

  • We put significant emphasis on understanding the mathematics behind the algorithms

  • We have plenty of hands-on programming exercises

• What will I learn after taking ECE 595?

• • You will know what a linear model is, such as Bayesian decision rule, perceptron algorithm, logistic regressoin, support vector machine, etc.

  • You will know how to understand a linear classifier from a geometric perspective.

  • You will know how to attack a classifier.

  • You will know the how much a machine learning algorithm can do, and what a machine learning algorithm cannot do.

  • You will know how to implement machine learning algorithms using Python and CVX.

• Can I audit the class?

  • Unfortunately, sorry, we do not allow audit in Spring 2021.

• Where can I get help for programming problems?

  • Please reach out to our teaching assistants.