Introduction to Probability for Data Science

Overview

I am fortunate to have the opportunity to witness and contribute to teaching several of the most important data science courses in Purdue ECE and college of engineering. This book is a collection of materials that I find fundamental, interesting, and practical. It is written based on three courses I have taught/created:

• ECE 20875 (was ECE 295) Introduction to Data Science with Python (Sophomore)

• ECE 302 Probabilistic Methods for Electrical and Computer Engineering (Junior-Senior)

• ECE 595ML Machine Learning (Graduate)

• ECE 645 Estimation Theory (Graduate)

As I write the book, I have done a fairly exhaustive search of the available textbooks on this subject. It was quite surprising to see that while there is a tsunami of data science books on the internet, many of them are written for programmers. I am not overlooking the importance of these books, but in my opinion college students need a more solid mathematical training so that they can pursue a more advanced career. However, on the other end of the spectrum, classical probability textbooks are everywhere. While these books offer great details, many of them do not have a soul. Why should we learn probability? How can flipping a coin be useful in modern data science? Can we help undergraduate students to appreciate measure theory? Why does the Gaussian have a bell shape? Where does the Poisson distribution come from? How to fit data with a line? How to tell whether a change is statistically significant?

I hope that the book will become a valuable asset to our community. The book is not yet finished, and I am actively revising the book. If you have any suggestions, I would appreciate you send me an email and let me know.

Stanley Chan, Feb 2021.

Chapters

• Front Matter
  

• Chapter 1 Mathematical Background (Update: 2-10-2021)
  

• Chapter 2 Probability (Update: 2-10-2021)
  

• Chapter 3 Discrete Random Variables (Update: 2-10-2021)
  

• Chapter 4 Continuous Random Variables (Update: 2-10-2021)
  

• Chapter 5 Joint Distributions (Update: 2-10-2021)
  

• Chapter 6 Sample Statistics (Update: 2-10-2021)
  

• Chapter 7 Regression (Update: 2-10-2021)
  

• Chapter 8 Estimation (Update: 2-10-2021)
  

• Chapter 9 Confidence and Hypothesis (Update: 2-27-2021)
  

• Chapter 10 Random Processes (Update: 2-27-2021)
  

Dataset

(Will expand)

• Chatper 3 English Alphabets

• Chapter 7 NHANES dataset

• Chapter 9 ROC example

• Chapter 10 Weiner filtering example
  
  

Videos and Slides

• Chapter 1

  • Lecture 1.1 Infinite Series (Video) (Slide)
    

  • Lecture 1.2 Approximation (Video) (Slide)
    

  • Lecture 1.3 Integration (Video) (Slide)
    

  • Lecture 1.4 Linear Algebra (Video) (Slide)
    

  • Lecture 1.5 Combinatorics (Video) (Slide)
    

• Chapter 2

  • Lecture 2.1 Set Theory (Video) (Slide)
    

  • Lecture 2.2 Probability Space (Video) (Slide)
    

  • Lecture 2.3 Axioms of Probability (Video) (Slide)
    

  • Lecture 2.4 Conditional Probability (Video) (Slide)
    

  • Lecture 2.5 Independence (Video) (Slide)
    

  • Lecture 2.6 Bayes Theorem (Video) (Slide)
    

• Chapter 3

  • Lecture 3.1 Random Variables(Video) (Slide)
    

  • Lecture 3.2 Probability Mass Function (Video) (Slide)
    

  • Lecture 3.3 Cumulative Distribution Function (Video) (Slide)
    

  • Lecture 3.4 Expectation (Video) (Slide)
    

  • Lecture 3.5 Moments and variance (Video) (Slide)
    

  • Lecture 3.6 Bernoulli random variables (Video) (Slide)
    

  • Lecture 3.7 Binomial random variables (Video) (Slide)
    

  • Lecture 3.8 Geometric random variables (Video) (Slide)
    

  • Lecture 3.9 Poisson random variables (Video) (Slide)
    

• Chapter 4

  • Lecture 4.1 Probability density function (Video) (Slide)
    

  • Lecture 4.2 Expectation (continuous) (Video) (Slide)
    

  • Lecture 4.3 Cumulative distribution function (continuous) (Video) (Slide)
    

  • Lecture 4.4 Mean, mode, median (Video) (Slide)
    

  • Lecture 4.5 Uniform random variables (Video) (Slide)
    

  • Lecture 4.6 Exponential random variables (Video) (Slide)
    

  • Lecture 4.7 Gaussian random variables (Video) (Slide)
    

  • Lecture 4.8 Transformation of random variables (Video) (Slide)
    

  • Lecture 4.9 Generating random numbers (Video) (Slide)
    

• Chapter 5

  • Lecture 5.1 Joint PMF, PDF, and CDF (Video) (Slide)
    

  • Lecture 5.2 Joint expectation (Video) (Slide)
    

  • Lecture 5.3 Correlation and covariance (Video) (Slide)
    

  • Lecture 5.4 Conditional distributions (Video) (Slide)
    

  • Lecture 5.5 Conditional expectation (Video) (Slide)
    

  • Lecture 5.6 Sum of two random variables (Video) (Slide)
    

  • Lecture 5.7 Examples for sum of two random variables (Video) (Slide)
    

  • Lecture 5.8 Random vector and covariance matrix (Video) (Slide)
    

  • Lecture 5.9 Multi-dimensional Gaussian (Video) (Slide)
    

  • Lecture 5.10 Gaussian whitening (Video) (Slide)
    

  • Lecture 5.11 Principal component analysis (Video) (Slide)
    

• Chapter 6

  • Lecture 6.1 Moment generating functions (Video) (Slide)
    

  • Lecture 6.2 Characteristic functions (Video) (Slide)
    

• Chapter 10

  • Lecture 10.1 Intro to random processes (Video) (Slide)
    

  • Lecture 10.2 Mean functions (Video) (Slide)
    

  • Lecture 10.3 Autocorrelation functions (Video) (Slide)
    

  • Lecture 10.4 Autocovariance functions, independent processes (Video) (Slide)
    

  • Lecture 10.5 Wide sense stationary processes (Video) (Slide)
    

  • Lecture 10.6 Power spectral density (Video) (Slide)
    

  • Lecture 10.7 Linear time invariant systems (Video) (Slide)
    

  • Lecture 10.8 Mean and autocorrelation through LTI systems (Video) (Slide)
    

  • Lecture 10.9 Cross-correlation through LTI systems (Video) (Slide)