Machine Learning DS-GA 1003 · Spring 2021 · NYU Center for Data Science

Instructors He He
Marylou Gabrié
Ming Liao (NYUSH)
Lecture Tuesday 5:20pm–7pm (blended)
Lab Wednesday 6:45pm–7:35pm (blended)
Wednesday 8am-8:50am (online)
Wednesday 9am-9:50am (blended)
Wednesday 7:45pm-8:35pm (blended)
Thursday 11:00am-12:00pm (in-person, Shainghai local time)
Office Hours All OH will be remote on Zoom. See calendar.
Location Rooms and Zoom links are on NYUClasses and the course calendar.

About This Course

This course covers a wide variety of introductory topics in machine learning and statistical modeling, including statistical learning theory, convex optimization, generative and discriminative models, kernel methods, boosting, latent variable models and so on. The primary goal is to provide students with the tools and principles needed to solve the data science problems found in practice. This course was designed as part of the core curriculum for the Center for Data Science's Masters degree in Data Science, and is intended as a continuation of DS-GA-1001 Intro to Data Science. This course also serves as a foundation on which more specialized courses and further independent study can build. A tentative syllabus can be found here.

For registration information, please contact Kathryn Angeles or Elisa Fox.

Prerequisites

Logistics

Grading

Resources

Related courses

Textbooks

The cover of Elements of Statistical Learning The cover of An Introduction to Statistical Learning The cover of Understanding Machine Learning: From Theory to Algorithms The cover of Pattern Recognition and Machine Learning The cover of Bayesian Reasoning and Machine Learning
The Elements of Statistical Learning (Hastie, Friedman, and Tibshirani)
This will be our main textbook for L1 and L2 regularization, trees, bagging, random forests, and boosting. It's written by three statisticians who invented many of the techniques discussed. There's an easier version of this book that covers many of the same topics, described below. (Available for free as a PDF.)
An Introduction to Statistical Learning (James, Witten, Hastie, and Tibshirani)
This book is written by two of the same authors as The Elements of Statistical Learning. It's much less intense mathematically, and it's good for a lighter introduction to the topics. (Available for free as a PDF.)
Understanding Machine Learning: From Theory to Algorithms (Shalev-Shwartz and Ben-David)
Covers a lot of theory that we don't go into, but it would be a good supplemental resource for a more theoretical course, such as Mohri's Foundations of Machine Learning course. (Available for free as a PDF.)
Pattern Recognition and Machine Learning (Christopher Bishop)
Our primary reference for probabilistic methods, including bayesian regression, latent variable models, and the EM algorithm. It's highly recommended, but unfortunately not free online.
Bayesian Reasoning and Machine Learning (David Barber)
A very nice resource for our topics in probabilistic modeling, and a possible substitute for the Bishop book. Would serve as a good supplemental reference for a more advanced course in probabilistic modeling, such as DS-GA 1005: Inference and Representation (Available for free as a PDF.)
Hands-On Machine Learning with Scikit-Learn and TensorFlow (Aurélien Géron)
This is a practical guide to machine learning that corresponds fairly well with the content and level of our course. While most of our homework is about coding ML from scratch with numpy, this book makes heavy use of scikit-learn and TensorFlow. Comfort with the first two chapters of this book would be part of the ideal preparation for this course, and it will also be a handy reference for practical projects and work beyond this course, when you'll want to make use of existing ML packages, rather than rolling your own.
Data Science for Business (Provost and Fawcett)
Ideally, this would be everybody's first book on machine learning. The intended audience is both the ML practitioner and the ML product manager. It's full of important core concepts and practical wisdom. The math is so minimal that it's perfect for reading on your phone, and I encourage you to read it in parallel to doing this class, especially if you haven't taken DS-GA 1001.

Other tutorials and references

Software

Lectures

Week 1

Topics Materials References

Lecture (HH) Feb 2

Topics

Materials

References

(None)

Lab Feb 3

Topics

Materials

References

Week 2

Topics Materials References

Lecture (HH) Feb 9

Topics

Materials

References

(None)

Lab Feb 10

Topics

Materials

(None)

References

(None)

Week 3

Topics Materials References

Lecture (HH) Feb 16

Topics

Materials

References

  • HTF Ch. 3.3-3.4

Lab Feb 17

Topics

Materials

References

(None)

Week 4

Topics Materials References

Lecture (HH) Feb 23

Topics

Materials

References

(None)

Lab Feb 24

Topics

Materials

(None)

References

(None)

Week 5

Topics Materials References

Lecture (HH) Mar 2

Topics

  • Kernel SVMs
  • The Representer Theorem and Kernelization

Materials

(None)

References

Lab Mar 3

Topics

(None)

Materials

(None)

References

(None)

Week 6

Topics Materials References

Lecture (MG) Mar 9

Topics

  • Maximum likelihood estimation
  • Probabilisitc models

Materials

(None)

References

(None)

Lab Mar 10

Topics

  • MLE review and exercise
  • Multivariate Gaussians

Materials

(None)

References

(None)

Week 7

Topics Materials References

Lecture (MG) Mar 16

Topics

  • Bayesian methods

Materials

(None)

References

  • Barber 9.1, 18.1
  • Bishop 3.3

Lab Mar 17

Topics

  • Review for Midterm

Materials

(None)

References

(None)

Week 8

Topics Materials References

Midterm Exam Mar 23

Topics

  • Cover material up to week 6

Materials

(None)

References

(None)

Lab Mar 24

Topics

  • Bayesian linear regression

Materials

(None)

References

(None)

Week 9

Topics Materials References

Lecture (HH) Mar 30

Topics

  • Multiclass classification
  • Structured prediction

Materials

(None)

References

  • SSBD Chapter 17

Lab Mar 31

Topics

  • Midterm solution discussion

Materials

(None)

References

(None)

Week 10

Topics Materials References

Lecture (HH) Apr 6

Topics

  • Classification and regression trees
  • Bagging and random forests
  • Adaboost

Materials

(None)

References

  • JWHT 8.1 (Trees)
  • HTF 9.2 (Trees)

Lab Apr 7

Topics

(None)

Materials

(None)

References

(None)

Week 11

Topics Materials References

Lecture (HH) Apr 13

Topics

  • Forward stagewise additive modeling
  • Gradient boosting

Materials

(None)

References

Lab Apr 14

Topics

  • Review on gradient boosting
  • (demo) L2 boosting
  • XGBoost

Materials

References

(None)

Week 12

Topics Materials References

Lecture (HH) Apr 20

Topics

  • Neural networks
  • Backpropagation and computation graph

Materials

(None)

References

Lab Apr 21

Topics

(None)

Materials

(None)

References

(None)

Week 13

Topics Materials References

Lecture (HH) Apr 30

Topics

  • Latent variable models
  • Gaussian mixture models
  • Expectiation Maximization

Materials

(None)

References

(None)

Lab May 1

Topics

(None)

Materials

(None)

References

(None)

Week 14

Topics Materials References

Lecture (HH) May 4

Topics

  • Course review
  • Advice on practical ML
  • Trustworthy ML

Materials

(None)

References

(None)

Lab May 5

Topics

  • Review for Final

Materials

(None)

References

(None)

Assignments

Late Policy: Homeworks are due at 11:59 PM EST on the date specified. You have seven late days in total which can be used throughout the semester without penalty. Once you run out of late days, each additional late day will incur a 20% penalty. For example, if you submit an assignment 1 day late after using all your late days, a score of 90 will only be counted as 72. Note that the maximum late days per homework is two days, meaning that Gradescope will not accept submissions 48 hours after the due date.

Collaboration Policy: You may form study groups and discuss problems with your classmates. However, you must write up the homework solutions and the code from scratch, without referring to notes from your joint session. In your solution to each problem, you must write down the names of any person with whom you discussed the problem—this will not affect your grade.

Submission: Homework should be submitted through Gradescope. If you have not used Gradescope before, please watch this short video: "For students: submitting homework." At the beginning of the semester, you will be added to the Gradescope class roster. This will give you access to the course page, and the assignment submission form. To submit assignments, you will need to:

  1. Upload a single PDF document containing all the math, code, plots, and exposition required for each problem.
  2. Where homework assignments are divided into sections, please begin each section on a new page.
  3. You will then select the appropriate page ranges for each homework problem, as described in the "submitting homework" video.

Feedback: Check Gradescope to get your scores on each individual problem, as well as comments on your answers. Regrading requests should be submitted on Gradescope.

Homework 0

Typesetting your homework

Due: January 1st, 11:59 PM EST

Homework 1

Error Decomposition and Polynomial Regression

Due: February 10th, 11:59 PM EST

Homework 2

Gradient Descent and Regularization

Due: February 22nd, 11:59 PM EST

Homework 3

SVMs and Kernel Methods

Due: March 15th, 11:59 PM EST

People

Instructors

Section Leaders

Graders