[expand]

[expand all]

[collapse all]

Overview

This course covers a selected set of topics in machine learning and data mining, with an emphasis on good methods and practices for deployment of real systems. The majority of sections are related to commonly used supervised learning techniques, and to a lesser degree unsupervised methods. This includes fundamentals of algorithms on linear and logistic regression, decision trees, support vector machines, clustering, neural networks, as well as key techniques for feature selection and dimensionality reduction, error estimation and empirical validation.
Prerequisites [click to expand the list]
This course requires programming skills (python) and basic knowledge of probabilities, calculus and linear algebra. For more information see the course prerequisites and restrictions at McGill's webpage.

Textbooks

• [Bishop] Pattern Recognition and Machine Learning by Christopher Bishop (2007)
• [Goodfellow] Deep Learning by Ian Goodfellow, Yoshua Bengio, and Aaron Courville (2016)
• [Murphy] Machine Learning: A Probabilistic Perspective by Kevin Murphy (2012)
• [Murphy'22] Probabilistic Machine Learning: An Introduction, by Kevin P. Murphy (2022)
Chapters from these four books are cited as optional reference materials for the slides.
There are several other related references. [click to expand the list]
The Elements of Statistical Learning: Data Mining, Inference, and Prediction by Trevor Hastie, Robert Tibshirani and Jerome Friedman (2009)
Information Theory, Inference, and Learning Algorithms, by David MacKay (2003)
Bayesian Reasoning and Machine Learning, by David Barber (2012).
Understanding Machine Learning: From Theory to Algorithms, by Shai Shalev-Shwartz and Shai Ben-David (2014)
Foundations of Machine Learning, by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar (2018)
Dive into Deep Learning, by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola (2019)
Mathematics for Machine Learning, by Marc Peter Deisenroth, A Aldo Faisal, and Cheng Soon Ong (2019)
A Course in Machine Learning, by Hal Daume III (2017)
Hands-on Machine Learning with Scikit-Learn and TensorFlow, by Aurelien Geron (2017)
Machine Learning, by Tom Mitchell (1997)
Introduction to Data Mining, by Pang-Ning Tan, Michael Steinbach, Anuj Karpatne, and Vipin Kumar (2020)
Machine Learning, Dynamical Systems and Control, by Steven L. Brunton and J. Nathan Kutz (2019)

Key Dates

• Thu Aug. 31: First Class
• Tue Sept. 12: Drop class deadline [make sure to have completed the 2 online pre-class quizzes by then, to determine if you have the required prerequisits for this course]
• Tue Sept. 19: Mini-Project 1 assignment given [you should have selected your group of 3 by then]
• Thu. Oct. 5: Mini-project 1 due, Mini-Project 2 assigned [groups for mini-project 2 should have been selected]
• Tue. Oct. 10: Reading week, no class
• Tue. Oct. 31: Mini-Project 2 due, Mini-Project 3 assigned
• Thu. Nov. 2: Tentative date for late Midterm
• Tue. Nov. 16: Mini-Project 3 due, 4 assigned
• Thu. Nov. 30: Make-up days, no class
• Tue. Dec. 5: Mini-Project 4 due, last day of class

Outline

Syllabus
slides
Introduction
slides, reference: 1 [Murphy22]
Parameter Estimation
slides, notebook (Colab), reference: 4 [Murphy22], 2-2.3 [Bishop], 3-3.5 [Murphy]
Linear regression
slides, notebook (Colab), reference: 7-7.3.3 [Murphy], 3-3.1.2 [Bishop]
Logistic and softmax regression
slides, notebook (Colab), reference: 8.1-8.3.3 [Murphy], 4.1-4.1.3 + 4.3-4.3.3 [Bishop]
Gradient descent methods
slides, notebook (Colab), reference: 8.3.2 [Murphy] and this overview by S. Ruder (in pdf )
Regularization
slides, notebook (Colab), reference: 3.1.4-3.3 [Bishop]
Generalization
slides, notebook for model selection (Colab), second notebook for curse of dimensionality (Colab)
Review 1
slides,
Perceptrons & multilayer perceptrons
slides, Perceptrons Colab, MLP demo, reference: 4.1.1-4.1.3 + 4.1.7 [Bishop], 6-6.5 + parts of 7 [Goodfellow]
Gradient computation and automatic differentiation
slides, notebook (Colab), reference: 6.5 + 8.2 [Goodfellow], blog post, visualization
Convolutional neural networks
slides, reference: 9 [Goodfellow], blog post, optional reading , interactive demo
Neural Networks for Sequences
slides, notebook (Colab), reference: 15 [Murphy'22], optional reading ,
For a quick understand of the LSTM, the prototypical Recurrent NeuralNet for sequeces, see this great blogpost. ,

Naive Bayes
slides, notebook (Colab), reference: 3.5-3.5.4 [Murphy]
Nearest neighbours
slides, notebook (Colab), reference: chapter 1 [Murphy]
Classification and regression trees
slides, notebook (Colab), reference: 16.1-16.2.6 [Murphy], 14.4 [Bishop]
Linear support vector machines
slides, notebook (Colab), reference: 4.1.1-4.1.3 + 4.1.7 + 7.1-7.1.4 excluding kernels [Bishop]
Bagging & boosting
slides, notebook (Colab), reference: 3.2 [Bishop], demos for Bias-Variance Tradeoff, Gradient Boosting explanation, and Interactive playground
Unsupervised learning
slides, notebook (Colab), reference: 25.5 [Murphy] and 9.1 [Bishop], demos for K-Means and DB-SCAN
Dimensionality reduction
slides, notebook (Colab), reference: 12.2 [Murphy], 12.1 [Bishop], demo

Evaluation

Weekly Practice Quizzes [10%]
One quiz per week to check the key topics discussed
Late Mid-term Exam [40%]
In person, tentatively on November 2nd
Closed book, but you can bring 5 pages of your hand written notes
Hands-on Mini-Projects [50%]
Four programming assignments to be done in groups of three*, *no exception to this given the grading load on TAs
Groups can stay the same between projects, you can also regroup when needed
All group members receive the same mark unless there are major complaints on not contributing, responding, etc. from group-mates, which will be resolved in a case by case basis. If a significant difficulty/conflict arises, please send an email to the course email, put 'Group-Issue' in the title
Work submitted for evaluation as part of this course may be checked with text-matching software within myCourses
Late submission policy
All due dates are 11:59 pm in Montréal, unless specified otherwise [e.g. check the due dates for quizzes].
No make-up quizzes will be given.
For mini-projects, 2^k% percent will be deducted per k days of delay.
If you experience barriers to learning in this course, submitting the projects, etc., please do not hesitate to discuss them with me directly, and please make sure to put "551 special" in the header to make sure I see your email [for general course correspondence, please use the course email: comp551mcgill@gmail.com]. As a point of reference, you can reach the Office for Students with Disabilities at 514-398-6009.

Academic Integrity

The ``McGill University values academic integrity. Therefore, all students must understand the meaning and consequences of cheating, plagiarism and other academic offenses under the Code of Student Conduct and Disciplinary Procedures'' (see McGill's webpage for more information). (Approved by Senate on 29 January 2003)

Other Useful Online Resources

Learning plan
metacademy
https://github.com/afshinea/stanford-cs-229-machine-learning
Video Playlists
StatQuest [💥😋 Loved by many past students]
FreeCodeCamp
Essence of linear algebra and Neural Networks by 3Blue1Brown
Mathematics for ML by David Rolnick
Courses with Playlist and/or Code
Introduction to Machine Learning by Google
Machine Learning by Stanford
Deep Learning by UC Berkeley
Hinton's Lectures on Neural Networks for Machine Learning
Deep Learning & Linear Algebra courses by fastai
Learning from Data by Caltech
Deep Learning (with PyTorch) playlist and course by NYU
Deep Learning by Stanford
Deep Learning by deeplearning.ai
Introduction to Deep Learning by MIT
Information Theory, Pattern Recognition, and Neural Networks by David MacKay
Good Blogs for Conceptual Understanding
Christopher Olah's Blog
Fast.AI
Distill.pub: Amazing Online and Interactive ML Journal, more accessible than other typical scientific publications
inFERENCE: Ferenc Huszár's Blog
Books with Code
Probabilistic Machine Learning: An Introduction by Kevin Murphy (book 1)
Dive into Deep Learning BY by Aston Zhang, Zachary Lipton, Mu Li, and Alexander J. Smola
Machine Learning Notebooks for O'Reilly book Hands-on Machine Learning with Scikit-Learn and TensorFlow
Similar Courses - Graduate Level
https://www.cs.toronto.edu/~rgrosse/courses/csc2515_2019/
https://www.cs.cornell.edu/courses/cs4780/2019fa/
Similar Courses - Undergraduate Level
hhttps://cs.mcgill.ca/~wlh/comp451/schedule.html
https://www.cs.toronto.edu/~rgrosse/courses/csc311_f20/
https://www.cs.toronto.edu/~rgrosse/courses/csc411_f18/
http://cs229.stanford.edu/syllabus-fall2020.html
https://cs230.stanford.edu/lecture/
Cheatsheets: https://stanford.edu/~shervine/teaching/
Similar Courses - Last Versions
Fall 2019
Winter 2020
Fall 2020
Winter 2023

FAQ

• Should I take this course or Comp 451? what is the difference?
If you are interested in graduate/advanced level Machine Learning, to for example be able to design new machine learning algorithms, consider taking Comp 451. This course gives you the fundamentals and in depth review of ML needed for a followup graduate level ML course. If you want a one-stop ML course to know how ML algorithm works and how to apply them, take Comp 551, which is alternative to Comp 451 + followup advanced ML course. That means Comp 451 is more appropriate for CS undergraduates interested in AI/ML.
• Class/waitlist is full, can I still register?
Unfortunately you will have to wait for the next semester. As an alternative, consider Fundamental of Machine Learning Course, Comp 451, and please check the list of free online courses below.
• Who to contact for department approval required for taking the course?
Please contact teresa.pian@mcgill.ca.
• Do I have the prerequisites to take the course?
This course requires strong Python programming skills and basic knowledge of probabilities, [multivariate] calculus and linear algebra. Please check this quiz to test if your background is strong enough for taking the course. It can also be used to diagnose where your background might be lacking and be used to self-study before taking the course. Most concepts covered in these questions will be used throughout the course in the slides.
• How similar is it to the last years?
Very similar, please check last year's websites to get a glimpse of the slides, expectations, etc. We will have an updated version and not exactly the same materials but very similar overall.
• Will there be lecture recordings?
Yes, and class participation is not mandatory.
• What do I learn in this course?
You will learn how the most common machine learning algorithms are designed, how they are implemented, and how to apply them in practice. This course has a heavy theory component, since it is important to understand the inner-workings of the algorithms in order to effectively utilize them in practice. Please check the Outline section above for more information but note that everything is tentative.
• This website is too long and complicated, can I just ask a chatbot instead?
Yes! Kellin Pelrine made an LLM chatbot with the info on this webpage, click here to try it out! (You might want to double-check the answers though)