Introduction to Machine Learning (CSC2515) - Fall 2022

CSC 2515 Fall 2022: Introduction to Machine Learning

Machine learning (ML) is a set of techniques that allow computers to learn from data and experience, rather than requiring humans to specify the desired behaviour manually. ML has become increasingly central both in AI as an academic field, and in industry. This course introduces the main concepts and ideas in machine learning, and provides an overview of many commonly used machine learning algorithms. It also serves as a foundation for more advanced ML courses.

By the end of this course, the students will learn about (roughly categorized)

Machine Learning Problems: Supervised (regression and classification), Unsupervised (clustering, dimension reduction), Reinforcement Learning
Models: Linear and Nonlinear (Basis Expansion and Neural Networks)
Loss functions: Squared Loss, Cross Entropy, Hinge, Exponential, etc.
Regularizers: l1 and l2
Probabilistic viewpoint: Maximum Likelihood Estimation, Maximum A Posteriori, Bayesian inference
Bias and Variance Tradeoff
Ensemble methods: Bagging and Boosting
Optimization technique in ML: Gradient Descent and Stochastic Gradient Descent

The students are expected to learn the intuition behind many machine learning algorithms and the mathematics behind them. Through homework assignments, they will also learn how to implement these methods and use them to solve simple machine learning problems. More details can be found in syllabus (to be posted) and piazza.

Announcements:

Teaching Staff:

Instructor: Amir-massoud Farahmand
- Email: csc2515-2022-09@cs.toronto.edu
- Office Hour: Tuesdays, 5-6PM
TAs: Claas Voelcker, Ekansh Sharma, Micaela Consens, Stephan Rabanser
- Office hours: Check the schedule of homework assignments as they vary. (maybe on Zoom)

Time & Location:

Tuesdays (Lectures):

Location: Emmanuel College, Room 1
Time: 3-5PM

Thursdays (Divided between Tutorials and Lectures)

Location: Claude T. Bissell Building, Room 205 (Might occasionally be on Zoom)
Time: 11AM-1PM

Lectures

This is a tentative schedule, especially the parts related to tutorials, and it will change.

Note on videos: The videos from Fall 2021 version of this course are available at YouTube. Use them if you have to miss a class or you need to review the material. The content of this year will mostly be the same (70-90%).

Week (date)	Topics	Material	Week’s schedule	Suggested reading
1 (Sept 12-16)	Lecture: Introduction to ML & K-Nearest Neighbours Tutorial: Probability review, Linear Algebra review, and preliminaries	Lecture: KNN slides video - part 1 video - part 2 Tutorial: Probability review slides & Linear Algebra review slides & preliminaries	Lec: Tue: Lec Tut: Thu	ESL 1. ESL 2.(1-3), 2.5, 13.3
2 (Sept 19-23)	Lecture: Decision Trees	DT slides video - part 1 video - part 2	Lec: Tue, Thu	ESL 9.2
3 (Sept 26-30)	Lecture: (Continued) Decision Trees Linear Models for Regression and Classification Tutorial: Optimization and Cross-Validation	Linear models slides video - part 1 video - part 2 video - part 3 video - part 4 Tutorials: Optimization and CV - notebook, worksheet (zip)	Lec: Tue Tut: Thu	ESL 3.(1,2,4), 4.(1,2,4); PRML 4.3
4 (Oct 3-7)	Lecture: (Continued) Linear Models for Regression and Classification Tutorial: Gaussian distribution (read on your own)	Gaussian distribution slides	Lec: Tue and Thu
5 (Oct 10-14)	Lecture: (Continued) Linear Models for Regression and Classification Bias-Variance Decomposition and Ensembles: Bagging	B&V slides video - part 1 video - part 2	Lec: Tue and Thu	ESL 7.3, 8.7, 15.1-2
6 (Oct 17-21)	Lecture: (Continued) Bias-Variance Decomposition and Ensembles: Bagging Support Vector Machines & Kernel Methods & Ensembles: Boosting	SVM, Kernels, & Boosting slides video - part 1 video - part 2 video - part 3	Lec: Tue and Thu	ESL 4.5.2, 12.(1-3), 10.(1-5)
7 (Oct 24-28)	Lecture: (Continued) SVM, Kernels, and Boosting Neural Networks	NN slides video - part 1 video - part 2	Lec: Tue and Thu	notes
8 (Oct 31 - Nov 4)	Lecture: (Continued) Neural Network		Lec: Tue and Thu
9 (Nov 14-18)	Lecture: Probabilistic Models Tutorial: NN with PyTorch (tentative)	Probabilisitic Models slides video - part 1 Tutorial: NN with PyTorch Colab (blank) Colab	Lec: Tue and Thu Tut: Thu (Zoom) at 2PM	ESL 2.6.3, 6.6.3, 4.3.0
10 (Nov 21-25)	Principal Component Analysis Ethics	PCA slides Ethics slides - Part 1 Ethics slides - Part 2	Lec: Tue and Thu	ESL 14.5.1
11 (Nov 28-Dec 2)	Lecture: K-Means Clustering Matrix Completion	KMeans slides	Lec: Tue, Thu	PRML 9.1-2
12 (Dec 5-9)	Lecture: Reinforcement Learning Tutorial: Policy gradient (read on your own)	RL slides Policy Gradient		RL 3, 4.1, 4.4, 6.1-6.5

Note on slides: Many have contributed to the design of these slides. Credit goes to several members of the ML Group at the U of T, and beyond, including (recent past, as far as I know): Roger Grosse, Murat Erdogdu, Richard Zemel, Juan Felipe Carrasquilla, Emad Andrews, and myself.

Assignments and Courswork

These are the main components of the course. The details are described below. You need to use MarkUs to submit your solutions (You need UTorID)

Three homework assignments (40%): 13.3% each
Take-home Test (10%): Release date: Nov 29; Due date: Dec 5
Project (30%)
Reading Assignments (10%): Due date: Nov 18 (2 papers) and Dec 7 (3 papers)
Questions & Answers (10%)
Bonus (5%): Finding typos in the slides, active class participation, evaluating the class, etc.

Homework Assignments

This is a tentative schedule of the homework assignments, and the exact date change according to the progress of the course. You have two weeks after the release date to submit your solutions.

Homework #	Out	Due	Materials	TA Office Hours
Homework 1	Sept 30	Oct 14, 16:59	Questions Dataset	(1) Oct 7 (Fri), 2PM at BA2270 (2) Oct 12 (Wed), 12PM (Zoom) (3) Oct 13 (Thu), 3PM (Zoom)
Homework 2	Oct 22	Nov 4, 16:59	Questions	(1) Oct 28 (Fri), 11AM-12PM at BA2270 (2) Nov 2 (Wed), 2-3PM at BA2270
Homework 3	Nov 9	Nov 22 25, 16:59	Questions Dataset	(1) Nov 14 (Mon), 3PM at BA3289 (2) Nov 18 (Fri), 11AM (Zoom) (3) Nov 23 (Wed), 11AM (Zoom)

Research Project

Read the instruction carefully!

Proposal (5%): November 14th
Project Report (25%): December 7 12th (extended)

Reading Assignments

The following papers (to be posted) are a combination of seminal papers in ML, topics that we didn’t cover in lectures, or active research areas. You need to choose five (5) papers out of them, depending on your interest. We will post the papers as the course progresses (so check here often). Please read them and try to understand them as much as possible. It is not important that you completely understand a paper or go into detail of the proofs (if there is any), but you should put some effort into it.

After reading each paper:

You should summarize it in a short paragraph (100-200 words). Highlight the main points of the paper. Ignore the less interesting aspects.
Try to come up with one or two suggestions on how the method/idea described in the paper can be used or extended.

Note: This list only covers a tiny fraction of all great papers published in various ML venues. Many of my favourites are not included.

There are two deadlines for Reading Assignments:

Nov 18 (for which you submit your summary of two papers)
Dec 7 (for which you submit the summary of three other papers)

Papers

Ming Yuan and Yi Lin, “Model selection and estimation in regression with grouped variables,” Journal of Royal Statistical Society (B), 2006. PDF
Ali Rahimi and Recht, “Random Features for Large-Scale Kernel Machines,” NIPS, 2007. PDF
Leon Bottou and Olivier Bousquet, “The Tradeoffs of Large Scale Learning,” Advances in Neural Information Processing Systems (NeurIPS), 2007. PDF
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in Neural Information Processing Systems (NeurIPS), 2012. PDF
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, “Deep Residual Learning for Image Recognition,” CVPR, 2016. PDF
Martin Zinkevich, “Online Convex Programming and Generalized Infinitesimal Gradient Ascent,” ICML, 2003. PDF
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, et al., “Human-level control through deep reinforcement learning,” Nature, 2015. PDF
Ian Goodfellow, Jonathan Shlens, and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” ICLR, 2015. PDF
Niranjan Srinivas, Andreas Krause, Sham Kakade, Matthias Seeger, “Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design,” ICML, 2010. PDF
Moritz Hardt, Eric Price, and Nathan Srebro, “Equality of opportunity in supervised learning,” Advances in Neural Information Processing Systems (NeurIPS), 2016. PDF
Ronan Collobert and Jason Weston, “A Unified Architecture for Natural Language Processing: Deep Neural Networks with Multitask Learning,” International Conference on Machine Learning (ICML), 2008. PDF
Jeffrey Pennington, Richard Socher, and Christopher D. Manning, “GloVe: Global Vectors for Word Representation,” Empirical Methods in Natural Language Processing (EMNLP), 2014. PDF

Questions & Answers

The goal is to encourage you to reflect on the content of lectures. You do this by writing down two questions based on the content of the assigned lecture(s). You also need to try to answer those questions and write your answers down. The questions should be beyond what is covered in the lecture or discussed in the class. The goal is to make you think deeper about the content. You can search on the Internet to try to find answer to your questions. You do not need to answer the questions successfully or completely. It is enough to show that you have seriously thought about them. There is no strict rule about how long your submission should be, but an equivalent of 0.5-1 page should be enough. You have almost 7 days after the lecture is finished to submit the assignment: If the lecture is finished on Tuesday, you have until 5PM of Monday, and if it is finished on Thursday, you have until 5PM of Wednesday.

Some of the lectures are grouped together, and you only need to submit one QA for them. The number of questions is still two: You do not need to submit four questions if two lectures are combined.

QA1: Lecture 1 (KNN) + Lecture 2 (DT) [Due date: Oct 3rd]
QA2: Lecture 3 (Linear Models) [Due date: Oct 19th]
QA3: Lecture 4 (Bias-Variance), Lecture 5 (SVM, Kernels, Boosting), and Lecture 6 (NN) [Due date: Nov 15th]
QA4: Lecture 9 (Ethics). Please read this and write a short essay. [Due date: Dec 5th]
QA5: Lecture 8 (PCA), Lecture 10 (Matrix Factorization), Lecture 11 (Clustering). [Due date: TBA]
QA6: Lecture 12 (RL). [Due date: TBA]

Computing Resources

For the homework assignments, we will use Python, and libraries such as NumPy, SciPy, and scikit-learn. You have two options:

The easiest option is probably to install everything yourself on your own machine.
- If you don’t already have python, install it. We recommend using Anaconda. You can also install python directly if you know how.
- Optionally, create a virtual environment for this class and step into it. If you have a conda distribution run the following commands:
```
  conda create --name csc2515
  source activate csc2515
```
- Use pip to install the required packages pip install scipy numpy autograd matplotlib jupyter sklearn
All the required packages are already installed on the Teaching Labs machines.