Introduction to Reinforcement Learning (Fall 2026)

You can find the Fall 2025 (PolyMtl) version of this course here and the Spring 2021 (U of T) version here.

This is an introductory course on reinforcement learning (RL) and sequential decision-making under uncertainty with an emphasis on understanding the theoretical foundation. We study how dynamic programming methods such as value and policy iteration can be used to solve sequential decision-making problems with known models, and how those approaches can be extended in order to solve reinforcement learning problems, where the model is unknown. Other topics include, but not limited to, function approximation in RL, policy gradient methods, model-based RL, and balancing the exploration-exploitation trade-off. The course will be delivered as a mix of lectures, tutorial sessions, and reading of classical and recent papers assigned to students. The deliverables includes homework assignments, reading assignment, a project, and a final exam. As the emphasis is on understanding the theoretical foundation, you should expect to go through mathematical detail and proofs. Required background for this course includes being comfortable with probability theory and statistics, calculus, linear algebra, optimization, and (supervised) machine learning. You should also be comfortable writing code.

---

Announcements:

• The Math tutorial will be online. We use this Zoom link.
• [Friday, September 4, 2026] If you are interested in taking the course but could not enrol it, please contact me as soon as possible (amirmassoud.farahmand at the free email service provided by Google), so I request the department to manually add you (before Monday, September 7th). The space is physically very limited and I cannot guarantee that you would be added. Also if you are thinking of dropping the course, I encourage you to do it sooner than later, so that other interested students can enrol.

---

Teaching Staff

• Instructor: Amir-massoud Farahmand
  • Email: amirmassoud.farahmand at the free email service provided by Google
• TAs: Ali Saheb Pasand and Thomas Mousseau

Time and Location:

• Lecture: Mondays, 12:45–3:30 at B-415
• Labs: Fridays, 11:30AM-2:30PM at M-1120
• Instructor’s Office Hour: Mondays, 4:00-5:00 at M-3424

Reading

The course material is based on Foundations of Reinforcement Learning. This is a live document and I will chapters as we progress through the course. If you want to take a look of the complete book, as of November 2025, you can take a look here Foundations of Reinforcement Learning (2025 version). If you find a typo or mistake, please let me know. I collect the list of reported ones here.

Some other useful textbooks (incomplete list):

• Richard Sutton and Andrew Barto, Introduction to Reinforcement Learning (2nd edition), 2018.
• Csaba Szepesvári, Algorithms of Reinforcement Learning, 2010.
• Dimitri Bertsekas, Abstract Dynamic Programming (2nd Edition)
• Dimitri Bertsekas and John Tsitsiklis, Neuro-Dynamic Programming, 1996.

Lectures

This is a tentative schedule, and will adaptively, based on progress, change.

Note on videos: The videos of a previous version of this course are publicly available on YouTube. The content is similar, but not exactly the same. You should be able to use the video to review the material or catch up a missed session, but they are not a substitute for class presence.

Week (date)	Topics	Lectures	Reading
1 (Aug 24)	Introduction to Reinforcement Learning (Part I) No Tutorial	slides (Intro) video (Intro) slides (Intro – annotated - Part I)	Chapter 1 of FRL
2 (Aug 31)	Introduction to Reinforcement Learning (Part II) Tutorial (Friday, Sept 4 – Online): Math Background Review	slides (Intro – annotated - Part II) Tutorials: Probability, Linear Algebra, Optimization
2’ (Sept 7)	(No Lecture – Labour Day) Tutorial (Friday, Sept 11): Q Learning
3 (Sept 14)	Structural Properties of Markov Decision Processes (Part I) Tutorial (Friday, Sept 18): PyTorch	slides (MDP) video (MDP - Part I), video (MDP - Part II)	Chapter 2 FRL
4 (Sept 21)	Structural Properties of Markov Decision Processes (Part II) Planning with a Known Model (Part I) Tutorial (Friday, Sept 25): RL Environments	video (Planning)	Chapter 3 FRL
5 (Sept 28)	Planning with a Known Model (Part II) Learning from a Stream of Data (Part I) Tutorial (Friday, Oct 2): TBD		Chapter 4 FRL
6 (Oct 5)	Learning from a Stream of Data (Part II) Tutorial (Friday, Oct 9): TBD
6’ (Oct 12)	(No Lecture or Tutorial – Reading Week, Oct 10–16)
7 (Oct 19)	Value Function Approximation (Part I) Tutorial (Friday, Oct 23): TBD	video (VFA - Part I), video (VFA - Part II), video (VFA - Part III)	Chapter 5 FRL
8 (Oct 26)	Value Function Approximation (Part II) Tutorial (Friday, Oct 30): TBD
9 (Nov 2)	Value Function Approximation (Part III) Tutorial (Friday, Nov 6): TBD
10 (Nov 9)	Policy Search Methods (Part I) Tutorial (Friday, Nov 13): TBD	video (PS - Part I), video (PS - Part II)	Chapter 6 FRL
11 (Nov 16)	Policy Search Methods (Part II) Tutorial (Friday, Nov 20): TBD
12 (Nov 23)	Model-based RL Tutorial (Friday, Nov 27): TBD		Chapter 7 FRL
13 (Nov 30)	Exploration–Exploitation OR Buffer/Catch-up Tutorial (Friday, Dec 4): TBD

Assignments and Coursework

These are the main components of the course. The details are described below. You need to use Gradescope, linked through Moodle, to submit your solutions.

• Four homework assignments: 30% (the lowest mark is excluded; the top three are 10% each)
• Project: 30% (Proposal: 5%; Report: 25%)
• Final Exam: 30% (Date: December 17)
• Reading Assignments: 10% (3 papers) Due date: Oct 30 (1 paper) and Nov 26 (2 papers)

Homework Assignments

There will be four homework assignments and your grade will be the average of the top three of them.

This is a tentative schedule of the homework assignments. A homework may be released earlier than the lecture is completed, but when enough material has been covered to start working on it. The deadline will be at two weeks after the final lecture covering the relevant topic. The deadline is 16:59. The exact date may change depending on the pace of lectures.

• Homework 1: Lectures 1 & 2 (Intro + Structural)
• Homework 2: Lectures 3 & 4 (Planning + Learning from Stream)
• Homework 3: Lecture 5 (Value Function Approximation)
• Homework 4: Lecture 6 (Policy Search)

Homework #	Out	Due	Materials	TA Office Hours
Homework 1	Sept 14	Sept 30	Questions	Friday (Sept 18, Sept 25)
Homework 2	Oct 5	Oct 23	Questions Code	Friday (Oct 9, Oct 23)
Homework 3	Oct 26 (or after HW2 closes)	Nov 17	Questions, Code - DQN, Code - QL Bias	Friday (Oct 30, Nov 6, Nov 13)
Homework 4	Nov 18	Nov 30	Questions Code	Friday (Nov 20, Nov 27)

HW2 contingency: If Learning from a Stream of Data is not completed before Reading Week and instead finishes on a later day (likely October 19), HW2 will be due on November 2. HW3 will then be released after HW2 closes.

Research Project

Read the instruction here!

• Proposal (5%): November 6th, 4:59PM
• Project Report (25%): December 11th, 4:59PM

Reading Assignments

The following papers are a combination of classic papers in RL, topics that we didn’t cover in lectures, or active research areas. You need to choose three (3) papers out of them, depending on your interest. Please read them and try to understand them as much as possible. It is not important that you completely understand a paper or go into detail of the proofs (if there is any), but you should put some effort into it.

After reading each paper:

• You should summarize it in a short paragraph (100-200 words). Highlight the main points of the paper. Ignore the less interesting aspects.
• Try to come up with one or two suggestions on how the method/idea described in the paper can be used or extended. Be thoughtful and creative! Don’t suggest the obvious improvements, such as “add more experiments” for a theory paper, or “prove some theory” for an empirical paper, or “add a NN as the function approximator” for a non-deep RL paper.

It is OK to discuss the paper with a Large Language Model (LLM), after you read it yourself. But do not use any LLM to write your summary. The act of writing itself tremendously helps you understand the paper better. If you delegate it to a machine, you will lose the opportunity to learn and solidify your understanding. Avoid using an LLM to come up with ideas. This is a practice for your creativity, not a machine’s.

These three assignments contribute 10% to your final mark. The reading assignments are only lightly evaluated, but we may randomly ask you to explain the papers to us. You should submit the summaries in two batches. The first batch should include one paper and is due on October 30th. The second batch should include two other papers and is due on November 26th. In both cases, submit only one PDF file, before 5PM.

We will post the papers as the course progresses. Please read and summarize them as we post them, so you won’t have a large workload close to the end of the semester.

Note: that this is an incomplete and biased list. I have many favourite papers that are not included in this short list.

• Sutton, Precup, Singh, “Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning,” Journal of Artificial Intelligence, 1999. PDF [AltMDP. Introduces the Options framework, as one of the major approaches to Hierarchical RL.]
• Bellemare, Dabney, Munos, “A Distributional Perspective on Reinforcement Learning,” ICML, 2017. PDF [AltMDP. How can we learn the distribution of returns instead of their expectation?]
• Barreto, Dabney, Munos, Hunt, Schaul, van Hasselt, Silver, “Successor Features for Transfer in Reinforcement Learning,” NeurIPS, 2017. PDF [Transfer in RL]
• Silver, Huang, Maddison, Guez, Sifre, et al., “Mastering the game of Go with deep neural networks and tree search,” Nature, 2016. PDF [Planning. This introduces alphaGo, which beat the human champion in the game of Go.]
• Mnih, Kavukcuoglu, Silver, Rusu, Veness, et al., “Human Level Control Through Deep Reinforcement Learning,” Nature, 2015. PDF [VFA. Deep Q-Networks learns how to play Atari.]
• Dann, Neumann, Peters, “Policy Evaluation with Temporal Differences: A Survey and Comparison,” JMLR, 2014. PDF [VFA. Survey on TD methods.]
• Farahmand, Ghavamzadeh, Szepesvari, Mannor, “Regularized Policy Iteration with Nonparametric Function Spaces,” JMLR, 2015. PDF [VFA. Regularization in RL]
• Mnih, Badia, Mirza, Graves, Lillicrap, Harley, Silver, Kavukcuoglu, “Asynchronous Methods for Deep Reinforcement Learning,” ICML, 2016. PDF [PS]
• Kakade, “A Natural Policy Gradient,” NIPS, 2001. PDF [PS]
• Schulman, Levine, Moritz, Jordan, Abbeel, “Trust region policy optimization,” ICML, 2015. PDF [PS]
• Haarnoja, Zhou, Abbeel, Levine, “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor,” ICML, 2018. PDF [PS]
• Deisenroth, Fox, Rasmussen, “Gaussian Processes for Data-Efficient Learning in Robotics and Control,” PAMI, 2014. PDF [PS & MBRL]
• Henderson, Islam, Bachman, Pineau, Precup, Meger, “Deep Reinforcement Learning that Matters,” AAAI, 2018. PDF [Criticism on reproducibility of deep RL research]
• Auer, Cesa-Bianchi, Fischer, “Finite-time Analysis of the Multiarmed Bandit Problem,” MLJ, 2002. PDF [Exploration-exploitation tradeoff]

Legend:

• AltMDP: Alternative models of MDP or similar models.
• VFA: Value Function Approximation
• PS: Policy Search
• MBRL: Model-based RL