Introduction to Reinforcement Learning

A course on reinforcement learning.

Introduction to Reinforcement Learning (Fall 2025)

You can find the Spring 2021 version of this course at here

Note: This website is being actively updated.

This is an introductory course on reinforcement learning (RL) and sequential decision-making under uncertainty with an emphasis on understanding the theoretical foundation. We study how dynamic programming methods such as value and policy iteration can be used to solve sequential decision-making problems with known models, and how those approaches can be extended in order to solve reinforcement learning problems, where the model is unknown. Other topics include, but not limited to, function approximation in RL, policy gradient methods, model-based RL, and balancing the exploration-exploitation trade-off. The course will be delivered as a mix of lectures and reading of classical and recent papers assigned to students. As the emphasis is on understanding the foundation, you should expect to go through mathematical detail and proofs. Required background for this course includes being comfortable with probability theory and statistics, calculus, linear algebra, optimization, and (supervised) machine learning.


Announcements:


Teaching Staff

Time and Location:

Reading

The course material is based on Foundations of Reinforcement Learning. This is a live document that will change as we progress through the course. If you find a typo or mistake, please let me know. I collect the list of reported ones here.

Some other useful textbooks (incomplete list):

Lectures

This is a tentative schedule, and may change.

Note on videos: The videos will be publicly available on YouTube. If you don’t feel comfortable being recorded, make sure to turn off your camera when asking questions (though I really prefer to see all your faces when presenting a lecture, so it doesn’t feel that I am talking to void!).

This will be updated soon!

Week (date) Topics Lectures Reading
1
(Aug 25)
Introduction to Reinforcement Learning (Part I) slides video Chapter 1 of FRL
1’
(Sept 1)
(No Lecture)
Tutorial: Math Review
   
2
(Sept 8)
Introduction to Reinforcement Learning (Part II)
Structural Properties of Markov Decision Processes (Part I)
Tutorial: Q Learning
slides video Chapter 2 FRL
3
(Sept 15)
Structural Properties of Markov Decision Processes (Part II)
Planning with a Known Model (Part I)
slides video Chapter 3 FRL
4
(Sept 22)
Planning with a Known Model (Part II)
Learning from a Stream of Data (Part I)
slides video (Part I) Chapter 4 FRL
5
(Sept 29)
Learning from a Stream of Data (Part II)
Value Function Approximation (Part I)
video (Part II of Learning from Stream)
slides (VFA) video (Part I of VFA)
Chapter 5 FRL
6
(Oct 6)
Value Function Approximation (Part II) video (Part II) Chapter 5 FRL
7
(Oct 20)
Value Function Approximation (Part III) video (Part III) Chapter 5 FRL
8
(Oct 27)
Policy Search Methods (Part I) slides video (Part I) Chapter 6 FRL
9
(Nov 3)
Policy Search Methods (Part II) video (Part II) Chapter 6 FRL
10
(Nov 10)
Model-based RL slides video Chapter 7 FRL
11
(Nov 17)
Exploration-Exploitation slides Chapter 8 FRL
12
(Nov 24)
Other Topics slides  
13
(Dec 1)
Presentations    

Assignments and Coursework

These are the main components of the course. The details are described below. You need to use … to submit your solutions.

Homework Assignments

There will be three homework assignments. The detail will be posted.

This is a tentative schedule of the homework assignments. Most of them will be released on a Monday (or late Sunday evening) and will be due on a Monday in two weeks. The deadline is 16:59.

This will be updated soon!

Homework # Out Due Materials TA Office Hours
Homework 1 Feb 22 March 8 Questions Code Feb 26 (Fri) and March 3 (Wed), both 12-1PM
Homework 2 March 12 March 26 Questions Code Mar 17 (Wed), 2-3PM and Mar 23 (Tue), 12-1PM
Homework 3 April 1 April 19 Questions Code Apr 7 (Wed) and Apr 14 (Wed), 10-11AM

Research Project

Read the instruction here!

This will be updated soon!

Reading Assignments

The following papers are a combination of seminal papers in RL, topics that we didn’t cover in lectures, or active research areas. You need to choose five (5) papers out of them, depending on your interest. Please read them and try to understand them as much as possible. It is not important that you completely understand a paper or go into detail of the proofs (if there is any), but you should put some effort into it.

After reading each paper:

These five assignments contribute 10% to your final mark. The reading assignments are only lightly evaluated. You should submit your summaries, all in one PDF file, before April 12th (Monday) at 5PM.

We will post the papers as the course progresses. Please read and summarize them as we post them, so you won’t have a large workload close to the end of the semester.

Note: that this is an incomplete and biased list. I have many favourite papers that are not included in this short list.

Legend: