Reinforcement-Learning Study Notes

About Reinforcement Learning

Reinforcement learning is not a specific algorithm, but a generic term for a class of algorithms. If used for comparison, it is similar to supervised learning and unsupervised learning, which is an umbrella term for learning.

The idea of a reinforcement learning algorithm is very simple: if a certain strategy can achieve a high score in a game, then it is further reinforced to continue achieving better results. This strategy is very similar to the various performance rewards in daily life. We often use such strategies to improve our game.

In the game Flappy bird, we need to simply click to control the bird, dodge various water pipes, and fly as far as possible, because the farther you fly, the higher the bonus points.

This is a typical reinforcement learning scenario.

• The machine has an explicit role for the bird - the agent
• The bird needs to be controlled to fly farther - the goal
• Need to dodge various water pipes throughout the game - the environment
• The way to avoid the pipes is to make the bird fly a little harder - the action
• The farther you fly, the more points you will get - the reward

You will find that the biggest difference between reinforcement learning and supervised and unsupervised learning is that it does not require a lot of data feeding. Instead, you learn certain skills by trying them out on your own.

Coursera Programming Assignments

• Course 1: Fundamentals of Reinforcement Learning

  • Bandits and Exploration / Exploitation

  • Optimal Policies with Dynamic Programming

• Course 2: Sample-based Learning Methods

  • Policy Evaluation with Temporal Difference Learning

  • Q-Learning and Expected Sarsa

  • Dyna-Q and Dyna-Q+

• Course 3: Prediction and Control with Function Approximation

  • TD with State Aggregation

  • Semi-gradient TD with a Neural Network

  • Function Approximation and Control

• Course 4: A Complete Reinforcement Learning System (Capstone)

  • MoonShot Technologies

  • Implement your agent

  • Completing the Parameter Study

Lecture Notes of CMPUT365 Reinforcement Learning

• Probabilities and Expectation

• K-Armed Bandits

• Markov Decision Processes

• Value Functions and Bellman Equations

• Dynamic Programming

• Monte Carlo Methods

• Temporal Difference Methods for Prediction

• Temporal Difference Methods for Control

• Planning, Learning, and Acting

• Function Approximation Prediction & Prediction with Approximation

• TD Update Stability & Neural Network

• Control with Approximation

• Review

Coursera Notes of CMPUT365 Reinforcement Learning

• 1. Fundamentals of Reinforcement Learning

  • Week 1: The K-Armed Bandit Problem & Explorations vs Exploitation Tradeoff

  • Week 2: Markov Decision Processes

  • Week 3: Value Functions & Bellman Equations

  • Week 4: Dynamic Programming (Policy Evaluation(Prediction), Policy Evaluation(Control), Generalized Policy Iteration)

• 2. Sample-based Learning Methods

  • Week 1: Monte Carlo Methods for Prediction & Control (Exploration Methods for Monte Carlo, Off-policy Learning for Prediction)

  • Week 2: Temporal Difference Learning Methods for Prediction

  • Week 3: Temporal Difference Learning Methods for Control

  • Weel 4: Planning, Learning, and Acting (Model)

• 3. Prediction and Control with Function Approximation

  • Week 1: On-policy Prediction with Approximation (Linear TD)

  • Week 2: Feature Construction for Linear Methods (Neural Networks, Training Neural Networks)

  • Week 3: Control with Approximation (Episodic Sarsa with Function Approximation, Exploration under Function Approximation, Average Reward)

Worksheet Questions

• Worksheets

Exam Reviews

• Midterm Review

• Final Review