Reinforcement LearningSummary
One page to revise from — the ideas first, then every formula you need.
- Reinforcement learning is learning by trial and error: an agent takes actions in an environment, sees the new state and collects a reward. Nobody gives it the right answers.
- A policy is the agent's rule from state to action. The goal is the policy that collects the most total reward over an episode.
- Explore or exploit: exploiting uses the best choice found so far; exploring tries choices that might be better. A greedy agent never explores and can get stuck on a second-best choice.
- The ε-greedy rule explores with chance ε and exploits otherwise. A small ε usually works best.
- The discount factor γ makes later rewards worth less. The return adds up the rewards, each one multiplied by one more γ.
- A Q-value estimates the return of taking action in state . Q-learning nudges it towards the reward plus the discounted best Q-value of the next state; the actions with the biggest Q-values form the policy.
- When the world changes, an old policy can fail, so the agent has to keep learning.
ε-greedy
chance ε: a random action
chance 1 − ε: the best estimate so far
Return
Q-learning update