Reinforcement LearningSimulation
The simplest RL problem has a single state and a few actions. Sina stands in front of the same three stalls every time, and each plate earns her a reward from 1 to 10 points. Every stall has a hidden quality — its average plate — but no two plates are exactly alike. This puzzle is called the multi-armed bandit, after a row of slot machines ("one-armed bandits") whose payouts a player has to discover.
Three Jalebi Stalls
You are Sina. Buy 20 plates, one at a time, and collect as many points as you can. The bars show your average plate at each stall so far.
Raju's first plate scores only 4, so many players never go back — and spend the mela at Meena's stall, which is good but not the best. To find the best stall you have to keep trying stalls, called exploring; to collect points you have to use what you already know, called exploiting. Balancing the two is the explore–exploit dilemma, and every RL agent faces it.
Explore or Exploit?
Sina keeps an estimate for each stall: the average of the plates she has eaten there. Raju's first three plates scored 4, 9 and 8, so her estimate for Raju's stall is (4 + 9 + 8) ÷ 3 =
A fourth plate scores 9. Instead of adding everything up again, Sina nudges the old estimate a step towards the new plate: new estimate = old + (plate − old) ÷ n = 7 + (9 − 7) ÷ 4 =
Ravi tasted Gupta's stall once, got a 6, and ate there for the next 19 rounds. What is the risk?
The ε-greedy rule
Before every plate, Sina picks a random number between 0 and 1.
- If it is below ε (epsilon), she explores: she picks a stall at random.
- Otherwise she exploits: she goes to the stall with the best estimate so far.
With ε = 0.1 she explores on about one plate in ten. With ε = 0 she never explores at all — she is greedy.
Let the Agent Play
Now an agent plays for Sina: 200 rounds at the same three stalls, following the ε-greedy rule. It always tries each stall once first — and, like many players, its first plate at Raju's scores less than its first plate at Meena's. The purple line is the agent's average so far, and the lower chart keeps the final average of every run, so you can compare different values of ε. Complete the three missions. Every move of the ε slider starts a new run.
Exploration rate ε =
With ε = 0 the agent is stuck at Meena's stall for ever: its only plate at Raju's scored 6, so Raju's estimate stays at 6 and it never goes back to check. With ε = 1 it buys at random every time and never uses what it has learned. A small ε — about 0.05 to 0.25 here — explores just enough to discover Raju's stall, and then exploits it for the rest of the mela.
Speed Round: Explore or Exploit?
Get five right to move on.