Reinforcement learning

Reinforcement learning (RL) is the branch of machine learning that learns from experience rather than from labelled examples. Nobody shows the learner the right answer. Instead an agent tries actions in an environment, and the environment hands back a number, the reward, saying how good the result was. Over many attempts the agent learns which actions lead to more reward.

It is close to how a coach teaches a student. The coach can’t move the student’s muscles for them. They can only say “better” or “worse”, and the student works out the rest.

The loop

Each step of the loop has four parts:

1. The agent receives an observation $s_t$ of the current state.
2. Its policy $\pi$ chooses an action $a_t$.
3. The environment moves forward one step and returns a reward $r_t$ and the next observation $s_{t+1}$.
4. The agent uses what happened to improve its policy.

A run of steps from start to finish is an episode. The goal is to find the policy that gets the most total reward, usually with later rewards counted a little less than earlier ones:

$$\pi^* = \arg\max_\pi \; \mathbb{E}_\pi\!\left[\sum_{t=0}^{T} \gamma^t r_t\right]$$

Here $\gamma$ is the discount factor (see markov decision process).

Why it is hard

• Credit assignment. A fall may happen a second or two after the mistake that caused it. The learner has to work out which earlier action was to blame. Reward shaping helps.
• Exploration. To find something better, the agent has to try things it isn’t sure about. A stochastic policy does this by adding randomness to its actions.
• The reward is all it knows. If the reward can be earned in an unintended way, the agent will find it. This is reward hacking.

In this project

The agent is the simulated fighter’s controller. The environment is a MuJoCo simulation of the body, the floor and the shoves. The reward pays for holding the coach’s stance and for not falling. The algorithm is PPO.