Value function

The value of a state is the expected return from that state if the agent follows its policy:

$$V^\pi(s) = \mathbb{E}_\pi\!\left[\sum_{k=0}^{\infty} \gamma^k r_{t+k} \;\middle|\; s_t = s\right]$$

It obeys a simple recursion, the Bellman equation: the value now is the reward now plus the discounted value of where you end up.

$$V^\pi(s) = \mathbb{E}\left[r + \gamma V^\pi(s')\right]$$

In deep RL a neural network called the critic learns $V(s)$ by regression on the returns it observes.

Advantage

The value function makes learning far less noisy. Instead of asking “was this episode good?”, the learner asks “did this action do better than expected?” That difference is the advantage:

$$A(s_t, a_t) = \underbrace{r_t + \gamma V(s_{t+1})}_{\text{what happened}} - \underbrace{V(s_t)}_{\text{what was expected}} = \delta_t$$

A positive advantage makes the action more likely; a negative one makes it less likely.

Generalised advantage estimation

The one-step $\delta_t$ relies heavily on the critic, which may be wrong. Summing actual rewards over many steps relies less on the critic but is noisier. GAE (Schulman et al. 2016) blends the two with a parameter $\lambda$:

$$\hat{A}_t = \sum_{l=0}^{\infty} (\gamma\lambda)^l \, \delta_{t+l}$$

With $\lambda = 0$ it is the one-step estimate; with $\lambda = 1$ it is the full return minus the baseline.

In this project

We use $\gamma = 0.99$ and $\lambda = 0.95$. The critic is a separate 2 × 256 MLP with the same inputs as the actor.

The critic is also a window into what the fighter expects. In the torque and neuron post it is drawn as a “forecast” line: 555 while the student distill9-196.6M stands, below half 40 ms after a 900 N push, down to −92 a second later, while the body only hits the floor 1.94 s after the push.