Actor-critic

An actor-critic method learns two functions at once:

• The actor is the policy $\pi_\theta(a \mid s)$. It decides what to do.
• The critic is a value function $V_\phi(s)$. It predicts how much reward is coming.

The critic’s prediction is the baseline the actor is judged against. After each step, the difference between what happened and what the critic expected is the advantage. Actions with a positive advantage are reinforced.

$$\nabla_\theta J \approx \mathbb{E}\!\left[\nabla_\theta \log \pi_\theta(a_t \mid s_t)\, \hat{A}_t\right]$$

Without a critic the actor would be judged against raw returns, which vary hugely from episode to episode for reasons that have nothing to do with any one action. The critic removes most of that noise.

The critic is only needed during training. A deployed policy uses just the actor.

Shared or separate networks

The two can share layers or be separate networks. Separate networks are simpler to tune, because the critic’s regression loss can’t disturb the actor’s features.

In this project

The actor and the critic are separate 2 × 256 MLPs with the same 193 inputs. The actor’s last layer starts at zero, so a brand-new policy outputs zero corrections and exactly follows the coach’s reference pose.