Glossary · Reinforcement learning
Action
Also called action space, continuous control
What the policy outputs on each step, such as a target angle for every motor.
An action is the policy’s output on one step. The set of all possible actions is the action space.
- A discrete action space has a few choices, like the buttons on a game controller.
- A continuous action space has real numbers, usually one per motor. Controlling a body is continuous control.
What an action means is a design choice, and it matters a lot. The same network can output joint torques, target joint angles, or corrections to a reference pose. Target angles sent to position servos are much easier to learn than raw torque, because the servo already does the low-level work of holding a joint still.
Scaling
Networks work best with outputs around . The environment scales each output to its physical range, for example multiplying by 0.4 rad. Outputs outside the range are clipped (see action clipping).
In this project
Each action is one number per motor: 37 on the simple body, and 43 on the muscular body, whose other 40 motors hold the fingers in a fist. Each is scaled to a correction of up to ±0.4 rad on the reference pose. The survive env also widens these ranges off balance, up to ±1.5 rad on the legs. The push-recovery env adds two extra actions: a clock speed for the recovery move and a correction to its size.
The actions then pass through a low-pass filter before reaching the servos.