Curriculum learning

If a task is too hard at the start, an agent never succeeds, so it gets no useful signal and never improves. Curriculum learning (Bengio et al. 2009) starts with an easy version and makes it harder step by step.

A curriculum has three parts:

1. A difficulty dial, such as how hard the shoves are or how big the movement is.
2. A promotion rule that decides when the agent is ready for the next level.
3. A step size for each promotion.

Getting the promotion rule right

Training statistics are noisy. A rule that looks at one iteration can promote on a lucky batch, and the agent then faces a level it can’t handle. Better rules:

• average over several iterations;
• wait after each promotion until the statistics come only from episodes at the new level;
• require both a low failure rate and long episodes.

In this project

Two curricula appear in the balance post, and a third in the punches post.

• Rhythm amplitude rose in steps of 0.1. After each raise it waited two episode lengths before judging again. Without the wait, the first run went from 0.2 to full amplitude in 15 iterations, judged on episodes that had started at the old level.
• Shove force. The top force is $F_{\max} = 80 + 520 \cdot \text{level}$ newtons. The level rises by 0.1 when the mean over the last 10–20 iterations has at most 20 % falls and an average episode of at least 8 of the 10 s. Single-iteration rules climbed on noise.
• Punch targets. In the punches post the level is the target itself: air at level 0, a pad on springs from 0.75 to 3 kg up to level 1, then a heavy bag from 12.5 to 35 kg up to level 2. It rises by 0.1 when, over 20 iterations, every punch hits at least 40 % of the time with at most 20 % falls. Four punches went from the air to the 35 kg bag in about 18 M steps.

In the blocking work the dial is the opponent’s punch speed: it starts at 50–60 % and the top speed rises as the fighter passes. With the bar at 80 % passed, the fighter never got there, and for 23 M steps nothing faster than 60 % was trained. Lowering the bar to 60 % let the speed climb (see Blocking).

A curriculum that waits a fixed number of iterations after each raise also decides how fast training can go. In Training on a rented GPU, the stance curriculum waits about two episodes for every copy of the world after each raise, so a solve takes about 350–500 iterations on any machine. Running 8,192 copies on a GPU gave five times the CPU’s steps a second but made each iteration slower, and it hadn’t solved the stance after 12 minutes; 1,024 copies solved it in 386 s.