Teachers: the uppercuts over distance

Both uppercuts, aimed up and down, left and right, and over distance: the scripted uppercut at depth 0 blending out to my long left from the capture (take_02) for the lead hand, and my short-to-long rear uppercut (take_02) for the rear hand. Both teachers are warm from distill4-131.1M in opponent_uppercut, and they trained side by side on two RunPod RTX 4090s for 3.2 hours each. That was about 261 M steps each at about 25,000 steps a second, $2.84 each.

The gate

	air hits, 9 aims × 6 seeds	bag hits	verdict
distill4-131.1M (the student, lead only)	37/54	0/54 (too slow at the bag)	–
upL1-261.3M (lead)	40/54; the far end (+0.19 m) never learnt, ~10 cm short	0/54 (3–5 m/s)	failed on video
upR1-209.7M (rear)	54/54, 2.4–3.4 cm, about 10 m/s	38/54	kept

After watching the long-uppercut videos I said: “After the uppercut the fighter turns 90 degree to the right. It needs to keep facing the opponent.” Measured on a check pod: upL1 turns −65° by 0.5 s and is still at −66° after 6 s. upR1 turns 20–27° during the punch and comes back within 5–15°. The student distill4 drifts too, most at the bag.

Cause: not the reference, which ends square at every depth. The home reward re-anchored its heading at every press and every shove recovery, and the opponent moved to wherever the fighter faced. So nothing paid for facing one direction. The fix (--face-fixed) keeps the home heading from the start of the episode, keeps the targets and the opponent in it, and adds a wide facing term. The gate now prints the hips’ turn 1–3 s after every technique. upL2 and upR2 are retraining with it. They are not counted here.

Where the money went

job	machine-hours	cost
upL1 and upR1 training (2 × RTX 4090)	6.39	$5.69
first gate pod: hung (below)	2.04	$1.21
second gate pod: both teachers’ evals and videos	0.94	$0.55
long-uppercut video and the facing check	0.29	$0.18
a pod image test and a --resume test (fake disconnect)	0.40	$0.18
total	10.06	$7.81

The first gate pod ran for 2 hours and produced nothing. On a RunPod pod nproc reports the host’s 128 cores, not the pod’s 16 vCPUs, so the eval started 128 processes with 64 BLAS threads each and hit the thread limit. The second pod passed the process count and one thread each explicitly.

Every run

From the run records in kakuto-ml: each run's machines, their billed hours and cost (EC2 runs are estimates from the run logs), plus the tests and renders this log covers.

• RunPod image tests, 25 October: 2 machines, 0.1 h, $0.07
• RunPod resume test: a pod killed by hand and resumed on a new one, 25 October: 2 machines, 0.3 h, $0.12

← All the work logs for “Shadow clones: four Muay Thai techniques learnt in parallel”