Work log · Stance · Fine-tuning
Work log for Shadow clones: four Muay Thai techniques learnt in parallel
The higher guard: retraining every skill onto a new stance
Training cost about $21.0436 rented GPUs (vast.ai RTX 5090s, then RunPod RTX 5090s from guard15) plus gate and clip jobs, 22–24 October 2026 · 18 training runs, 36 machines, about 24.5 machine-hours
I asked for a new guard: both wrists at the brow, palms facing forward and turned about 25° in, the forearms framing the face with the eyes clear. Before, the fists sat at eye and nose level. Every reference table was rebuilt from the new pose (rhythm, recovery, the five punches and their aim grids, blocks, walking). I wanted the current models fine-tuned onto it rather than retrained from scratch, and balance, walking, punches and blocks all had to still work.
The parent was opp15-14.5M, the blocking keeper on the old guard.
The runs
| run | what changed | outcome |
|---|---|---|
| guard1 | opp15 fine-tuned in the opponent env on the new guard | stopped at 34.2 M: punches, walk, blocks short |
| guard2 | more punch, block and walk episodes | stopped at 159 M: everything held but the punches |
| guard3 | level-head and palm-to-ear references | lead hook about 1 m/s, 0/12 (it got ~3 % of episodes) |
| guard4 | three times the lead-hook presses | 56 M at the 3 h cap: no lead hook, punches 52 → 44 |
| guard5 | no walk teacher while punching (teacher-free block-punch) | kept 26.2 M: every row at or above opp15, opponent 125/160 |
| guard6 | the lead hook from my capture | trained on a body with the ankle widened for the retarget: walk 10/40, not kept |
| guard7 | guard6 with the ankle back | lead hook 11/12 but punches 47 → 38, walk 31–33/40; guard5 itself lands the captured hook 10/12, so it stays |
| guard8–10 | the lead hook’s form: impact and swing rewards, a follow-through | the swing still ~25 cm low |
| guard11 | the reference’s feed-forward torque on the hook arm | swing −14 to −18 cm, blocks and opponent lost |
| guard12 | guard5 as the teacher outside the lead hook, an arm-pose term | other skills held (blocks 31, opponent 118), elbow still 10 cm low |
| guard13 | the arm term narrower, with the trunk joints | swing −10, elbow −9 cm |
| guard14 | the swing term ×3 | kept 45.9 M: elbow level with the fist, lead hook 12/12, every row at guard5’s |
| guard15 | the jab’s aim | stopped at 0.7 M when I asked for the original elbows |
| guard16–18 | the original elbows, the jab at the reference’s pace, its pose and line terms | guard18-124.5M: the base for new training (jab 2–4 cm off; blocks 27/32) |
guard14 is the parent of the first body-straight and knee teachers; guard18 is the base of the distillation and every later clone.
Where the money went
| machines | machine-hours | cost | |
|---|---|---|---|
| guard1–14 on vast.ai, plus gate, clip and render jobs | 29 | about 21.6 | about $17.62 |
| guard15–18 on RunPod, plus an angle render and the gate | 7 | about 2.9 | about $3.42 |
| total | 36 | about 24.5 | about $21.04 |
Several vast.ai boxes never came up (created stopped, no SSH after 4 minutes, or still loading after 10). They are billed for the minutes they existed and are in the totals. RunPod first refused every request filtered to CUDA 13 (“no longer any instances available”) while 5090s were in stock; without the filter it deployed at once.
Every run
From the run records in kakuto-ml: each run's machines, their billed hours and cost (EC2 runs are estimates from the run logs), plus the tests and renders this log covers.
| Run | Outcome | Machines | Hours | Cost |
|---|---|---|---|---|
guard1-2026102222 October · jab, cross, lead hook, rear hook, lead uppercut, blocks · kept guard1-9.8M | Kept | 2 | 2.6 | $2.06 |
guard2-2026102222 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 1 | 2.5 | $1.79 |
guard3-2026102222 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 1 | 0.8 | $0.51 |
guard4-2026102222 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Not kept | 1 | 3.1 | $2.50 |
guard5-2026102222 October · jab, cross, lead hook, rear hook, lead uppercut, blocks · kept guard5-26.2M | Kept | 2 | 1.6 | $1.51 |
guard6-2026102323 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Not kept | 4 | 1.5 | $1.10 |
guard7-2026102323 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 1 | 1.1 | $0.79 |
guard8-2026102323 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 1 | 1.2 | $1.23 |
guard9-2026102323 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 1 | 0.7 | $0.60 |
guard10-2026102323 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 2 | 1 | $0.65 |
guard11-2026102423 October · jab, cross, lead hook, rear hook, lead uppercut, blocks | Stopped | 2 | 1.2 | $0.82 |
guard12-2026102423 October · lead hook · kept guard12-39.3M | Kept | 5 | 1.5 | $1.34 |
guard13-2026102423 October · lead hook · kept guard13-29.5M | Kept | 1 | 1 | $1.05 |
guard14-2026102423 October · lead hook · kept guard14-45.9M | Kept | 4 | 1.8 | $1.65 |
guard15-2026102524 October · jab | Stopped | 1 | 0.1 | $0.13 |
guard16-2026102524 October · jab, cross, lead hook, rear hook, lead uppercut · kept guard16-34.1M | Kept | 2 | 0.6 | $0.68 |
guard17-2026102524 October · jab, cross, lead hook, rear hook, lead uppercut · kept guard17-42.6M | Kept | 1 | 0.6 | $0.70 |
guard18-2026102524 October · jab, cross, lead hook, rear hook, lead uppercut · kept guard18-124.5M | Kept | 2 | 1.6 | $1.86 |
| 18 training runs | 34 | 24.4 | $20.95 |
- lead-hook reference renders for the higher guard, 22 October: 2 machines, 0.1 h, $0.09
← All the work logs for “Shadow clones: four Muay Thai techniques learnt in parallel”