Muscles, stage 1: joint muscles instead of motors

Every fighter so far has been driven by motors in its joints: the policy picks a target angle for each joint and a servo gets it there, so an action of zero already tracks the reference motion. A real body has muscles, which only pull, lag behind the command, and are stiff only when opposing muscles both work. On 27 October I asked how much longer training would take with muscles, whether it works at all, and how much of the existing work could be reused.

The first stage keeps everything else the same: the same skeleton, masses, contacts and reference motions, with a pair of MuJoCo muscles across each joint in place of its motor (86 in all, each as strong as the motor it replaces). That makes the comparison with the servo body direct. Every run learns the stance env from scratch on a RunPod GPU, 1,024 worlds at once; “solved” is the trainer’s test: falls at most 5 %, episodes at least 490 steps of 500, over 20 iterations at the full rhythm.

The runs

run	what	result
muscle-check	GPU against CPU on the muscle body, step by step; speed	matches (qpos within 2e-6); the muscles cost no GPU time (45.5k steps/s against the servo body’s 36.6k)
mbase	the servo body from scratch, two seeds: the baseline	solved at 11.7 M and 12.1 M steps, about 5 minutes each
muscle1s0, muscle1s1	the muscle body from scratch, seeds 0 and 1, with the servo body’s action filter	never solved: falls plateau at 7–9 %; seed 0 collapsed to 100 % falls at 185 M steps
muscle1t	the muscle body with the servo policy as a torque teacher, fading out over 200 iterations	3–4× faster start, then the same plateau once the teacher faded
muscle1t2	the same with the teacher never fading below 0.2	held at about 20 % falls
muscle1f	from scratch, lighter action filter (0.5 instead of 0.8 in balance)	solved at 37.8 M steps, 882 s
muscle1f2	the same, seed 1	solved at 26.6 M steps, 602 s
mbase2	the servo body on the lighter filter, two seeds: like for like	solved at 12.9 M and 12.7 M steps
volcheck	a 1-minute pod, so the RunPod volume’s growth from 21 to 50 GB took effect

What it showed

• Possible: yes. 86 pull-only muscles with activation lag and force-length-velocity curves learn the stance, its rhythm and light shoves from scratch.
• How much longer: 2.1–2.9× the steps of the servo body on the same filter, seed against seed; 1.8–2.6× the time. The physics is no slower on the GPU: the extra is all samples.
• The action filter decided it. The servo body’s filter adds about 80 ms of lag on top of the muscles’ own; with it the muscle runs never solved. At about 20 ms one solved in 15 minutes. No muscle saturated before any fall: the falls were timing, not strength.
• Reuse: everything but the checkpoints carried over unchanged. A servo policy as a teacher only speeds up the start.

Kept: muscle1f-37.8M and muscle1f2-26.6M. The next stage puts real leg muscles on the body (stage 2).

Every run

From the run records in kakuto-ml: each run's machines, their billed hours and cost (EC2 runs are estimates from the run logs), plus the tests and renders this log covers.

• a 1-minute RunPod pod to make the volume resize to 50 GB take effect, 27 October: 1 machines, 0 h, $0.01
• GPU against CPU parity check of the muscle body on a RunPod RTX 4090, 27 October: 1 machines, 0.1 h, $0.08

← All the work logs for “Muscles instead of motors”