Muscles instead of motors

Motors in the joints

The fighter in every other post is a MuJoCo body with 43 moving joints, each with a motor. The motors are position servos: the policy gives each joint a target angle, as a correction to the reference motion, and the servo pushes the joint there. An action of zero already follows the reference. Holding a pose is the default; the policy only has to learn what to change.

A real body works differently. A muscle can only pull, so every joint needs at least two working against each other. It doesn’t respond instantly: the nerve signal (the excitation) takes about 10 ms to switch it on and 40 ms to let it go. Its force depends on how stretched it is and how fast it is moving. And a joint is stiff only when the muscles on both sides of it pull at once. On 27 October I asked: if the fighter were driven by muscles, how much longer would training take, is it possible at all, and how much of the existing work could be reused?

Stage 1: a pair of muscles on every joint

The cleanest first test changes only the drive. Same skeleton, masses, contacts, reference motions and rewards; each joint’s motor is replaced by two MuJoCo muscles, one pulling each way, as strong as the motor at their best length. That’s 86 muscles. MuJoCo’s muscles have the three properties that make muscle control hard: activation that lags the command, a force that depends on length and speed, and pulling only. What this stage leaves out is anatomy: each muscle acts on one joint with a fixed lever.

The policy now outputs one excitation per muscle (86 numbers instead of 43), and gets nothing for free: at zero, every muscle is at half excitation, which turns no joint and holds no posture. Left alone, the servo body stands for about 2 s on its servos; the muscle body folds in 0.6 s.

Every run learnt the stance env from scratch: hold the stance, then sway with its rhythm, then take shoves, until it passes the trainer’s test (falls at most 5 % and nearly full-length episodes, over 20 iterations at the full rhythm). Each ran on one rented GPU (RunPod, RTX 4090 or 5090), simulating 1,024 fighters at once with MuJoCo Warp.

• The muscles cost nothing on the GPU. The muscle body simulated at 45.5k steps a second against the servo body’s 36.6k; the contacts dominate the cost, not the muscles.
• On the servo body’s settings it never solved. Two seeds plateaued at 7–9 % falls; one ran on and collapsed to 100 % falls at 185 M steps. No muscle was at full strength before any fall. The problem was timing: the policy’s actions pass through a smoothing filter that adds about 80 ms of lag, on top of the muscles’ own.
• With a lighter filter (about 20 ms) it solved: 37.8 M and 26.6 M steps on two seeds. The servo body on the same filter: 12.9 M and 12.7 M. 2.1–2.9 times the steps, seed for seed, and 1.8–2.6 times the time (10–15 minutes against 5–6).
• A servo policy as a teacher (its joint torques turned into target excitations) made the start 3–4 times faster, but once the teacher faded the run went back to the same plateau. It isn’t a substitute for learning on the muscle body.

Everything except the trained models carried over unchanged: body, references, rewards, the GPU trainer, the cloud tooling, the evaluation tools.

Stage 2: real leg muscles

Joint muscles are a simplification. Real leg muscles run along paths over the bones, so their lever on a joint changes with the angle, and several cross two joints (the hamstrings bend the knee and extend the hip). I took the leg muscles of MyoSuite, an open-source musculoskeletal model built for this kind of research, and mapped their paths onto the fighter’s legs, scaled to his bones. Stage 2a used MyoLeg26 (13 muscles a leg, 26 paths in all); 2b the full MyoLeg (40 a leg, wrapping round the bones). Everything above the hips kept stage 1’s joint pairs, so the full body was 96 and then 142 muscles. I checked every muscle’s lever arm against MyoSuite’s own model at every joint it crosses: within 2.2 cm everywhere, most within 1 cm.

It took three fixes before the 2a body learnt the stance, and none of them was needed by the servo body.

A neutral zero action. At half excitation, real leg muscles don’t cancel: the hip extensors, for one, are much stronger than the flexors, and the leg was pushed up to about 180 N m off the stance. From there the policy never found the stance at all: it stood tall and straight-legged with its arms out. I set each leg muscle’s resting excitation to the values that hold the still stance, so a zero action means “stand”, as it does for the servo body.

A cost for resting on joint limits. With that, both seeds passed the trainer’s test, but in the wrong posture: the rear leg straight and swung back, the trunk bent to make up for it (Figure 1, middle). My first guess was a weak muscle: the stance turns the rear hip out 45°, where the hip abductors lose half their lever, in MyoSuite’s own models too. Making them 1.5 and 2 times stronger changed nothing, to the degree. The real cause: the rear leg was pressed against its hip and knee joint limits, with its muscles idle. A straight leg on its limits is held up by the simulated bone, for free. The policy found that and kept it. A small cost for resting within 10° of a limit took the free support away: the rear knee went to 33–35° (the reference: 43°), and the servo body didn’t notice the cost at all.

Smaller policy updates. In the right posture, the balance plateaued at 7–15 % falls and then collapsed. With a third of the learning rate and half the PPO passes over each batch, it solved within 6.6 M more steps.

With all three from the start, the 2a body solved from scratch in 33.1 M and 49.6 M steps, and the full 2b body in 63.3 M and 73.5 M, first time, with nothing body-specific added. Figure 2 puts the four bodies side by side.

So: possible, yes, at 2–6 times the training, and the more anatomy, the more training. On wall time the muscle bodies took 10 to 37 minutes on one GPU against about 6 for the servos. The muscle stances are a little looser (pose score 0.86–0.89 against the servos’ 0.95), and stage 1’s held still more smoothly than the servos (0.04 joint reversals a second against 0.15).

Stage 3: the punches

Standing is one thing; the point of the fighter is to fight. The punch env is the one the five punches (jab, cross, both hooks, the uppercut) were learnt in: punches pressed on command at targets in the air and at a heavy bag, with shoves in between. A punch counts as a hit when the fist passes within 5 cm of the target at 3.5 m/s or faster. Each body started from its own stance, on the same settings.

The servo body learnt the punches without help: within 10 M steps the punch left the guard, and over 145 M steps its hits rose from 4 % to 9 % and its misses shrank from 25 to 10 cm, with under 3 % falls. The 2a muscle body never landed one. In the first runs its arm barely left the guard (misses of 51–68 cm), and the shoves took up the whole run.

A stretch reflex

The servo body gets a rough punch for free: its targets follow the punch’s reference, so the arm goes out even before the policy has learnt anything. The muscle body has to find the punch from 96 excitations and the reward alone. In a person, the spinal cord does some of this work: a stretch reflex excites a muscle when it is stretched, which pulls a limb back toward where it was going. I gave each muscle one: extra excitation in proportion to how far it is stretched past its length at the target pose, plus a little for how fast. The arm came out at once (misses 28–31 cm, where the servo body started).

It had to be there from the start, though. Switched on under a stance trained without it, the body fell in 97 % of episodes at first and 25–30 % after 30 M steps. A stance trained with the reflex from scratch solved in 56.0 M steps, with the tidiest pose of all (0.93), but three times the jitter (0.30 joint reversals a second against 0.10). The punches from that stance: still no hits.

Why it didn’t score

Tested without exploration noise, the muscle fighter didn’t fall at all, and some punches came close: uppercuts 3–10 cm from the target, left hooks 10–16 cm. But its fist moved at 0.5–2.9 m/s. The servo body’s: 3 to 11. The punches weren’t short so much as slow.

Recording the arm muscles through a jab showed why. Both muscles of the shoulder’s and the elbow’s pairs sat between 0.3 and 0.7 excitation the whole time: a lot of co-contraction. A muscle that is switched on while being stretched resists more than it can pull, so the opposing muscles were braking every punch. To check it was that and not a lack of strength, I forced the jab’s muscles (the shoulder’s forward muscle and the elbow’s straightener) fully on from the stance, open loop, with everything else the same:

The body is strong enough: 8.6 m/s with the brakes off. With them at half, as the policy held them, 3.0 m/s, under what a hit needs. And the policy never learnt to switch them off: half excitation was the resting point of every arm muscle, and moving off it in the right muscles at the right moment is a lot to find by trial and error.

Relaxed arms

So I changed the arms’ resting point: every arm muscle rests at 10 % instead of 50 %, with the policy’s action still covering the whole range from off to full on. That’s a different starting point, not a different body: the same muscles and strengths. The stance on it solved in 47.1 M steps (pose 0.92). In the punches, the opposing muscles now switch off (the reflex pushes them to zero while the punching muscle goes to full), and the fighter landed its first muscle-body hit within 4 M steps: an uppercut at 4.3 m/s.

Then it stopped improving. Within about 10 M steps the policy settled on slow punches that keep it on its feet (a local optimum), 1.5 m/s for the jab, and it stayed there for another 120 M steps (Figure 4). With exploration noise on, as in training, it fell in about a quarter of episodes even with the shoves at their weakest; without it, never. With a third of the noise, punches came out sooner (2 % hits early) but it fell in nearly every episode and never recovered. Fast punches knock this body over. The servo body’s legs absorb them.

I stopped there. The next steps are bigger pieces of work, not more steps: a teacher that shows the muscle arms the servo body’s punch, or a hit-speed bar that starts low and rises as the fighter gets better.

What it means

• Possible: yes, for the stance. A body with 142 muscles, real leg anatomy among them, learns the Muay Thai stance, its rhythm and shoves, from scratch, in under 40 minutes on one GPU.
• How much longer: 2–3 times with joint muscles, 5–6 times with the full leg, measured against the servo body on the same settings. The simulation isn’t slower; the extra is all learning.
• What carries over: everything except the trained models: the body, the references, the rewards, the training code and the tooling. A servo policy as a teacher helps only at the start.
• What a muscle body needs that a motor body doesn’t: a resting point that holds the pose; a reason not to hang on its joint limits; smaller learning steps near its balance limit; for the arms, a resting point low enough to let them move fast.
• What it hasn’t learnt: to punch fast and stay on its feet. The servo body gets the punch’s shape for free and its legs absorb the swing; the muscle body has to find both. Its arms can move fast enough. Putting speed and balance together is the open problem.

What I think, as a coach

My question was a simple one: if this fighter had muscles instead of motors, how much longer would he take to learn, could he learn at all, and how much of what we’ve built would still be useful? I let the experiment run on its own recommendations (“go with all your recommendation”), and from the second day I told it to make the decisions itself on a $50 budget and report back. I watched the videos of each new stance as they came in.

So far we’ve used servo motors for the body because they’re convenient and quick to learn. But the goal really is to fight Muay Thai as a human. So it’s very important to understand how muscles move, and eventually the energy those movements cost.

This is a very early experiment. It doesn’t cover all the techniques, and we didn’t try much on redundancy, or on injury to muscles and hyperextension. Eventually we’ll do Brazilian Jiu-Jitsu, where a joint can be hyperextended and a ligament broken. So this is literally my first experiment in understanding how a servo robot translates to muscles and a more human-like structure.

The work logs

The runs behind this post, stage by stage. The costs are each RunPod pod’s billed hours times its hourly price, from the RunPod ledger, so they are estimates, not the bill. They include the short GPU checks that the simulation of each new body matched the CPU one.