Learning to balance in a Muay Thai stance

1. Why start with the stance

In Muay Thai, as in most striking arts, everything comes from the stance. A kick, a punch or a check starts and finishes there. A fighter who can be pushed off it can’t strike or defend. So the stance is the first thing our simulated fighter learns, and balance is the first thing we test.

Our coach is the author, Tony Yum, who teaches Muay Thai. Every step here was judged the way a coach judges a student: does it look right? The reward functions can’t measure that. So the training is human-in-the-loop by design. The coach specified the stance and the movement. He watched the progress videos, rejected motions that looked wrong (a few results that scored well looked wrong) and chose what to try next. Section 9 lists the decisions that came from that judgement.

There is no single correct Muay Thai stance. Guards vary (a long guard, a high guard), and so does the weight split between the feet. The stance here carries more weight on the rear leg. That is a stylistic choice, the coach’s starting point, not a claim that it is the right one.

2. The simulated fighter

All experiments use MuJoCo 3.16 with two bodies that share one mass model.

• Muscular body. An average adult male, 1.75 m and 77 kg (welterweight). Segment masses come from de Leva (1996) and limb girths from the ANSUR II survey means. It has 83 actuated joints, with ranges taken from the AAOS range-of-motion tables, and 14 touch sensors. Its contacts use three materials: soft muscle (about 8 mm of sink at 50 N and 14 mm at 400 N under a palm-sized probe), hard bone (under 1 mm) and grippy hands. The stance, rhythm, scripted-recovery and first survival experiments ran on this body.
• Simple body (v2). 37 motors and 23 colliding shapes: one capsule per segment, box fists, and box feet with a heel sphere and a toe capsule. It keeps the joints that matter for standing and striking: waist, neck, clavicles, shoulders, elbows, wrists, hips, knees, ankles and toes. Each segment’s mass, centre of mass and inertia are copied from the muscular body, so the two bodies have the same overall mass distribution. A physics step costs 161 µs, against 423 µs for the muscular body.

Neither body has contracting muscles. Every motor is driven as a torque-capped position servo, so tensing doesn’t make the body stiffer.

3. Specifying the stance

The coach described the stance and we turned it into numbers:

property	value
weight split	70 % rear foot, 30 % lead foot (orthodox: left foot forward)
rear foot	turned 45° out, flat
lead foot	pointing forward, heel raised 35°
stance size	0.51 m long, 0.16 m wide
trunk	upright, chest square
guard	high: elbows down by the ribs, gloves in front of the forehead

No set of joint angles was written by hand. solve_stance.py hill-climbs the leg, trunk and arm joints until the body meets these targets, with the centre of mass (COM) where the weight split needs it. The solved stance became the reference that every later policy tracks while it is in balance.

The rhythm

A Muay Thai fighter is seldom still, so the reference also moves. At one beat per second the hips rock about 10 cm forward along the line between the feet, from the stance (70 % rear) to 50/50 and back, and sink 2 cm on the way back. The feet roll on their balls with the weight. With the weight forward, the lead foot is flat and the rear heel is up 25°. With the weight back, the rear foot is flat and the lead heel is up 35°. The lead foot taps 2 cm, the chest turns ±2.5° while the head keeps facing forward, and the lead hand paws. An amplitude dial scales all of this from 0 (still stance) to 1 (full rhythm).

The rhythm was changed three times at the coach’s request. At first it rocked around the stance (60–77 % on the rear foot). Then it rocked from the stance forward to 50/50. Last came the heel roll, because the coach saw that the lead foot always stayed on its toes.

4. Learning to track the stance

Policy and actions

The policy is a PPO (Schulman et al. 2017) actor-critic with a 2 × 256 MLP. It doesn’t output joint targets directly. Each output is a residual of up to ±0.4 rad that the env adds to the reference pose for the current rhythm phase. The result drives the position servos at 50 Hz. Episodes last 10 s. Each episode draws its own start phase, a tempo of 0.8–1.3 beats per second and an amplitude. The 193 observations are: joint errors against the reference and joint velocities; pelvis orientation, velocity and offset from its reference; the COM relative to the feet and the foot forces; the phase, tempo and amplitude; the previous and filtered actions; and the balance gate described below.

The reward is mostly about tracking (85 %). It covers joint angles, the positions of the head, hands and feet relative to the pelvis, the pelvis position over the rear foot, an upright chest, the rear-foot share of the weight, the lead-foot tap, a planted rear ball of the foot and both heel heights. The other 15 % rewards smoothness. A fall ends the episode with −1. A fall is the pelvis dropping 0.3 m, the head going below 1.1 m, or any body part other than a foot touching the floor.

A curriculum raises the rhythm amplitude in 0.1 steps once the episodes are long and seldom fall. After each raise it waits two episode lengths before judging again. Without that wait, the first run went from 0.2 to full amplitude in 15 iterations on episodes that had started at the old level.

Jitter, and how it was removed

The first policy (35 minutes and 7 M steps on one cloud machine, under $1) held the full rhythm. But it was visibly twitchy. We measured jitter as direction reversals per joint per second. It scored 1.7 standing calmly and 4.4 when nudged. The reference motion scores 0.02. A coach would not accept a stance that trembles, so the next run was rebuilt around smoothness:

• A balance gate b ∈ [0, 1]. It combines how far the capture point is from the reference’s (6 to 15 cm) with chest tilt (10° to 25°). The capture point is the COM plus its velocity divided by ω = √(g / z), the point where the body would have to step to come to rest (Pratt et al. 2006). The gate rises at once and eases back over 0.5 s.
• An adaptive action filter f ← α·f + (1 − α)·a, with α = 0.8 in balance (about 1.8 Hz) and 0.2 off balance. The body is smooth when it is calm and can react fast when it stumbles.
• Smoothness rewards on the real joint motion, not just the network output: joint velocities that match the reference’s, no reversals, no extra acceleration, and gentle changes in the servo targets. Off balance they relax to 20 % strength.
• CAPS regularisation (Mysore et al. 2021) in the PPO loss: one term keeps consecutive actions close, and one keeps the actions for nearby states close.
• A gym-mat floor. The coach’s idea. With the foam’s compliance (the feet sink about 9 mm), the unchanged old policy dropped from 1.7 to 0.15 reversals per joint per second, 11× less, because the feet stopped chattering on a rigid floor.

After retraining, calm jitter was 0.01 reversals per joint per second (Figure 3). The final heel-roll policy scores about 0.1–0.2 under light nudges. It holds the still stance under nudges with 73 % of its weight on the rear foot. We froze this policy as the parent of everything that follows.

5. The shove test

When a student’s stance looks right, the coach checks it with a push. If the student is pushed off balance easily, the stance isn’t working yet. We did the same.

Light nudges (20–60 N on the chest every 1.5–4 s) were part of stance training from the start. Stronger shoves (40–160 N) at that stage made 70 % of episodes fall, so they were left for later. Out of curiosity we then kept pushing harder, and the stance eventually broke.

The shove model used from here on:

• one shove every 2–4 s, lasting 0.10–0.20 s, from any direction, mostly on the chest and sometimes on a shoulder or the hips
• force from 30 N up to a top force set by a curriculum level: F_max = 80 + 520 · level newtons, so 600 N at level 1.0 and 860 N at level 1.5
• the level rises by 0.1 when the mean over the last 10–20 iterations at that level has at most 20 % falls and an average episode of at least 8 of the 10 s. Single-iteration rules climbed on noise.

The way it failed is instructive. Shoves from the front and back of 150 N were absorbed, because the stance is long in that direction. Sideways shoves of 100–150 N started a slow drift (about 0.2 m/s) that the stance couldn’t stop. The capture point crept to the edge of the feet, and the body toppled 0.6–2.4 s after the shove, never during it. The frozen stance policy fell in 15 of 30 episodes with 30–150 N chest shoves, and in 0 of 30 without shoves. A stance can’t hold these shoves on its own. The feet have to move.

6. Approach A: recovery by scripted moves

The first approach teaches the body how to recover, in the coach’s style.

Reference recovery moves

We designed keyframed recovery motions. Inverse kinematics solves the legs and monotone cubic splines join the keyframes. Each has a size dial from 0 to 1.

• The sideways hop (recover_side.py). A sideways shove piles the weight onto one foot, which then can’t simply step. Instead, the light leg lifts and reaches out as a counterweight while the trunk leans over the loaded foot, in a T shape like a skater’s. The calf drives the loaded foot up onto its ball, and the foot hops sideways (up to 35 cm out and 5 cm up). It lands ball first and the heel comes down as the hop is absorbed. The free foot comes down wide (up to 45 cm), and once balance is back the feet shuffle back into the stance. The guard rides on the chest throughout. A shove to the left hops off the lead foot. A shove to the right is the mirror: the rear foot hops.
• The forward and backward steps (recover_sagittal.py): a lunge step forward or a step back on the ball of the foot, modelled on photos of the coach stepping.

Each move is tabled at 5 sizes in 0.02 s frames. An early version had an inverse-kinematics bug that crossed the legs. The previews now check the gap between the limbs and the IK error.

When to recover, and how the policy takes part

Recovery is triggered by the capture point: when it leaves the outline of the feet in contact with the mat, or when for 80 ms it sits within 2 cm of that edge and moves outward at 0.08 m/s or more, or drifts at 0.15 m/s or more. The direction it left in picks the move, and the distance picks the size. Recovery then blends the servo targets from the stance to the move with a weight w. The policy gets two extra outputs: a clock speed for the move (0.5–2×), and a size correction of ±0.3 during the first 0.15 s.

The reward blends as well, r = (1 − w)·r_stance + w·r_recovery. The recovery reward pays for the capture point inside the support, the COM settling, an upright trunk, the guard, style against the reference move, foot and heel heights, and no skating. Off balance the residual ranges change: legs ±0.6 rad, trunk ±0.4 rad, and arms only ±0.15 rad, so the gloves stay at the face.

What went wrong, and the fixes

run	problem	fix
balance	stuck at the first level (150 N top force), about 49 % falls after 11 M steps. Recovery started too late (a 100 N sideways shove triggered at 1.1 s) and the moves were too small	every move at least half size, full size at 15 cm outside the support; the earlier drift trigger above (it now fires 160–200 ms after a 100 N shove); first level eased to 80 N
balance2	the level climbed on lucky single iterations	raise only on a 10–20-iteration mean
balance3	reward hacking: with no shoves at all, the policy stayed in recovery mode 95 % of the time. It kept the capture point near the edge so recoveries kept re-triggering, because a wide, upright recovery pose (about 0.7 per step) paid about as well as the stance it could actually hold (about 0.75). Training then diverged to NaN	recovery reward scaled to at most about 0.6, below a good stance step; recovery ends once the body is calm; during a recovery a new one starts only on a real escape; the trainer rolls back any update with non-finite numbers

With these fixes, balance4 reached level 1.0 (600 N) at about 68 M steps and stayed stable to 115 M. Training falls were 21–48 %, and the body spent 0 % of shove-free time in recovery mode, so the hack was gone. In a fixed test of 12 deterministic episodes with the same shove sequences, it fell 3 of 12 times at level 1.0. On the controlled shove grid (8 directions × 2 forces) it held 9 of 16. It failed on shoves from behind and back-left at both forces, and from the right at 600 N. A rightward shove has to move the loaded rear foot 35 cm, and it often chained 2–4 different moves after one shove.

The scripted moves gave the recoveries a recognisable style. But they also limited the body to those moves, and the hardest shoves needed something they didn’t include.

7. Approach B: survive, with the guard up

The second approach, run in parallel with the first, removes the scripted moves. Once off balance the body may do whatever keeps it up, with two rules from the coach: the hands stay up, and once it has recovered, it returns to the stance.

Survive mode

The capture-point trigger, the shoves and the curriculum are the same as in Approach A. Only what happens off balance changes:

• No reference move. The servo targets are the still stance plus residuals. As w rises these widen to ±1.5 rad on the legs and ±0.8 rad on the trunk, limited by the joint ranges, so the hips and legs are free.
• The guard is enforced twice. The arm residuals stay at ±0.15 rad, so the gloves physically can’t leave the face. The reward also includes a guard term, exp(−50 · ‖g − g₀‖²), on the glove positions.
• The reward is only survival: 0.4 per step for staying up plus up to 0.1 for the guard, and −1 for a fall. There are no pose, style, foot or smoothness terms, and the action filter opens. At most 0.5 per step, being off balance never pays more than a good stance step (about 0.7–0.8).
• Return to the stance. In balance, the full stance reward applies again (90 % stance and 10 % guard), so the policy gains by getting back into its stance.

Both approaches started from the same policy: the frozen stance-and-rhythm policy from Section 4. So survive mode’s recoveries were never shown the scripted hop. Whatever style they have comes from where they start (the coach’s stance, guard and footwork, which they must return to) and from the physics.

Making it train

run	problem	fix
survive	the mode ended only when the body matched the stance again, and a free step rarely does. So the mode stayed on while the body was calm, nothing steered the posture, and the feet drifted to a 0.9 m split until it toppled. About 24 M steps at the first levels	calm exit: end the mode after at least 0.3 s once the capture point is 2 cm inside the feet, the COM is slower than 0.1 m/s and the chest is within 10° of upright, for 0.2 s
survive2	the −1 for a fall arrived 1–2 s after the mistake that caused it	potential-based shaping (Ng et al. 1999): F = 0.99·φ(s′) − φ(s), where φ penalises the capture point outside the feet, tilt past 10° and the pelvis dropping past 3 cm. It doesn’t change which policy is optimal, and it is about 0 in a calm stance. Levels 0.2 → 0.7 in 7.4 M steps
survive2	collapse: the action means drifted beyond ±1 (mean |μ| 17). Actions are clipped, so nothing pulled them back and every joint stuck at its limit	a loss on |μ| > 1
survive3–6	resuming or warm-starting the mature policy wrecked it within about 20 updates (100 % falls)	lower learning rate: 5 × 10⁻⁵ instead of 3 × 10⁻⁴ for any mature policy

With these fixes (run survive7), the curriculum went on past 1.0 at the coach’s request, to level 1.5, an 860 N top force. Training falls at that level settled at about 10–11 % with exploration noise and random shoves. The deterministic policy:

fixed test, falls out of 12	no shoves	0.7 (444 N)	0.9 (548 N)	1.0 (600 N)	1.5 (860 N)
A: scripted recovery (balance4, 115 M steps)	0	2	3	3	–
B: survive (survive3, 17.6 M steps)	0	0	1	1	–
B: survive, low learning rate (survive7, 55 M steps)	0	–	–	0	0

At 75 M steps, survive7 held 16 of 16 on the shove grid at level 1.5. The two approaches also differ in reward details and training length, so this compares two complete recipes rather than isolating a single factor.

8. From scratch on the simple body

The muscular body is built for grappling later on. For balance and striking we moved to the simple body and trained the survive task from scratch: no stance or rhythm parent, with the same curriculum from level 0. The first simple body had no toe joints and no clavicles. It learned the task, but the coach rejected it. In the heel-up stance the rigid foot balanced on its front edge like a plank, and without clavicles the shoulders couldn’t shape the guard. Toes and clavicles went back in (v2, 37 motors). The coach chose realism over the extra speed.

	simple v1 (no toes or clavicles)	simple v2
motors	31	37
level 1.5 (860 N) reached	37.3 M steps, 41 min	33.6 M steps, 39 min
fixed test at 1.5, falls out of 12	1	0 (also 0 at 1.0)
shove grid at 1.5	14–15 of 16	15–16 of 16; final grid 16 of 16
total	81 M steps, about 1.5 h, about $1.50	75.6 M steps, about 1.5 h, about $1.50

Figure 6 sets that fixed test next to the two approaches of Sections 6 and 7.

For comparison, the muscular body needed about 89 M steps over five runs to reach level 1.0, while its environment was still being fixed. We stop a run on a plateau. We average the training falls at the top level over 10 M-step windows, and stop when the newest window isn’t at least 1.5 points better than the one two before it.

9. What the coach decided

Several results that scored well looked wrong to a martial artist. These decisions came from the coach watching the videos, not from a metric:

• The stance itself, and its rear-weighted style.
• The rhythm, changed three times, including the heel roll that the reward hadn’t asked for.
• The gym mat, which cut jitter 11× before any retraining.
• Shove testing as in a real class, then pushing past the breaking point to see what fails.
• Bigger recovery steps (“a big step can recover a late reaction”), and step references from photos of the coach stepping.
• The survive experiment: let the body find its own recovery, but keep the hands up and come back to the stance.
• Going on past 600 N to 860 N.
• Rejecting the toeless simple body because its foot looked like a plank, even though it trained well.

10. Training setup and cost

Training never ran on our development machine. Every run used one EC2 spot instance in us-east-1. The default is a 64-vCPU c8a.16xlarge at about $0.93–1.23 an hour, against $3.28 on demand. It runs about 29 M environment steps an hour, so a typical run of 0.5–1.5 h costs $1–2. Simulation is spread over 52 worker processes, each running 8 environments. Each worker runs its own copy of the policy for a whole rollout, so collection overlaps the update (decoupled PPO). That raised throughput from about 14 M to 24 M steps an hour on the same machine (CPU use from 28 % to 65 %). Checkpoints go to S3 every 60 s, a watcher relaunches the run after a spot reclaim, and the instance terminates itself after its deadline. Each time the shove level rises, the instance renders a grid video of controlled shoves for the coach to review.

What the balance work cost

The stance, rhythm and shove runs of 15 and 16 October used 14 EC2 runs. Adding up their start and stop times gives about 15 hours on 64-vCPU machines and 13 hours on 32-vCPU ones. At those days’ spot prices in us-east-1 (about $1.05 an hour for a c7a.16xlarge or c8a.16xlarge, $0.55–0.72 for the 32-vCPU c7a.8xlarge, c8a.8xlarge and m7a.8xlarge), that comes to roughly $24: about $16 on the large machines and $8 on the small ones. One more session ran free on the owner’s gaming PC. These are estimates from the run logs and AWS’s spot price history, not from the bill.

The choice of machine came down to a few decisions:

• Start on c7a, move to c8a. The first runs used a c7a.16xlarge. On 15 October the owner asked for the newer c8a.16xlarge. It runs about 29 M steps an hour against the c7a’s 22 M (Figure 7), for a similar spot price, so it became the default.
• Two half-size machines for two experiments. The account could run 64 spot vCPUs at a time. The scripted-recovery and survive approaches (Sections 6 and 7) were trained side by side, each on a 32-vCPU machine. When no c8a or c7a was free, an m7a.8xlarge filled in.
• Spot over on demand. Spot was about a third of the on-demand price. Amazon reclaimed a machine eight times over the two days. Thanks to the checkpoints and the watcher, each reclaim cost minutes rather than the run.
• Long runs cost more than they taught. The two longest runs, recover v3 and survive3, ran 5–6 hours each on 32-vCPU machines, about a quarter of the bill. Late in the evening the login the watcher used expired, so both ran unwatched to their deadlines. survive3 had already collapsed after a reclaim. The simple body, trained from scratch on a c8a.16xlarge, reached the top shove level in 39 minutes, and its whole run cost about $1.50.

Two operational lessons:

• Progress videos must sample actions. The deterministic policy hid falls that training was still having.
• Resume a mature policy at a low learning rate. Several of our collapses came from the first updates after a restart, not from the task.

11. Limitations and next steps

• The bodies have no muscle activation. Stiffness can’t change with tension.
• “Looks right” is judged by one coach in one style. Other stances (a long guard, different weight splits) would need their own references.
• Training measures balance by the number of falls. The style of the survive recoveries is judged only by eye, from video.
• A real fighter also falls safely (ukemi). That is a later stage.

Next we build on this stable base: striking from the stance, starting with an aimed jab.

References

1. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347.
2. Ng, A. Y., Harada, D. and Russell, S. (1999). Policy invariance under reward transformations: theory and application to reward shaping. ICML.
3. Pratt, J., Carff, J., Drakunov, S. and Goswami, A. (2006). Capture point: a step toward humanoid push recovery. IEEE-RAS Humanoids.
4. Mysore, S., Mabsout, B., Mancuso, R. and Saenko, K. (2021). Regularizing action policies for smooth control with reinforcement learning. ICRA.
5. Hilton, J., Cobbe, K. and Schulman, J. (2022). Batch size-invariance for policy optimization. NeurIPS.
6. Todorov, E., Erez, T. and Tassa, Y. (2012). MuJoCo: a physics engine for model-based control. IROS.
7. de Leva, P. (1996). Adjustments to Zatsiorsky–Seluyanov’s segment inertia parameters. Journal of Biomechanics 29(9).
8. Gordon, C. C. et al. (2014). 2012 Anthropometric Survey of U.S. Army Personnel (ANSUR II). U.S. Army Natick.
9. American Academy of Orthopaedic Surgeons (1965). Joint Motion: Method of Measuring and Recording.