Stretch reflex

When a muscle is suddenly stretched, sensors in it (muscle spindles) signal the spinal cord, which excites the same muscle within a few tens of milliseconds, long before the brain could respond. The tap of a doctor’s hammer below the knee is the famous example: the tendon is stretched, the quadriceps fire, the leg kicks. In everyday movement the same loop holds posture and corrects small errors automatically.

A simple controller version adds excitation in proportion to how far a muscle is stretched past its length at a target pose, and how fast:

$$u_i = u_i^{\text{policy}} + \frac{k_p\, (\ell_i - \ell_i^*) + k_d\, \dot\ell_i}{\ell_i^{\text{opt}}}$$

where $\ell_i^*$ is the muscle’s length at the target pose and $\ell_i^{\text{opt}}$ its optimal fibre length. Stretched muscles get excited and pull back; shortened ones get less, which also switches off the muscle opposing a movement, as the spinal cord does. It is the muscle body’s version of a servo’s pull toward its target.

In this project

In Muscles instead of motors each muscle got this reflex toward the reference pose, with $k_p$ = 5 and $k_d$ = 0.2 (chosen on a CPU test: a zero action held the still stance 2.4 s against 1.4 s without it; higher gains were worse). In the punches it brought the muscle fighter’s arm out at once, misses of 28–31 cm against 51–68 without it. Under a stance trained without it, the body fell (97 % of episodes at first); a stance trained with it solved in 56.0 M steps with the tidiest pose (0.93) and three times the jitter.