Kakuto AI

Glossary

The machine learning, robotics and physics simulation terms used on this site, explained. Terms underlined with dots in a post link here.

A

Actionaction space, continuous control
What the policy outputs on each step, such as a target angle for every motor. Reinforcement learning
Action clippingclipping, saturation, action bound
Cutting off actions that fall outside the allowed range. It's needed for safety, but it can hide drift in the policy's outputs. Reinforcement learning
Actor-criticactor, critic
An RL design with two learned parts: the actor (the policy) chooses actions, and the critic (a value function) judges how good things are going. Reinforcement learning
Actuatormotor, actuated joint
Anything that makes a robot or simulated body move, such as a motor at a joint. Robotics and control
AnthropometryANSUR II, de Leva, segment parameters, body segment parameters
The measurement of the human body: sizes, proportions, and the mass and inertia of each body segment. Used to build realistic simulated bodies. Biomechanics
Auxiliary lossimitation loss, supervised loss, auxiliary task, button imitation
An extra training signal added beside the main reward, such as a label saying what the right answer was, to speed up learning. Machine learning

C

CAPS regularisationCAPS, Conditioning for Action Policy Smoothness
Extra loss terms that make a policy's actions smooth: similar from one step to the next, and similar for similar states. Reinforcement learning
Capture pointinstantaneous capture point, divergent component of motion, DCM
The spot on the floor where a moving body would have to step to come to a complete stop. If it's outside the feet, the body must step or fall. Robotics and control
Catastrophic forgettingforgetting, catastrophic interference
When a neural network trained on something new loses a skill it already had, because the same weights are reused for both. Machine learning
Centre of massCOM, CoM, center of mass, centre of gravity
The average position of a body's mass, weighted by how much mass is where. Gravity acts as if it pulls on this one point. Physics simulation
Checkpointsnapshot, saved model, frozen policy
A saved snapshot of a model during training, so training can resume after a stop or an earlier version can be used again. Machine learning
Coding agentAI agent, agent, AI harness, agent harness
An AI assistant that doesn't just answer questions but writes code, runs it, reads the results and keeps going, on a real computer. Machine learning
Collision geometrycollider, geom, capsule, collision shape
The simple shapes a physics engine uses to work out what is touching what: spheres, capsules, boxes and meshes. Physics simulation
CUDA graphCUDA graphs, graph capture, graphed, torch.cuda.graph
A recording of a sequence of GPU kernels that can be replayed as one unit, so the CPU doesn't have to launch each kernel again. Computing
Curriculum learningcurriculum, automatic curriculum
Training on easy versions of a task first and making it harder as the agent improves, like a coach raising the level for a student. Reinforcement learning

D

Decoupled PPOasynchronous PPO, batch size-invariant PPO
A variant of PPO that lets data collection run on a slightly older copy of the policy while the learner updates, so the two can overlap. Reinforcement learning
Degrees of freedomDoF, DOF, joint, hinge joint, ball joint
The number of independent ways something can move. Each simple joint adds one; a free body floating in space has six. Robotics and control

E

Early stoppingplateau, stopping rule
Ending a training run once it stops improving, instead of running for a fixed time. Machine learning
Environmentenv
Everything the agent interacts with: it takes the agent's action, moves the world forward one step, and returns the next observation and a reward. Reinforcement learning
Environment stepstep, sample, timesteps, M steps
One tick of the agent–environment loop: one observation, one action, one reward. Training length is counted in these steps. Reinforcement learning
Episodetrajectory, termination
One attempt at the task, from reset to the end: a fall, a time limit or success. Reinforcement learning

F

Feed-forward controlfeed-forward, feedforward, servo feed-forward, velocity feed-forward
Telling a motor in advance what a planned movement will need, instead of waiting for an error to build up and then correcting it. Robotics and control
Fine-tuningwarm start, warm-starting, resume, from scratch
Continuing to train an already-trained model, often on a new or harder task, instead of starting from scratch. Machine learning
Floating pointfloat32, float64, 32-bit, single precision, double precision, rounding error, FP32
How computers store real numbers: a fixed number of significant digits plus an exponent, so every result is rounded slightly. Computing
Frame stackingstacked history, history window, stacked frames, observation history
Giving a network the last few frames of its input side by side, so it can see motion without having a memory of its own. Machine learning
Frictioncoefficient of friction, skating, friction cone
The force that resists surfaces sliding over each other. It's what lets feet push on the floor without skating. Physics simulation

G

GPUgraphics card, CUDA, graphics processing unit
A graphics processor: a chip with thousands of small cores that do the same sum on lots of numbers at once. Fast for neural networks, less useful for CPU physics. Computing
GPU kernelkernel, kernel launch, launch, CUDA kernel, thread block
One small program sent from the CPU to the GPU, run by thousands of GPU threads at once; every launch has a small fixed start-up cost. Computing
Gradient descentSGD, stochastic gradient descent, Adam, backpropagation, optimiser
The basic method for training neural networks: repeatedly nudge every parameter a small step in the direction that reduces the loss. Machine learning

H

Hill climbinglocal search, random search
A simple search method: make a small random change, keep it if it improves the score, and repeat. Machine learning
Human-in-the-loopHITL, human feedback
Training where a person judges results and steers what happens next, instead of relying only on an automatic score. Machine learning
Hyperparameterhyperparameters, tuning
A setting chosen by the person training the model, such as the learning rate or network size, rather than learned during training. Machine learning

I

Inertiamoment of inertia, inertia tensor, rotational inertia
How hard it is to change an object's motion. For rotation, the moment of inertia depends on how far the mass is from the axis. Physics simulation
Inverse kinematicsIK
Working out the joint angles that put a hand or foot at a chosen position, the reverse of forward kinematics. Robotics and control

J

Jitterchatter, twitching, direction reversals
Small, fast, back-and-forth twitching of the joints. Common in RL policies; we measure it as direction reversals per joint per second. Robotics and control

K

Keyframekeyframing, keyframed motion, interpolation
A pose set at a specific moment in a motion. The frames between keyframes are filled in automatically by interpolation. Robotics and control
Keypointkeypoints, pose estimation, skeleton, landmarks
A tracked point on a body, such as a wrist, elbow or hip, used to describe a pose with a few numbers instead of a whole image. Robotics and control
Kinematicsforward kinematics, kinematic, dynamics
The geometry of motion: where each body part is, given the joint angles, without asking what forces caused it. Robotics and control
Kinetic chainkinematic chain, power chain, hip drive
The linked segments of the body (feet, legs, hips, trunk, shoulder, arm) that pass force and speed from one to the next in a movement such as a punch. Biomechanics

L

Latencyvision delay, reaction time, delay, lag
The delay between something happening and the system knowing about it; for a fighter, the reaction time. Robotics and control
Learning ratestep size, η, lr
How big a step training takes each time it updates the weights. Too big and training becomes unstable; too small and it's slow. Machine learning
Local optimumlocal optima, local minimum, local maximum, plateau
A solution that is better than everything close to it but not the best overall, so small improvements can't get away from it. Machine learning
Loss functionloss, objective, cost function
A number that measures how wrong a model is. Training adjusts the model to make it smaller. Machine learning
Low-pass filteraction filter, exponential moving average, EMA, smoothing filter, cutoff frequency
A filter that lets slow changes through and smooths away fast ones. Used on a policy's actions to stop twitching. Robotics and control

M

Machine learningML
Getting computers to do a task by learning patterns from data or experience, rather than by following rules a programmer wrote by hand. Machine learning
Markov decision processMDP, discount factor, return
The maths framework behind reinforcement learning: states, actions, rewards, and rules for how one state leads to the next. Reinforcement learning
Mass matrixinertia matrix, factorisation, factorization, Cholesky, LDL
A table that says how hard each joint is to accelerate, given everything attached to it; the physics has to factorise it every step. Physics simulation
MomentumN s, N·s, linear momentum, impact
Mass times velocity: how much motion something carries. A punch's momentum is what it can pass on to the target. Physics simulation
Monotone cubic splinePCHIP, Fritsch–Carlson interpolation, cubic Hermite spline, spline
A smooth curve through a set of points that never overshoots them, so it rises where the points rise and stays flat where they're flat. Robotics and control
Motion capturemocap, stock motion, motion clip, animation clip
Recording a real person's movement, usually with markers or cameras, so a computer can replay it on a digital body. Biomechanics
Motion plannerplanner, footstep planner, step planner
The part of a system that decides which movement to make next to reach a goal, leaving the details of making it to a controller. Robotics and control
MuJoCoMulti-Joint dynamics with Contact
A fast, accurate physics engine for simulating bodies with joints and contacts, widely used in robotics and reinforcement learning research. Physics simulation
MuJoCo WarpMJWarp, mujoco-warp, Warp, NVIDIA Warp, GPU physics
A version of the MuJoCo physics engine that runs on a GPU and simulates thousands of copies of a world at once. Physics simulation
Multilayer perceptronMLP, fully connected network, feed-forward network, 2 × 256
The simplest kind of neural network: a stack of fully connected layers, where every unit in one layer feeds every unit in the next. Machine learning

N

Neural networkartificial neural network, deep learning, weights
A function built from layers of simple units, each multiplying its inputs by learned weights, adding them up and passing the result through a nonlinearity. Machine learning
Newton (unit)N, force, impulse
The SI unit of force. One newton accelerates one kilogram by one metre per second, every second. A 77 kg person weighs about 755 N. Physics simulation

O

Observationstate, observation space, observation normalisation
The numbers the policy sees at each step, describing the state of the body and the task. Reinforcement learning
Occlusionline of sight, ray casting, blindness, occluded, hidden
When something is hidden from view because something else is in the way, such as your own arms in front of your eyes. Robotics and control
Oracleoracle baseline, upper bound, perfect reader
A hand-made baseline that is given information or abilities the learner doesn't have, to show the best result possible. Machine learning

P

Parallel environmentsvectorised environments, vectorized environments, copies of the world, worlds, num envs, batch of environments
Many copies of the same training world run side by side, so one policy collects experience from all of them at once. Reinforcement learning
Phase variablephase, motion clock, punch clock, clock, time scaling
A clock that says how far through a movement the body is, from 0 at the start to 1 at the end, so a motion can be played faster or slower. Robotics and control
Physics simulationphysics engine, simulator, rigid-body dynamics, sim-to-real
Computing how objects move under forces, one small time step at a time, so a virtual body behaves like a real one. Physics simulation
Policyπ, controller, actor
The agent's decision rule: given what it observes, which action it takes. In deep RL, the policy is a neural network. Reinforcement learning
Policy distillationdistillation, teacher, teacher policy, student, anchor, behaviour cloning
Training one policy to copy another's actions. Here a frozen copy of an older policy acts as a teacher, so new training doesn't wipe out skills it already had. Reinforcement learning
Position servoPD controller, PD control, servo, position actuator, stiffness, damping
A motor controller that pushes a joint toward a target angle, with a force that grows with the error, up to a limit. Robotics and control
Profilingprofiler, profile, bottleneck, torch profiler
Measuring where a program spends its time, part by part, to find what to speed up. Computing
Proximal Policy OptimizationPPO
A widely used reinforcement learning algorithm that improves a policy in small, safe steps by limiting how far each update can move it. Reinforcement learning
Push recoveryshove recovery, perturbation, balance recovery
Staying on your feet after an unexpected shove, by leaning, stepping or hopping. Robotics and control

R

Range of motionROM, joint limits, joint range
How far a joint can move, in degrees, from one end of its travel to the other. Biomechanics
Recurrent neural networkRNN, GRU, LSTM, gated recurrent unit, memory
A neural network with a memory: it carries a hidden state from one step to the next, so it can use what it saw earlier. Machine learning
Reference motionmotion tracking, motion imitation, reference, tracking reward
A target motion the policy is rewarded for following, such as a stance, a rhythm or a keyframed step. The policy adds the physics. Robotics and control
Reference state initialisationRSI, reference state initialization, mid-motion starts, mid-walk starts
Starting some training episodes partway through a reference motion, in its pose and speed, so the policy practises every part of it and not only the beginning. Reinforcement learning
Regularisationregularization, weight decay
Anything added to training that prefers simpler or smoother solutions, usually an extra penalty in the loss. Machine learning
Reinforcement learningRL
A way of training a program by trial and error: it acts, gets a score called a reward, and slowly learns to act in ways that score more. Reinforcement learning
Residual policyresidual action, residual control
A policy that outputs corrections to a reference motion rather than whole new motions, so it only learns the difference. Reinforcement learning
Reward functionreward, reward design
The rule that scores each step of the agent's behaviour. The agent learns to get as much reward as it can, so the reward defines the task. Reinforcement learning
Reward hackingspecification gaming, reward exploitation
When an agent finds a way to score well that isn't what the designer meant, exploiting a gap in the reward. Reinforcement learning
Reward shapingpotential-based shaping, shaping reward
Adding extra reward terms to guide learning. Potential-based shaping does this in a way that provably doesn't change which policy is best. Reinforcement learning
Rolloutbatch, horizon, iteration
Running the current policy in the environment for a fixed number of steps to collect experience for one training update. Reinforcement learning

S

Shared memorymultiprocessing.shared_memory, pipes, inter-process communication, IPC
A block of memory that several processes can read and write directly, so data passes between them without being copied. Computing
Shortcut learninggiveaway, giveaways, spurious correlation, Clever Hans, tell, data leak
When a model learns an easy cue that happens to predict the answer in training, instead of the thing you meant it to learn. Machine learning
Soft contactcompliance, contact model, penetration, solref, solimp
Modelling contacts as slightly springy, so surfaces can sink into each other a little. Used both for stability and to model soft things like skin and foam. Physics simulation
Spot instanceEC2 spot, spot pricing, on-demand
A cloud computer rented cheaply from spare capacity, which the provider can take back at short notice. Computing
Stochastic policyGaussian policy, deterministic policy, exploration noise, sampled actions
A policy that outputs a probability distribution over actions and samples from it, so the agent explores. Taking the most likely action instead makes it deterministic. Reinforcement learning
Support polygonbase of support, support region
The outline on the floor around all the points where the body touches the ground. Standing still, the body balances if its centre of mass is above it. Robotics and control

T

Timesteptime step, physics step, control frequency, substeps, Hz
The small slice of time a physics simulation advances in one step. The policy usually acts less often, once every several physics steps. Physics simulation
Torquemoment, N·m, newton-metre
A twisting force: what makes something rotate. A motor turns a joint by applying torque, measured in newton-metres. Physics simulation
Training divergenceNaN, collapse, policy collapse, instability
When training breaks down instead of improving: the loss blows up, the numbers become NaN, or a good policy suddenly collapses. Machine learning

V

Value functioncritic, V(s), advantage, GAE
An estimate of how much future reward the agent can expect from a given state. The advantage says whether an action did better or worse than that. Reinforcement learning
vCPUvirtual CPU, core, worker process, parallelism, hyperthreading, hyperthread, SMT
A virtual CPU: one unit of processor capacity in a cloud machine. A 64-vCPU machine can run about 64 things at once. Computing