Hyperparameter

Parameters are what training learns: the weights of the network. Hyperparameters are what you choose before training starts. They control how learning happens.

Typical hyperparameters for PPO:

hyperparameter	what it controls	our value
Learning rate	size of each update	$3 \times 10^{-4}$, or $5 \times 10^{-5}$ for mature policies
discount $\gamma$	how far ahead the agent looks	0.99
GAE $\lambda$	bias and variance of the advantage	0.95
clip $\epsilon$	largest policy change per update	0.2
horizon	steps per environment per rollout	256
epochs	passes over each batch	8
minibatch	samples per gradient step	512
network size	capacity of the MLP	2 × 256

Tuning

There is no formula for the best values. They are found by experiment: try settings, compare results, keep what works. Because RL results vary from run to run with the random seed, a single run per setting can mislead.

In practice a lot of what looks like hyperparameter tuning in RL is really environment and reward design. In our posts most fixes were changes to the task, not to these numbers.