Stats

What the training cost in the cloud, how often it fails, and how the GPU providers compare. Models were first trained on Amazon EC2 spot machines, now mostly on GPUs rented on vast.ai and RunPod; nothing heavy runs on our own computers. Every run is counted once: in the post that tells its story, in awork log that isn't in a post yet, or below as not sorted yet.

Total so far (est.)$282

Training runs195

Machine-hours294

By post

Post	Runs	Machine-hoursHours	Cost	Running total
Learning to balance in a Muay Thai stance16 October 2026 · in the post	19	29.8	$22.98	$22.98
The jab: the simplest punch, and a long lesson18 October 2026 · in the post	22	15.9	$16.01	$38.99
Five punches: the cross, the hooks and the uppercut18 October 2026, 10:00 UTC · in the post	10	14.2	$21.95	$60.94
Faster punches: from trained amateur towards professional18 October 2026, 16:27 UTC · in the post	7	4.5	$8.07	$69.01
Footwork: walking to a spot in a Muay Thai stance19 October 2026, 15:44 UTC · in the post	22	26.5	$35.03	$104.04
Blocking: reading a punch you see a tenth of a second late20 October 2026, 17:43 UTC · in the post	20	16.7	$33	$137.04
Training on a rented GPU: four and a half times cheaper21 October 2026, 07:54 UTC · in the post	–	9.1	$5.42	$142.46
Two fighters and vision: GPU against CPU, optimised23 October 2026, 05:08 UTC · in the post	–	5.5	$6.43	$148.89
Shadow clones: four Muay Thai techniques learnt in parallel24 October 2026, 20:30 UTC · in the post ·	71	108.1	$81.04	$229.93
The higher guard: retraining every skill onto a new stance	18	24.5	$21.04
Teacher: the rear straight to the body	4	5.7	$4.01
Teachers: the rear knee and the switch knee	6	9.2	$6
Distillation: six runs to the student distill4	6	22.9	$16.27
Teacher: the pull counter	1	3.1	$3.45
Teachers: slips, rolls and the lean back	9	20	$12.97
Teachers: the uppercuts over distance	2	10.1	$7.81
Fast stepping and the step-jab (abandoned)	25	12.6	$9.49

Not in a post yet

Machines that trained nothing: renders and tests25–26 October 2026	0	5	$3.28
Teachers: the uppercuts, second round25–26 October 2026	4	11.1	$10.06
Teacher: the rear knee, landing upright26 October 2026	1	2.1	$1.29
Teacher: the dash26–27 October 2026	6	10.1	$9.91
Teachers: the kicks26–27 October 2026 · still going	6	15.7	$13.98
Distillation: distill7 and distill826–27 October 2026 · still going	2	10	$6.24
Teachers: the elbows26 October 2026 · still going	2	9.5	$6.83
The muscle body: muscles instead of joint motors27 October 2026 · still going	3	0.1	$0.08

Not sorted yet

Tests, renders and shakedowns that are no model's: a 1-minute RunPod pod to make the volume resize to 50 GB take effect, 27 October.

Not in a work log

$0.01

By technique

The same total, split by what was trained. A run that trained one technique (or a left and right pair) counts under it. A run that trained several at once (a base model, a retrain onto a new stance, a distillation into the student) counts once under "shared", not split between them. Stance, balance, walking and stepping are the foundation, and machine tests and renders are infrastructure.

Technique	Runs	Machine-hoursHours	Cost
The jab	32	19.7	$18.19
The cross	0	0	$0
The hooks	3	4.3	$4.04
The uppercuts	6	20.8	$17.67
The body straight	4	5.7	$4.01
The knees	7	11.3	$7.28
The elbows	2	9.5	$6.83
The low kicks	2	5.3	$4.71
The mid kicks	2	5.3	$4.71
The high kicks	2	5.1	$4.57
The pull counter	1	3.1	$3.45
The slip and uppercut	3	6.6	$4.39
The lean back	2	4.2	$2.42
The rolls	4	9.2	$6.16
The dash	6	10.1	$9.91
Shared: several techniques at once	84	107	$132.39
Foundations: stance, balance, walking, stepping	35	34.4	$26.51
Infrastructure: machine tests, benchmarks, renders	0	32.2	$24.35
Total	195	293.8	$281.58

What came of the runs

Most runs are experiments: a change is tried, and the gate or the video decides whether the model is used. "Kept" means a model from the run is used (a keeper, a teacher, or the parent of the next run). "Not kept" ran as planned but wasn't needed or wasn't better. "Failed" was judged a failure (usually by me, on the video) or crashed. "Stopped" was stopped early, usually because it had stopped improving or the plan changed.

Outcome	Runs	Machine-hoursHours	Cost	Share
Kept	133	179	$191.12	74 %
Not kept	20	37.5	$33.22	13 %
Failed	18	28.9	$21.15	8 %
Stopped	20	13.3	$9.82	4 %
Training	4	2.9	$1.92	1 %

vast.ai and RunPod

Small samples. These are 153 vast.ai machines (22–27 October 2026) and 104 RunPod rents (23–27 October 2026), a few days of one project. Read them as early signs, not a verdict. Every machine rented goes into the ledgers, so the samples grow with the project and this section is rebuilt from them each time.

One row per machine rented, and on RunPod also per rent it refused (no machine, no charge). Amazon EC2 isn't here: it ran on CPUs, before the GPU trainer existed, so it isn't a like-for-like comparison.

	vast.ai	RunPod
Rents tried	153	104
Did its job	103 (70 %)	42 (40 %)
Refused	0 (0 %)	40 (38 %)
Never came up	34 (23 %)	1 (1 %)
Slow host	4 (3 %)	0 (0 %)
Failed or stopped	7 (5 %)	21 (20 %)
Not recorded	5	0
Spent	$81.54	$52.14
On machines that never came up or ran slow	$8.18 (10 %)	$1.21 (2 %)
On jobs that failed or were stopped	$2.20 (3 %)	$7.84 (15 %)
RTX 5090 per hour (median)	$0.69 (n = 142)	$1.19 (n = 21)
Renting to logged in (median)	2 min (n = 65)	33 s (n = 63)

Percentages are of the rents whose outcome was recorded. "Failed or stopped" machines came up fine; the job on them failed or was stopped, almost always on our side: a bug in our scripts, a copy back to my PC that broke, a run stopped by hand because it had plateaued. They are counted so the money is complete, not held against the provider.

How the rents ended

What an RTX 5090 costs an hour

Each dot is one machine; the black line is the median. Hover a dot for its price and run.

From renting to logging in

Seconds from asking for a machine to its ssh answering, any GPU. Machines that never came up aren't in it.

So far

vast.ai

Cheaper: an RTX 5090 at a median $0.69 an hour against $1.19 on RunPod.
There is nearly always a 5090 to rent: it is a market of many hosts.
34 machines never came up (22 of them because vast.ai's command-line tool created them stopped), and 4 hosts ran far below other 5090s (a weak CPU, a GPU held at low power). That costs money and, worse, waiting.
Slower to start: a median 2 min from renting to logging in.

RunPod

Machines start fast (a median 33 s) and, once rented, came up every time but 1.
A refused rent costs nothing, and cheap L4s suit renders and checks.
Often out of stock: 40 of 104 rents were refused, and the 4090 filled in when no 5090 was free.
Dearer for the same GPU.

For now: long training runs on vast.ai 5090s, where the lower price adds up and a bad host is dropped early; RunPod for short jobs (gates, videos, renders) where a fast start matters more than the hourly price. Training speed isn't compared yet: the ledgers record each trainer's steps a second, but the two providers have mostly run different trainers so far.

EC2 figures are estimates from each run's start and stop times and the spot price of its zone at the time, not from the AWS bill; vast.ai and RunPod figures come from each machine's billed hours and hourly price. A run's evaluation, gate and video machines count with the run. Prices are in US dollars.