Stats
What the training cost in the cloud, how often it fails, and how the GPU providers compare. Models were first trained on Amazon EC2 spot machines, now mostly on GPUs rented on vast.ai and RunPod; nothing heavy runs on our own computers. Every run is counted once: in the post that tells its story, in a work log that isn't in a post yet, or as not sorted yet.
- Cost by post
Newest: “How full is a fighter's brain? A network-size experiment”, $7.65. Every post, newest first, with its work logs and a running total.
- By technique
The same total, split by what was trained: each technique, the shared runs, the foundations.
- What came of the runs
161 of 260 runs kept. What the kept, failed and stopped runs each cost.
- vast.ai and RunPod
511 rented machines compared: how often they fail, what a 5090 costs, how fast they start.
- Inside the brain
The neurons each technique and push wakes, the techniques' neural fingerprints, and which joints do the work.
Latest post with a cost: How full is a fighter's brain? A network-size experiment, .
EC2 figures are estimates from each run's start and stop times and the spot price of its zone at the time, not from the AWS bill; vast.ai and RunPod figures come from each machine's billed hours and hourly price. A run's evaluation, gate and video machines count with the run. Prices are in US dollars.