Spot instance

Cloud providers such as Amazon Web Services (AWS) rent out computers by the hour. An on-demand instance is yours until you stop it, at full price. A spot instance uses capacity that would otherwise sit idle, at a large discount, often 60–90 % off. The catch: when the provider needs the capacity back, it reclaims the machine, on AWS with two minutes’ warning.

Making spot work for training

Interruptions are fine if a run can resume:

• save checkpoints often, to storage that outlives the machine;
• have something watch for the reclaim and relaunch the run on a new machine;
• make the run terminate itself at a deadline, so a forgotten machine can’t run up a bill.

In this project

Every training run uses one Amazon EC2 spot instance in the us-east-1 region. The default is a 64-vCPU c8a.16xlarge at about $0.93–1.23 an hour, against $3.28 on demand. Checkpoints go to Amazon S3 every 60 s, a watcher relaunches the run after a reclaim, and the instance shuts itself down after its deadline. A typical run of 0.5–1.5 hours costs $1–2.

In the 22 walking runs of 18–19 October, Amazon reclaimed the machine five times, and once the zones it tried had no spare machine of that size for 20 minutes. The launch script falls back from the newest generation (c8a) to its Intel twin (c8i) and then the generation before. Each reclaim cost a few minutes of training, because the run carried on from its last checkpoint.

Rented GPUs were cheaper still for the stance. On vast.ai, a market of GPU machines rented by the second, the same L40S card costs about a third of EC2’s spot price, and a solve of the stance cost less than a quarter of what it cost on the CPU spot machine (Training on a rented GPU).