Spot (or preemptible) GPU capacity is spare capacity that a provider sells at a discount and can take back, usually with little warning. The discount is real enough to be tempting. The catch is that a training job written for a stable machine will lose hours of work the first time one disappears. Whether spot is a good deal depends almost entirely on how well your job handles being killed.

Start by Accepting That Interruption Is Normal

The mistake is treating a reclaim as an incident. With spot capacity it’s the expected operating mode, and the question is only how much it costs each time. If your job can restart and lose five minutes, an interruption is a minor cost. If it loses six hours, you’ve paid more than you saved.

So design for this: every run should be killable at any moment and resumable from a known good point. Do that, and you gain something beyond cost savings, because the same machinery protects you from ordinary hardware failures on any instance type.

What a Resumable Job Actually Saves

A checkpoint that holds only model weights is a common trap. Resuming from weights alone often changes the training behavior in subtle ways. A proper resumable checkpoint usually includes:

• Model weights.
• Optimizer state (momentum and similar buffers), which can be as large as the model itself.
• Learning-rate scheduler position.
• The step or epoch counter.
• Random number generator state, if you want reproducibility.
• Your position in the data, so you don’t repeat or skip samples.

That last one gets neglected. If a restart replays the dataset from the beginning with the schedule already advanced, you’ve quietly trained on an unbalanced mix. Frameworks differ in how they support resuming data position, so check your framework’s current documentation for its recommended approach.

Where You Write Checkpoints Matters

Local instance disk disappears when the instance is reclaimed. Write checkpoints to durable storage that outlives the machine, such as object storage or a network file system, and verify the write finished before you treat it as safe.

Two habits prevent the nastiest failures:

1. Write atomically. Save to a temporary name and rename on success, so a kill mid-write doesn’t leave a corrupted file that your restart logic then loads.
2. Keep more than one. Retain the last few checkpoints rather than overwriting a single file, so a bad one doesn’t strand you.

Choosing a Checkpoint Interval

Frequent checkpoints cost time and storage; infrequent ones cost lost work on interruption. A reasonable way to think about it is to measure how long a checkpoint takes to write, decide what fraction of runtime you’re willing to spend on it, and see how much work you’d lose in the worst case. For large models with big optimizer state, writes can be slow enough that asynchronous saving is worth looking into.

Providers typically signal an impending reclaim in some way, such as a notice delivered through the instance metadata or an operating-system signal. The details and the amount of warning vary, so check the vendor’s current documentation. You can use the signal to trigger a final save, but never depend on it alone, since the notice may be short or not arrive at all.

Automatic Restart Is Half the Job

A resumable job still needs something to resume it. Otherwise the interruption happens at 2 a.m. and nobody restarts it until morning. Common approaches:

• A job scheduler or orchestrator that requeues failed work automatically.
• A managed training service’s built-in retry behavior, where available.
• A small wrapper script that, on startup, looks for the latest valid checkpoint and resumes from it.

Whatever you use, make the startup logic idempotent: launching the job twice, or on a fresh machine, should land in the same place.

A Hypothetical Example

A team trains a mid-sized model over roughly a day on a single multi-GPU node. They first try spot with checkpoints every few hours saved to local disk. The instance is reclaimed late in the run, the disk goes with it, and they restart from the beginning.

After switching, they save every thirty minutes to object storage using write-then-rename, keep the last three, and add a wrapper that resumes from the newest valid checkpoint. They simulate a reclaim by killing the process mid-save during a short test run. The restart ignores the half-written file, loads the previous one, and continues. The next real interruption costs them under half an hour of work.

A Readiness Checklist

Check	What to verify
Kill test	Terminate the job mid-run; it resumes without manual steps
Mid-write kill	Interrupt during a save; restart ignores the partial file
Full state	Optimizer, scheduler, step and data position all restored
Durable storage	Checkpoints survive instance loss
Auto-restart	Something relaunches the job without a human
Metric continuity	Loss curve after resume matches the pre-interruption trend

When Spot Is the Wrong Choice

Not every workload fits. Jobs that can’t checkpoint, tightly coupled multi-node runs where one lost node halts everything, and deadline-driven work with no slack are poor candidates. A mixed strategy is common: keep a core on stable capacity and use spot for the portion that tolerates it. Interactive development and latency-sensitive serving generally call for stable instances.

Frequently Asked Questions

How much can I actually save with spot? It varies by provider, hardware and demand, and prices change. Look at current published pricing rather than assuming a figure, and factor in the cost of lost work.

Will a resumed run match an uninterrupted one exactly? Not necessarily bit-for-bit, especially if random state or data order isn’t restored. It should be statistically equivalent if you restore full state.

Do multi-node jobs work on spot? They can, but losing any node usually stops the whole job, so interruptions are more disruptive. Elastic training features exist in some frameworks; check their current documentation.

Should I checkpoint inference workloads? Generally no. Serving jobs are better handled by running multiple replicas and letting the load balancer route around lost ones.

What if I get no warning? Plan for that case. Your regular checkpoint interval is your real protection, and the warning signal is a bonus.

Next Step

Run a deliberate kill test on a short job today: stop it mid-save, restart it, and confirm it resumes from the right place. If that passes, you’re ready to try spot on a real run.