How to Resume GPU Training After a Spot Instance Interruption (Without Hand-Rolling a Checkpoint Rig)
Somewhere on your box there's a cron job that copies a .pt file to S3 every fifteen minutes. You wrote it so you could rent spot GPUs at a fraction of the on-demand price, and it works right up until the run you actually cared about gets reclaimed at minute fourteen.
I build the infrastructure underneath this, not the training runs on top of it, so I've spent more time than is healthy reading how people resume GPU training after a spot instance interruption. The pattern is always the same. Everyone hand-rolls a checkpoint. Almost everyone loses a run anyway.
Here's the shift I want to argue for. Resuming after a reclaim shouldn't mean replaying your last checkpoint into a box you have to rebuild first. It should mean one command that brings the whole machine back, the weights and the code and the CUDA stack and the staged dataset, on whatever GPU is cheapest and in stock. The checkpoint is a thing you should never have had to build.
TL;DR: Spot GPUs are 50 to 80 percent cheaper than on-demand, so practitioners hand-roll checkpoint scripts to survive reclaims. Those scripts save your model weights but not your environment, always miss the work since the last save, and rot the moment someone forgets them. Resuming reliably means snapshotting the whole box, not just the training loop, so you can restore it on any provider instead of rebuilding from a bare instance.
Why do spot GPUs force you to hand-roll a checkpoint?
Because spot capacity is borrowed. Providers rent you idle hardware at a steep discount and take it back the moment someone pays full price, usually with about two minutes of warning. Nothing about your running job survives that reclaim, so the only way to make spot safe is to persist your own state, on your own schedule.
That discount is real, which is why nobody wants to give it up. Spot and interruptible instances run 50 to 80 percent below on-demand pricing, enough that a solo researcher or a small team can afford compute they otherwise couldn't touch. The catch is the two-minute notice. AWS itself, describing spot training, admits the do-it-yourself version of this "can be very tricky," which is a strong word coming from the vendor whose docs you are following. Two minutes is enough to flush one more checkpoint. It is nowhere near enough to save a working environment.
The rig everyone ends up building
Open any spot-GPU thread and the same stack appears, assembled independently by people who have never met.
There's the loop that copies model files to object storage on a timer, the literal while true; rsync *.pt; sleep 300 that gets copy-pasted from one project to the next. There's the decision to checkpoint every fifteen minutes specifically because that's the interval that makes spot pricing feel survivable. There's a SIGTERM handler wired to the termination notice so the job flushes a final save before the box dies. There's an nvidia-smi watchdog that shuts the instance down when the card goes idle so you stop paying for nothing. And there's the boot script, the EC2 user-data that git-pulls, dvc-pulls, and rsyncs on startup so a fresh instance can pick the run back up.
Each piece is reasonable on its own. Together they're a small distributed system that you maintain by hand, on top of the actual work you were trying to do. And it's a system whose entire job is to paper over the fact that the box underneath it is disposable.
What does a spot instance interruption actually cost you?
More than the hours you can see. A checkpoint every N minutes means the interruption lands, on average, N/2 minutes after your last save, and that delta is gone. On a long run at cluster prices, the discarded work between checkpoints runs to hundreds or thousands of dollars, before you count the hours spent finding a new box.
The stories are consistent and they are not cheap. One practitioner writing up a preemptible run put it plainly: "This is what happened to me on a $10,000 training run. We lost 12 hours of compute." Another engineering writeup on sub-millisecond checkpointing points out the cruel arithmetic of coarse intervals: if you save every four hours and the reclaim hits at three hours and fifty-nine minutes, you discard hundreds of dollars of work and, if the interruption catches you mid-write, you can corrupt the checkpoint you were counting on. And a Vast.ai user summed up the failure mode that no checkpoint interval fixes: a "machine simply went offline after 10 days and $1,000 spent generating data."
Now watch what happens when the problem isn't your interval at all:
"We DO use checkpointing. It doesn't matter when the interruption rate spikes to 40% in one zone. We lost two full model training runs (36 hours of compute). Total damage: about $12k." Practitioner on the Gather ML community
Their checkpoints were fine. The zone was the problem. When a provider's spot pool dries up, restarting into the same pool just gets you reclaimed again, and a local checkpoint can't save you from that. You need to land somewhere else.
Why isn't checkpointing enough?
Because a checkpoint saves your training loop, not your machine. Even the careful playbooks that tell you to checkpoint "the whole state" mean the optimizer, the learning-rate schedule, and the RNG seed. None of that includes the Python environment, the pinned CUDA version, the custom kernels you compiled, or the dataset you staged on local disk. Restore a checkpoint onto a bare box and you still have to rebuild the box.
This is the part the genre quietly skips. A good checkpoint-and-resume playbook for ephemeral GPUs will walk you through serializing model weights, optimizer state, and RNG so training resumes deterministically. That's correct and it's worth doing. But "whole state" there means the state of the loop, not the state of the environment the loop needs to run. The 30 to 60 minutes it takes to reinstall drivers, resolve dependencies, and re-download your models is exactly the reinstall tax I've written about before, and your training checkpoint does nothing about it.
The second reason is more human. A hand-rolled rig only works if someone keeps it working, and someone always stops.
"You SSH in every Monday to check for model updates, manually copy checkpoints to backup storage, and clear old logs. Eventually you forget, the disk fills up, and the inference server crashes at 2 AM." gigagpu, on hand-rolled backup cron jobs
The script that saved you for three months is the script nobody touched for the fourth, and the reclaim that finally lands is the one that catches the cron job pointing at a full disk. The failure is never the code you were reviewing. It's the plumbing you stopped looking at.
What actually resumes GPU training after a spot instance interruption
The right answer is layered, and the first layer is the one you already have. Keep the framework checkpoint. It's the correct tool for training-loop state, it's cheap, and losing the optimizer and RNG would make a resume non-deterministic. Shorten the interval to match the interruption rate you actually see, and write it to storage that outlives the box rather than the local disk that dies with it.
The layer that's missing is the box itself. If a resume is going to be a restore instead of a rebuild, something has to capture the whole environment, the filesystem and the dependencies and the staged data, not just the weights. And that capture has to be portable, because the $12k lesson is that surviving a bad zone means moving to a different one, not fighting for capacity in the pool that just reclaimed you.
This is the primitive we're building in the open with ogre: a single binary that snapshots an entire GPU box, the full filesystem and not just a bolted-on data volume, and restores it with one command. It's open source, it runs on a box you already rent, and we're proving the cross-provider restore in public rather than asking you to take it on faith. The goal is simple to state. Your framework keeps checkpointing the run. The box stops being something you rebuild by hand every time the run gets interrupted. And because the snapshot moves with you, the answer to a 40 percent interruption rate is to restore somewhere cheaper instead of losing the run.
What I'd actually do
If you're renting spot today, don't rip anything out. Keep the checkpoint, tighten the interval, and get it off local disk. That covers the training loop, which is the part your framework is genuinely good at persisting.
Then stop treating the environment as free to rebuild. Snapshot the whole box separately, so a resume brings back the CUDA stack and the venv and the dataset without a 40-minute cold start. Keep that snapshot portable, because region-locked state that can't leave its datacenter fails you at exactly the moment you need to switch providers.
The honest version is that you can build every layer of this yourself. Most people do, and most people still lose a run eventually, because the piece that fails is never the piece you were watching. It's the cron job pointing at a full disk, the SIGTERM handler that flushed weights onto a box you then had to rebuild from scratch, the boot script that assumed a provider still had capacity. A checkpoint you have to hand-roll is a checkpoint that fails the way hand-rolled things fail: quietly, on the run you cared about most.
About the author
I'm on the team at Aquanode, where we build ogre, an open-source snapshot and restore tool for GPU boxes. I don't run the training jobs myself, I build the layer underneath them, and most of what's here comes from reading how practitioners on r/LocalLLaMA, the Vast.ai forums, and ML communities actually survive spot reclaims. If your checkpoint rig has ever failed you at minute fourteen, that's the problem we're working on in the open.
Sources
- What Are Spot GPUs? A Practical Guide, Northflank (spot pricing 50 to 80 percent below on-demand; auto-checkpoint-every-15-min pattern)
- Train Deep Learning Models on GPUs Using Amazon EC2 Spot Instances, AWS ML Blog (DIY spot checkpointing "can be very tricky")
- Backup and Sync Scripts, TensorRigs (the hand-rolled rsync loop)
- EC2 Spot user-data with git, dvc, and rsync, colllin gist (auto-resume boot script)
- Spot Instances for Preemptible ML Training, ADHDecode ("$10,000 training run, lost 12 hours")
- Sub-Millisecond LLM Checkpointing on Spot Instances, Stacklab (coarse-interval discard cost; corrupt-mid-write)
- Vast.ai user review, Trustpilot ("machine simply went offline after 10 days and $1,000 spent")
- Spot interruptions and the real cost, Gather ML community (checkpointing does not help when the zone interruption rate spikes; $12k lost)
- A Checkpoint-and-Resume Playbook for Ephemeral GPUs, dev.to ("whole state" means the training loop, not the environment)
- Cron Jobs for Model Backups, gigagpu ("eventually you forget")
