How to Recover Your Environment After a Cloud GPU Reclaim (and Stop Rebuilding It From Scratch)

Look at everything expensive in a modern ML stack and notice one thing: it is all versioned and portable now. Your code lives in git. Your containers live in a registry. Your infra lives in Terraform state you can re-apply anywhere. Every one of those you can lose and rebuild in seconds, because a copy exists somewhere that is not the machine. Then there is the single most expensive thing you rent, the GPU box itself, and it is the one thing with no copy anywhere. So when a host reclaims it, and on cheap providers it will, you do not recover. You rebuild from scratch.

I went through a stack of provider docs, marketplace comparison guides, and Trustpilot reviews to understand what a reclaim actually costs, and the pattern is the same everywhere. On Vast.ai a spot instance can be pulled back with about fifteen seconds of notice and no uptime SLA. On RunPod's cheaper on-demand tier it is around five minutes. The compute you rent to save money is exactly the compute that disappears without warning. That is the reclaim tax, and the answer people reach for, pay a pricier provider, is the wrong one. The fix is not a more expensive box. It is making the box something you can get back.

TL;DR: When a cloud GPU host reclaims your box, the local disk is wiped and you lose the whole environment, not just your latest work. The standard advice, push checkpoints to object storage and auto-restart, recovers your data but not your install, your venv, your custom nodes, or your models, and it stays locked to one provider. To actually recover after a reclaim you need a portable snapshot of the entire box that restores at its original paths on whatever provider you land on next. Then a reclaim becomes a restore instead of a rebuild.

What actually gets wiped when a host reclaims your GPU?

Everything on the box's local disk. On most cloud GPUs that disk is ephemeral, so a reclaim clears your models, your Python environment, your cloned custom nodes, and any uncommitted work in one shot. Data you already pushed to external storage survives. The running environment that made the box useful does not.

The exact mechanics differ by provider, but the outcome converges. On a managed cloud like Google Cloud, a host maintenance event stops the instance, and data on any Local SSD is unrecoverable once that happens. On a marketplace provider the notice window is far shorter and the reason is economic, not maintenance: a higher bidder or the host's own owner wants the card back. And there is a third way to lose the box that has nothing to do with reclaims at all, running your balance dry. RunPod's own support documentation is blunt about it:

"RunPod cannot restore data once a resource has been terminated due to insufficient balance. RunPod does not maintain backups of pod data." Source: RunPod support documentation

So there are three doors out of your environment: a maintenance stop, a marketplace reclaim, and an empty wallet. All three land in the same room. The box is gone and the local disk went with it. What you get to keep is exactly what you had the foresight to copy somewhere else first.

The official answer, and where it stops

The standard, correct advice for a managed cloud VM is well documented. Configure the workload to move in-progress work to a Cloud Storage bucket, write anything you need to survive to a secondary persistent disk, set the instance to restart automatically after the event, and watch the maintenance-event metadata so you get advance notice to checkpoint before the box goes down. Google Cloud spells all of this out. If you run a single managed VM on a provider that gives you a polite heads-up, follow it. It works.

It stops in two places, and both matter for anyone renting cheap compute.

First, it recovers your data, not your environment. A bucket holds your dataset and your checkpoints. It does not hold your install at the commit that actually worked, your virtual environment with its resolved dependencies, your custom nodes pinned to versions that do not throw errors, or your twenty gigabytes of downloaded models. Restore the data onto a fresh box and you still have to rebuild everything around it before the data is usable.

Second, it is provider-locked. A persistent disk lives in one provider's one region. It cannot follow you to a cheaper card on a different provider, which is the entire reason the price-hopping crowd rents the way they do. And on a marketplace box there is no maintenance-event metadata to poll. There is a fifteen-second notice, or none. The official playbook assumes a world of planned maintenance and stable single-cloud tenancy. The bursty single-GPU practitioner does not live in that world.

What people actually do to survive a reclaim

Because no product owned this problem, practitioners built the recovery layer themselves, and you can see the whole spectrum of it in the wild.

The disciplined version is a checkpoint rig: a cron job that pushes state to S3 every fifteen minutes, so a reclaim costs you at most one interval. Done well it genuinely works. Pinterest reported checkpointing every fifteen minutes to save $4.8 million a year, a 72% cost reduction on production workloads. But read that carefully: it works because they wired the checkpoint-and-resume logic themselves, at Pinterest scale, with Pinterest engineers. It recovers the training state, not the box.

The less disciplined version is the reclaim you did not see coming, on a card you rented to save a few dollars an hour. One Vast.ai reviewer put the whole failure mode in a single sentence:

"machine simply went offline after 10 days and $1,000 spent generating data." Source: Vast.ai Trustpilot review

Ten days and a thousand dollars, gone with the box. And even when you survive the reclaim, the rebuild that follows is its own tax. A team at Lunit described the loop precisely: each new spot instance meant over ten minutes of context switching to set it up, and a chain of preemptions could trap the training process in an endless cycle of starts and stops. The reclaim is not one event. It is a treadmill.

Some people pay their way off the treadmill with a network volume, and it half-works. The volume outlives the box, so your files are still there when you spin up again. But it bills you while the GPU is off, and it is pinned to one datacenter, so it cannot follow you to a cheaper provider either. I walked through exactly where that breaks in our breakdown of network-volume alternatives. The common thread across all of these is that the best case is still a faster rebuild, not an actual recovery of the box you were running.

Is recovering your work the same as recovering your box?

No, and the gap is the whole problem. A checkpoint recovers your training state: weights, optimizer, step count. It does not recover the environment that ran it, the install at its exact commit, the venv, the custom nodes, the multi-gigabyte models. Restore a checkpoint onto a bare box and you still rebuild for thirty to sixty minutes before it runs.

That distinction is why "recover your work" and "recover your box" quietly became two different jobs. Everything in the section above recovers your work. The box, the thing that took real time to assemble, gets rebuilt every single time. The reclaim survival stories all share this shape: the data was saved, and the environment was not, so recovery meant starting the setup over on a new machine.

Real recovery is restoring the whole box. Capture the entire filesystem, the install at its commit, the venv, the custom nodes at their pinned versions, and the model files, as one portable snapshot, then restore that snapshot at its original paths on whatever GPU you rent next. That is the approach we are building Aquanode around, and it is what turns a reclaim into a restore. The snapshot is not anchored to a provider or a region, so the card you land on after a reclaim does not have to be the same provider, or even the same one you started on. I covered how the git verbs map onto this, snapshot, restore, push, pull, in our explainer on snapshotting a GPU environment across providers.

One honest boundary, because this is pre-launch and I would rather state it than have you find it. We have validated a full round-trip of a real GPU environment, the install at its commit, the venv, the custom nodes, and a model checkpoint, restored bit-for-bit on a new box. The broader claim, that this holds across many provider pairs at scale, is exactly what we are proving in public right now rather than asserting. This is the standard we are building openly, not a finished product to buy. Take the boundary as stated.

What you should actually do

Pick by how far your box needs to travel after a reclaim.

• You run a single managed VM with planned maintenance. Follow the provider's playbook. Move in-progress work to a bucket, write survival state to a persistent disk, enable auto-restart, and poll the maintenance-event signal for advance notice. Inside that one provider, it genuinely solves the recovery problem. Do not over-build past it.
• You rent marketplace or on-demand cards and hop providers for price. The persistent disk cannot follow you and the reclaim notice is seconds, so the managed playbook does not reach you. You want a portable full-box snapshot so recovery is a restore on the next provider, not a rebuild. Pair it with pausing to zero when you step away, so an idle box is not the next thing that quietly drains your balance. I wrote up the pause side in how to pause a GPU instance without losing work.
• You are hand-rolling a checkpoint rig already. Keep it, it is the right instinct, but know its ceiling: it recovers your run, not your environment. If you find yourself rebuilding the same box after every reclaim, that is the layer to replace. The full playbook for surviving reclaims on spot capacity is in how to resume GPU training after a spot instance interruption.

The reclaim itself is not the failure. Cheap compute comes with reclaims, and no pricier provider makes that go away for the price you actually want to pay. The failure is having no copy of the box anywhere, so that losing it means rebuilding it. Version the box the way you already version everything else expensive in your stack, and a reclaim stops being a disaster and starts being a restart.

About the author

I'm on the team at Aquanode. I have not lived every provider's reclaim personally, so this is written from a stack of provider docs, marketplace guides, and reviews rather than a claim about your exact setup. What I can speak to is the recovery side: we are building and validating in public a snapshot that captures a full GPU box, environment and models included, and restores it on a different provider. The sources below are where the reclaim numbers and quotes come from.

Sources

• Google Cloud, Handle GPU host maintenance events: Local SSD data unrecoverable on maintenance, move work to a bucket, auto-restart, maintenance-event metadata
• PromptQuorum, Cloud GPU Rental Guide 2026: Vast.ai spot reclaim with ~15 seconds notice and no uptime SLA, RunPod on-demand ~5 minutes
• RunPod support documentation: "cannot restore data once a resource has been terminated... Does not maintain backups"
• Vast.ai Trustpilot review: "machine simply went offline after 10 days and $1,000 spent generating data"
• Lunit, Optimizing GPU Costs by Leveraging Spot Instances: over 10 minutes of context switching per new instance, the endless start-stop preemption loop
• Medium/@velinxs, GPU Cloud Pricing Is a Scam: Pinterest checkpoint-every-15-minutes, $4.8M/year, 72% reduction