How to Snapshot and Restore a GPU Environment Across Cloud Providers (Git for GPU Boxes)

Go read almost any r/LocalLLaMA thread about surviving a spot reclaim and you find the same thing: people have already built a worse version of git for their GPU box. A cron job that rsyncs the working directory to S3 every 15 minutes. A 200-line setup script that re-clones custom nodes and re-pulls model weights. A Docker image somebody rebuilds by hand every time a dependency drifts. I keep seeing it across the community forums, and it is the tell. When hundreds of practitioners independently hand-roll the same duct tape, they are filling a product-shaped hole.

The hole is this. You version-control your code, your containers, and your infrastructure, then you rebuild your GPU box from scratch every time a host reclaims it. This post is about how to snapshot and restore a GPU environment across cloud providers instead, so a resume is a restore and not a rebuild. The mental model is the one you already use every day: treat the whole machine the way you treat a repo.

TL;DR: A cloud GPU is stateless by design, so when a host reclaims your box you lose the environment, not just the data. Disk snapshots and network volumes bring state back but stay locked to one provider and one region. To snapshot and restore a GPU environment across cloud providers, you need a portable, full-filesystem snapshot of the whole box (code, venv, pinned nodes, and models) that restores at the same paths on whatever card is cheapest and in stock next.

What people actually do today, and why it is git done badly

The workflow for a bursty single-GPU renter has a predictable shape. Price-shop across 5 to 10 providers before a job. Spin up a raw box wherever is cheapest. Then spend 30 to 60 minutes rebuilding: git pull, re-download 2 to 20 GB of model weights, re-clone custom nodes, resolve dependencies, restart. To survive the reclaim that eventually kills the box with no warning, people bolt on a coping stack of checkpoint crons and sync scripts.

That coping stack is a partial, hand-built version of version control for the machine. It captures a slice of the state, on a schedule, to a place you hope to reach later, and it is fragile in the way improvised version control is fragile. The interruption lands between saves, and the setup rots because a human has to maintain it. One team running spot instances described the loop plainly:

"Each attempt to initiate a new instance often leads to a chain of continuous preemptions, and the time required for context switching, typically over 10 minutes to set up each new instance, causes significant training interruptions. Such a continuous loop can trap the training process in an endless cycle of starts and stops." Source: Lunit Engineering, "Optimizing GPU Costs by Leveraging Spot Instances"

The reason multi-cloud has become the default is the same: renters hop to chase price and stock, and hopping means rebuilding. Industry write-ups now describe manually scanning several providers before each job as a survival strategy rather than an optimization. The rebuild is the tax you pay for cheap, available compute. Git for your box is the idea that you should not have to.

Why "snapshot" means three different things

Search for how to snapshot and restore a GPU environment and you get results that talk past each other, because the word covers three separate things.

GPU memory snapshots capture the live contents of GPU memory and the CUDA context so a model can skip cold-start initialization. This is what Modal's memory snapshot work and NVIDIA's Dynamo snapshot are about. It is real and useful, and it is a different problem. It shaves seconds off inference startup. It does not carry your environment to a new provider.

Disk snapshots and network volumes capture the bytes on a disk. This is the GCP restore-from-snapshot genre, and RunPod-style network volumes. They persist your files past a terminate, which is a genuine improvement. The catch is where they live: a volume is pinned to one provider and one datacenter region.

Full-box environment snapshots capture the whole filesystem of the running machine (the repo at its commit, the virtualenv, custom nodes at their exact pins, and the model files) then restore all of it at the original paths on a different box. This is the layer nobody in the first two categories delivers, and it is the one that lets you move.

Most tooling solves the first or second and quietly implies it solved the third.

What does it actually mean to snapshot and restore a GPU environment?

It means capturing everything a fresh box would make you rebuild and putting it back byte-for-byte on the next machine. Not a data directory and not GPU memory, but the environment: the app at its working commit, the venv, the pinned custom nodes, and the model weights, restored at the same paths so the box boots into your setup.

A checkpoint file is not your box. Your box is the checkpoint plus the 2 hours of setup that make the checkpoint runnable.

The git mental model, applied to a box

If you already think in git, the mapping does most of the teaching:

• snapshot is commit: freeze the current, working state of the whole machine as a restorable point.
• restore is checkout: bring that exact state back on demand, on a box that has never seen it.
• push and pull are how the snapshot leaves the machine so it is not stranded on hardware you no longer control.

The payoff is the same one git gives you for code. You stop fearing the reset. A reclaim, a cheaper card on another provider, a region running out of stock: each becomes a checkout to a new box instead of a from-scratch rebuild. The environment is an artifact you own and move, not a thing you re-derive from the internet every session.

Why can't a network volume just do this?

Because a network volume solves persistence but not portability, and it keeps billing at roughly $0.10/GB per month while the GPU is off. It survives a terminate on the same provider and region, but it cannot follow you to a cheaper card somewhere else, and it charges rent even when nothing is attached to it.

The region lock is not a footnote, it is the design. Provider docs are direct about it: a volume is region-specific, and changing regions means manually transferring the data to a new volume. That manual transfer is re-downloading by another name, and it is exactly the cost you were trying to kill. The region-lock behavior is documented in the RunPod network volume guides that the community leans on, and the harder edge shows up when a balance runs dry:

"RunPod cannot restore data once a resource has been terminated due to insufficient balance. RunPod does not maintain backups of pod data." Source: RunPod support documentation

So a volume is a good same-provider answer and a non-answer the moment you want to move. For an audience that switches providers specifically to chase price and availability, a persistence layer that forbids switching is half a fix.

How to snapshot and restore a GPU environment across providers, in practice

The steps are the same regardless of which tool you reach for. The point is to capture the box, not a slice of it.

1. Get your box into a known-good state once. Install the app, pin your custom nodes to the commits that actually work with your workflow, pull your models, and confirm it runs. This is the state you never want to rebuild again.
2. Snapshot the whole filesystem, not just a data volume. The snapshot has to include the install at its commit, the venv, the pinned nodes, and the model files, at their real paths. If it only grabs /data or a models folder, you will be reassembling the environment around it on restore.
3. Restore onto the cheapest available card, on any provider. This is the step a volume cannot do. The snapshot lands the full environment on a box that has never seen your setup, at the same paths, so nothing re-downloads from Hugging Face or CivitAI.
4. Relaunch and resume. Bring the app back up and pick your work up where it stopped. The compute you pay for goes to the job, not to a card sitting idle waiting on a download.

If you want the deeper walkthrough of keeping model files specifically across sessions, we broke that down in how to stop re-downloading your models every time you spin up a GPU, and the cost math of treating compute as portable and resumable is in how to reduce GPU cost by more than 40 percent.

What breaks, and how to not get burned

A few honest edges, because this audience has been lied to by restore buttons before.

Restore brings the environment back, not a running server. A good restore lands your files, nodes, and models on the new box. You still relaunch the app once before you are generating. Better stated than discovered.

Driver and CUDA compatibility still applies across hardware. Moving from one GPU generation to another can mean a framework rebuild if you were compiled against a specific stack. Restoring the environment removes the setup tax; it does not repeal physics.

Large model libraries take real time to sync the first time. The win is that it happens once, not every session. After the first snapshot, restores move bytes you already own instead of re-fetching them at GPU-hour rates.

And the failure this whole approach exists to prevent is the one people describe over and over:

"Machine simply went offline after 10 days and $1,000 spent generating data." Source: Vast.ai reviewer, Trustpilot

Cheap GPUs are not the problem in that story. Losing the box when it dies is. A portable snapshot is the difference between "start over" and "restore and keep going."

What you should actually do

Pick by how far your environment needs to travel.

• You only ever use one provider and one region. A network volume with symlinks genuinely solves the re-download tax inside that box. Use it and do not over-engineer.
• You hop providers to chase cheap or available cards. A volume cannot follow you and managed hosts lock you in. You want a provider-agnostic, full-filesystem snapshot that restores the whole environment at its original paths on whatever card you rent next.
• You run spot and eat reclaims. Spot is 50 to 80 percent cheaper, which is why people tolerate the reclaims. Stop hand-rolling the checkpoint rig and start treating the box as the unit of state. A reclaim should be a checkout, not a rebuild.

The principle under all three is the one git taught you for code: hard-won state should be a restorable artifact you own, not something you re-derive by hand.

Where this is going

We are building ogre as an open-source standard for exactly this: a single-binary CLI that snapshots the whole GPU box and restores it on a different provider in one command. It is pre-launch and we are proving it in public, not selling it. The one thing validated end to end so far is a single full round-trip, a real pause then resume that brought back the entire environment (the install at its commit, the venv, the pinned nodes, and a model checkpoint) byte-for-byte identical on the new box. Cross-provider reliability at scale is the part we are still measuring in the open before we make bigger claims. "Git for GPU boxes" is the standard we want to get right in the open.

About the author

I am on the team at Aquanode. I am not going to claim I have lived your exact reclaim horror story; what I can speak to is the snapshot side. We built and validated a full-box snapshot that captures a complete GPU environment, models and pinned nodes included, and restores it byte-for-byte on a new box, and we are building it into an open standard in public. The sources below are where the pain quotes and provider behavior come from.

Sources

• Lunit Engineering, "Optimizing GPU Costs by Leveraging Spot Instances": the preemption loop and the "over 10 minutes to set up each new instance" quote
• Modal, "GPU Memory Snapshots": GPU memory snapshotting for inference cold starts, a different problem from environment portability
• NVIDIA Technical Blog, "Dynamo Snapshot": CRIU plus cuda-checkpoint for single-GPU inference startup
• Google Cloud, "Restore from a snapshot": disk snapshot restore, single-provider by design
• Next Diffusion, RunPod network volume guide: the region-lock and manual cross-region transfer behavior of network volumes
• RunPod support documentation: "cannot restore data once a resource has been terminated," no pod backups
• Vast.ai reviewer, Trustpilot: "machine simply went offline after 10 days and $1,000 spent generating data"