GPU Volume Snapshot vs Full Filesystem Snapshot: One Restores Your Data, the Other Restores Your Work

Picture the version of this where losing a GPU box costs you nothing. The host reclaims your instance mid-run, you rent the next cheapest one on a different provider, you type one restore command, and a minute later you are looking at the exact machine you had before. Same venv, same pinned custom nodes, same models on disk, same shell history. The box became a thing you version and move, the way you already version a repo. That is the state everyone renting a GPU actually wants, and almost nobody renting a single GPU today lives in it. I build the infrastructure layer that sits under rented boxes rather than the models on top, and after a week reading r/LocalLLaMA, r/comfyui, and a stack of provider docs, the same confusion kept surfacing: people reach for a snapshot to get there, but a GPU volume snapshot and a full filesystem snapshot are not the same tool, and the gap between them decides whether your environment actually comes back.

TL;DR: A volume snapshot captures the data on one disk, at the block level, scoped to a single provider's region. It brings your models and files back, not your working machine. A full filesystem snapshot captures the whole box, the operating system state, the venv, the installed packages, the pinned commits, and the models at their real paths, so a restore is your environment running again instead of a folder of files you rebuild around. Data versus work is the entire distinction.

What a volume snapshot actually captures

A volume snapshot, sometimes called a disk snapshot, is a copy of the blocks on one storage volume at a point in time. The major clouds are precise about this, and the precision is the whole point. Google Cloud's documentation describes a disk snapshot as a copy of the data on a disk, and draws a hard line between that and a machine image, which instead stores "all the configuration, metadata, permissions, and data from multiple disks". Amazon's EBS snapshots are block-level and incremental: the first is a full copy, each one after that stores only the changed blocks. Kubernetes formalizes the same idea, noting a volume snapshot runs in either filesystem or block mode against a single volume.

Two properties fall straight out of that definition. First, it is one disk, so anything living outside that disk is simply not in the snapshot. Second, it is provider-scoped and region-scoped by default. On RunPod a network volume is pinned to its datacenter and keeps charging you while the GPU is off, which I wrote about in more detail in our breakdown of why a network volume bills you while idle and cannot leave its datacenter. So a volume snapshot is data persistence, for one disk, in one place. It is genuinely useful. It is also not what most people think they are buying.

Your environment lives outside the volume

Here is the part that catches people. Mount a network volume at /workspace, keep your models and outputs there, and they survive a restart. But your working environment is not the files in /workspace. It is the Python virtualenv and its resolved dependency tree, the custom nodes pinned to the exact commits that do not throw red errors, the system packages you apt-installed onto the root filesystem, the driver and CUDA context, the environment variables, and the config. Most of that sits on the operating system filesystem, outside the data volume, on the disk that gets wiped when the box goes away.

So the models come back from the volume, and then the rebuild starts anyway. Thirty to sixty minutes of it, on every cold start, before the box is usable again. One engineering team running ML training on interruptible GPUs described the recurring cost of that reset plainly:

"time required for 'context switching'... causes significant training interruptions. Such a continuous loop can trap the training process in an endless cycle of starts and stops." Lunit engineering team, on running ML training on spot GPUs

That loop is not a data problem. Their data was fine. What they lost, over and over, was the machine that knew how to use it. A volume snapshot cannot end that loop because it was never holding the thing that broke.

The head-to-head, in one table

Put the three common approaches side by side and the boundary gets obvious. This is qualitative on purpose. What each one captures is a design fact, not a benchmark.

A machine image gets you closer than a bare volume, because it does capture the OS and configuration. But it is built as a provider primitive, tied to that cloud's format and registry, so it moves within a provider far more easily than across one. The row that matters for a price-shopping renter is the third column, and specifically the two middle rows: restoring a working environment, and being able to take it somewhere cheaper.

Why doesn't a volume snapshot restore my GPU environment?

Because a volume snapshot is block-level and scoped to a single disk, usually your data disk, while your environment is spread across the operating system filesystem: the venv, the pinned custom-node commits, the system packages, the driver context. Restore the volume and you get your models back, not the machine that ran them.

This is the quiet reason the entire "one-command ComfyUI setup" script genre exists. Those scripts are a workaround for the fact that the persistence layer people attach only knows about one disk, so everything else has to be reconstructed by hand each time. A snapshot that captures one volume can only ever hand you back that one volume. The environment is a tree of dependencies that only works at specific versions, and it is precisely that tree, not the checkpoint files, that costs an hour to rebuild. If you want the full ladder of how each layer fails, I walked through it in our guide to keeping a persistent environment across providers.

Can you move a full filesystem snapshot to another provider?

In principle, yes. A full filesystem snapshot captures the machine's state by path rather than as one provider's block device, so it can restore onto any host you rent, in any region. That portability is the entire reason it beats a region-locked volume. It is also the hard part, which is why we are still proving it in public.

Portability is a property of the design, not of the vendor. Switching providers does not make a volume portable, because a volume on any provider is still one region-locked disk. But when the whole box is the unit you snapshot, the question stops being "which provider do I commit to" and becomes "which box is cheapest and in stock right now." Practitioners already treat multi-cloud as a survival strategy, scanning five to ten providers before a job. Portable full-box state is what makes acting on that scan cheap, because moving no longer means rebuilding. The runtime primitives for capturing complete machine state already exist in open source, for example CRIU, which checkpoints a process's full memory, file descriptors, and open connections. The open question is doing the same for an entire GPU box and restoring it on a different provider, reliably, at scale.

What I'd actually do

Use a volume for the models you are actively touching inside a session, so you are not re-downloading twenty gigabytes mid-project. That is what a volume snapshot is good at, and you should use it for exactly that. But do not treat the volume as your source of truth for the environment, because it is not one and was never designed to be. Treat the whole box as the thing you snapshot and restore. When a host reclaims your machine with no warning, and on cheap spot capacity it will, the recovery should be a restore command, not another hour of setup.

This is the gap we are building ogre to close, in the open. It is an open-source tool we describe as "git for GPU boxes": snapshot a full GPU box and restore the whole environment, not just a data volume, on whatever provider you rent next. It is early, and we are proving the cross-provider restore in public rather than asking you to take the claim on faith, because "git for GPU boxes" only means anything if you can watch it move a real environment between providers. Treat it as one option in a young space, not a finished pitch. The idea underneath it is simple and provider-neutral: the most expensive artifact in your stack, the box itself, should be as portable as your code.

About the author

I am Ansh Saxena. I work on the infrastructure layer under rented GPU boxes, the plumbing beneath the models rather than the models themselves, and I spend most of my time on the unglamorous problem of making machine state portable across providers. I do not rebuild ComfyUI environments for a living, but I have read enough teardown threads to take the difference between "your data came back" and "your box came back" personally.

Sources

• Machine images. Google Cloud Compute Engine. A machine image stores all configuration, metadata, permissions, and data from multiple disks, unlike a single-disk snapshot.
• About disk snapshots. Google Cloud Compute Engine. A disk snapshot is a copy of the data on a disk, not a bootable machine template.
• How Amazon EBS snapshots work. AWS documentation. Snapshots are block-level and incremental after the first full copy.
• Volume Snapshots. Kubernetes documentation. A volume snapshot runs in either filesystem or block mode against a single volume.
• Network volumes. RunPod documentation. A network volume is pinned to its datacenter and keeps billing while the GPU is off.
• CRIU. Checkpoint/Restore in Userspace. Open-source project that captures a process's complete runtime state: memory, file descriptors, and open connections.
• Lunit engineering team, "Optimizing GPU Costs by Leveraging Spot Instances". Medium. The context-switching cost of setting up each new instance traps training in a start-stop loop.