GPU snapshots (Beta)

Beta: this feature may change before general availability

A GPU class spends most of its cold start in @nodus.enter(): loading weights onto the GPU, compiling kernels, warming caches. With gpu_snapshot=True, the first worker snapshots its process and GPU memory once its enter hooks finish. A later worker of the same class on a machine that holds the snapshot restores it and resumes right after enter, with the model already in GPU memory.

"""A GPU class whose model loads once, then is snapshotted so later workers restore it instead (Beta).

Run it with `nodus run examples/functions/gpu_snapshot/app.py`. The first worker runs `load` and offers a snapshot
of itself; a later worker on a machine that holds the snapshot resumes right after `load`, with the model already on
the GPU. Every worker answers the same, snapshot or not.
"""

import nodus

app = nodus.App("gpu-snapshot")
image = nodus.Image.debian_slim().pip_install("torch==2.6.0", "transformers==4.53.3")
MODEL = "Qwen/Qwen2.5-0.5B-Instruct"


@app.cls(gpu="A10", image=image, gpu_snapshot=True, scaledown_window="30s", max_cost=1)
class Generator:
    @nodus.enter()
    def load(self) -> None:
        # Runs once per worker that starts cold; a restored worker resumes after it.
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer

        self.tok = AutoTokenizer.from_pretrained(MODEL)
        self.model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float16).to("cuda").eval()

    @nodus.method()
    def generate(self, prompt: str, tokens: int = 24) -> str:
        import torch

        ids = self.tok(prompt, return_tensors="pt").to("cuda")
        with torch.no_grad():
            out = self.model.generate(**ids, max_new_tokens=tokens, do_sample=False)
        return self.tok.decode(out[0][ids["input_ids"].shape[1] :], skip_special_tokens=True)


@app.local_entrypoint()
def main() -> None:
    print("generated:", Generator().generate.remote("The capital of France is"))

$ nodus run examples/functions/gpu_snapshot/app.py
generated:  Paris. It is the largest city in France and ...

gpu_snapshot is accepted only on an @app.cls class that requests a gpu. It sets the Function’s spec.snapshot.gpuMemory; turning it on or off restarts the workers once.

In the console, open the Function’s Overview → GPU snapshots (Beta) section to request or stop requesting snapshots. Saving this preference rolls workers after their active calls finish; the next workers may start cold. It does not enable organization or host support.

The same section shows observed node reports separately from the requested setting. GPUSnapshotTaken means a worker reported taking a snapshot. GPUSnapshotRestoredLocal and GPUSnapshotRestoredFleet distinguish a restore on the host from one fetched from storage. GPUSnapshotDeclined and GPUSnapshotFailed describe an operation that was declined or failed. Missing history leaves runtime support and restore use unknown.

These events also appear on the worker’s Attempt. A benchmark can correlate the FunctionCall’s status.worker.attempt and status.worker.epoch with the Attempt and its events. Repeated reports aggregate into Event series; their counts and timestamps are not per-operation trial counts or restore durations. Stable operation IDs and timings are not yet exposed. No snapshot event proves physical GPU allocation release, provider billing cessation, workload quality, SLO compliance or useful output per physical GPU-hour.

When a worker restores

A worker restores a snapshot only when everything that shaped it is the same:

the same organization and the same Function;
the same code, image digest, Python version, environment and resources;
the same secrets delivered to the worker, at the same versions, so a rotated secret starts a fresh snapshot;
the same GPU model and driver version;
a machine that holds the snapshot: the machine the earlier worker ran on, which a new worker reuses while it is kept warm, a machine of your own pool that an earlier worker of the class used, or, with restores on other machines enabled for your organization, any of the machines Nodus rents for your organization with the same GPU, driver and machine software.

Anything else, and any failure on the way, starts the worker normally. If a restore fails after it began, that worker is replaced by one that starts normally, and the snapshot is not tried again on that machine.

A restore reads the whole snapshot back, so the machine gives it time in proportion to the snapshot’s size, but never longer than the cold start it replaces. A restore that runs out of that time fails, that worker is replaced by one that starts normally, and the snapshot is not tried again on that machine. A snapshot that could not be read back within 10 minutes is not kept (too_large) and the worker serves without one; on most machines every snapshot fits.

Restore on another machine

With it enabled, a worker’s snapshot is also stored, encrypted, in Nodus storage once it is taken, and a later worker of the same class that lands on another of your organization’s machines downloads and restores it instead of running its enter hooks. Nodus downloads a snapshot only when the download and the restore are expected to take well under the cold start it replaces, so it pays off for classes whose enter hooks spend their time compiling kernels and capturing CUDA graphs, such as vLLM engines, and not for classes that mostly read weights from disk, which a download cannot beat. Nodus learns how fast your organization’s machines of each GPU model and region upload, download and restore snapshots, from your own machines only. Until it has measured them it assumes a slow network and a slow restore, so a large snapshot can at first stay on the machine that took it. Newer measurements count more than older ones, and machines of a GPU model and region that stopped fetching after slow downloads try one again each hour, so they go back to restoring within hours of the network recovering.

The same machine software. The other machine must have the same GPU model and driver version, and the same kernel, CPU features and snapshot tools. Machines of one offering usually match; a worker on a machine that does not starts normally.
What it costs. A stored snapshot counts as your organization’s storage and is billed as storage of the Function, at the storage rate. Nothing is stored when it would take your organization’s storage (stored snapshots, checkpoints and Volumes together) past its storage quota, or while your account is restricted or unpaid; nothing is stored either for a snapshot that could never restore faster than the class starts cold. Snapshots on the machine keep working.
When it is deleted. A stored snapshot is deleted after 7 days without a restore, when a newer version of the class replaces it, and when the Function is deleted. When a secret it was taken with is rotated, its key is destroyed at once, so it can no longer be read, and its data is deleted within about 2 hours; a deleted secret’s snapshots go the same way within about 2 hours.
How many are kept. A Function keeps at most 8 stored snapshots and 256 GiB of them; past either, the ones used least recently (stored or restored) are deleted first.
The same weights download once per machine. A weights file kept out of the snapshot, such as a vLLM engine’s, is stored apart from the rest. A machine that already holds the same weights, from a snapshot of your organization it downloaded within the last hour, downloads only the rest and gives the worker a copy of the weights it checked when it first kept them. That helps most after you deploy a new version of a class that loads the same model.
What to expect. On two A10 machines, a vLLM engine with its weights and KV cache kept out of the snapshot made a 5.39 GB snapshot of Qwen3-0.6B. A worker on the other machine downloaded and restored it, onto another GPU, and served 16.3 to 17.0 s after it started, against 91.5 to 95.5 s for a cold start, with the same output. With Qwen3-8B the snapshot was 21.64 GB, and a worker on the other machine served 68 s after it started against 2 min 11 s to 2 min 13 s cold; when that machine already held the weights it downloaded 5.08 GB and served 42 to 44 s after it started. The snapshots were stored in a MinIO test store, not in Nodus storage, so downloads from Nodus storage can take a different time.

Spend less on warm workers

A restored worker skips everything its enter hooks did, so you can keep fewer workers warm: lower min_workers and scaledown_window and let a worker that comes back on a warm machine restore instead of loading.

A restore reads the snapshot back from disk, and the snapshot holds the GPU memory the worker filled, so its time grows with the model unless you keep the model out of it. On an A10, for a class whose enter hook loads a model from its image onto the GPU:

Model	Snapshot	Cold start, enter hook done	Restored
0.5B parameters	2.7 GB	5.7 to 5.9 s	3.1 s
3B parameters	8.2 GB	8.0 to 8.5 s	7.1 s
8B parameters	18.2 GB	11.4 to 11.6 s	14.2 to 14.8 s

With the weights in the snapshot, loading weights that sit on a local disk is about as fast as reading a snapshot of them, so such snapshots pay off when your enter hooks do more than that: download the weights, compile kernels, capture CUDA graphs or warm caches. Each restored worker answered greedy decoding exactly as the cold one did.

Take turns on rented machines

With it enabled, the machine of a worker that leaves stays with your organization for its Function’s scaledown_window, and the next worker of any of your GPU classes that fits starts on it. The machine keeps the snapshot of every class that ran there, so several classes can take turns on one rented GPU, each restoring its own snapshot. With snapshot placement enabled, a new worker prefers the held machine that holds its snapshot, among held machines within 2 % of the cheapest price.

Take turns on your own machines

A machine you enrolled in a pool stays yours after a worker leaves it, and so does its snapshot cache. Name the pool with pool= and several classes can take turns on the same GPU, each restoring its own snapshot instead of loading its model again:

@app.cls(gpu="A10", image=image, pool="model-lab", gpu_snapshot=True, scaledown_window="0s")
class Writer:
    ...

The machine keeps the snapshot of every class of your organization that ran there, until it has gone unused for an hour or the machine’s snapshot space runs out, when the least recently used is deleted first. With snapshot placement enabled, a new worker prefers a free machine of the pool that holds its snapshot. On an A10, four vLLM models taking turns this way each came back in 6.8 s on average, against 83 s for a cold start (report 034).

A snapshot taken on your own machine stays on that machine. It is never stored in Nodus storage or restored on another machine, even with restores on other machines enabled, because you control the machine that took it.

Keep turns fast with the swap budget

Every class that takes turns on a GPU adds a restore to each cycle, so too many classes on one GPU make calls wait. With the swap budget enabled, Nodus checks two numbers each of your machines reports from its recent restores before it places a new worker where other classes take turns on the GPU. The machine is within the budget:

while few classes take turns: the classes there, the new one included, number at most 15 s divided by its 95th percentile restore time (three, with restores of about 5 s);
or, beyond that, while restoring keeps the GPU less busy: it spent at most 40 % of the last 10 minutes restoring.

A machine over the budget is chosen last among the places the worker’s placement profile treats as equally good, so the worker starts on another of your machines or on a new one among them if there is one, and on the busy GPU if there is not. Under the Balanced and Cost profiles those are the places within 2 % of the cheapest price the worker could get, so a worker may pay up to 2 % more than on the busy machine; under Speed they are those within 2 % of the best score, at up to 1.5 times the cheapest expected cost. A new machine first boots, billed as any new machine, and a worker moved off a machine your organization holds for GPU handoff also leaves that machine billed until its window ends. The budget never holds a worker back waiting for a machine within it. Your calls have no latency setting, so the budget aims at 15 s at the 95th percentile, the target it was measured against, but it does not guarantee it: on an A10, six small vLLM models taking turns this way answered within 15 s at the 95th percentile up to 6 requests a minute in all, four up to 8 and two at every rate tried, on a sixth to a half of the GPU time of one GPU per model.

A worker’s Attempt shows what the budget decided in status.placement.explain.swapBudget: WithinModels or WithinShare when its machine was within the budget, Moved when the budget sent it elsewhere, and OverBudget when it joined a busy GPU because no other place was as good. On your own pool, OverBudget means the pool needs another machine for that many classes.

Keep two classes on one GPU of your pool

When every machine of your pool is busy, a worker of one class can run beside a worker of another class of your organization on the same GPU instead of waiting for its turn, when both fit in the GPU’s memory. Each machine measures how much GPU memory a class’s worker used while it had the GPU to itself, and the second worker is placed beside the first only if both together leave 1 GiB of the GPU free. When two classes already share a GPU and a third needs it, the one whose last call ended longest ago and that is not serving a call leaves, as it would when it scales down, and the third takes its place. A worker serving a call is not chosen, and a call it takes meanwhile still finishes.

To use it:

Cap each engine’s GPU memory so that two fit. For vLLM, pass gpu_memory_utilization=0.45 on an A10. An engine at vLLM’s default of 0.9 never fits beside another, so its class keeps taking turns.
Keep a scaledown_window, such as the default 60 s, so a class’s worker stays on the GPU after its call and serves the next one without a restore. Another class that needs the GPU makes the least recently used one leave.

Two workers on one GPU share its compute, so when both are busy a call can take longer than on a GPU of its own. Their GPU memory is not isolated from each other beyond both being your organization’s: a worker that grows past the memory it was measured at can run the other out of memory. Each machine’s routing fee is charged once: the worker that was there first pays it, and the second worker is charged from when the first leaves. In Nodus research on one A10, six small vLLM models at 0.45 each kept p95 within 15 s up to at least 48 requests a minute with two resident, against 24 with one, on the same GPU time, and four models up to at least 64 against 32. That ran its own scheduler on the machine, so your pool’s results depend on how your classes’ calls arrive.

Fill the GPU with a Job between workers (Beta)

A class that scales to zero leaves its GPU idle between its workers. With gap-fill enabled, a one-GPU Job of your organization on the same pool can use that idle GPU. The Job must keep a checkpoint and ask for it: set spec.placement.interruptible to Prefer. Allow, the default for a Job that keeps a checkpoint, does not opt a Job in. When the class needs the GPU again, Nodus pauses the Job’s process with one snapshot of its memory, GPU memory included, kept on that machine. The class’s new worker then restores its own snapshot. Once the class has scaled back to zero, the Job resumes from its snapshot where it stopped, as the same attempt.

In the console, open Batch jobs → New job, or Run again on an existing Job. Choose your private Pool, then use Batch work in serving gaps (Beta) in the resources step. The server checks the Job’s shape and organization support before offering Use serving gaps. Review the resulting manifest and current estimate before launching. This preference does not turn on organization features or reserve a GPU.

The preview requires a digest-pinned image, one GPU, checkpointed recovery and a supported payload. A tag-only image, unavailable estimate or unsupported shape leaves eligibility unknown or unavailable; the manifest editor can supply your verified image digest. Choosing the preference preserves your command, recovery paths and image. Stopping the request still allows ordinary interruption. Your program must save and load its declared checkpoint paths for recovery after a lost host. Existing Jobs keep their immutable placement; use Run again to change it.

Your class gets the GPU back from the Job each time it needs it, with one exception: a call that arrives while the Job is being resumed waits for that resume to finish, then for the pause. If the Job cannot be paused, for example because it holds a network connection, Nodus stops it with its checkpoint first. The Job then resumes from that checkpoint as a plain Job that fills no GPU again, and that stop does not count against its recovery.maxAttempts.

On an A10, a training Job filling a GPU this way kept 91 % of a dedicated GPU’s steps at 1 request a minute, 60 % at 4 and 34 % at 8. Its results were bit-identical to an uninterrupted run. Each pause cost it about 10 s, plus the time your class’s worker stays up after its last call (scaledown_window).

The trade is latency. A call that finds the Job on the GPU waits for the pause and the restore: on an A10 about 7 s at the median and 9 s at the 95th percentile, against about 4 s when classes only take turns. Use gap-fill for classes that already scale to zero and accept a restore.

Nodus stops filling a GPU whose class pauses the Job too often to leave it at least half of each hour. Your machines report each pause and how long it kept the Job off the GPU. A Job is then placed on another of your pool’s machines, or waits, while that GPU serves your class alone.

The Job’s Events show GPUSnapshotPaused and GPUSnapshotResumed for each turn, and GPUSnapshotNotUsed with the reason when it was stopped with its checkpoint instead. A GPU pays one pool fee. While your class’s worker uses a GPU the Job holds, the Job’s attempt pays the fee and the worker’s time is not charged. If the Job ends first, the worker pays the fee from that moment. A GPU that fills with a Job never keeps a second class resident beside it.

Keep the model out of the snapshot

A large model’s GPU memory makes the snapshot large, and copying it out and back can take longer than loading the model again. Call nodus.exclude_from_gpu_snapshot at the end of an enter hook to keep a model’s weights out of the snapshot:

@app.cls(gpu="A10", image=image, gpu_snapshot=True)
class Generator:
    @nodus.enter()
    def load(self) -> None:
        self.model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.float16).to("cuda").eval()
        nodus.exclude_from_gpu_snapshot(self.model)

It takes torch.nn.Module objects, whose parameters and buffers it keeps out, and tensors; only tensors on the GPU count. When the snapshot is taken, the worker writes their memory once to a file on the machine and frees it from the GPU, so the snapshot holds the rest of the worker. The worker that was snapshotted, and every worker restored from it, reads the weights back into the same tensors before it takes a call. Without the call nothing changes, and outside a class with gpu_snapshot=True it does nothing.

Eager code only. The weights come back in the same tensors with the same shapes, but at new GPU addresses. Code that kept a raw GPU address breaks: CUDA graphs, torch.compile(mode="reduce-overhead"), and engines such as vLLM that capture CUDA graphs. Leave the weights in the snapshot for those; a vLLM engine can keep them out its own way.
What it costs. A copy of the weights stays on the machine’s disk, and in its memory cache while there is room, for as long as the snapshot is kept, and is deleted with it. The worker that is snapshotted writes the weights once and reads them back before it serves. When the weights would not fit on the machine’s disk or in its snapshot space, the worker writes nothing and keeps them on the GPU; a snapshot that then holds them is usually too large as well and is declined (no_space), and the worker serves either way. While one worker of a class is being snapshotted on a machine, others that start beside it skip their snapshot (cached).
What to expect. The snapshot holds the CUDA context and the rest of the worker’s memory instead of the weights, so a restore reads the weights from that file instead of moving them through the snapshot. Measure your own class before you rely on it: nodus: gpu snapshot excluded … bytes and nodus: gpu snapshot refilled … in the worker’s log show how much it kept out and how long reading it back took.

On an A10, with a Transformers model loaded from the image, keeping the weights out made a restore at 8B parameters about 2.9× faster than with the weights in the snapshot and about 2.3× faster than a cold start, comparing medians (5.1 s against 14.5 s and 11.8 s). These are three runs of 3, separate from the table above on the same machine, so their times differ a little:

Model	Cold start	Restored, weights in the snapshot	Restored, weights kept out
0.5B parameters	5.5 to 7.1 s	3.1 to 3.3 s	2.5 to 2.8 s
8B parameters	11.2 to 12.4 s	14.2 to 16.3 s	5.0 to 6.2 s

A worker whose weights cannot be read back exits instead of serving without them, and Nodus replaces it. When it was restored, its snapshot is not used again on that machine. If the weights cannot be written, the worker keeps them in the snapshot and logs why, as in nodus: gpu snapshot keeps the weights in (write_failed).

Keep a vLLM engine out of the snapshot

A vLLM engine reserves most of the GPU for its KV cache, so its snapshot holds the KV cache’s whole reservation as well as the weights. Pass the engine to nodus.exclude_from_gpu_snapshot to keep both out:

@app.cls(gpu="A10", image=image, gpu_snapshot=True)
class Server:
    @nodus.enter()
    def load(self) -> None:
        from vllm import LLM

        self.llm = LLM(MODEL, enable_sleep_mode=True, worker_extension_cls="nodus.runtime.VLLMWorkerExtension")
        nodus.exclude_from_gpu_snapshot(self.llm)

On an A10, a vLLM 0.30 engine serving Qwen3-0.6B in a class restored in about 6 s with its weights and KV cache kept out, against about 15 s with them in the snapshot and about 90 s for a cold start, with the same output; its snapshot fell from 17.1 GB to 5.4 GB. That was measured with vllm.LLM; LLMEngine and AsyncLLM take the same path and have not been measured yet.

Create the engine with enable_sleep_mode=True and worker_extension_cls="nodus.runtime.VLLMWorkerExtension". If your engine already has a worker extension, make it a subclass of nodus.runtime.VLLMWorkerExtension. It takes a vllm.LLM, its LLMEngine, or an AsyncLLM (AsyncLLMEngine) created in an async enter hook, one per class, and not together with PyTorch models or tensors. An engine that does not meet these requirements raises nodus.errors.Invalid in the enter hook, with the fix.

When the snapshot is taken, the engine writes its weights once to a file on the machine and goes to sleep, which frees its weights and its KV cache from the GPU, so the snapshot holds the rest of the worker. Going to sleep aborts any request the engine is still running, so let your enter hooks finish their own requests before they return. The worker that was snapshotted, and every worker restored from it, wakes the engine and reads the weights back before it takes a call.

The same GPU addresses. vLLM’s sleep mode keeps the GPU addresses of the weights and the KV cache reserved while it sleeps and maps memory back at the same addresses when it wakes, so the addresses the engine’s CUDA graphs captured point at the weights again once they are read back. The weights are read back byte for byte.
The KV cache comes back empty. Its prefix cache is reset, so the first request with a cached prefix computes it again. Nothing in the KV cache is written to the machine’s disk.
A spawned engine process. vLLM runs the model in an engine process of its own. In a class with gpu_snapshot=True, Nodus sets VLLM_WORKER_MULTIPROC_METHOD=spawn unless you set it yourself, because an engine process that vLLM forks keeps GPU device files open that a snapshot cannot hold. If you set it to fork and vLLM keeps it, taking the snapshot fails and the worker serves as it would without one, whether or not you keep the engine out. A spawned engine process receives its arguments by pickling, so pass the engine settings that can be pickled: a function defined inside your class or a lambda in its arguments makes LLM() raise. Any other library that forks a process after the GPU is in use, such as a PyTorch DataLoader with workers, makes the snapshot fail the same way.
What it costs. As for a model kept out: a copy of the weights stays on the machine with the snapshot, the snapshotted worker writes it once, and every worker reads it back. If the machine cannot keep the file, the engine stays awake with everything in the snapshot, and the worker serves either way.
What to expect. nodus: gpu snapshot excluded … bytes of vLLM weights and nodus: gpu snapshot refilled … in the worker’s log show how much it kept out and how long waking took. Measure your own class before you rely on it.

A worker whose engine cannot read its weights back exits instead of serving without them, and Nodus replaces it, as for a model kept out. If the engine fails to go to sleep, the worker reads back whatever it freed and serves without a snapshot (nodus: gpu snapshot skipped).

What a snapshot holds

A snapshot is the worker’s process memory and GPU memory after its enter hooks, and the files those hooks wrote to the worker’s filesystem; with exclude_from_gpu_snapshot, the weights it kept out are a file beside it. It can hold anything your hooks read, including secret values, so it is readable only by Nodus, on your organization’s machines, and is deleted on a machine when the Function changes, after an hour unused and when the machine is released. With restores on other machines enabled, it is also stored, encrypted with a key of its own, in Nodus storage; it is restored only onto your organization’s own machines, never downloadable, checked against what was stored before it is used, and its key is destroyed when a secret it holds is rotated.

Keep in mind when you write the enter hooks:

Files. A restored worker sees the files your enter hooks wrote outside Volumes, such as compiled kernel caches, as they were when the snapshot was taken. Up to 8 GiB of them is kept; a worker that wrote more is not snapshotted (no_space), so keep downloaded weights in the image or a Volume. Nodus extracts your code again in a restored worker.
Connections. A worker that still holds an open network connection when its enter hooks finish is not snapshotted and keeps serving. Close connections after loading, or open them in a method. Listening sockets are fine.
Memory. While the snapshot is taken, the worker’s GPU memory is also held in its process memory. A worker whose memory floor cannot hold both is not snapshotted.
Randomness. Every restored worker starts from the same memory. Reseed your own random generators in a method if calls must differ between workers; Nodus reseeds its own. A vLLM engine’s sampling state is in its engine process, where your code does not reach, so pass a seed in each request’s sampling parameters if sampled output must differ between restored workers.
Engines. PyTorch and Transformers models loaded in your class restore. vllm serve on one GPU, in its own container, restores with the kernels it compiled and its CUDA graphs, since its engine process and its semaphores are part of the snapshot. A vLLM engine created in your class gets a spawned engine process, as vllm serve does, restores too, and can keep its weights and KV cache out. vLLM spread over several GPUs has not been tested with snapshots yet.

See what happened

Each worker’s log says what happened at its snapshot point: nodus: gpu snapshot taken, restored, or declined with a reason such as not_enabled, cached, connected (the worker kept a connection to Nodus open), memory_limit, memory_unreadable (the machine could not read the memory in use, so the next worker offers again), no_space or too_large (the snapshot could not be read back within 10 minutes). A worker that could have restored another machine’s snapshot and started normally did so for a reason such as too_slow (the download would not have beaten the cold start, or did not finish in time), corrupt (what was downloaded did not match what was stored, and the snapshot is not used again), download_failed, no_space, too_large, incompatible or kept_weights (the machine’s own copy of the model’s weights could not be used, so its next start downloads them); a snapshot not stored for your storage quota or account is quota. A restore that failed for a reason of the machine’s, not the snapshot’s, is launch_failed: that worker is replaced, and the snapshot is tried again.

A worker that keeps its weights in its snapshot, or its vLLM engine, logs why once, before the snapshot is taken: nodus: gpu snapshot keeps the weights in (…) with one of these reasons. The snapshot is then larger and restores more slowly, but the worker serves either way.

not_registered: nothing was passed to nodus.exclude_from_gpu_snapshot, or nothing passed was on the GPU.
node_declined:…: the machine would not keep the weights file, with its reason, such as node_declined:no_space, node_declined:memory_limit, node_declined:cached (another worker of the class is being snapshotted) or node_declined:incompatible.
no_room: the machine’s disk had no room for the file.
write_failed: the file could not be written, or the GPU memory could not be freed after it was.
engine_kept: the vLLM engine cannot be kept out, for example an engine of another vLLM release or one already asleep; the log says which.
plan_failed: the vLLM engine could not report what it would write.
probe_failed: the worker could not ask the machine whether it keeps the file.
disabled: nodus.exclude_from_gpu_snapshot was called while NODUS_GPU_SNAPSHOT, which Nodus sets to 1 in a class with gpu_snapshot=True, had another value, so it did nothing.

Inspect a batch Job’s serving-gap history (Beta)

Open Runs → your Job → Details → Serving-gap observations. For a GPU Job on your private Pool, the view separates the requested interruption preference from reported snapshot pauses and resumes. It reads only that Job’s identity; running another Job with the same name does not inherit the old reports.

A generic snapshot recovery report may concern reclaim or serving gaps and is labelled separately. Missing history leaves actual use unknown. Repeated reports aggregate and do not establish whether the Job is paused now, count benchmark trials, measure pause duration, or prove physical GPU release, billing cessation or more useful output per GPU-hour.