Skip to content

nodus.runtime

View Markdown

In-container helpers that speak nodusd’s sockets (resources.md §8); each is a no-op outside Nodus.

Telemetry goes to /run/nodus/events.sock as JSON lines. Without the socket, inside a Nodus container, events are printed as nodus.event {json} lines, which nodusd also parses; outside Nodus nothing is emitted.

class VLLMWorkerExtension

The vLLM worker extension that lets exclude_from_gpu_snapshot keep a vLLM engine out of a GPU snapshot (Beta).

Name it as the engine’s worker_extension_cls, or subclass it in your own extension. vLLM mixes it into the worker in the engine’s process, where its methods write the engine’s weights to the snapshot’s weights file and read them back; it changes nothing until the worker’s snapshot point calls them.

emit(event: dict[str, Any]) -> None

Send one telemetry event; ids are unique per process so the node’s per-attempt dedup keeps every event.

exclude_from_gpu_snapshot(*objs: Any) -> None

Keep the GPU memory of these models and tensors, or of one vLLM engine, out of this worker’s GPU snapshot (Beta).

Call it in an @nodus.enter() hook of an @app.cls(gpu=..., gpu_snapshot=True) class once the model is on the GPU, as in nodus.exclude_from_gpu_snapshot(self.model). A torch.nn.Module contributes its parameters and buffers, a torch.Tensor its storage; only CUDA tensors count, and memory several of them share is kept once. At the snapshot point the worker writes that memory to a file on the machine and frees it, so the snapshot holds the rest of the worker; the worker that was snapshotted and every worker restored from it read it back into the same tensors before they serve. Tensors keep their identity, shape and dtype but not their device address, so use it only with eager code: not with CUDA graphs, torch.compile(mode="reduce-overhead") or anything else that kept a raw device pointer.

A vLLM 0.30.x engine (vllm.LLM, LLMEngine or AsyncLLM) on one GPU keeps its weights and its KV cache out, and both come back at the same device addresses, which its CUDA graphs captured: create it with enable_sleep_mode=True and worker_extension_cls="nodus.runtime.VLLMWorkerExtension", as in vllm.LLM(model, enable_sleep_mode=True, worker_extension_cls="nodus.runtime.VLLMWorkerExtension"), and register it alone. Its weights go to the file and come back byte for byte; its KV cache comes back empty, with its prefix cache reset. Its sleep aborts any request still in flight, so finish them in the hook. Outside a worker with GPU snapshots on it does nothing.

in_nodus() -> bool
progress(completed: int | float, total: int | float | None = None) -> None

Report progress; it appears in status.progress and the console, and drives Restartable cursors.

restored() -> bool

True when this attempt started from a restored checkpoint.

self() -> dict[str, Any]

{kind, name, project, attempt, epoch, index} of the object this container runs for, from api.sock.

state_dir() -> Path

The checkpointed directory: write model, optimizer and progress files here.