nodus.runtime
View MarkdownIn-container helpers that speak nodusd’s sockets (resources.md §8); each is a no-op outside Nodus.
Telemetry goes to /run/nodus/events.sock as JSON lines. Without the socket, inside a Nodus container, events are
printed as nodus.event {json} lines, which nodusd also parses; outside Nodus nothing is emitted.
VLLMWorkerExtension
Section titled “VLLMWorkerExtension”class VLLMWorkerExtensionThe vLLM worker extension that lets exclude_from_gpu_snapshot keep a vLLM engine out of a GPU snapshot (Beta).
Name it as the engine’s worker_extension_cls, or subclass it in your own extension. vLLM mixes it into the
worker in the engine’s process, where its methods write the engine’s weights to the snapshot’s weights file and
read them back; it changes nothing until the worker’s snapshot point calls them.
emit(event: dict[str, Any]) -> NoneSend one telemetry event; ids are unique per process so the node’s per-attempt dedup keeps every event.
exclude_from_gpu_snapshot
Section titled “exclude_from_gpu_snapshot”exclude_from_gpu_snapshot(*objs: Any) -> NoneKeep the GPU memory of these models and tensors, or of one vLLM engine, out of this worker’s GPU snapshot (Beta).
Call it in an @nodus.enter() hook of an @app.cls(gpu=..., gpu_snapshot=True) class once the model is on the
GPU, as in nodus.exclude_from_gpu_snapshot(self.model). A torch.nn.Module contributes its parameters and
buffers, a torch.Tensor its storage; only CUDA tensors count, and memory several of them share is kept once.
At the snapshot point the worker writes that memory to a file on the machine and frees it, so the snapshot holds
the rest of the worker; the worker that was snapshotted and every worker restored from it read it back into the
same tensors before they serve. Tensors keep their identity, shape and dtype but not their device address, so
use it only with eager code: not with CUDA graphs, torch.compile(mode="reduce-overhead") or anything else that
kept a raw device pointer.
A vLLM 0.30.x engine (vllm.LLM, LLMEngine or AsyncLLM) on one GPU keeps its weights and its KV cache out,
and both come back at the same device addresses, which its CUDA graphs captured: create it with
enable_sleep_mode=True and worker_extension_cls="nodus.runtime.VLLMWorkerExtension", as in
vllm.LLM(model, enable_sleep_mode=True, worker_extension_cls="nodus.runtime.VLLMWorkerExtension"), and register
it alone. Its weights go to the file and come back byte for byte; its KV cache comes back empty, with its prefix
cache reset. Its sleep aborts any request still in flight, so finish them in the hook. Outside a worker with GPU
snapshots on it does nothing.
in_nodus
Section titled “in_nodus”in_nodus() -> boolprogress
Section titled “progress”progress(completed: int | float, total: int | float | None = None) -> NoneReport progress; it appears in status.progress and the console, and drives Restartable cursors.
restored
Section titled “restored”restored() -> boolTrue when this attempt started from a restored checkpoint.
self() -> dict[str, Any]{kind, name, project, attempt, epoch, index} of the object this container runs for, from api.sock.
state_dir
Section titled “state_dir”state_dir() -> PathThe checkpointed directory: write model, optimizer and progress files here.