nodus.checkpoint

The checkpoint handshake and gang checkpoints (resources.md §8, distributed-training.md §10.3).

For file snapshots, Nodus owns cadence and storage; programs write their files into nodus.state_dir(). Before a snapshot the node sends checkpoint.request: on_request(callback) saves, then ack() tells the node the files are consistent. Restoring files never restores process memory: a restarted program loads what it saved. The collective DCP helper requests step-scoped storage access and publishes only after every process finishes. For a distributed loop, call dcp.save_if_requested(state, step) on every global worker at every optimizer boundary. It saves when any worker has a pending request, and acknowledges captured requests only after publication. Use force=True for a scheduled save. Never run distributed collectives in an on_request background callback.

ack

ack(seq: str | None = None) -> None

Tell the node the checkpoint files are complete for request seq (the latest one by default).

Prefer the ID captured by pending_request() before saving: acknowledging it leaves a newer request pending.

handle

handle(message: dict[str, Any]) -> None

Process one message from the events socket.

on_request

on_request(callback: Callable[[], None]) -> None

Call callback (then ack()) whenever the node asks for a checkpoint; a no-op outside Nodus.

pending_request

pending_request() -> str | None

Capture the pending request ID before saving; pass that ID to ack after the save completes.

A request arriving during the save remains pending. Distributed loops must coordinate saves across every worker; dcp.save_if_requested performs that coordination at a boundary every worker reaches.

requested

requested() -> bool

True while a checkpoint request is pending; poll it at safe points in a training loop.

subscribe

subscribe() -> None

Listen for checkpoint requests without a callback: poll requested() and call ack() after saving.