nodus.integrations.transformers

NodusCallback for the Hugging Face Trainer and every TRL trainer built on it (ADR-103).

from nodus.integrations.transformers import NodusCallback

trainer = SFTTrainer(..., callbacks=[NodusCallback()])
trainer.train(resume_from_checkpoint=nodus.integrations.transformers.last_checkpoint())

Point output_dir at nodus.state_dir() so the Trainer’s checkpoint-<step> folders are the recovery state.

NodusCallback

class NodusCallback(phase: str = 'Training') -> None

Metrics, progress, phases and the checkpoint handshake for a transformers.Trainer.

NodusCallback.on_log

on_log(args: Any, state: Any, control: Any, logs: dict[str, Any] | None = None, **kwargs: Any) -> None

NodusCallback.on_save

on_save(args: Any, state: Any, control: Any, **kwargs: Any) -> None

NodusCallback.on_step_end

on_step_end(args: Any, state: Any, control: Any, **kwargs: Any) -> Any

NodusCallback.on_train_begin

on_train_begin(args: Any, state: Any, control: Any, **kwargs: Any) -> None

complete_checkpoints

complete_checkpoints(directory: str | os.PathLike[str] | None = None, *, require_training_state: bool = True, process_index: int | None = None) -> list[str]

Complete Trainer checkpoints, newest first, for selecting a step present on every distributed worker.

Supply the worker’s global process_index to require its own RNG state in a distributed checkpoint. Each worker inspects its own state directory; callers coordinate the common step before resuming.

last_checkpoint

last_checkpoint(directory: str | os.PathLike[str] | None = None, *, require_training_state: bool = True, process_index: int | None = None) -> str | None

The newest complete checkpoint-<step> under directory (default the state dir), for resume_from_checkpoint.

A folder counts only when it holds everything the Trainer reloads to continue that step: its trainer_state.json at that step, the model weights, and the optimizer, scheduler and RNG state, in the layout the Trainer wrote (single process or DDP, FSDP or DeepSpeed). An attempt stopped mid-save resumes from the checkpoint before it. Set require_training_state=False only to intentionally warm-start from a weights-only save; that starts fresh optimizer and RNG state and does not preserve the training trajectory.