Skip to content

Spot capacity and direct SSH

View Markdown

Choose Instances → Launch instance → Capacity → Spot (interruptible) for interruptible compute. Existing managed offerings retain their terminal, VS Code and Jupyter access; dedicated VM and GPU-container quotes show Direct SSH, with the environment shown before launch. Quotes do not guarantee launch capacity. Direct-SSH instances are organization-wide and appear in each project’s Instances list.

For a temporary disk, Location → Lowest eligible price is the console default. Nodus checks matching locations at launch; the summary shows the lowest current eligible quote. Select a specific location to keep placement there. Your GPU count, hourly ceiling and transfer consent still apply. Retained Workspaces keep their separate placement settings; selecting another region does not copy existing files.

The direct-SSH API gives your scripts a Nodus SSH command for the dedicated node. Nodes can be preempted at any time. Your own scripts remain responsible for saving progress and restarting work during the pilot. The API exposes execution_environment (vm or container). A GPU container does not provide host administration or nested VMs. Retained storage is available only when supports_workspace_storage is true; always check that field before planning checkpoint retention.

The controlled staging GPU-container pilot allocates a fixed 512 GB temporary disk per node. It does not support retained Workspace storage, and files on that disk do not survive instance release. Save environments, caches and checkpoints externally when they must survive replacement. This staging pilot is not general production availability.

Terminal window
pip install --upgrade nodus-compute
nodus login
nodus create sshkey laptop --from-file ~/.ssh/id_ed25519.pub

You can also save public keys in the console’s Account → SSH keys. Keep the private key on your machine. Spot uses the same organization, API keys and billing account as your other Nodus work.

A quoted price is not a reservation. A location can be unavailable when no admitted price ceiling fits your bid, even if its displayed market price is lower. The request’s maximum falls back to your account default; without either, a finite platform limit still applies. If capacity checks fail, launch is refused safely.

A GPU route that fails to become ready is temporarily excluded for 15 minutes, including when startup reaches its 15-minute deadline. Its price remains visible with unavailable status. Other eligible routes can be selected within your maximum and storage constraints; no on-demand fallback is used. Releasing a node yourself or reaching your price limit does not trigger this cooldown. Failed startup incurs no compute charge; retained storage is billed separately.

Only routes enabled for public launch can receive new requests. A listed price alone does not mean a GPU and location is available to launch. Existing instances keep their access, billing and release behavior when new launches for their route are disabled.

Terminal window
nodus spot prices
nodus spot up h200x8 --max-price YOUR_NODE_PRICE_CAP --wait

For training, create a retained Workspace below and pass --workspace WORKSPACE_ID. A launch without it has only a temporary boot disk; releasing or preempting it does not retain training files.

Prices and ceilings are USD per complete node-hour. The quote table shows available GPU configurations and named locations; a quote does not reserve capacity. The command prints an idempotency key, waits for capacity and SSH readiness, then prints an SSH command. Run that command from the machine with your private key.

Use --ssh-key PATH.pub to override saved keys, --regions eu-1 to restrict placement to Finland, and -o json for machine-readable output. Use the stable location ID shown by nodus spot prices, or a region class such as eu. Locations keep distinct storage: Finland (eu-1), France (eu-2), Iceland (eu-3) and United States 1 (us-1) are separate choices when quoted. Without --wait, the command returns the accepted node’s current state immediately.

Terminal window
nodus spot ls
nodus spot rm NODE_ID

Release ends compute billing immediately and starts deletion of the instance and its temporary disk. A preempted node is cleaned up automatically. Save needed files outside the boot disk before releasing it. A terminal node status reports the end of compute; resource cleanup can continue in the background.

Choose transfer billing explicitly (pilot)

Section titled “Choose transfer billing explicitly (pilot)”

Some container quotes offer or require a separate data transfer contract. Run nodus spot prices -o json and read transfer_billing_disclosure, transfer_markup_percent and transfer_billing_required before accepting it. The console displays that disclosure with an unchecked consent checkbox. Changing launch choices clears consent.

For a container that becomes ready, this contract charges reported transfer costs over its entire lifetime, including startup and shutdown traffic, plus the transfer service fee frozen at launch (7.5% for this contract). Charges are provisional and may be debited or credited later, including after deletion. Never-ready startups are not charged. When a startup ends without becoming ready, its unused compute and transfer reserves return to your available balance. Resource cleanup can continue in the background; this does not authorize another launch automatically. The hourly compute ceiling does not cap transfer charges; monitored spending controls are not a guaranteed total cap.

Accept only within your approved transfer spending scope:

Terminal window
nodus spot up h100x1 --max-price YOUR_NODE_PRICE_CAP --transfer-billing actual_cost_plus_markup_v1 --wait

Python takes transfer_billing="actual_cost_plus_markup_v1" in client.spot.request(). MCP takes the same transfer_billing value in spot_request, separately from confirmed. An available quote does not grant consent. Omitting the option keeps the legacy contract; a quote requiring it cannot launch without acceptance. Waiting, capacity retries and idempotent replays preserve the exact selection. CLI resume uses the saved contract.

Node details show transfer_cost_total as the net lifetime transfer charge, separately from compute-only cost_so_far. transfer_cost_provisional indicates that later reports can revise it. The accepted contract, service fee and disclosure remain on the node. Usage date filters apply to compute intervals; transfer totals remain lifetime totals.

Terminal window
nodus spot ssh NODE_ID -- nvidia-smi
nodus spot ssh NODE_ID --identity ~/.ssh/id_ed25519
nodus spot ssh NODE_ID --config

The command carries SSH through the authenticated Nodus endpoint. Your private key stays on your computer. Use --identity for a nondefault key that matches one installed at launch. Review the SSH host key on first connection. --config prints an entry you can add to ~/.ssh/config for plain SSH, SCP, rsync or VS Code. Keep your Nodus CLI session or NODUS_API_KEY available when using that entry. The key needs both spot:read and spot:write; project-restricted keys cannot access these organization-wide nodes.

Upgrade the CLI and Python SDK before using this connection contract. The deprecated ip field is empty; read ssh_command after the node becomes running. Scripts that connect directly to node["ip"] must use nodus spot ssh NODE_ID instead. SSH port forwarding works through the same connection. Releasing the node closes existing relay sessions after the next authorization check.

Connect your agent using the MCP setup guide. Use an organization-wide key with spot:read and spot:write, and save your public SSH key first. The agent can use the same hosted connection or nodus mcp as other Nodus work.

  1. Call spot_prices and choose a GPU configuration within your approved budget. Prices are for the entire node.
  2. Call spot_request with gpu, max_price_per_node_hr, a unique idempotency_key, and confirmed: true once the request is within your approved spending scope. Omit regions for the cheapest eligible location. Pass workspace_id to retain files; without it, the node has temporary storage.
  3. Save the returned node ID. Poll spot_get until it is running with ssh_command, then run that command on the machine holding your private SSH key. MCP returns connection instructions; it does not upload private keys.
  4. Use spot_usage to inspect charges and spot_release when finished. Poll spot_get for cleaned_at to confirm compute cleanup. Retained Workspace storage is deleted separately and keeps billing while retained.

A no-capacity error includes a retry delay. An uncertain request must be inspected with spot_list before requesting again; retain its original key. These tools do not run a background retry or training recovery loop.

A Workspace in Spot mode owns regional storage mounted at /workspace on linked nodes. Creating it starts storage allocation and billing. It does not start a machine. This differs from a managed development Workspace, which runs a session and saves its home directory.

In the console, open Workspaces → New retained storage. Choose the project, location, capacity and access, review the storage rate and reserved credit, then create it. Capacity choices include 512 GiB, 1 TiB, 2 TiB and 4 TiB, plus Custom for any whole number from 512 through 4096 GiB. For example, 1222 GiB is valid and the estimate uses exactly 1222 GiB. One instance uses a Block slot; Shared by instances uses SharedFilesystem. Each retained slot is billed for its allocated capacity. Open the Workspace to see retained regional slots and charges, and choose Launch linked instance. Only compatible regional Workspaces appear in a direct-SSH launch. Managed home storage cannot attach to these VMs. Capacity and access mode are fixed at creation; storage does not grow automatically. Compute release keeps linked storage and its charges; deleting the Workspace requires a separate confirmation. The launch form lets you reuse a slot and choose whether to wait for home-region capacity. Choose Temporary disk for disposable files, or Keep files in a workspace to attach retained storage.

apiVersion: nodus.dev/v1
kind: Workspace
metadata:
name: spot-training
spec:
mode: Spot
regionalStorage:
provider: nodus
homeRegion: eu-1
sizeGiB: 512
mode: Block

Apply the manifest, read its metadata.uid with nodus get workspace spot-training -o json, then launch:

Terminal window
nodus spot up h200x8 --workspace WORKSPACE_ID --placement local_only --max-price YOUR_NODE_PRICE_CAP --wait --timeout 30m

Save files under /workspace. Releasing the node retains those files and their storage charge. A replacement in the same region and slot mounts the retained filesystem before becoming ready. A block slot permits one writer; 409 volume_in_use means its previous attachment has not cleared. For independent block disks use --workspace-slot worker-2; each slot allocates and bills its own capacity. SharedFilesystem supports simultaneous writers to the default slot in a region when enabled for the pilot.

The default local_first placement waits up to five minutes for home-region capacity before considering other regions. Change the bound with --local-wait 5m (zero to 15 minutes), choose local_only to keep files local, or choose cheapest to select by price immediately. Placement always respects your compute price ceiling. A different region receives separate empty storage; files are not copied automatically.

Python accepts the same options: client.spot.request("h200x1", workspace_id="wsp_...", workspace_slot="default", placement="local_only", local_wait_s=300, max_price=3).

Storage bills allocated GiB over retained time, separately from GPU usage, including while no node is running. The ordinary object storage free allowance does not apply. Storage quota and funded credit must cover each allocation; For 429 storage_quota, an Owner or Admin can raise the storage limit in Settings › Quotas up to the account’s approved ceiling, release storage that is no longer needed, or request a higher ceiling. Five separate 512 GiB workspaces require at least 2,560 GiB of quota. Stopping compute does not delete retained files or free their storage allocation. Settings › Compute separately shows active Spot nodes, simultaneous starts and their editable account limits. Nodes awaiting cleanup remain counted; raising a limit does not guarantee available capacity.

To remove the data, release linked nodes and run nodus delete workspace spot-training --confirm. The API requires DELETE .../workspaces/spot-training?confirm=true; Python supports client.api.delete(..., confirm=True). Deletion waits for attachments and confirmed storage removal. This is permanent deletion of the Workspace’s regional files, including any model checkpoints saved there. In the console, deletion requires typing the Workspace name and confirming that its files will be lost.

Retained files do not restart a training process. During this pilot your command must write valid checkpoints, load them on restart, and request replacements. Managed checkpoint recovery is a separate next stage.

Retry initial capacity acquisition from the CLI

Section titled “Retry initial capacity acquisition from the CLI”

Retry is opt-in and bounded. This command permits at most three accepted startup attempts within one 30-minute deadline, including the first attempt. It uses saved SSH keys and temporary boot storage:

Terminal window
nodus spot up h100x8 --max-price 12 --wait --retry-startup --max-attempts 3 --timeout 30m --state ./h100-acquisition.json

Only a confirmed capacity_unavailable failure before SSH readiness can be retried, and only after cleaned_at confirms compute cleanup. Each attempt has a new node ID and request key; previous nodes remain in history with zero compute charge. Quotes, limits and funding are checked again. Supply is not guaranteed. The command honors the location, GPU and price constraints; it never falls back to on-demand or reserved compute. Explicit --ssh-key public keys remain unchanged. If omitted, each attempt uses the account’s currently saved keys.

Keep the state file and its .lock companion. They contain public request and node data, not API credentials, and are private to your user. After an interruption, use the printed resume command with the same API/account:

Terminal window
nodus spot up h100x8 --wait --retry-startup --state ./h100-acquisition.json --resume

After a node is accepted, CLI waiting retries temporary connection failures and HTTP 429, 502, 503 or 504 by reading the same node. It honors Retry-After within the original timeout. These read retries do not request another node or release the accepted node. Authentication errors and other HTTP failures stop the wait.

Resume retains the original deadline and attempt limit. Time spent with your computer asleep counts toward that deadline. A lost POST response without a saved node ID stops for manual inspection with nodus spot ls; the CLI cannot safely resend that request. Timeout or cancellation does not release an accepted node. Inspect it and use nodus spot rm NODE_ID when appropriate.

With --workspace, retries stay in the first accepted attachment’s location and keep the same slot; files are never moved to another location. Use --placement local_only to keep initial placement at the Workspace’s home. There is no retry after a node has ever become ready, and no automatic restart of training or recovery from running-node preemption. The console retains its single-accepted-node behavior. Python waiting uses the separate timeout-bounded behavior below.

Set SPOT_WORKSPACE_ID to the retained Workspace UID, SPOT_MAX_NODE_PRICE to your approved price ceiling, and SPOT_REQUEST_KEY to a unique identifier for one logical request. Keep the key for inspection and recovery. After an ambiguous initial POST, inspect node history before resending. This example polls every 30 seconds and releases compute after inspecting its GPUs and mount.

examples/spot/poll.py
"""Request one node with retained storage, inspect its GPUs, then release compute."""
import os
import subprocess
import nodus
client = nodus.Client()
node = client.spot.request(
"h200x8",
max_price=float(os.environ["SPOT_MAX_NODE_PRICE"]),
workspace_id=os.environ["SPOT_WORKSPACE_ID"],
placement="local_only",
poll=30,
timeout=1800,
idempotency_key=os.environ["SPOT_REQUEST_KEY"],
)
try:
subprocess.run(
[
"nodus",
"spot",
"ssh",
node["id"],
"--",
"nvidia-smi && nvidia-smi topo -m && df -h /workspace",
],
check=True,
)
finally:
client.spot.release(node["id"])

client.spot.request() waits by default, with a 15-minute timeout. Set wait=False for one immediate POST, or set timeout=1800 for a longer wait. Waiting polls the accepted node, with the same temporary read-error retries and Retry-After handling as the CLI. A typed capacity_unavailable startup diagnosis on a start_failed or capacity_unavailable node permits another attempt only after cleaned_at is present. The replacement uses a fresh request key, preserves the original timeout and price ceiling, and keeps retained storage in the first accepted location. A key you supply identifies the first attempt; save node IDs for later inspection. Other ended nodes, including preemptions after readiness, raise nodus.errors.SpotNodeEnded; a timeout raises nodus.errors.SpotTimeout with node and idempotency_key for recovery. Timeout or cancellation does not release the node. Inspect or release it before starting a different request. Async callers use await client.spot.request.aio(...).

Other methods are client.spot.prices(), list(), get(node_id), release(node_id) and usage(from_="2026-10-24T00:00:00Z", to="2026-10-25T00:00:00Z"). Results are ordinary JSON dictionaries and lists. Optional schemas are available in nodus.api.models.spot.

In Settings → Compute → Preemption webhooks, add your public HTTPS URL and save the signing secret shown once. The form selects spot.preempted; existing endpoints subscribed to that type, spot.* or all events also receive it. Owners and admins can edit, disable or rotate these endpoints using the same webhook controls.

The signed notification arrives after Nodus observes preemption. Its data.object contains the node’s id, gpu, status: "preempted" and ended_at billing cutoff. It carries no advance eviction warning and does not restart work. Verify the signature and deduplicate webhook-id; failed deliveries retry for up to three days, and each endpoint exposes delivery history and replay controls.

Send your existing bearer token to the platform API. The Spot OpenAPI document describes the REST schemas and the SSH WebSocket handshake. SSH returns HTTP 101 and carries binary SSH traffic after the upgrade; use nodus spot ssh rather than a generated JSON HTTP client for this connection.

Method Path Result
GET /v1/spot/prices Current node-hour quotes
POST /v1/spot/nodes Accepted node or a capacity/payment refusal
GET /v1/spot/nodes Your nodes
GET /v1/spot/nodes/{id} Status, Nodus SSH command and accrued cost
GET /v1/spot/nodes/{id}/ssh Authenticated binary WebSocket carrying SSH (spot:write)
DELETE /v1/spot/nodes/{id} Release and cleanup status
GET /v1/spot/usage?from=...&to=... Per-node price intervals and charges

Include Idempotency-Key or idempotency_key in every POST. While the key is retained, a retry returns the same node. Preemption and failed-start cleanup can release the key, so it does not permanently prevent a replacement. After a running node is preempted and cleaned up, use a new key for a replacement. A diagnosed pre-start capacity failure retains its key until physical cleanup is confirmed. If a launch response is lost or unreadable, the console locks the launch form and offers Check instances. Inspect node history before making another request; a terminal state alone is not proof of cleanup. A POST without a key reuses an existing provisioning node of the same GPU type in your account.

503 no_capacity means no node was created. Read its message before retrying: a quote can fit your maximum while its available price limit does not. In that case the message gives a minimum compatible maximum in USD per node-hour, separate from the current quote. Choose another location, wait, or raise your maximum only within your approved budget. Raising the maximum does not guarantee capacity or change the current rate. For a capacity retry, wait retry_after seconds and reuse the same key. 503 price_above_max requires a cheaper quote within your approved ceiling. 402 payment_required needs action in billing, while 429 node_limit requires releasing a node or changing your account limit.

An accepted node can still fail before SSH is ready. When capacity failure is confirmed, its JSON includes startup_failure_code: "capacity_unavailable" and a plain-language startup_failure_message; the console and CLI show that explanation. The node remains failed with no compute charge. A retryable failure can have end_reason: "start_failed" or "capacity_unavailable"; only the typed diagnosis plus cleaned_at permits a replacement. The cleanup of "capacity_unavailable" also exposes retry_after. Missing diagnostic fields mean the specific cause is unknown. By default the CLI does not replace an accepted failed node. The opt-in initial-acquisition mode waits for confirmed cleanup before a capacity retry; an available quote does not guarantee that the next startup will succeed.

Compute billing begins when SSH is ready. Price changes appear as separate intervals in usage, and charges appear under Spot GPU in the existing billing history. The instance detail page shows accrued cost; posted billing usage updates on the metering cycle. Provisioning failures are unbilled. A billing warning includes a release time so you can save work and restore funding before the node is released.

Organizations approved for invoice terms see their credit limit, outstanding invoices, payment links and due dates in Usage & billing. Invoice credits become available when the approved invoice is issued; paying it settles that advance without adding the credits a second time. ACH processing can take several days. Overdue invoices or returned payments pause new launches and can trigger a release warning for running nodes.