# Sandboxes

> Create an isolated container, run commands in it with streaming output, read and write files, and stop it to stop paying for compute.

Source: https://www.nodus-compute.ai/docs/guides/sandboxes/
Build revision: 8ce15960614b1b611b14233ed51ec773ebaf8c7d

A Sandbox is an isolated, long-running container for agents and untrusted code. You drive it with commands and file requests. Choose a persistent directory with `continuity.stateDir` to keep files across stop/start (Beta). Only that directory is saved; the rest of the image filesystem and process memory are not captured. CPU Sandboxes without an explicit directory use the legacy `/nodus/state` checkpoint before shutdown. GPU Sandboxes default to ephemeral storage.

Sandboxes run on CPU machines or dedicated on-demand GPU machines, each in its own isolated runtime with no network access unless you open it. [Sandbox isolation](https://www.nodus-compute.ai/docs/concepts/sandboxes-isolation/) explains what keeps one Sandbox from another.

## Quick start

This manifest is the whole Sandbox. It asks for 1 vCPU and 2 GiB of memory, stops after 5 idle minutes and caps its spending at 1 USD:

sandbox.yaml

```yaml
apiVersion: nodus.dev/v1
kind: Sandbox
metadata:
  name: hello
spec:
  image: nodus/agent-tools # required: Python 3.12, Node 22 and git
  resources:
    cpu: "1"
    memory: 2Gi
  continuity:
    mode: Snapshotted
    stateDir: /workspace # Beta: save this folder after shutdown and restore it on the next start
  network:
    egress:
      policy: Deny # the default: no outbound traffic
  lifecycle:
    idleTimeout: 5m # stop after 5 minutes with no exec, file request or running process
    onIdle: Stop # keep /workspace and stop paying for compute; Delete removes the Sandbox instead
    maxLifetime: 2h # delete it 2 hours after creation whatever it is doing
  maxCostUSD: "1.00"
```

Create it, run a command, copy a file in and read it back:

Terminal window

```console
$ nodus apply -f sandbox.yaml
sandbox/hello created
$ nodus exec sb/hello -- echo hello from a sandbox
hello from a sandbox
$ nodus cp ./notes.txt sb/hello:/nodus/state/notes.txt
$ nodus exec sb/hello -- cat /nodus/state/notes.txt
written from my laptop
$ nodus stop sb/hello
$ nodus exec sb/hello -- cat /nodus/state/notes.txt    # starts it again; verify the saved state file
written from my laptop
$ nodus delete sb/hello
```

From Python, the same steps are in the [Python Sandboxes guide](https://www.nodus-compute.ai/docs/guides/python/sandboxes/).

## Create a Sandbox

`nodus apply -f sandbox.yaml`, `nodus create sandbox <name>` and the SDK all create the same object. The API requires an image after resolving any startup profiles. The CLI, SDK and console request the API’s `Interactive` defaults, which use `nodus/agent-tools` (Python 3.12, Node 22 and git) when neither a profile nor the request supplies an image.

|Field|Default|Accepted values|
|-|-|-|
|`spec.image`|— (required)|Any image reference, or a Nodus catalog image such as `nodus/agent-tools`|
|`spec.resources.cpu`|`1`|0.25 to 64 vCPU|
|`spec.resources.memory`|`1Gi`|128 MiB to 256 GiB|
|`spec.resources.disk`|`10Gi`|1 GiB to 200 GiB|
|`spec.resources.gpu`|none|Supported accelerators on a dedicated, on-demand VM; never Spot/preemptible|
|`spec.placement.regions`|any|Optional Nodus region classes such as `us`; a nonempty list is enforced for quote and placement|
|`spec.fuseMount`|`false`|`true` only with a GPU request; enables guest FUSE, but does not provision JuiceFS|
|`spec.virtualization.profile`|Unset|`gpu-child-v1` requests a qualified single-GPU parent for child VMs (Alpha); availability is checked by the API|
|`spec.network.egress.policy`|`Deny`|`Deny` (no outbound traffic), `AllowList` (named HTTP/HTTPS hosts), or `Open` (public internet)|
|`spec.lifecycle.idleTimeout`|`5m`|`0s` (never idle) or 1 minute to 24 hours|
|`spec.lifecycle.onIdle`|`Stop`|`Stop` or `Delete`|
|`spec.lifecycle.maxLifetime`|`24h`|1 minute to 720 hours|
|`spec.continuity.mode`|`Snapshotted` for CPU; `Ephemeral` for GPU|GPU `Snapshotted` requires an explicit `stateDir` (Beta)|
|`spec.continuity.stateDir`|Unset|CPU or GPU Snapshotted (Beta): an immutable local directory, such as `/workspace`|
|`spec.workingDir`|`/workspace`|An absolute path|
|`spec.maxCostUSD`|none|A USD amount such as `"5.00"`; the Sandbox stops when it has cost this much|
|`spec.state`|`Running`|`Running`, `Stopped`, or `Paused` (CPU memory pause, Beta)|

`nodus get sb/hello` shows `PHASE` (`Pending`, `Starting`, `Running`, `Pausing`, `Paused`, `Resuming`, `Stopping`, `Stopped`, `Recovering`, `Terminating`, `Failed`), `ACTIVITY` (`Busy` or `Idle`), the `CPU`, `MEMORY` and `GPU` you asked for, and `COST` so far.

Before creating a Sandbox through the REST API, submit the same request with `?dryRun=All` to inspect `status.estimate`. It includes the hourly price, initial funding hold, minimum charge and selected offering when capacity is available. Send the returned `ETag` as `If-Match` when creating that unchanged request. A quote does not reserve a machine; check the Sandbox’s readiness before sending work.

In the console, the price summary shows the hourly rate. Expand **Pricing details** to see the initial credit reservation and minimum charge. If something prevents the Sandbox from starting, the summary explains what to change.

### Startup profiles (Alpha)

An organization owner or admin can create a `StartupProfile` containing reusable Sandbox defaults, then select it for a launch or bind it as the organization or project default. Profiles contain settings and Secret names, not credential values. In **New sandbox → Advanced**, choose a profile or keep your organization and project defaults. Your explicit settings take priority. Expand **Applied startup profiles** in the summary for revisions, or **Full configuration** for the resolved settings. Existing Sandboxes keep their resolved configuration after profile edits.

```yaml
apiVersion: nodus.dev/v1
kind: StartupProfile
metadata:
  name: research
spec:
  defaults:
    image: nodus/agent-tools
    resources:
      cpu: "2"
      memory: 4Gi
    lifecycle:
      idleTimeout: 5m
```

Apply that profile, then review a launch without allocating compute:

Terminal window

```sh
nodus apply -f profile.yaml
nodus create sandbox research-1 --startup-profile research --dry-run=server -o yaml
```

The resolved request and its estimate must be reviewed before launch. Selecting a profile does not reserve or warm a machine. After review, omit `--dry-run=server` to create the Sandbox; normal budget and capacity checks apply.

Defaults resolve in this order: platform compatibility defaults, organization profile, project profile, selected profile, then explicit launch fields. Nested objects merge by field; lists replace inherited lists. Explicit `false` and an empty list override inherited values. Only `resources.gpu` and `virtualization` accept `null` to clear those optional objects; other invalid clears are rejected.

Set `spec.startup.inherit: false` (CLI `--inherit-startup=false`) to skip organization/project bindings. A selected `spec.startup.profile` still applies. Raw API clients can omit `spec.startup.defaults`; without an image in the request or an applied profile, the API rejects the launch as before. Use `defaults: Interactive` for the same convenience defaults as the CLI, SDK and console.

Bindings are organization-scoped `StartupProfileBinding` resources. For the organization, use the fixed name `organization` and `spec.profile: research`. For a project named `research`, use name `project-research` with `spec.project: research` and `spec.profile: research`. See the generated reference for long project names and all supported fields. A disabled or missing bound profile rejects new launches until the binding is corrected.

Jobs, Functions and Workspaces do not inherit Sandbox profiles. Agent-created Sandboxes do not automatically inherit them. Profiles do not install a filesystem, run setup commands or establish application readiness: verify your own HTTP, CUDA and filesystem checks after Sandbox Ready.

Image preparation can overlap saved-file restoration for supported, digest-pinned Sandbox images on the already allocated host. Application startup still waits for both operations. Freshly deleted hosts do not retain an image cache; a startup profile is not a warm pool or a startup-time guarantee.

### Pause process memory (Beta)

For a short break, open the Sandbox menu and choose **Pause memory**. On a compatible exclusive CPU machine, Nodus freezes the running processes and retains their memory for up to **30 minutes**. Wait for **Paused** before leaving it. The overview shows the retention deadline. Choose **Resume** to continue the same processes.

Compute billing stops when the freeze is acknowledged. Retained storage remains billable against the Sandbox’s budget, including the backing machine’s disk. Resume requires available credits and budget; compute billing restarts when the machine resumes. Minimum compute increments apply to total active time across pauses. The reservation alone is not a compute charge.

GPU Sandboxes and shared or incompatible CPU machines cannot use memory pause. A refused request leaves the Sandbox running, with its normal compute charges and idle policy. Ordinary stop/start preserves only the configured file state, not RAM. Retained memory expires after 30 minutes and is also lost on machine or guest restart failure; Nodus reports the loss and requires a new start rather than silently replacing the process. Stopping or deleting a paused Sandbox discards its retained memory without taking a new file checkpoint. After memory loss or expiry, choose **Start** explicitly. Terminal, file, and reconnect requests cannot start a replacement process automatically.

The API accepts a merge patch on an existing running Sandbox:

pause.json

```json
{"spec":{"state":"Paused"}}
```

Patch `spec.state` back to `Running` to request resume. Check `status.phase` and `status.pauseUntil`; a submitted request is not proof that the machine has paused or resumed. Creating a Sandbox directly in `Paused` is rejected.

### Create by name reconnects

If you set `spec.command`, a successful main-command exit stops the Sandbox with `stopReason: Completed`. `status.command` retains the successful Attempt, exit code and finish time. It stays stopped until you start it or use a wake trigger; the next start runs the command again. A clean exit does not prove that files were saved. Use a Job when you want a run-to-completion resource with a `Succeeded` phase.

Creating a Sandbox whose name already exists in the project does not make a second one:

* **Same spec.** You get the existing Sandbox back, and if it is stopped it starts again. This is how an agent reconnects to its Sandbox after a restart: it creates the same manifest every time.
* **A `maxCostUSD` that is added or raised.** The new budget is applied; nothing else changes. A budget can only be raised: a lower one, or none when the Sandbox has one, counts as a different spec.
* **Any other change.** The request fails with `409 AlreadyExists` and lists the fields that differ in `details.diff` and `details.causes`. Delete the Sandbox and create it again to change its image, shape or lifecycle.

GPU Sandboxes use a dedicated rented GPU VM and are non-interruptible. They default to Ephemeral continuity; a stopped GPU Sandbox starts empty. `fuseMount: true` is an opt-in guest capability for FUSE clients such as JuiceFS. Nodus does not create the filesystem or mount it for you, and unsupported GPU/runtime combinations fail closed.

### GPU child virtual machines (Alpha)

This capability is under qualification. A GPU listing by itself does **not** promise child-VM support. Check `spec.supports.sandboxVirtualizationProfiles` on an Offering; an absent or empty list means no qualified profile is advertised for it. The console reads that same field. The API validates the full request and returns the selected profile’s quote; the ordinary offering price and stock are not a child-VM availability guarantee.

The shared request field is `spec.virtualization.profile: gpu-child-v1`. The CLI convenience flag is `--virtualization-profile gpu-child-v1`, and Python accepts `virtualization_profile="gpu-child-v1"` on `Sandbox.create`. MCP accepts the identical Sandbox manifest through `estimate` and `apply`. An unsupported profile fails before allocation instead of silently creating an ordinary Sandbox.

This enables a parent for VMs you build and manage. Keep your own parent container image and run your VMM inside it; Nodus does not replace your entrypoint with a VM manager. The parent exposes `/dev/kvm`, `/dev/fuse`, `/dev/net/tun`, `/dev/vhost-net` and its assigned NVIDIA devices within a reserved user namespace. Your VMM must work within those permissions and the parent’s resource and network limits. It receives no host networking, VFIO device passthrough or `/dev/vhost-vsock` access. Forward a child service to a parent loopback port to use the existing authenticated Sandbox HTTP or port-forward route.

GPU access inside a child requires a compatible GPU transport and guest driver stack; KVM alone does not provide it. The reference path uses software forwarding with pinned components. The installed, verified companion includes an optional launcher and example guest; you do not have to use that launcher or guest to manage your own VMs. Its `--rootfs` option accepts a guest image with its required init and compatible userspace, while retaining the reference kernel and GPU stack. That helper’s requirements do not apply to a separately managed VMM, whose kernel and GPU integration need their own compatibility validation. Arbitrary VMM/kernel combinations have not been verified.

The profile requires one GPU and cannot change after creation. Child VMs share that GPU within the exclusive parent; they do not receive hardware-isolated GPU partitions. Stop/start can preserve the parent’s declared local state directory, but does not preserve child RAM or GPU memory. Keep remote filesystem mounts outside the saved directory. Existing Sandboxes continue to use their current runtime when this field is omitted.

### Stop and start a GPU Sandbox

GPU Sandboxes support stop/start with `Ephemeral` continuity. Stop releases the GPU allocation and keeps the Sandbox definition, image reference and Secret references. Customer compute occupancy ends when the execution controller accepts the stop; Nodus pays subsequent allocation cleanup. The minimum billing increment still applies to usage before the stop, and API-to-controller queue delay remains billable. Start launches fresh processes from the image; it does not restore process memory, local files or a previous FUSE mount. The Sandbox still expires at its original `maxLifetime`.

These commands are supported by CLI 0.1.72. Replace `gpu-dev` with your existing Sandbox name:

Terminal window

```sh
nodus stop sb/gpu-dev
nodus wait sb/gpu-dev --for=jsonpath='{.status.phase}'=Stopped --timeout 20m
nodus start sb/gpu-dev
nodus wait sb/gpu-dev --for=condition=Ready --timeout 20m
```

Data successfully saved to your external JuiceFS filesystem remains there. Put its mount and readiness checks in the image’s startup command so each start remounts it using the existing Secret. A mount made only by an interactive `exec` is not replayed. If your image instead contains a manual setup script, run it after readiness:

Terminal window

```sh
# This script must exist in your image and use your configured Secret environment.
nodus exec --no-wake sb/gpu-dev -- sh /opt/app/mount-juicefs.sh
```

Verify a known remote file before restarting application work. `Ready` alone does not prove that your filesystem is mounted. A fresh GPU VM may need to download the image again; stop/start does not promise an image cache or an instant start. Use the bounded prepared Sandbox workflow below when you need compute ready ahead of work.

### Preserve a local state directory (Beta)

Choose the directory when you **create a new Sandbox**. Existing Sandboxes cannot change continuity in place:

```yaml
spec:
  # Keep your existing image, GPU, secrets, FUSE, lifetime and budget settings.
  continuity:
    mode: Snapshotted
    stateDir: /nodus/state
```

The Python equivalent is `nodus.Sandbox.create(..., gpu="A100", state_dir="/nodus/state")`. For a CPU development sandbox, use `nodus.Sandbox.create(..., state_dir="/workspace")` and keep your project and its local dependencies inside `/workspace`. The directory starts empty and hides any image files already at that path. Write the files you need to retain there (`NODUS_STATE_DIR` names the same path). System directories and credential mounts are rejected. Only this local directory is retained; installed packages elsewhere, `/workspace` outside that directory, process memory and GPU memory are not saved. Do not mount JuiceFS inside it: keep the remote mount separate and remount it in startup. Nodus snapshots the local host directory, not the remote FUSE contents.

Flush application state on shutdown. Nodus saves the directory after the process exits during stop and periodically while running, then restores it before launching a fresh process on start. Stop remains bounded: if saving fails, the previous useful snapshot is retained and `StateSaved=False` reports the failure. Check `nodus describe sb/<name>` before relying on a final save. An empty directory cannot replace a saved checkpoint: it reports `StateSaved=False`, and a later start restores the previous snapshot rather than treating the deletion as saved. Snapshot size is bounded by the requested disk size. Startup and restore take time, and storage remains billed while the saved data is retained. This is disk recovery, not GPU pause or an instant warm restart.

A complete bounded manifest and verification commands are in `examples/sandboxes/gpu-state/` in the repository.

## Prepare one Sandbox for one job

The tested prepared-queue example creates at most one Sandbox, waits for the server’s `Ready` condition, runs an image-specific setup command, and only then admits it to a local queue. By default, a consumer takes that Sandbox once; an empty queue fails without creating a cold replacement. The manifest sets `idleTimeout: 0s` so the prepared Sandbox stays Running and `maxLifetime: 20m` so the server deletes it even if the client exits. `maxCostUSD` is a second per-Sandbox limit.

Run the example with CPU in compose, or pass `--gpu <type>` and a GPU-capable image on an account with that offering. Pass `--regions us` to keep the request in the US; if no US offer is available, it waits rather than falling back to another region. GPU defaults to 24 vCPU and CPU can be changed with `--cpu`. Pass `--fuse-mount` to enable the guest FUSE path for a supported GPU runtime. Egress defaults to `Deny`; configure an explicit allow list with `--egress AllowList --allow-host <host>`, or opt in to unrestricted egress with `--egress Open` when the image’s setup needs it. For a customer-owned remote filesystem, pass an existing Nodus Secret with `--secret <secret-name>`; the image must include the client and mount configuration. Use `--prepare-command` to mount and initialize the filesystem, then `--ready-command` to verify a data read and any required CUDA check before queue admission. The example does not supply credentials or mount configuration. Use `--prepare-only` to prepare ahead of traffic, then `--run-prepared` in a later invocation to claim the saved entry once. Both invocations use the same `--state-file`; its file lock coordinates callers on one host. Checkout rechecks ownership, readiness and expiration and submits `nodus exec --no-wake`. A stopped or expired Sandbox fails immediately without starting replacement compute. Use `--cleanup` for an unused entry. Prepared GPUs continue billing while idle; keep the lifetime and spending cap bounded.

This is a customer-owned queue example on one host: it is not a managed autoscaler, a shared host pool, a provider image cache, or a latency guarantee. Region choice is opt-in and `--regions us` is scoped to this request. It also does not retain model memory after the setup command exits.

### Reuse a prepared Sandbox within one trust boundary

For repeated jobs from the same trusted application, opt in with `--reuse-max-jobs` (2–100), a `--reset-command` or `--reset-script`, and a `--ready-command` or `--ready-script`. Supply the same reset and readiness checks when preparing and consuming the saved entry. The example checks the configured commands against the saved entry, runs reset and readiness checks before returning it to the queue, and retains the original lifetime and cost caps. A failed job or check triggers deletion. An interrupted dispatcher leaves a claimed entry that requires `--cleanup` before further use; it is not returned for reuse. The final permitted job also deletes it. Single-use behavior remains the default.

Reuse keeps the same running Sandbox and its already-pulled image and mounted filesystems. Idle time remains billable. Your reset command must clean application state; this does not isolate mutually untrusted users. It does not provide a fresh-host image cache, GPU memory snapshots, or stop/start continuity. Store durable data in your external filesystem. Use `--image-pull-secret` for registry credentials and `--secret` for guest credentials; they have different purposes.

### Customer-owned JuiceFS filesystem (Beta)

This self-service recipe mounts a customer-owned, already-formatted JuiceFS filesystem from a Nodus Secret, checks a customer-selected sentinel file, and runs a small CUDA smoke check before the one-item queue admits the Sandbox. It does not create or format a filesystem. GPU Sandboxes use dedicated on-demand capacity with `Never` interruption; choose the GPU, vCPU count, image and spend limits for your Job. Start with `--regions us`; if Canada is acceptable after US placement is unavailable, retry with `--regions ca`.

Create a protected local env file with the required `KEY=VALUE` entries from this table (plus backend-specific variables if needed), then create the Secret from the file. Keep the file out of source control; see the [Secrets guide](https://www.nodus-compute.ai/docs/guides/secrets/) for how Nodus delivers Secret values. JuiceFS documents its credential environment variables and FUSE flags in the [FUSE mount options reference](https://juicefs.com/docs/community/fuse_mount_options/).

|Secret key|Required|Purpose|
|-|-|-|
|`JUICEFS_META_URL`|Yes|Metadata URL for the existing filesystem; keep passwords out of the URL.|
|`JUICEFS_READY_PATH`|Yes|Relative path to an existing readable sentinel file.|
|`JUICEFS_READY_SHA256`|Yes|Expected SHA-256 digest of the sentinel file.|
|`META_PASSWORD`|If needed|JuiceFS metadata password.|
|`ACCESS_KEY`, `SECRET_KEY`|If needed|Credentials for the configured object-storage backend.|
|`JUICEFS_FUSE_OPTIONS`|Optional|Comma-separated FUSE mount options, such as `allow_other`.|

Terminal window

```console
touch .juicefs.env && chmod 600 .juicefs.env
$EDITOR .juicefs.env
nodus secret create customer-juicefs --from-env-file .juicefs.env && rm .juicefs.env
```

Save the two source blocks below as `juicefs_prepare.sh` and `ready_queue.py` in the same local directory. The command below prepares one GPU Sandbox; adjust the GPU, CPU, image, region and budget as needed. Replace the Job command with your application’s command and use an image containing that application; the sample `/workspace/train.py` path must exist in your chosen image.

Terminal window

```sh
python3 ready_queue.py --prepare-only \
  --image 'ghcr.io/nodus-compute/catalog/workspace-pytorch-cuda@sha256:5ce9e93b5e347b64000d12dc728b82743718c3be0593aa402d4d5f4ae0a2f8ba' \
  --gpu A100 --cpu 30 --fuse-mount --regions us --egress Open \
  --secret customer-juicefs --prepare-script juicefs_prepare.sh \
  --max-lifetime 20m --max-cost 5.00

python3 ready_queue.py --run-prepared \
  --job-command 'python /workspace/train.py --data-dir /mnt/juicefs'
```

The prepared GPU remains billable while idle until its bounded lifetime or spend cap ends. This is a local, one-Sandbox queue for one later job, not an autoscaling service or proof that a particular customer filesystem or GPU offering is qualified. For the general FUSE API, see the [Python Sandbox guide](https://www.nodus-compute.ai/docs/guides/python/sandboxes/#customer-owned-juicefs-beta).

juicefs_prepare.sh

```bash
#!/usr/bin/env bash
# Mount an already formatted JuiceFS filesystem and admit this Sandbox only after its data and CUDA checks pass.
set +x
set -eu

exec 3>&1
exec >/dev/null 2>&1

mountpoint_path=${JUICEFS_MOUNTPOINT:-/mnt/juicefs}
ready_path=${JUICEFS_READY_PATH:-}
ready_sha256=${JUICEFS_READY_SHA256:-}
phase=configuration
mounted=0
mount_owned=0
tmpdir=

cleanup() {
  status=$?
  trap - EXIT
  if [ "$status" -ne 0 ]; then
    if [ "$mount_owned" -eq 1 ] && command -v juicefs >/dev/null 2>&1 && command -v mountpoint >/dev/null 2>&1 && mountpoint -q "$mountpoint_path"; then
      juicefs umount "$mountpoint_path" >/dev/null 2>&1 || true
    fi
    if [ -n "$tmpdir" ]; then rm -rf -- "$tmpdir" >/dev/null 2>&1 || true; fi
    printf 'JuiceFS Sandbox preparation failed at %s\n' "$phase" >&3
  fi
  exit "$status"
}
trap cleanup EXIT

if command -v mountpoint >/dev/null 2>&1 && mountpoint -q "$mountpoint_path"; then
  phase=mountpoint_already_mounted
  exit 2
fi

if [ -z "${JUICEFS_META_URL:-}" ] || [ -z "$ready_path" ] || [ -z "$ready_sha256" ]; then
  phase=required_secret_missing
  exit 2
fi
case "$ready_path" in
  /*|*..*) phase=invalid_ready_path; exit 2 ;;
esac
if ! [[ "$ready_sha256" =~ ^[0-9a-fA-F]{64}$ ]]; then
  phase=invalid_ready_digest
  exit 2
fi

phase=dependencies
if ! command -v juicefs >/dev/null 2>&1; then
  command -v apt-get >/dev/null 2>&1
  command -v curl >/dev/null 2>&1
  command -v sha256sum >/dev/null 2>&1
  tmpdir=$(mktemp -d)
  apt-get update -qq
  apt-get install -qq -y --no-install-recommends ca-certificates curl fuse3
  curl --fail --silent --show-error --location --retry 3 \
    --output "$tmpdir/juicefs.tar.gz" \
    https://github.com/juicedata/juicefs/releases/download/v1.3.4/juicefs-1.3.4-linux-amd64.tar.gz
  printf '%s  %s\n' 98111cc7a67660eee9dd35915c0d33529e519282c9a5adf44c85a9e6dabfa676 \
    "$tmpdir/juicefs.tar.gz" | sha256sum --check --status
  tar -xzf "$tmpdir/juicefs.tar.gz" -C "$tmpdir" juicefs
  install -m 0755 "$tmpdir/juicefs" /usr/local/bin/juicefs
fi
phase=juicefs_version
juicefs version | grep -Eq '(^|[[:space:]])v?1\.3\.4([+[:space:]]|$)'
phase=fuse_dependencies
command -v fusermount3 >/dev/null 2>&1 || command -v fusermount >/dev/null 2>&1
command -v mountpoint >/dev/null 2>&1
command -v python3 >/dev/null 2>&1

if mountpoint -q "$mountpoint_path"; then
  phase=mountpoint_already_mounted
  exit 2
fi

phase=mount
mkdir -p "$mountpoint_path"
# The background mount must not retain the preparation Process output pipe.
if [ -n "${JUICEFS_FUSE_OPTIONS:-}" ]; then
  juicefs mount -d -o "$JUICEFS_FUSE_OPTIONS" "$JUICEFS_META_URL" "$mountpoint_path" >/dev/null 2>&1 3>&-
else
  juicefs mount -d "$JUICEFS_META_URL" "$mountpoint_path" >/dev/null 2>&1 3>&-
fi
mount_owned=1
for _ in $(seq 1 30); do
  if mountpoint -q "$mountpoint_path"; then mounted=1; break; fi
  sleep 1
done
if [ "$mounted" -ne 1 ]; then phase=mountpoint_check; exit 1; fi

phase=filesystem_data_check
JUICEFS_MOUNTPOINT="$mountpoint_path" python3 - "$ready_path" "$ready_sha256" <<'PY'
import hashlib
import hmac
import os
import pathlib
import sys

root = pathlib.Path(os.environ["JUICEFS_MOUNTPOINT"]).resolve()
candidate = (root / sys.argv[1]).resolve()
if root not in candidate.parents or not candidate.is_file():
    raise SystemExit(1)
digest = hashlib.sha256(candidate.read_bytes()).hexdigest()
if not hmac.compare_digest(digest, sys.argv[2].lower()):
    raise SystemExit(1)
PY

phase=cuda_compute_check
python3 - <<'PY'
import torch

if not torch.cuda.is_available():
    raise SystemExit(1)
matrix = torch.arange(64 * 64, dtype=torch.float32, device="cuda").reshape(64, 64)
result = matrix @ matrix.T
torch.cuda.synchronize()
if result.shape != (64, 64) or not torch.isfinite(result).all().item():
    raise SystemExit(1)
PY

if [ -n "$tmpdir" ]; then rm -rf -- "$tmpdir"; tmpdir=; fi
trap - EXIT
printf 'JuiceFS data and CUDA smoke checks passed; Sandbox remains mounted\n' >&3
```

ready_queue.py

```python
"""Prepare one bounded Sandbox ahead of a request, then claim it once without a cold fallback."""

from __future__ import annotations

import argparse
import fcntl
import hashlib
import json
import os
import re
import shlex
import subprocess
import sys
import tempfile
import uuid
from collections.abc import Iterator
from contextlib import contextmanager
from datetime import datetime, timezone
from decimal import Decimal
from pathlib import Path

_DURATION_PART = re.compile(r"(\d+(?:\.\d+)?)(ns|us|µs|μs|ms|s|m|h)")
_DURATION_SECONDS = {
    "ns": Decimal("0.000000001"),
    "us": Decimal("0.000001"),
    "µs": Decimal("0.000001"),
    "μs": Decimal("0.000001"),
    "ms": Decimal("0.001"),
    "s": Decimal(1),
    "m": Decimal(60),
    "h": Decimal(3600),
}


def duration_seconds(value: str) -> Decimal | None:
    """Parse the compound Go duration strings emitted by metav1.Duration JSON."""
    total = Decimal(0)
    offset = 0
    while offset < len(value):
        part = _DURATION_PART.match(value, offset)
        if part is None:
            return None
        total += Decimal(part.group(1)) * _DURATION_SECONDS[part.group(2)]
        offset = part.end()
    return total if offset else None


def lifecycle_matches(lifecycle: dict[str, object], requested: str) -> bool:
    actual_lifetime = duration_seconds(str(lifecycle.get("maxLifetime", "")))
    requested_lifetime = duration_seconds(requested)
    idle_timeout = duration_seconds(str(lifecycle.get("idleTimeout", "")))
    return (
        actual_lifetime is not None
        and requested_lifetime is not None
        and requested_lifetime > 0
        and actual_lifetime == requested_lifetime
        and idle_timeout == 0
    )


def guest_command(command: str | None, script: str | None, label: str) -> tuple[list[str], bytes | None]:
    if script is not None:
        if command is not None:
            raise ValueError(f"choose either --{label}-command or --{label}-script")
        return ["bash", "-s"], Path(script).read_bytes()
    argv = shlex.split(command) if command is not None else None
    if argv is not None and not argv:
        raise ValueError(f"--{label}-command must not be empty")
    return argv or [], None


def prepare_commands(
    args: argparse.Namespace,
) -> tuple[tuple[list[str], bytes | None], tuple[list[str], bytes | None] | None]:
    prepare = guest_command(args.prepare_command, args.prepare_script, "prepare")
    if not prepare[0]:
        prepare = (["python", "-c", "print('prepared-image-ready')"], None)
    ready = None
    if args.ready_command is not None or args.ready_script is not None:
        ready = guest_command(args.ready_command, args.ready_script, "ready")
    return prepare, ready


def exec_guest(name: str, command: list[str], stdin: bytes | None = None) -> None:
    args = ["nodus", "exec", "--no-wake"]
    if stdin is not None:
        args.append("--stdin")
    args.extend([f"sb/{name}", "--", *command])
    subprocess.run(args, check=True, input=stdin)


class ReadyQueue:
    """A bounded queue operation that admits only prepared Sandbox names."""

    def __init__(self, desired_ready: int, hard_max: int) -> None:
        if hard_max != 1 or desired_ready != 1:
            raise ValueError("this example supports desired-ready=1 and hard-max=1")
        self.desired_ready = desired_ready
        self.hard_max = hard_max
        self.created = 0
        self.ready: list[str] = []

    def prepare(self, args: argparse.Namespace) -> None:
        while len(self.ready) < self.desired_ready:
            if self.created >= self.hard_max:
                raise RuntimeError("ready queue reached its hard maximum")
            self.ready.append(self._create_and_prepare(args))
            self.created += 1
        print(f"ready queue: {len(self.ready)}/{self.hard_max}")

    def take(self) -> str:
        if not self.ready:
            raise RuntimeError("no prepared Sandbox is ready; cold fallback is disabled")
        return self.ready.pop(0)

    @staticmethod
    def _create_and_prepare(args: argparse.Namespace) -> str:
        run_id = uuid.uuid4().hex
        name = "ready-" + run_id[:12]
        if args.cpu is not None and not 1 <= args.cpu <= 64:
            raise ValueError("--cpu must be between 1 and 64")
        if not args.gpu and args.fuse_mount:
            raise ValueError("--fuse-mount requires --gpu")
        if args.egress == "AllowList" and not args.allow_host:
            raise ValueError("--egress AllowList requires at least one --allow-host")
        if args.egress != "AllowList" and args.allow_host:
            raise ValueError("--allow-host requires --egress AllowList")
        prepare, ready = prepare_commands(args)
        if not 1 <= args.reuse_max_jobs <= 100:
            raise ValueError("--reuse-max-jobs must be between 1 and 100")
        reset = guest_command(args.reset_command, args.reset_script, "reset")
        if args.reuse_max_jobs > 1 and (not reset[0] or ready is None):
            raise ValueError("reuse requires explicit reset and readiness commands/scripts")
        state_file = Path(args.state_file)
        # Record ownership before POST so cleanup can resolve a lost response without adopting a foreign object.
        state: dict[str, object] = {
            "sandbox": name,
            "run": run_id,
            "phase": "creating",
            "maxLifetime": args.max_lifetime,
            "maxCostUSD": args.max_cost,
            "maxJobs": args.reuse_max_jobs,
            "completedJobs": 0,
            "returnCheck": return_check_digest(reset, ready) if args.reuse_max_jobs > 1 else None,
        }
        write_state(state_file, state)
        resources: dict[str, object] = {
            "cpu": str(args.cpu if args.cpu is not None else 24 if args.gpu else 1),
            "memory": "2Gi",
            "disk": "10Gi",
        }
        if args.gpu:
            resources["gpu"] = {"type": [args.gpu], "count": 1}
        egress: dict[str, object] = {"policy": args.egress}
        if args.allow_host:
            egress["allow"] = args.allow_host
        secret_mounts = [{"name": name} for name in args.secret]
        manifest = {
            "apiVersion": "nodus.dev/v1",
            "kind": "Sandbox",
            "metadata": {
                "name": name,
                "labels": {"example.nodus.dev/role": "prepared-queue", "example.nodus.dev/run": run_id},
            },
            "spec": {
                "image": args.image,
                **({"imagePullSecrets": args.image_pull_secret} if args.image_pull_secret else {}),
                "resources": resources,
                "fuseMount": args.fuse_mount,
                "network": {"egress": egress},
                **({"placement": {"regions": args.regions}} if args.regions else {}),
                "lifecycle": {
                    "idleTimeout": "0s",
                    "onIdle": "Delete",
                    "maxLifetime": args.max_lifetime,
                },
                "maxCostUSD": args.max_cost,
                **({"secrets": secret_mounts} if secret_mounts else {}),
            },
        }
        with tempfile.NamedTemporaryFile(mode="w", suffix=".json", encoding="utf-8") as f:
            json.dump(manifest, f)
            f.flush()
            subprocess.run(["nodus", "create", "-f", f.name], check=True)

        # Ready is the server's Running condition; the setup command then verifies this image's own prerequisites.
        subprocess.run(
            ["nodus", "wait", f"sandbox/{name}", "--for=condition=Ready", "--timeout=10m"],
            check=True,
        )
        obj = get_owned(name, state)
        status = obj.get("status") or {}
        conditions = status.get("conditions") or []
        server_ready = any(c.get("type") == "Ready" and c.get("status") == "True" for c in conditions)
        lifecycle = (obj.get("spec") or {}).get("lifecycle") or {}
        if (
            status.get("phase") != "Running"
            or not server_ready
            or not status.get("expirationTime")
            or not lifecycle_matches(lifecycle, args.max_lifetime)
        ):
            raise RuntimeError("Sandbox did not report Running, Ready and the requested server lifetime")
        exec_guest(name, *prepare)
        if ready is not None:
            # Storage mounts and device checks must pass before this Sandbox enters the ready queue.
            exec_guest(name, *ready)
        obj = get_owned(name, state)
        status = obj.get("status") or {}
        if status.get("phase") != "Running":
            raise RuntimeError("Sandbox stopped during setup; it was not admitted to the ready queue")
        state["phase"] = "ready"
        state["uid"] = (obj.get("metadata") or {}).get("uid")
        state["expirationTime"] = status.get("expirationTime")
        if not state["uid"] or not state["expirationTime"]:
            raise RuntimeError("Sandbox did not report its UID and expiration time")
        validate_prepared(obj, state)
        write_state(state_file, state)
        print(f"prepared Sandbox {name}")
        return name


def get_owned(name: str, state: dict[str, object]) -> dict[str, object]:
    current = subprocess.run(
        ["nodus", "get", f"sandbox/{name}", "-o", "json"],
        check=True,
        capture_output=True,
        text=True,
    )
    obj = json.loads(current.stdout)
    meta = obj.get("metadata") or {}
    labels = meta.get("labels") or {}
    if (
        meta.get("name") != name
        or labels.get("example.nodus.dev/run") != state.get("run")
        or (state.get("uid") is not None and meta.get("uid") != state.get("uid"))
    ):
        raise RuntimeError("Sandbox operation refused: name or unique run label did not match")
    return obj


def delete_exact(name: str, state: dict[str, object]) -> None:
    # Check the unique run label before deleting by name; a selector could delete multiple objects.
    try:
        get_owned(name, state)
    except subprocess.CalledProcessError as exc:
        if state.get("phase") in {"ready", "claimed"} and state.get("uid") and wait_until_deleted(name):
            return
        raise RuntimeError(
            f"cleanup refused: could not verify ownership of Sandbox {name}; no delete was sent and "
            "the state file was retained for inspection"
        ) from exc
    except RuntimeError as exc:
        raise RuntimeError(
            f"cleanup refused: could not verify ownership of Sandbox {name}; no delete was sent and "
            "the state file was retained for inspection"
        ) from exc
    subprocess.run(
        [
            "nodus",
            "delete",
            f"sandbox/{name}",
            "--wait",
            "--timeout=5m",
            "--ignore-not-found",
        ],
        check=True,
    )


def wait_until_deleted(name: str) -> bool:
    """Use the CLI's structured delete-wait result to confirm absence after a failed get."""
    result = subprocess.run(
        ["nodus", "wait", f"sandbox/{name}", "--for=delete", "--timeout=1s"],
        check=False,
        stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL,
    )
    return result.returncode == 0


def read_state(state_file: Path) -> dict[str, object]:
    if not state_file.exists():
        raise RuntimeError(f"no prepared Sandbox state exists at {state_file}")
    state = json.loads(state_file.read_text(encoding="utf-8"))
    name = state.get("sandbox")
    run_id = state.get("run")
    if (
        not isinstance(name, str)
        or not name.startswith("ready-")
        or not isinstance(run_id, str)
        or len(run_id) != 32
        or any(char not in "0123456789abcdef" for char in run_id)
        or name != "ready-" + run_id[:12]
    ):
        raise ValueError("state does not contain a generated Sandbox name")
    return state


def write_state(state_file: Path, state: dict[str, object]) -> None:
    state_file.parent.mkdir(parents=True, exist_ok=True)
    with tempfile.NamedTemporaryFile(
        mode="w", suffix=".tmp", prefix=state_file.name + ".", dir=state_file.parent, delete=False, encoding="utf-8"
    ) as f:
        temp_path = Path(f.name)
        json.dump(state, f)
        f.flush()
        os.fsync(f.fileno())
    os.replace(temp_path, state_file)
    directory = os.open(state_file.parent, os.O_RDONLY)
    try:
        os.fsync(directory)
    finally:
        os.close(directory)


@contextmanager
def state_lock(state_file: Path) -> Iterator[None]:
    """Serialize operations sharing one local prepared-entry file."""
    state_file.parent.mkdir(parents=True, exist_ok=True)
    lock_path = state_file.with_name(state_file.name + ".lock")
    fd = os.open(lock_path, os.O_CREAT | os.O_RDWR, 0o600)
    try:
        try:
            fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
        except BlockingIOError as exc:
            raise RuntimeError("prepared Sandbox state is already being used") from exc
        yield
    finally:
        try:
            fcntl.flock(fd, fcntl.LOCK_UN)
        finally:
            os.close(fd)


def cleanup_state(state_file: Path) -> None:
    if not state_file.exists():
        return
    state = read_state(state_file)
    delete_exact(str(state["sandbox"]), state)
    state_file.unlink(missing_ok=True)


def cleanup(args: argparse.Namespace) -> None:
    state_file = Path(args.state_file)
    with state_lock(state_file):
        cleanup_state(state_file)


def prepare_only(args: argparse.Namespace) -> None:
    state_file = Path(args.state_file)
    with state_lock(state_file):
        if state_file.exists():
            raise RuntimeError("an earlier prepared Sandbox still needs a job or cleanup")
        queue = ReadyQueue(args.desired_ready, args.hard_max)
        try:
            queue.prepare(args)
        except Exception:
            # Delete when ownership can be proved; retain state if creation or deletion is ambiguous.
            cleanup_state(state_file)
            raise
        print(
            "prepared Sandbox is available for one later invocation"
            if args.reuse_max_jobs == 1
            else f"prepared Sandbox supports at most {args.reuse_max_jobs} same-trust jobs"
        )


def validate_prepared(obj: dict[str, object], state: dict[str, object]) -> None:
    status = obj.get("status") or {}
    conditions = status.get("conditions") or []
    ready = any(c.get("type") == "Ready" and c.get("status") == "True" for c in conditions)
    spec = obj.get("spec") or {}
    lifecycle = spec.get("lifecycle") or {}
    expiration = status.get("expirationTime")
    recorded_expiration = state.get("expirationTime")
    if (
        state.get("phase") != "ready"
        or not isinstance(state.get("uid"), str)
        or not state.get("uid")
        or status.get("phase") != "Running"
        or not ready
        or not isinstance(expiration, str)
        or expiration != recorded_expiration
        or not lifecycle_matches(lifecycle, str(state.get("maxLifetime", "")))
        or spec.get("maxCostUSD") != state.get("maxCostUSD")
    ):
        raise RuntimeError("prepared Sandbox is no longer Running, Ready, or within its recorded lifetime")
    try:
        expires_at = datetime.fromisoformat(expiration.replace("Z", "+00:00"))
    except ValueError as exc:
        raise RuntimeError("prepared Sandbox expiration time is invalid") from exc
    if expires_at.tzinfo is None or expires_at <= datetime.now(timezone.utc):
        raise RuntimeError("prepared Sandbox has expired")


def return_check_digest(reset: tuple[list[str], bytes | None], ready: tuple[list[str], bytes | None] | None) -> str:
    """Bind return checks without persisting command text, script contents or credentials."""
    checks = [reset, ready]
    digest = hashlib.sha256()
    for check in checks:
        if check is None:
            raise ValueError("reuse requires a readiness check")
        argv, stdin = check
        digest.update(json.dumps(argv).encode())
        digest.update(b"\x00")
        digest.update(stdin or b"")
        digest.update(b"\x00")
    return digest.hexdigest()


def run_prepared(args: argparse.Namespace) -> None:
    state_file = Path(args.state_file)
    with state_lock(state_file):
        state = read_state(state_file)
        name = str(state["sandbox"])
        obj = get_owned(name, state)
        validate_prepared(obj, state)
        job_command = shlex.split(args.job_command)
        if not job_command:
            raise ValueError("--job-command must not be empty")
        max_jobs = state.get("maxJobs", 1)
        completed = state.get("completedJobs", 0)
        if (
            type(max_jobs) is not int
            or not 1 <= max_jobs <= 100
            or type(completed) is not int
            or not 0 <= completed < max_jobs
        ):
            raise RuntimeError("prepared Sandbox job limit is invalid or exhausted")
        reset = guest_command(args.reset_command, args.reset_script, "reset")
        ready = guest_command(args.ready_command, args.ready_script, "ready")
        if max_jobs > 1 and (
            not reset[0] or not ready[0] or return_check_digest(reset, ready) != state.get("returnCheck")
        ):
            raise RuntimeError("reuse requires the original reset and readiness checks")
        state["phase"] = "claimed"
        write_state(state_file, state)
        print(f"claimed prepared Sandbox once: {name}")
        returned = False
        try:
            exec_guest(name, job_command)
            state["completedJobs"] = completed + 1
            if completed + 1 < max_jobs:
                exec_guest(name, *reset)
                exec_guest(name, *ready)
                state["phase"] = "ready"
                validate_prepared(get_owned(name, state), state)
                write_state(state_file, state)
                returned = True
                print(f"returned prepared Sandbox: {completed + 1}/{max_jobs} jobs completed")
        finally:
            # Failed/uncertain execution or return is never eligible for reuse. A killed dispatcher leaves
            # durable phase=claimed, which requires explicit cleanup; it cannot execute another job.
            if not returned:
                cleanup_state(state_file)


def demo(args: argparse.Namespace) -> None:
    state_file = Path(args.state_file)
    with state_lock(state_file):
        queue = ReadyQueue(args.desired_ready, args.hard_max)
        if state_file.exists():
            raise RuntimeError("an earlier queue item still needs cleanup; run with --cleanup first")
        try:
            queue.prepare(args)
            name = queue.take()
            print(f"handed out once: {name}")
            subprocess.run(
                ["nodus", "exec", "--no-wake", f"sb/{name}", "--", *shlex.split(args.job_command)], check=True
            )
            try:
                queue.take()
            except RuntimeError as exc:
                print(f"queue empty; {exc}")
            else:
                raise AssertionError("the prepared Sandbox was handed out more than once")
        finally:
            cleanup_state(state_file)


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("--desired-ready", type=int, default=1)
    parser.add_argument("--hard-max", type=int, default=1)
    parser.add_argument("--image", default="nodus/agent-tools")
    parser.add_argument(
        "--image-pull-secret",
        action="append",
        default=[],
        metavar="NAME",
        help="Registry Secret for image pulls only; never mounted in the guest",
    )
    parser.add_argument(
        "--reuse-max-jobs", type=int, default=1, help="opt-in same-trust job limit, 1..100; idle compute stays billed"
    )
    parser.add_argument("--reset-command", help="same-trust guest cleanup required before returning a reusable Sandbox")
    parser.add_argument(
        "--reset-script", help="local cleanup script streamed to bash; required check is pinned by digest"
    )
    parser.add_argument("--gpu", help="optional accelerator family; omit to run the compose example on CPU")
    parser.add_argument("--regions", nargs="+", help="optional Nodus region classes, for example us")
    parser.add_argument("--cpu", type=int, help="vCPU count (defaults to 1 for CPU or 24 for GPU)")
    parser.add_argument("--fuse-mount", action="store_true", help="enable guest FUSE support; requires --gpu")
    parser.add_argument(
        "--egress",
        choices=("Deny", "AllowList", "Open"),
        default="Deny",
        help="container outbound policy; Open is an explicit opt-in",
    )
    parser.add_argument(
        "--allow-host", action="append", default=[], help="allowed DNS host (requires --egress AllowList)"
    )
    parser.add_argument("--max-lifetime", default="20m")
    parser.add_argument("--max-cost", default="1.00")
    parser.add_argument("--state-file", default=".ready-queue-state.json")
    parser.add_argument(
        "--prepare-command",
        help="quoted command string parsed as argv without a shell; can mount/init customer storage before admission",
    )
    parser.add_argument(
        "--prepare-script",
        help="local script streamed to bash -s inside the Sandbox before it is admitted",
    )
    parser.add_argument(
        "--ready-command",
        help="optional quoted data-read/CUDA readiness check, parsed as argv and run before queue admission",
    )
    parser.add_argument(
        "--ready-script",
        help="optional local script streamed to bash -s inside the Sandbox before it is admitted",
    )
    parser.add_argument(
        "--secret",
        action="append",
        default=[],
        metavar="NAME",
        help="existing Nodus Secret to expose to the Sandbox (repeatable; secret values are not copied)",
    )
    parser.add_argument(
        "--job-command",
        default="python -c \"print('prepared-queue-job-ok')\"",
        help="quoted command string parsed as argv without a shell",
    )
    modes = parser.add_mutually_exclusive_group()
    modes.add_argument("--cleanup", action="store_true", help="delete the exact Sandbox recorded for cleanup")
    modes.add_argument(
        "--prepare-only", action="store_true", help="prepare one Sandbox and leave it ready for one later invocation"
    )
    modes.add_argument(
        "--run-prepared", action="store_true", help="claim and run one prepared Sandbox without creating or waking it"
    )
    args = parser.parse_args()
    try:
        if args.cleanup:
            cleanup(args)
        elif args.prepare_only:
            prepare_only(args)
        elif args.run_prepared:
            run_prepared(args)
        else:
            demo(args)
    except subprocess.CalledProcessError as exc:
        raise SystemExit(exc.returncode) from exc
    except (RuntimeError, ValueError) as exc:
        print(str(exc), file=sys.stderr)
        raise SystemExit(2) from exc


if __name__ == "__main__":
    main()
```

## Run commands

`nodus exec` runs one command in the running Sandbox and streams its standard output and standard error back as the command writes them. The exit code of `nodus exec` is the command’s. Every command is recorded as a Process of the Sandbox, whatever started it (the CLI, an agent or the MCP tools): its name, such as `sb-hello-12`, comes back with the result, and `nodus get processes` lists it with how it ended.

Terminal window

```console
$ nodus exec sb/hello -- python -c 'print(6 * 7)'
42
$ nodus exec sb/hello -- sh -c 'pip list 2>/dev/null | head -3; exit 3'; echo "exit $?"
Package    Version
---------- -------
pip        24.2
exit 3
```

A command runs in `spec.workingDir` (`/workspace` unless you set it) with the Sandbox’s environment, as a non-root user. Pass a single string to `sh -c` to use pipes and redirection. Every command counts as activity, so a Sandbox never stops as idle while a command is running.

Pass `--no-wake` when a command must use an already running Sandbox. It refuses a stopped, starting or expired Sandbox without waking compute or waiting for startup. Without that flag, the usual implicit wake remains.

### Open a shell

`nodus shell` is `nodus create sandbox` and `nodus exec -it` in one step: it opens an interactive terminal in a Sandbox and creates the Sandbox first when the name is new. It takes the same flags as `nodus create sandbox`, and the image is `nodus/agent-tools` unless you pass `--image`.

Terminal window

```console
$ nodus shell sandbox/dev --cpu 2 --memory 4Gi
sandbox/dev created
sandboxes/dev is starting; waiting for it
```

You are now in a terminal in the Sandbox, in `/workspace`. When you exit, `nodus shell sandbox/dev` opens it again on the current running attempt. After a stop, only the selected persistent directory is restored. Without an explicit `stateDir`, CPU checkpoints retain `/nodus/state`; `/workspace` is then not saved.

The Sandbox stays after you exit, and stops by itself when idle. Pass `--rm` to delete a Sandbox this command created when the shell exits; `--rm` refuses a Sandbox that already exists, and so does a flag such as `--cpu`, because neither changes a Sandbox that is there. `nodus shell` with no name gives the shell a Sandbox of its own and deletes it on exit unless you pass `--keep`. Put a command after `--` to run it instead of `bash`: `nodus shell sandbox/dev -- zsh`. With input that is not a terminal, the shell runs without one, so `echo 'make test' | nodus shell sandbox/dev` works in a script. The exit code is the shell’s.

### Timeouts

Each command has its own timeout: 10 minutes unless you set one, and at most 24 hours. When the timeout passes, the process is killed with `SIGKILL` and the command ends with the reason `DeadlineExceeded`. The Sandbox keeps running. Set the timeout per command, for example `sb.exec("make", "test", timeout="2m")` in Python or `"timeout": "2m"` in the API request.

How a command ended is in its result:

|Reason|Meaning|
|-|-|
|none|Exit code 0|
|`NonZeroExit`|The command exited with another code|
|`Signaled`|A signal ended it; the exit code is 128 plus the signal number|
|`DeadlineExceeded`|Its timeout passed and it was killed|
|`StartFailed`|It could not start, for example because the program does not exist|
|`NodeLost`|The machine running the Sandbox was lost while the command ran|
|`OutputLimitExceeded`|It wrote more output than a Process may keep|
|`ParentStopped`|The Sandbox stopped or was deleted while the command ran|
|`Cancelled`|You cancelled it|

A command line holds at most 1,024 arguments and 64 KiB in total, plus up to 256 extra environment variables. A Sandbox runs at most 1,024 commands at once; the next one fails with `429 TooManyRequests` until one ends.

## Files

In the console, open **Files** on a Sandbox, or **Browse files** inside an agent run. Upload files or a folder, open a text file to edit it, or download it. The editor supports UTF-8 text up to 1 MiB; console uploads support up to 32 MiB per file. Uploads refuse to replace existing content. If an agent changes a file while you edit, your save is refused and your draft stays available to copy or merge. **Load latest and replace draft** replaces your local draft with the current file. A stopped or restarted Sandbox cannot be edited through a stale panel. Choose **Wake and browse** explicitly to start stopped compute.

`nodus cp` copies files in both directions, and works on every image because the file service is part of the Sandbox runtime, not of the image:

Terminal window

```console
$ nodus cp ./notes.txt sb/hello:/nodus/state/notes.txt     # laptop to Sandbox
$ nodus cp sb/hello:/workspace/result.json ./result.json  # Sandbox to laptop
```

The same operations are one request each on `/apis/nodus.dev/v1/namespaces/{project}/sandboxes/{name}/files`:

|Request|Does|
|-|-|
|`GET …/files?path=/workspace/a.txt`|Returns the file’s bytes|
|`PUT …/files?path=/workspace/a.txt`|Replaces the file with the request body, creating parent directories|
|`DELETE …/files?path=/workspace/a.txt`|Deletes a file or an empty directory|

A write replaces the file atomically: a reader sees the old content or the new content, never half of it. A file is at most 64 MiB per request; a larger one fails with `413 FileTooLarge`. Paths are absolute, and a path that does not exist answers `404 NotFound`.

For an already running Sandbox, add `wake=false` to refuse an implicit start. Add both `expectedUID` and `expectedAttempt` from the Sandbox’s current metadata and status to refuse a request if the Sandbox was replaced or restarted. A stale target returns `409 Conflict`; no new compute starts. Without `wake=false`, the existing automatic wake behavior applies.

### Writing without overwriting someone else’s change

When two writers share a file, send the digest of the content you started from as `expectedSHA256`:

Terminal window

```console
$ curl -X PUT --data-binary @plan.md "$API/sandboxes/hello/files?path=/workspace/plan.md&expectedSHA256=$(sha256sum old-plan.md | cut -d' ' -f1)"
```

The write happens only if the file still has that digest. If it changed, nothing is written and the request fails with `412 SHA256Mismatch`: read the file again, merge, and retry. The digest is 64 lowercase hex digits. The digest of empty content (`e3b0c442…b855`) matches a file that does not exist yet, so it means “create only”.

## Stop, start and wake

A Sandbox is `Running` while it holds compute and `Stopped` while it does not. With an explicit `continuity.stateDir`, CPU and GPU Sandboxes save that directory after the process exits and report `StateSaved`. Check the save result before relying on it, then verify the expected files after starting. A stopped phase alone does not prove a successful save.

CPU `Snapshotted` continuity without a stateDir retains its legacy behavior: it checkpoints `/nodus/state` before sending the process its stop signal. Flush files before requesting stop; shutdown-handler writes, `/workspace`, installed packages elsewhere and the rest of the root filesystem are not captured. This legacy mode does not expose `StateSaved` or `status.snapshot`.

Terminal window

```console
$ nodus stop sb/hello        # checkpoints /nodus/state, then releases compute
$ nodus start sb/hello       # starts a fresh process; verify the state files
$ nodus get sb/hello
NAME    PHASE     ACTIVITY   CPU   MEMORY   GPU   COST      AGE
hello   Stopped   Idle       1     1Gi      -     $0.0042   12m
```

`nodus stop` sets `spec.state: Stopped` and the Sandbox stays stopped until you start it. Every other stop is made by the platform, shows its reason in `status.stopReason`, and ends the next time something needs the Sandbox:

|`stopReason`|Why it stopped|
|-|-|
|`User`|You ran `nodus stop`|
|`Idle`|Nothing used it for `idleTimeout`|
|`InsufficientCredits`|Your balance could not fund the next period|
|`BudgetExceeded`|A Budget on the project reached its limit|
|`MaxCostReached`|The Sandbox cost as much as `spec.maxCostUSD`|

A stopped Sandbox starts again on any of these wake triggers:

* a command (`nodus exec`, or an exec request),
* a file request,
* `nodus start`,
* creating a Sandbox with the same name and an identical spec.

Credits and budgets still apply to a wake. A Sandbox stopped for `InsufficientCredits`, `BudgetExceeded` or `MaxCostReached` stays stopped until the balance, the Budget or `maxCostUSD` allows the next period of compute. Until then a command, a file request and `nodus start` answer `402` instead of `SandboxStarting`: `InsufficientCredits` states the hold the start needs and what you have, and `BudgetExceeded` the Budget or `maxCostUSD` that is full. Add credits, raise the Budget or raise `maxCostUSD`, then send a trigger again. A create by name records its wake too, but returns the Sandbox as it is: still stopped, with `Funded` false in its conditions.

### `SandboxStarting` and `Retry-After`

A command or file request that finds the Sandbox stopped starts it and answers `503 SandboxStarting` with a `Retry-After` header (2 seconds), because the container is not ready yet. The CLI and the SDK wait and retry for you, so `nodus exec` on a stopped Sandbox just takes a few seconds longer. If you call the API directly, retry after the time in `Retry-After`; each retry is also a wake trigger, so it does no harm to send them early.

Other reasons a request can fail while the Sandbox is not usable:

|Status and reason|Meaning|
|-|-|
|`409 SandboxNotRunning`|The Sandbox is being deleted|
|`409 SandboxFailed`|The Sandbox failed to start or lost its machine; `status.reason` says why|
|`402 InsufficientCredits`|The Sandbox is stopped for lack of credits and cannot start yet|
|`402 BudgetExceeded`|The Sandbox is stopped because a Budget or its `maxCostUSD` is full and cannot start yet|
|`404 NotFound`|There is no Sandbox with that name|

Each code has a page in the [error reference](https://www.nodus-compute.ai/docs/reference/errors/).

## Idle, `onIdle` and `maxLifetime`

A Sandbox is **idle** when no command is running, no file request arrived for `idleTimeout` and no terminal had input in that time. A running process, including one that prints nothing for hours, counts as busy; a terminal that is open but silent does not count as activity.

When the idle time passes, `onIdle` decides what happens:

* `Stop` (the default) requests a state-directory checkpoint, releases compute and sets `stopReason: Idle`. The next wake trigger starts it again.
* `Delete` removes the Sandbox with its files. Use it for one-off work that leaves nothing worth keeping.

`idleTimeout: 0s` turns idle detection off, so the Sandbox runs until you stop it, until `maxLifetime` or until a budget ends it.

`maxLifetime` is a hard deadline counted from creation, 24 hours by default. When it passes, the Sandbox is deleted whatever it is doing, running or stopped. It is the backstop that keeps a forgotten Sandbox from living forever. Create a new Sandbox, or one with a longer `maxLifetime`, to carry on.

## Network access

A Sandbox has no outbound network access unless you open it. `spec.network.egress.policy` is one of:

* `Deny` (the default): nothing leaves the Sandbox, so code in it cannot download or upload anything.
* `AllowList`: set `spec.network.egress.allow` to explicit DNS names, such as `github.com`, or `*.domain` patterns. HTTP and HTTPS tools must use the injected `HTTP_PROXY` and `HTTPS_PROXY` settings. Other destinations and private addresses stay blocked. Presets are not accepted for Sandboxes.
* `Open`: the Sandbox can reach the public internet. It still cannot reach other Sandboxes, the machine it runs on or Nodus’s own services.

A Sandbox accepts no public inbound connections. Use commands, file requests, or `nodus port-forward` to reach its applications through your authenticated Nodus session.

Your application can listen on `127.0.0.1` inside the Sandbox. HTTP requests and port forwarding connect to that Sandbox’s loopback, including GPU Sandboxes with `fuseMount: true`.

For a backend making short HTTP calls, the **Beta** API provides a stable authenticated endpoint: `POST /apis/nodus.dev/v1/namespaces/PROJECT/sandboxes/NAME/http`. It returns the application’s status, headers and base64 body inside JSON. The Sandbox must already be running; the call never starts compute.

```python
import nodus

box = nodus.Sandbox.from_name("dev", wake=False)
response = box.http_request(8000, "GET", "/health")
response.raise_for_status()
print(response.text)
```

For asynchronous callers, use `await box.http_request.aio(...)`. Existing Nodus login or API-key configuration authenticates the request. API keys need `processes:write` in the Sandbox’s project. The MCP `sandbox_http_request` tool uses this same API; its `request` fields are `port`, `method`, `path`, optional `headers` (arrays of values), and optional base64 `body`. API and SDK responses retain application errors and redirects; they do not follow redirects or retry. A connection failure can occur after the application acted, so check its state before retrying.

Calls last at most 30 seconds, with 1 MiB request and 2 MiB response bodies. Use a custom header such as `X-App-Token` for application authentication. Nodus credentials, cookies and routing headers are never forwarded. Use port-forward for streaming, WebSockets, browser access, or larger payloads. See the [backend HTTP example](https://github.com/nodus-compute/nodus-platform/tree/main/examples/sandboxes/http).

### Browser previews

In an Agent run, open **Sandbox → Previews**, add your app’s port, and choose **Open preview**. The same controls are in a Sandbox’s **Ports** tab. Ports are private by default: project execution access is required. The app opens in a separate tab on an isolated user-content domain. Adding a port, viewing its address, and minting a sign-in link do not start compute.

You can declare ports through the API or CLI:

Terminal window

```sh
nodus patch sandbox/dev --type merge -p '{"spec":{"ingress":{"ports":[{"port":3000}]}}}'
nodus open sandbox/dev --port 3000
```

Or use Python:

```python
box = nodus.Sandbox.from_name("dev")
preview = box.tunnels.open(3000)
# Open preview.url in your browser. Do not log or share private sign-in URLs.
```

Private sign-in links expire after 60 seconds; browser sessions last up to one hour and remain subject to current project permissions and credential revocation. API keys need `processes:write`. The preview endpoint requires `{port, expectedUID}` so an old panel cannot mint access to a replacement Sandbox. `status.endpoints` contains stable addresses without credentials, not an app readiness check.

Choose **Make public** only for an app you intend anyone with its URL to access. Public links normally leave idle-stopped compute asleep. Enabling **Allow visitors to wake** permits those requests to restart compute and incur charges. Authenticated preview requests can wake idle-stopped compute; an explicit Stop, memory pause, expired Sandbox, or lost retained memory requires explicit Start. Ordinary funding and budget admission still apply. Removing a port revokes new requests; an already-open stream may remain until it closes or the runtime stops.

Declare up to 16 ports, from 1024–65535 excluding reserved ports 7777, 18080 and 49983. If preview routing is unavailable, or you prefer a local tunnel to port 8000, run:

Terminal window

```sh
nodus port-forward sandbox/dev 8000:8000
```

Keep the command running and connect to `http://127.0.0.1:8000`. The connection uses your Nodus credentials; the local listener defaults to loopback. This provides a local tunnel, not a public or persistent preview URL.

## Spending limits

Review the estimate before starting a Sandbox.

`nodus get sb/hello` shows the running total in the `COST` column. Set `spec.maxCostUSD` to cap one Sandbox: it stops when it has cost that much, and you can raise the cap with another create by name. Without a cap, your credit balance and any Budget on the project are the limits.

## Next steps

* [Sandbox isolation](https://www.nodus-compute.ai/docs/concepts/sandboxes-isolation/) for what separates your Sandbox from the machine and from other Sandboxes.
* [Sandboxes from Python](https://www.nodus-compute.ai/docs/guides/python/sandboxes/) for the SDK, including terminals and previews.
