Benchmark GPU Functions on your pool (Beta)

Beta: enabled per organization

The benchmark runs GPU Functions on your pool with GPU snapshots. Both are in Beta and enabled separately for your organization.

IdleBench is a versioned, reproducible benchmark of the useful AI work a GPU does per hour when requests are sparse. Its suite s2 serves several models that each get a few requests a minute. Instead of a GPU for each, the models take turns on one GPU: a worker leaves once its calls finish, and the next model’s worker restores its snapshot there. examples/pools/idlebench/ runs suite s2 on your own pool, so you can measure what taking turns gives on your machines and compare runs over time.

Run the benchmark

Enroll a GPU host into a pool named idlebench (Enroll a host). The example’s pool.yaml creates it with mode: Only and burstToMarket: Deny, so no request is served outside your pool. From examples/pools/idlebench/, give the number of GPUs in the pool:

nodus apply -f pool.yaml
nodus run app.py --gpus 1

The app asks for A10 GPUs; set IDLEBENCH_GPU to your pool’s GPU model, and IDLEBENCH_POOL for another pool.

examples/pools/idlebench/app.py

"""IdleBench s2 on your own pool: four models take turns on its GPU while a seeded trace of requests arrives (Beta).

`nodus run app.py --gpus <GPUs in the pool>` calls each model once, so its first worker loads it and is snapshotted,
then replays the trace on schedule; a worker leaves the GPU once its calls finish and the next model restores there.
"""

import hashlib
import json
import os
import time
from concurrent.futures import ThreadPoolExecutor

import nodus

POOL = os.environ.get("IDLEBENCH_POOL", "idlebench")
GPU = os.environ.get("IDLEBENCH_GPU", "A10")
SLO_S = 15.0  # a request counts when it is answered within 15 s of its time in the trace
MAX_NEW_TOKENS = 64
# Trace model m is MODELS[m], IdleBench's s2 models in its order, pinned so every run serves the same weights.
MODELS = [
    ("Qwen/Qwen3-0.6B", "c1899de289a04d12100db370d81485cdf75e47ca"),
    ("Qwen/Qwen2.5-1.5B-Instruct", "989aa7980e4cf806f80c7fef2b1adb7bc71aa306"),
    ("Qwen/Qwen3-1.7B", "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"),
    ("Qwen/Qwen2.5-Coder-1.5B-Instruct", "2e1fd397ee46e1388853d2af2c993145b0f1098a"),
]

app = nodus.App("idlebench")
image = nodus.Image.debian_slim().pip_install("torch==2.6.0", "transformers==4.53.3")
OPTIONS = dict(gpu=GPU, image=image, pool=POOL, gpu_snapshot=True, scaledown_window="0s", max_cost=1)


class Model:
    INDEX = 0

    @nodus.enter()
    def load(self) -> None:
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer

        repo, rev = MODELS[self.INDEX]
        self.tok = AutoTokenizer.from_pretrained(repo, revision=rev)
        net = AutoModelForCausalLM.from_pretrained(repo, revision=rev, torch_dtype=torch.float16)
        self.net = net.to("cuda").eval()

    @nodus.method()
    def generate(self, prompt: str) -> str:
        import torch

        ids = self.tok(prompt, return_tensors="pt").to("cuda")
        with torch.no_grad():
            out = self.net.generate(**ids, max_new_tokens=MAX_NEW_TOKENS, do_sample=False)
        return self.tok.decode(out[0][ids["input_ids"].shape[1] :], skip_special_tokens=True)


# One GPU class per model, Model0, Model1, ..., so each model has its own workers and its own snapshot.
SERVERS = [app.cls(**OPTIONS)(type(f"Model{i}", (Model,), {"INDEX": i, "__module__": __name__}))
           for i in range(len(MODELS))]


@app.local_entrypoint()
def main(trace: str = "s2-zipf-k4-r16-s1.json", out: str = "runs", gpus: int = 1) -> None:
    import idlebench as ib

    tr, ref = ib.load_trace(trace)
    if tr["models"] > len(SERVERS):
        raise SystemExit(f"{trace} has {tr['models']} models; this app serves {len(SERVERS)}")
    names = [repo for repo, _ in MODELS[: tr["models"]]]
    served = {"engine": "transformers 4.53.3, float16, eager", "decoding": "greedy", "max_new_tokens": MAX_NEW_TOKENS,
              "models": [{"repo": r, "revision": v} for r, v in MODELS[: tr["models"]]]}
    env = {"pool": POOL, "gpu": GPU, "gpus": gpus, "sdk": nodus.__version__}
    run = ib.Run(out, "s2", "pool-snapshot-swaps", "nodus", tr["seed"], served=served, trace=ref,
                 slo={"e2e_s": SLO_S}, env=env, provenance={"driver": "examples/pools/idlebench/app.py"})
    answers: dict[tuple[int, int], list[str]] = {}  # SHA-256 digests of each model's answers to each prompt

    def call(m: int, p: int, timeout: float) -> tuple[bool, str | None]:
        try:
            text = SERVERS[m]().generate.spawn(ib.PROMPTS[p]).get(timeout=timeout)
        except Exception as exc:
            return False, type(exc).__name__
        answers.setdefault((m, p), []).append(hashlib.sha256(text.encode()).hexdigest())
        return True, None

    with run:
        for m, name in enumerate(names):  # a first worker may build its image and download its model
            start = time.time()
            ok, error = call(m, 0, timeout=1800)
            end = time.time()
            run.event("transition", what="warmup", model=name, start=start, end=end, dur_s=end - start, ok=ok,
                      error=error)
        run.check("every model answered its warm-up call", len(answers) == len(names), f"{len(answers)} answered")

        lags: list[float] = []
        dones: list[float] = []
        t0_mono, t0 = time.monotonic(), time.time()

        def replay(i: int, a: dict) -> None:
            arrive = t0 + a["t"]
            lags.append(time.time() - arrive)
            ok, error = call(a["m"], a["p"], timeout=600)
            dones.append(time.time())
            run.event("request", id=f"r{i}", model=names[a["m"]], arrive=arrive, done=dones[-1],
                      e2e_s=dones[-1] - arrive, ok=ok, prompt=a["p"], error=error)

        with ThreadPoolExecutor(max_workers=max(1, len(tr["arrivals"]))) as executor:
            for i, a in enumerate(tr["arrivals"]):
                time.sleep(max(0.0, t0_mono + a["t"] - time.monotonic()))
                executor.submit(replay, i, a)
        # The GPU time scored runs from the trace's start to its last answer, and covers at least the whole trace.
        run.meta["params"] = {"t0": t0, "measured_s": max(tr["duration_s"], max(dones, default=t0) - t0)}
        run.check("every arrival was sent", len(lags) == len(tr["arrivals"]), f"{len(lags)} sent")
        latest = max(lags, default=0.0)
        run.check("every arrival was sent within 1 s of its time", latest <= 1, f"latest {latest:.3f} s")
        # Greedy decoding gives one answer per model and prompt, whether the worker started cold or restored.
        run.event("identity", what="greedy answers of one model to one prompt, against its first answer",
                  compared=sum(len(d) - 1 for d in answers.values()),
                  mismatches=sum(x != d[0] for d in answers.values() for x in d[1:]))

    print(json.dumps(ib.score(str(run.dir)), indent=1))

What a run measures

• Warm-up. Each of the four models is called once, in turn. Its first worker starts cold, loads the model and is snapshotted; the time to its answer is the model’s first_answer_s.
• Replay. The 113 requests of s2-zipf-k4-r16-s1.json are sent at their times over 6 minutes, without waiting for earlier answers, most of them for the first model. Each is one call, with greedy decoding of up to 64 tokens.
• Time to answer. From a request’s time in the trace to its answer reaching the app: waiting for the GPU, restoring a snapshot and answering. A request answered within 15 s counts toward the score.
• GPU time. The GPUs you gave times the replay’s duration, from the trace’s start to its last answer and at least 6 minutes, since the pool’s GPUs serve the benchmark whether or not a model is on them.

The app prints score.json: within_slo_per_gpu_hour is the score, beside attainment (the share of requests answered within 15 s), e2e_p50_s and e2e_p95_s, first_answer_s and gpu_hours. Greedy decoding gives one answer per model and prompt, so greedy_mismatches counts answers that differ from that model’s first answer to the prompt, cold or restored; the app keeps only a SHA-256 digest of each answer, never the text.

Keep and compare runs

Each run is a directory runs/s2/<run id>/. run.json names the trace by its SHA-256 digest, the models by their revisions, and the pool, its GPU model and count; events.jsonl has one line per warm-up call and per request. A run is valid when every model answered its warm-up call and every request was sent within 1 s of its time; one that is not says which check failed in run.json, and its score is not comparable. Two valid runs are comparable when their trace, models and GPU model are the same: python3 idlebench.py check s2-zipf-k4-r16-s1.json regenerates the trace from its seed and confirms the file matches byte for byte. To make turns faster, read Keep turns fast with the swap budget.

What it costs

Each worker is charged the pool’s routing fee for each GPU it asked for while it holds the machine; the machine itself costs nothing. max_cost=1 caps each model’s workers at $1. The app runs as an ephemeral App, so its classes are deleted when the run ends and each run takes its snapshots anew in its warm-up.

Check prerequisites separately from results

The Pool capabilities API (Beta), GET /apis/nodus.dev/v1/namespaces/{project}/pools/{pool}/capabilities, reports organization support separately from current host prerequisites for GPU Function snapshots, two resident Function workers and serving-gap batch work. A host with no current connection capability observation remains unknown. Meeting prerequisites does not reserve capacity or establish that your model fits or that a restore succeeded.

Compare public protocols and evidence at Nodus Benchmarks. Your Pool’s GPU activity, memory and busy-time readings are diagnostic measurements. Useful workload output per physical GPU-hour also needs the workload’s quality and SLO gates and its physical allocation intervals. A feature setting or a snapshot outcome alone does not prove an improvement.