# Benchmark GPU Functions on your pool (Beta)

> Run IdleBench suite s2 on your own pool and score how many requests its GPUs answer within 15 s per GPU-hour.

Source: https://www.nodus-compute.ai/docs/guides/pools/benchmark/
Build revision: 4c89a51d7dcfd856f1373531796fbb15831d8da3

**Beta:** this feature may change.

Beta: enabled per organization

The benchmark runs [GPU Functions on your pool](https://www.nodus-compute.ai/docs/guides/pools/#run-gpu-functions-on-the-pool-beta) with [GPU snapshots](https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/). Both are in Beta and enabled separately for your organization.

IdleBench is a versioned, reproducible benchmark of the useful AI work a GPU does per hour when requests are sparse. Its suite s2 serves several models that each get a few requests a minute. Instead of a GPU for each, the models take turns on one GPU: a worker leaves once its calls finish, and the next model’s worker restores its snapshot there. `examples/pools/idlebench/` runs suite s2 on your own pool, so you can measure what taking turns gives on your machines and compare runs over time.

## Run the benchmark

Enroll a GPU host into a pool named `idlebench` ([Enroll a host](https://www.nodus-compute.ai/docs/guides/pools/#enroll-a-host)). The example’s `pool.yaml` creates it with `mode: Only` and `burstToMarket: Deny`, so no request is served outside your pool. From `examples/pools/idlebench/`, give the number of GPUs in the pool:

Terminal window

```sh
nodus apply -f pool.yaml
nodus run app.py --gpus 1
```

The app asks for `A10` GPUs; set `IDLEBENCH_GPU` to your pool’s GPU model, and `IDLEBENCH_POOL` for another pool.

examples/pools/idlebench/app.py

```python
"""IdleBench s2 on your own pool: four models take turns on its GPU while a seeded trace of requests arrives (Beta).

`nodus run app.py --gpus <GPUs in the pool>` calls each model once, so its first worker loads it and is snapshotted,
then replays the trace on schedule; a worker leaves the GPU once its calls finish and the next model restores there.
"""

import hashlib
import json
import os
import time
from concurrent.futures import ThreadPoolExecutor

import nodus

POOL = os.environ.get("IDLEBENCH_POOL", "idlebench")
GPU = os.environ.get("IDLEBENCH_GPU", "A10")
SLO_S = 15.0  # a request counts when it is answered within 15 s of its time in the trace
MAX_NEW_TOKENS = 64
# Trace model m is MODELS[m], IdleBench's s2 models in its order, pinned so every run serves the same weights.
MODELS = [
    ("Qwen/Qwen3-0.6B", "c1899de289a04d12100db370d81485cdf75e47ca"),
    ("Qwen/Qwen2.5-1.5B-Instruct", "989aa7980e4cf806f80c7fef2b1adb7bc71aa306"),
    ("Qwen/Qwen3-1.7B", "70d244cc86ccca08cf5af4e1e306ecf908b1ad5e"),
    ("Qwen/Qwen2.5-Coder-1.5B-Instruct", "2e1fd397ee46e1388853d2af2c993145b0f1098a"),
]

app = nodus.App("idlebench")
image = nodus.Image.debian_slim().pip_install("torch==2.6.0", "transformers==4.53.3")
OPTIONS = dict(gpu=GPU, image=image, pool=POOL, gpu_snapshot=True, scaledown_window="0s", max_cost=1)


class Model:
    INDEX = 0

    @nodus.enter()
    def load(self) -> None:
        import torch
        from transformers import AutoModelForCausalLM, AutoTokenizer

        repo, rev = MODELS[self.INDEX]
        self.tok = AutoTokenizer.from_pretrained(repo, revision=rev)
        net = AutoModelForCausalLM.from_pretrained(repo, revision=rev, torch_dtype=torch.float16)
        self.net = net.to("cuda").eval()

    @nodus.method()
    def generate(self, prompt: str) -> str:
        import torch

        ids = self.tok(prompt, return_tensors="pt").to("cuda")
        with torch.no_grad():
            out = self.net.generate(**ids, max_new_tokens=MAX_NEW_TOKENS, do_sample=False)
        return self.tok.decode(out[0][ids["input_ids"].shape[1] :], skip_special_tokens=True)


# One GPU class per model, Model0, Model1, ..., so each model has its own workers and its own snapshot.
SERVERS = [app.cls(**OPTIONS)(type(f"Model{i}", (Model,), {"INDEX": i, "__module__": __name__}))
           for i in range(len(MODELS))]


@app.local_entrypoint()
def main(trace: str = "s2-zipf-k4-r16-s1.json", out: str = "runs", gpus: int = 1) -> None:
    import idlebench as ib

    tr, ref = ib.load_trace(trace)
    if tr["models"] > len(SERVERS):
        raise SystemExit(f"{trace} has {tr['models']} models; this app serves {len(SERVERS)}")
    names = [repo for repo, _ in MODELS[: tr["models"]]]
    served = {"engine": "transformers 4.53.3, float16, eager", "decoding": "greedy", "max_new_tokens": MAX_NEW_TOKENS,
              "models": [{"repo": r, "revision": v} for r, v in MODELS[: tr["models"]]]}
    env = {"pool": POOL, "gpu": GPU, "gpus": gpus, "sdk": nodus.__version__}
    run = ib.Run(out, "s2", "pool-snapshot-swaps", "nodus", tr["seed"], served=served, trace=ref,
                 slo={"e2e_s": SLO_S}, env=env, provenance={"driver": "examples/pools/idlebench/app.py"})
    answers: dict[tuple[int, int], list[str]] = {}  # SHA-256 digests of each model's answers to each prompt

    def call(m: int, p: int, timeout: float) -> tuple[bool, str | None]:
        try:
            text = SERVERS[m]().generate.spawn(ib.PROMPTS[p]).get(timeout=timeout)
        except Exception as exc:
            return False, type(exc).__name__
        answers.setdefault((m, p), []).append(hashlib.sha256(text.encode()).hexdigest())
        return True, None

    with run:
        for m, name in enumerate(names):  # a first worker may build its image and download its model
            start = time.time()
            ok, error = call(m, 0, timeout=1800)
            end = time.time()
            run.event("transition", what="warmup", model=name, start=start, end=end, dur_s=end - start, ok=ok,
                      error=error)
        run.check("every model answered its warm-up call", len(answers) == len(names), f"{len(answers)} answered")

        lags: list[float] = []
        dones: list[float] = []
        t0_mono, t0 = time.monotonic(), time.time()

        def replay(i: int, a: dict) -> None:
            arrive = t0 + a["t"]
            lags.append(time.time() - arrive)
            ok, error = call(a["m"], a["p"], timeout=600)
            dones.append(time.time())
            run.event("request", id=f"r{i}", model=names[a["m"]], arrive=arrive, done=dones[-1],
                      e2e_s=dones[-1] - arrive, ok=ok, prompt=a["p"], error=error)

        with ThreadPoolExecutor(max_workers=max(1, len(tr["arrivals"]))) as executor:
            for i, a in enumerate(tr["arrivals"]):
                time.sleep(max(0.0, t0_mono + a["t"] - time.monotonic()))
                executor.submit(replay, i, a)
        # The GPU time scored runs from the trace's start to its last answer, and covers at least the whole trace.
        run.meta["params"] = {"t0": t0, "measured_s": max(tr["duration_s"], max(dones, default=t0) - t0)}
        run.check("every arrival was sent", len(lags) == len(tr["arrivals"]), f"{len(lags)} sent")
        latest = max(lags, default=0.0)
        run.check("every arrival was sent within 1 s of its time", latest <= 1, f"latest {latest:.3f} s")
        # Greedy decoding gives one answer per model and prompt, whether the worker started cold or restored.
        run.event("identity", what="greedy answers of one model to one prompt, against its first answer",
                  compared=sum(len(d) - 1 for d in answers.values()),
                  mismatches=sum(x != d[0] for d in answers.values() for x in d[1:]))

    print(json.dumps(ib.score(str(run.dir)), indent=1))
```

## What a run measures

* **Warm-up.** Each of the four models is called once, in turn. Its first worker starts cold, loads the model and is snapshotted; the time to its answer is the model’s `first_answer_s`.
* **Replay.** The 113 requests of `s2-zipf-k4-r16-s1.json` are sent at their times over 6 minutes, without waiting for earlier answers, most of them for the first model. Each is one call, with greedy decoding of up to 64 tokens.
* **Time to answer.** From a request’s time in the trace to its answer reaching the app: waiting for the GPU, restoring a snapshot and answering. A request answered within 15 s counts toward the score.
* **GPU time.** The GPUs you gave times the replay’s duration, from the trace’s start to its last answer and at least 6 minutes, since the pool’s GPUs serve the benchmark whether or not a model is on them.

The app prints `score.json`: `within_slo_per_gpu_hour` is the score, beside `attainment` (the share of requests answered within 15 s), `e2e_p50_s` and `e2e_p95_s`, `first_answer_s` and `gpu_hours`. Greedy decoding gives one answer per model and prompt, so `greedy_mismatches` counts answers that differ from that model’s first answer to the prompt, cold or restored; the app keeps only a SHA-256 digest of each answer, never the text.

## Keep and compare runs

Each run is a directory `runs/s2/<run id>/`. `run.json` names the trace by its SHA-256 digest, the models by their revisions, and the pool, its GPU model and count; `events.jsonl` has one line per warm-up call and per request. A run is valid when every model answered its warm-up call and every request was sent within 1 s of its time; one that is not says which check failed in `run.json`, and its score is not comparable. Two valid runs are comparable when their trace, models and GPU model are the same: `python3 idlebench.py check s2-zipf-k4-r16-s1.json` regenerates the trace from its seed and confirms the file matches byte for byte. To make turns faster, read [Keep turns fast with the swap budget](https://www.nodus-compute.ai/docs/guides/functions/gpu-snapshots/#keep-turns-fast-with-the-swap-budget).

## What it costs

Each worker is charged the pool’s routing fee for each GPU it asked for while it holds the machine; the machine itself costs nothing. `max_cost=1` caps each model’s workers at $1. The app runs as an ephemeral App, so its classes are deleted when the run ends and each run takes its snapshots anew in its warm-up.

## Check prerequisites separately from results

The Pool capabilities API (Beta), `GET /apis/nodus.dev/v1/namespaces/{project}/pools/{pool}/capabilities`, reports organization support separately from current host prerequisites for GPU Function snapshots, two resident Function workers and serving-gap batch work. A host with no current connection capability observation remains unknown. Meeting prerequisites does not reserve capacity or establish that your model fits or that a restore succeeded.

Compare public protocols and evidence at [Nodus Benchmarks](https://benchmarks.nodus-compute.ai/). Your Pool’s GPU activity, memory and busy-time readings are diagnostic measurements. Useful workload output per physical GPU-hour also needs the workload’s quality and SLO gates and its physical allocation intervals. A feature setting or a snapshot outcome alone does not prove an improvement.