vastai run benchmarks

Benchmark a template against one or more GPUs

Usage

vastai run benchmarks [OPTIONS]

Options

--template_hash · string

the template to benchmark; provide either —template_hash or —template_id

--template_id · integer

the template to benchmark; provide either —template_hash or —template_id

--gpus · string

comma-separated GPU names to benchmark (e.g. RTX_4090,RTX_3090). Prefix a name with a count to set GPUs per instance, e.g. “2x RTX_4090” (overrides —num_gpus)

--num_gpus · integer

GPUs per instance for tokens without an Nx prefix (default 1); overridden by inline Nx in —gpus

--timeout · integer · default: "1800"

max seconds to wait for a benchmark before giving up (default 1800)

--no-cache · boolean

re-measure instead of reusing recent cached benchmark results

--yes · boolean

Skip confirmation prompt

Description

Rents one instance per GPU in parallel, measures perf, tears down. Each rental runs for up to —timeout seconds and costs real money.

Specs benchmarked recently (same template, GPU, and count, by any user) are served from the benchmarks table as an estimated perf, and you’re asked whether to reuse that or run a fresh benchmark. Pass —no-cache to always re-measure, or -y to always reuse the cache.

Perf/$uses the GPU$/hr, so cached and freshly measured rows are comparable. Cached rows recorded before that price was tracked show a perf with no price.

Examples

# auto-sweep the default GPUs against TGI
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5

    # specific GPUs against vLLM
    vastai run benchmarks --template_hash 393fa8572e6c73c927c8275fe4dffd53 --gpus RTX_4090,RTX_3090

    # multi-GPU configurations via inline Nx prefix against ComfyUI
    vastai run benchmarks --template_hash 40ef49becc953aa910ee05bd4653b9b3 --gpus "2x RTX_4090, 2x RTX_3090"

    # default count for tokens without an Nx prefix
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5 --gpus RTX_4090,RTX_3090 --num_gpus 2

    # shorter timeout (10 min), skipping the cost prompt
    vastai run benchmarks --template_hash 393fa8572e6c73c927c8275fe4dffd53 --timeout 600 -y

    # ignore cached results and re-measure everything
    vastai run benchmarks --template_hash 79ebdd2ebfb9d42cedf7a221c42d37a5 --no-cache

    # raw JSON output for piping into another tool
    vastai run benchmarks --template_hash 40ef49becc953aa910ee05bd4653b9b3 --raw

Global Options

The following options are available for all commands:

Option	Description
--url URL	Server REST API URL
--retry N	Retry limit
--raw	Output machine-readable JSON
--explain	Verbose explanation of API calls
--api-key KEY	API key (defaults to ~/.config/vastai/vast_api_key)