RTX PRO 6000 Blackwell Max-Q: how many tokens per second?

On our server, Qwen3-8B Q4_K_M generated 202.21 tokens/s at a depth of 2,048 tokens and 111.73 tokens/s at 32,768 tokens. These are llama-bench engine measurements: means of three repetitions after warmup, with the model on one RTX PRO 6000 Blackwell Max-Q GPU. They are not application response times or model-quality scores.

The practical takeaway when choosing a server is that a GPU name and one “tokens/s” figure are insufficient. You need the model, quantization, context length and measurement type. We provide those alongside downloadable results you can inspect.

Test configuration

This engine build is a pinned research version marked as a pre-release by the project, not a claim to use the latest stable release. Revisions, model-file hashes and individual repetitions are included in the JSON.

Qwen3-8B Q4_K_M results

“±” is the standard deviation of three repetitions reported by the tool, not a confidence interval or guaranteed range. In prompt processing, the stated length is the input-token count. In generation, it is an already populated context, followed by 256 new tokens.

In the Qwen3-8B series, generation at a depth of 32,768 tokens was about 45% slower than at 2,048. That does not mean a long-document application will be exactly 45% slower: it must also prepare prompts, possibly retrieve documents and deliver responses. It does show why a very short-input benchmark is a poor description of long conversation histories.

Qwen3-32B Q4_K_M results

We also completed all six Qwen3-32B cases: prompt processing and generation at each of the three context lengths. We kept the same llama.cpp 0.5.0-dev b11146 build, F16 KV cache, Flash Attention and 256 generated tokens. Each value below is the mean of three measured repetitions; “±” is their standard deviation reported by the tool.

Tables are rounded to two decimal places; more precise means, deviations and individual repetitions are in the downloads. We used Qwen3-32B-Q4_K_M.gguf from the publisher's repository at revision 938a7432affaec9157f883a87164e2646ae17555. That model revision uses the Apache-2.0 license; we do not extend it automatically to other models or environment components.

An important difference: cooling between cases

The 8B series had no additional wait for cooling between cases. For 32B, before each of the six separate cases, we waited for a GPU reading no higher than 40°C. The tool then performed its warm-up and three measured repetitions, without additional cooling pauses within that case. The waits between cases are outside the reported engine throughput.

These are observations of two disclosed configurations, not a controlled model comparison at the same thermal state or a measurement isolating parameter count. Cooling before a process starts does not mean thermal equilibrium under continuous load. This series neither establishes stability after four hours of operation nor measures answer quality. The procedural difference is also retained in the public JSON and CSV.

The 72B attempt: completed cases and the stopping point

Qwen2.5-72B-Instruct Q4_K_M completed prompt processing and generation at depths of 2,048 and 8,192 tokens — four cases, each with three measured repetitions. We allowed cooling to no more than 40°C before each case. During processing of 32,768 tokens, the harness's 85°C threshold stopped the test; generation at that depth was not run. The full 72B matrix did not complete, so it is not added to the tables or chart above. The JSON retains the completed-case scope and the interrupted run's record checksum.

This is the earlier series with a harness threshold, not a finding of protective GPU shutdown. In the later campaign with a 90°C threshold, we did not repeat the 72B test: the stop occurred during Qwen3-8B API testing. The account of both new API attempts keeps those results separate from the engine measurements presented here.

This model is from a different generation than Qwen3-8B and Qwen3-32B. It also has a separate Qwen license at the pinned repository revision, not Apache-2.0. Completing short cases alone establishes neither sustained operation nor answer quality.

Two speeds for two different stages

Prompt processing, or prefill, and generating subsequent tokens, or decode, are measured separately. More than 10,000 input tokens/s must not be presented as the speed of writing an answer. Qwen3-8B generation measurements were approximately 112–202 tokens/s, depending on context depth; Qwen3-32B produced approximately 44–54 tokens/s with the cooling procedure described above.

llama-bench uses synthetic tokens and excludes tokenization and sampling. The method is described in the documentation for the exact llama.cpp revision used. This is also not a test of Polish-language accuracy, coding ability or RAG quality.

Reproducing the settings

This Qwen3-8B example assumes the pinned llama-bench build and identical model file are already available. For a depth of 8,192, we ran input processing and generation separately:

llama-bench -m Qwen3-8B-Q4_K_M.gguf -ngl 999 -fa on -t 16 \
  -b 2048 -ub 512 -ctk f16 -ctv f16 -r 3 -o json \
  -p 8192 -n 0 -d 0

llama-bench -m Qwen3-8B-Q4_K_M.gguf -ngl 999 -fa on -t 16 \
  -b 2048 -ub 512 -ctk f16 -ctv f16 -r 3 -o json \
  -p 0 -n 256 -d 8192

Built-in warmup remained enabled. For Qwen3-32B, we used the same flags with Qwen3-32B-Q4_K_M.gguf, additionally waiting for cooling to at most 40°C before each separate invocation, as described above. We did not drop the OS file cache or run other GPU workloads concurrently. The original 8B weights come from the Qwen3-8B GGUF publisher; the datasets identify the revision and SHA-256 for both models.

What this means for a deployment

This measurement provides an initial view of computation speed for the stated model and context. Before selecting a configuration, also check answer quality on your tasks, time to first token and concurrent-request performance. Engine figures alone do not establish how many employees can use a service or a guaranteed response time.

We do not compare these numbers with another provider without running an identical test. These are first-party measurements described by the service provider, not independent certification. Our methodology and update policy explain evidence retention and the distinction between a measured test system and the commercial specification.

Explore the server configuration or tell us your model, document lengths and expected concurrency.