More tokens per second for a server does not necessarily mean a faster answer for one person. With 2,048 input tokens in our Qwen3-8B test, moving from one to four concurrent requests increased aggregate throughput from 171.22 to 312.65 tokens/s. Median complete-response time also increased, from 1.49 to 3.26 seconds.
This matters when designing an internal assistant or API: throughput describes the amount of work completed, while latency describes the wait for a particular response. One cannot be directly substituted for the other.
Qwen3-8B streaming API results
Each request generated 256 tokens. TTFT is measured from the client's request start to its first received generated-token event. Complete-response time includes finishing that response. Both times are medians; throughput is the mean of three rounds.
| Input tokens per request | Concurrent requests | Median TTFT | Median complete response | Aggregate output throughput |
|---|---|---|---|---|
| 2,048 | 1 | 0.19 s | 1.49 s | 171.22 tokens/s |
| 2,048 | 4 | 0.70 s | 3.26 s | 312.65 tokens/s |
| 8,192 | 1 | 0.85 s | 2.37 s | 107.84 tokens/s |
| 8,192 | 4 | 2.49 s | 6.56 s | 154.86 tokens/s |
The single-request cases contain three measured responses each; the four-request cases contain twelve each. This series therefore comprises 30 successfully completed responses. Downloads also include the 95th percentile, but with such small samples it only describes the sample, not a dependable production p95 guarantee.
Interpreting the results
With 8,192 input tokens, four concurrent requests delivered higher aggregate throughput than one, but median first-token time increased from about 0.85 to 2.49 seconds. Increasing concurrency therefore had a latency cost. Prompt length has a separate effect: with one request, increasing input from 2,048 to 8,192 tokens increased TTFT from about 0.19 to 0.85 seconds.
This does not mean the server supports only four users. We measured simultaneous requests, not accounts or people. An application whose users type occasionally differs from a batch process that continuously keeps four generations active. We did not test more than four concurrent requests here.
Round throughput is total generated tokens divided by the time to finish the entire round. That time includes prompt processing, waiting and generation. It is therefore different from decode-only speed in our llama-bench engine microbenchmark.
Settings and experimental scope
We used one RTX PRO 6000 Blackwell Max-Q with 96GB VRAM, a 300W power limit, an EPYC 4565P and 96GB system RAM. OS: Rocky Linux 10.2; driver: NVIDIA 595.91.07; engine: pinned llama.cpp 0.5.0-dev, b11146; model: Qwen3-8B Q4_K_M.
The server had four slots with 8,704 context tokens each, F16 KV cache and Flash Attention enabled. That allocation remained unchanged when only one request was active. Automatic configuration fitting and context shifting were disabled.
We tokenized public test text, repeated its tokens and truncated them to exactly 2,048 or 8,192 tokens. One unmeasured warmup request on slot 0 preceded each measurement group; it was not a round of four concurrent clients. We then ran three measured rounds for that group. Prompt caching was disabled and response counters checked so that reusing a prompt could not masquerade as full input processing. Output was fixed at 256 tokens, ignoring earlier EOS. This is a synthetic infrastructure test, not an evaluation of meaningful answers.
The client and server communicated over localhost. These times exclude a client's connection from Poland, Germany or any other country, VPNs, external proxies and the public network. The /completion protocol is documented in the exact llama-server revision used.
Longer context: two further attempts
We repeated the extended Qwen3-8B Q4_K_M test with 2,048, 8,192 and 32,768 input tokens and 1 or 4 concurrent requests. This time, the harness would stop the workload at 90°C or a T.Limit margin of 2°C or less, with GPU readings every second. These are our test thresholds, not a change to factory protections. The 300W power limit and fan settings remained unchanged.
Each attempt used four slots of 33,280 tokens, or 133,120 allocated context tokens, rather than the 34,816 used for the original table. We do not pool results or memory readings from these different configurations.
| Attempt with the new threshold | Complete cases | Completed rounds / responses | Outcome |
|---|---|---|---|
| First attempt | 6 of 6 | 18 / 45 | Stopped during the final process check; highest sampled temperature 87°C |
| Repeat | 5 of 6 | 17 / 41 | Stopped at the harness's 90°C threshold |
The first attempt finished all measured cases but did not complete the overall campaign successfully. In the repeat, the 32,768-token / 4-request case completed only two of its three rounds. The record contains 148 GPU readings at a nominal one-second interval. At the stop, temperature was 90°C and T.Limit margin was 3°C: the temperature threshold triggered, not the margin threshold. In that sample, SW/HW thermal slowdown were “Not Active”, while power capping was “Active”. A harness stop alone does not establish a hardware fault or protective GPU shutdown.
We did not launch further tests after that stop. 32B, 72B and the four-hour test were not run under the new threshold. Both separate JSON records are available to download; we neither construct a complete series from fragments of different attempts nor replace the original table and chart with them.
Earlier attempts with the 85°C threshold
The short, completed 8B series in the table above reached a maximum sampled temperature of 73°C. A separate, earlier extension to 32,768 tokens stopped at the harness's 85°C threshold. That is not a temperature limit declared by the GPU manufacturer.
- 32B API: completed 2,048/1, 2,048/4 and 8,192/1. The 85°C threshold stopped the 8,192/4 warmup with a margin of 7°C; neither that warmup nor a measured round of the case completed. This was not a reported out-of-memory (OOM) error. In the stop sample, SW/HW thermal slowdown were inactive and power capping was active at the stock 300W limit.
- 72B API: completed three rounds of 2,048/1 and one round of 2,048/4, containing three and four responses respectively. The 85°C threshold stopped the attempt before the 8,192-token cases.
- Planned four-hour 32B test: with 8,192 input tokens, 4 requests and 256 output tokens, the 85°C threshold stopped the attempt after three rounds, or 12 responses. Total campaign time was 105.11 seconds including warmup and cleanup, not 105.11 seconds of continuous generation. The four-hour test did not complete.
We do not add incomplete series to the completed-results table or present their average throughput as a comparison result. None of these short attempts establishes multi-hour stability.
What this test does not establish
We did not measure answer quality, RAG accuracy, service availability or an application SLA. Production validation needs your prompts, a realistic request arrival pattern and longer observation of queues and errors.
Before choosing a configuration, identify typical and maximum prompt lengths, output length, simultaneous generations and acceptable time to first token. Describe your workload, or start with the local API and SSH-tunnel guide.