Whether an LLM fits on a GPU depends on more than its download size. In our completed Qwen3-8B Q4_K_M test, the weights file was approximately 4.68 GiB. The highest observed GPU-memory reading for the running server was about 9.88 GiB, with four slots, 8,704 context tokens per slot and F16 KV cache.
That difference is not “wasted memory.” Planning must account for context, computation buffers and the runtime as well as weights. Comparing GGUF size alone with a GPU's “96GB” label leaves those components out.
Completed request-serving tests
| Model and format | Total weights-file size | Highest sampled GPU-memory use | Verified scenario |
|---|---|---|---|
| Qwen3-8B, Q4_K_M | 5,027,783,488 B / 4.68 GiB | 10,114 MiB / 9.88 GiB | 2,048/8,192 input tokens; 1/4 concurrent requests; 256 output tokens per request |
Byte counts come from verified files; GPU readings come from nvidia-smi, sampled every two seconds while the server was running. Here, GiB means 2³⁰ bytes and MiB means 2²⁰ bytes. The GPU's marketed 96GB VRAM capacity is not a reading of free memory for a particular process.
The maximum includes global GPU usage: the model, KV allocation and other runtime buffers. It is not a model's minimum VRAM requirement or an exact measurement of its weights alone. Sampling can also miss brief peaks. One model server used the GPU during these computations, without concurrent rendering.
What we also verified for a 32B model
Qwen3-32B Q4_K_M completed all six short llama-bench cases on this GPU: prompt processing and generation at depths of 2,048, 8,192 and 32,768 tokens, with 256 output tokens in each generation case. The GPU was allowed to cool to no more than 40°C before each case. This confirms that this particular configuration fits in the short engine test, not sustained stability or a maximum user count. See the engine microbenchmark for details.
The earlier 32B API attempt completed cases 2,048/1, 2,048/4 and 8,192/1 (input tokens / concurrent requests). The harness stopped during the 8,192/4 warmup at 85°C with a thermal margin of 7°C, before either the warmup or a measured round of that final case completed. The harness threshold triggered rather than an OOM report. The table above therefore remains a record of the completed 8B configuration; we do not infer peak VRAM usage for a 32B API server from short engine tests.
What about a 72B model?
Qwen2.5-72B-Instruct Q4_K_M loaded on this GPU and completed short engine cases for 2,048 and 8,192 input tokens, plus 256-token generation at both depths. Each case followed cooling to no more than 40°C. Processing 32,768 tokens stopped at the harness's then-configured 85°C threshold; this is neither a passed full matrix nor confirmation of sustained long-context serving. The engine test describes that scope. This 72B version has a separate Qwen license, not Apache-2.0.
The earlier, separate 72B API attempt completed three measured rounds with 2,048 input tokens and one request. With 2,048 tokens and four concurrent requests, only one round containing four responses completed; the 8,192-token cases were not started. The harness stopped the test at its 85°C threshold. This is an incomplete series: we do not add it to the completed-tests table or present its memory reading as verified capacity across the full set of 72B API workloads.
The new attempts with a 90°C threshold tested only the extended 8B API configuration. The repeat stopped before 32B, 72B or the four-hour workload could run, so it provided no new measurements for those models. Its context allocation was 133,120 tokens, not the 34,816 used for the table above. The separate account and data for both API attempts preserve that distinction; we do not substitute memory readings between configurations.
Why context and slots change the result
This llama-server configuration reserved 34,816 context tokens in total: four slots of 8,704. That allocation remained active even when the test submitted a single request. The table should therefore not be read as the minimum memory needed for one short prompt.
Inputs contained up to 8,192 tokens and outputs 256. Remaining space provided headroom within each slot. We disabled automatic configuration fitting, context shifting and prompt caching. Context and parallelism options are described in the pinned llama-server documentation.
More slots or a different KV format can change memory use. We do not estimate maximum user count by simply dividing 96GB by weights-file size: a user is not a permanent allocation of a separate model, and requests vary in length and frequency.
RAM and VRAM are different resources
VRAM belongs to the graphics card. System RAM serves the CPU, application and, among other things, the OS file cache. Increasing system RAM does not increase the GPU's physical VRAM. CPU execution or offloading has a different performance profile and needs a separate test.
The test system contained an RTX PRO 6000 Blackwell Max-Q with 96GB VRAM, an EPYC 4565P and 96GB system RAM. We used Rocky Linux 10.2, NVIDIA driver 595.91.07 and pinned llama.cpp 0.5.0-dev b11146 with CUDA 12.8. The data also records exact Linux-visible RAM, which differs from the installed module capacity.
Starting a model is not a storage benchmark
For Qwen3-8B, we observed approximately 2.03 seconds from process creation to the first successful /health response. This is a single observation, with readiness checked once per second. We verified the model-file checksum before startup, warming the OS file cache.
We do not describe this as a cold NVMe read benchmark or a drive's gigabytes-per-second figure. It also includes runtime initialization, weight loading and context allocation. Startup is worth measuring separately from an already running API, especially when an application changes models frequently.
Planning a test of your model
Record the exact model and revision, weights format, maximum input and output lengths, concurrent requests and KV-cache format. Then verify not only that the process starts, but also that the longest expected request completes and that the service behaves under load.
Q4_K_M in this series identifies a specific quantization variant; results do not automatically apply to FP16, BF16 or other formats. Capacity testing also does not assess answer quality. Qwen3-8B comes from the publisher's pinned repository revision. Check licensing separately for the model you choose.
See the API latency and concurrency results and the token-generation microbenchmark. If you already know your model and expected workload, ask about matching it to a server.