Running your own model does not have to begin with a full container platform. A first step can be simpler: download specific files, verify them, start a GPU process and connect through SSH. This guide uses Qwen3-8B in Q4_K_M format with llama.cpp.
It is a deployment walkthrough based on artifacts and settings used in our tests. It is not a model-quality comparison, an application security assessment or a complete public multi-user service. The API remains local to the server; an SSH tunnel makes it reachable from your computer.
We checked the one-slot configuration below: local /health and /completion returned HTTP 200, and the stream completed 32 generated tokens. We verified the process on the intended GPU and the offload of 37/37 layers to the GPU. This is a startup check, not a performance measurement or a test of a customer's internet connection.
Environment and scope
| Component | Configuration used in our test series |
|---|---|
| GPU | NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition, 96GB VRAM |
| Operating system | Rocky Linux 10.2, x86-64 |
| NVIDIA driver | 595.91.07 |
| System RAM | 96GB |
| Engine | llama.cpp 0.5.0-dev, build b11146, CUDA 12.8 |
| Model | Qwen3-8B-GGUF, Q4_K_M, 5,027,783,488-byte file |
Our benchmark methodology explains how measurements are taken.
We pin llama.cpp to commit 7fe450e19305b828c199d602c23a8337aaa1f03b. The project labels build b11146 a pre-release. It is selected to reproduce this series, not presented as the latest stable production release. Although the archives have Ubuntu in their names, these exact binaries and libraries ran on the stated Rocky environment. That does not establish compatibility between every Ubuntu package and every distribution.
Before downloading the model
You need a working GPU driver, an unprivileged account, Bash, curl, tar, sha256sum and SSH access. Run the preparation commands in the same Bash session on the server. Allow space for both downloaded archives and their extracted contents, as well as the model.
First check the GPU:
nvidia-smi --query-gpu=name,driver_version,memory.total,power.limit --format=csv
If this fails or does not show the intended card, resolve the driver installation first. Downloading CUDA libraries does not replace the kernel driver. The “CUDA Version” field in nvidia-smi is also not proof that the same toolkit version is installed: it describes driver support. See NVIDIA-SMI documentation and CUDA compatibility rules.
The following steps do not install system packages or change the driver, firmware or SSH configuration. LD_LIBRARY_PATH makes the extracted libraries available only to the process being launched.
Download pinned versions and verify checksums
Create a new private working directory. Running this command again creates another directory instead of overwriting the previous environment.
set -euo pipefail
umask 077
llm_dir="$(mktemp -d "$HOME/qwen3-api.XXXXXX")"
cd "$llm_dir"
mkdir downloads engine cuda models
Download the two official release archives: the application and its CUDA libraries. The --proto options restrict downloads and redirects to HTTPS.
llm_release='https://github.com/ggml-org/llama.cpp/releases/download/b11146'
curl --fail --location --proto '=https' --proto-redir '=https' \
"$llm_release/llama-b11146-bin-ubuntu-cuda-12.8-x64.tar.gz" \
--output downloads/llama.tar.gz
curl --fail --location --proto '=https' --proto-redir '=https' \
"$llm_release/cudart-llama-b11146-bin-ubuntu-cuda-12.8-x64.tar.gz" \
--output downloads/cudart.tar.gz
printf '%s %s\n' \
'c2ab9e19838513ff69d1af8d999ad717dd3c7ee4714ac04c7ed5ab9077c50e4e' downloads/llama.tar.gz \
'1466daea60aad1144819e151b2bae19d54556cf1da6c129c4f55a5ded2637c25' downloads/cudart.tar.gz \
| sha256sum --check
Both entries must report OK. Do not extract a file or continue after a download or checksum error. A checksum establishes a match with the pinned artifact; it does not replace trust in its publisher.
Download the model from its publisher's repository at revision 7c41481f57cb95916b40956ab2f0b139b296d974, rather than the changing main branch:
llm_model_source='https://huggingface.co/Qwen/Qwen3-8B-GGUF/resolve/7c41481f57cb95916b40956ab2f0b139b296d974'
curl --fail --location --proto '=https' --proto-redir '=https' \
"$llm_model_source/Qwen3-8B-Q4_K_M.gguf" \
--output models/Qwen3-8B-Q4_K_M.gguf
printf '%s %s\n' \
'd98cdcbd03e17ce47681435b5150e34c1417f50b5c0019dd560e4882c5745785' \
models/Qwen3-8B-Q4_K_M.gguf | sha256sum --check
curl --fail --location --proto '=https' --proto-redir '=https' \
"$llm_model_source/LICENSE" --output models/LICENSE
curl --fail --location --proto '=https' --proto-redir '=https' \
"$llm_model_source/README.md" --output models/README.md
Again, require OK before loading the model. Retain its license and model card. Q4_K_M identifies the chosen weight variant; it does not imply the same quality or size as other quantisations.
Extract the runtime and confirm CUDA access
Extract only the verified archives into the empty directories created earlier:
tar --extract --gzip --file downloads/llama.tar.gz \
--directory engine --no-same-owner --no-same-permissions
tar --extract --gzip --file downloads/cudart.tar.gz \
--directory cuda --no-same-owner --no-same-permissions
llm_server="$llm_dir/engine/llama-b11146/llama-server"
llm_libraries="$llm_dir/engine/llama-b11146:$llm_dir/cuda/cudart-llama-b11146-bin-ubuntu-cuda-12.8-x64"
env LD_LIBRARY_PATH="$llm_libraries" CUDA_VISIBLE_DEVICES=0 \
"$llm_server" --list-devices
The output should identify a CUDA device and the intended GPU. Merely starting a CPU process does not confirm GPU operation. If a library is missing, check both archives and the paths rather than copying unrelated .so files into system directories.
Bind the API only to server localhost
In the same session, start the process on the first GPU. We use one slot and an 8704-token context, with automatic parameter fitting disabled. This is a starting configuration for integration checks, not a setting tuned for every application.
env LD_LIBRARY_PATH="$llm_libraries" CUDA_VISIBLE_DEVICES=0 \
"$llm_server" \
-m "$llm_dir/models/Qwen3-8B-Q4_K_M.gguf" \
--host 127.0.0.1 --port 8080 \
-ngl 999 --fit off -fa on \
-t 16 -b 2048 -ub 512 -ctk f16 -ctv f16 \
-np 1 -c 8704 \
--cache-ram 0 --no-cache-idle-slots --no-context-shift
Leave this terminal open. After the earlier CUDA device check, run nvidia-smi in a separate SSH session: you should see the llama-server process and allocated GPU memory, with GPU activity during a request. This build's default logging level may omit the number of layers offloaded to the GPU; the absence of that line alone does not imply CPU-only execution. --host 127.0.0.1 matters here: do not replace it with 0.0.0.0 or open port 8080 on the router. The options and endpoints are described in the llama-server documentation for this revision.
Connect from your computer
In a second terminal on your own computer, start a tunnel. Replace gpu-server.example, the username, key path and port 22 with your server's details. Use the SSH port chosen at installation, including 3414 or a random port if applicable. Do not copy your private key to the server.
ssh -N -T -a \
-i "$HOME/.ssh/gpu-server" -p 22 \
-o ExitOnForwardFailure=yes \
-o ServerAliveInterval=30 -o ServerAliveCountMax=3 \
-L 127.0.0.1:18080:127.0.0.1:8080 \
gpu@gpu-server.example
Verify the host key through a known, trusted channel on the first connection. Do not disable host identity checking. The tunnel binds port 18080 only to your computer's localhost and forwards it to port 8080 as seen by the server. See the OpenSSH documentation for -L and -N.
In a third terminal, also on your own computer, check readiness:
curl --fail --show-error http://127.0.0.1:18080/health
Readiness means HTTP 200 with {"status":"ok"}. HTTP 503 can occur while the model is loading. Then send a short streaming request:
curl --fail --show-error --no-buffer \
http://127.0.0.1:18080/completion \
-H 'Content-Type: application/json' \
--data '{"prompt":"Write one sentence about GPU memory.","n_predict":32,"stream":true,"cache_prompt":false,"temperature":0}'
Here, /completion checks local generation and response transport. It is not a complete chat template or an assessment of model quality. The parameters of this short check are not recommended Qwen settings for every conversation. Choose your application's chat integration, template and generation settings using the Qwen model card and server documentation.
Troubleshooting
| Symptom | Next check |
|---|---|
| SHA-256 mismatch | Do not run the file; check the completed download and exact version URL. |
| Missing CUDA library or device | Check the driver, both archives and LD_LIBRARY_PATH; CPU execution is not GPU validation. |
| Tunnel reports an occupied port | Choose another local port instead of 18080 and use it in curl. |
| SSH denies forwarding | Check the account's forwarding policy with the administrator; do not work around it by exposing the API publicly. |
| HTTP 503 | Inspect the model-loading log; if the process exited, read its error instead of waiting indefinitely. |
| Out of memory | Check other GPU processes, context and slot count; change one setting at a time. |
To end the trial, press Ctrl+C in the server terminal and then in the tunnel terminal. No machine restart is needed. Downloads remain in the separate directory for a later run.
Security and licensing boundaries
Loopback plus a tunnel restricts network access, but does not authenticate other local processes or people with access to the account. Use test data at this stage. A persistent service needs a separate design for authentication, request limits, updates and log retention; making the API available to customers is another deployment stage, not a one-flag address change.
Qwen3-8B-GGUF uses Apache-2.0 in the pinned revision, while llama.cpp uses the MIT license. NVIDIA libraries have separate CUDA terms. Do not assume this model's license also covers other Qwen variants or other environment components. Weights are downloaded directly from their publisher; GPU Server Hub does not redistribute them through this guide.
To move from a trial to your own application, check the server configuration or tell us about your model, context and expected load. Our tests and guides distinguish a working installation from a measurement of its performance.