You do not need the full technical vocabulary to describe your idea. This short glossary helps explain what an AI application needs and what the numbers in our tests mean.
GPU and CPU
A GPU, or graphics processing unit, can perform many similar calculations in parallel. That makes it useful for AI models as well as graphics. The CPU is the server’s main processor: it runs the operating system, applications and some data preparation. See how the work is divided.
LLM and names such as 8B, 32B and 72B
An LLM is a large language model: software trained to work with text. The B in a model name refers to billions of parameters, not gigabytes of memory. More parameters do not automatically guarantee better answers for every task.
MoE: a model with experts
A mixture-of-experts model divides some calculations among modules called experts. Only a selection works on each new piece of text, a little like a team assigning a task to selected people. This describes the division of work, not evidence that the model thinks like a person. Fewer active parameters can reduce computation, but the rest of the model data must still be available. The active parameter count therefore does not describe the whole model's memory footprint.
Token
A token is a piece of text used by the model: it can be a word, part of a word or a symbol. A thousand tokens therefore need not mean a thousand words. The count depends on the language, content and the model’s method of dividing text.
Context
Context is the text a model uses to prepare an answer: instructions, a question, documents and conversation history. The context limit also includes space for the generated answer. Longer context can need more memory and time. See our measurements for different input lengths.
RAM and VRAM
RAM is the server’s system memory; VRAM is memory on the GPU. They are not one shared pool. VRAM must hold more than model data: the running model also needs working memory. Compare measured configurations and explore the role of RAM.
API
An API is a way for programs to communicate. Your application sends the model a task and receives the result instead of requiring someone to work manually in a chat window. Our private API guide provides a working example.
TTFT: time to first token
TTFT measures how long it takes to receive the first generated piece of text. It differs from the time needed to finish the whole response. Some models first generate reasoning content that an application does not display as the answer. For those models, we distinguish the start of generation from the first visible part of the answer shown to the user.
Tokens per second, prefill and decode
Tokens per second describe text-processing speed. Prefill processes the input; decode generates successive parts of the answer. The rate for one request differs from the aggregate throughput of several requests: these are not interchangeable measures.
Concurrency
Concurrency is the number of tasks running at the same time. Four concurrent requests do not mean a limit of four user accounts. More simultaneous tasks can increase total throughput while making an individual user wait longer.
Quantisation and Q4_K_M
Quantisation represents the numbers describing a model at lower precision, which can reduce file size and memory requirements. Q4_K_M is one such format. A change of format can affect speed and answer quality, so comparisons should identify the exact model version.
KV cache
KV cache is working memory that stores information about text the model has already processed. It helps avoid recalculating everything for each new token. Its capacity becomes more important with long documents and several concurrent requests.
Offloading: sharing the work
Offloading moves some calculations or data between CPU and GPU. A model can, for example, use GPU VRAM and the server’s RAM together. That does not turn system RAM into equally fast GPU memory: capacity and speed need to be checked separately.
Our same-model comparison with different placements measures the effect on response time. The larger 235B model test applies this approach to a mixture-of-experts model.
From terminology to your use case
Start with what the application should do, how much text it needs to process and how many tasks it should handle at once. Our tests and guides help translate that into a configuration. You can also explore the server offer or tell us about your idea.