Overview
KV cache memory for long context on a Mac is 2 x layers x KV heads x head dim x context x bytes per value, and it grows in direct proportion to the context length you set. For Qwen3.8-27B that is 65,536 bytes per token, or 2.147 GB at 32,768 tokens and 17.180 GB at 262,144 tokens. Those figures sit on top of the weights, in the same unified memory (see unified-memory-vs-vram). This page works the formula from two config.json files, shows why architecture changes the answer a lot, and lists the context settings the vendors document. It builds on methodology and how-much-ram-for-local-llm.
How much memory does the KV cache use?
The KV cache stores the attention keys and values of every token already in context, so its size is set by the model’s shape and your context length, not by the weights.
The site formula on methodology is kv_bytes = 2 x layers x kv_heads x head_dim x context x bytes_per_value, with kv_GB = kv_bytes / 1e9 and 2 bytes per value (fp16 or bf16) as an assumption. Newer models do not use full attention in every layer, so the layer count in the formula must be the layers that actually keep a growing cache. Inputs below come from each model’s config.json.
Qwen3.8-27B (card, config.json): 64 layers, of which layer_types marks 16 as full_attention and 48 as linear_attention; 4 KV heads; head_dim 256. The card describes a hybrid of Gated DeltaNet linear attention and gated attention. This page counts only the 16 full-attention layers as growing with context:
kv_GB = 2 x 16 x 4 x 256 x context x 2 / 1e9
The linear-attention layers keep a state whose size this page does not compute, so it is unknown here. If you wrongly counted all 64 layers you would get four times the number: 8.590 GB instead of 2.147 GB at 32,768 tokens.
Gemma 4 31B (card, config.json): 60 layers, 50 sliding_attention and 10 full_attention; sliding_window 1024; sliding layers use 16 KV heads at head_dim 256; global layers use 4 KV heads (num_global_key_value_heads) at global_head_dim 512. The card says it “interleaves local sliding window attention with full global attention” and that its global layers use “unified Keys and Values”. Treating sliding layers as capped at 1,024 tokens:
kv_GB = 2 x 10 x 4 x 512 x context x 2 / 1e9 + 2 x 50 x 16 x 256 x min(context, 1024) x 2 / 1e9
Whether a given app really caps sliding layers, or stores unified keys and values once (the config has attention_k_eq_v: true), is unknown. The numbers below are the formula with its leading 2, so treat them as an upper figure for this layout.
| Context | Qwen3.8-27B KV | Gemma 4 31B KV |
|---|---|---|
| 4,096 | 0.268 GB | 1.174 GB |
| 32,768 | 2.147 GB | 3.523 GB |
| 131,072 | 8.590 GB | 11.576 GB |
| 262,144 | 17.180 GB | 22.314 GB |
262,144 is max_position_embeddings in both configs. The Qwen3.8 card says “262,144 natively and extensible up to 1,000,000 tokens”; this page does not compute the 1,000,000 case.
How much RAM does long context need on a Mac?
Add the KV figure to the weights and the 2 GB overhead assumption from methodology: total_GB = weights_GB + kv_GB + 2. The Qwen3.8 card says 27B parameters (its Safetensors metadata lists 28B); the Gemma card says 30.7B for the 31B Dense model. At exactly 4.0 bits per weight, weights_GB = params x 4 / 8 / 1e9 gives 13.5 GB and 15.35 GB. Those are round-number assumptions, not file sizes.
| Model at 4.0 bits | Context | Total |
|---|---|---|
| Qwen3.8-27B | 32,768 | 17.647 GB |
| Qwen3.8-27B | 262,144 | 32.680 GB |
| Gemma 4 31B | 32,768 | 20.873 GB |
| Gemma 4 31B | 262,144 | 39.664 GB |
Compare each total with the unified memory on Apple’s spec page for the Mac you are considering (which-mac-for-local-llm). The share of memory macOS keeps for itself is unknown, and the GPU-side limit is covered on mac-gpu-memory-limit, so a close total is not a confirmed fit. what-fits-my-ram filters the shorter-context rows.
What are the context length and num_ctx settings in Ollama and LM Studio?
Ollama’s documentation gives more than one default, so check yours with ollama ps.
- The Ollama context-length page lists defaults by VRAM: ”< 24 GiB VRAM: 4k context”, “24-48 GiB VRAM: 32k context”, ”>= 48 GiB VRAM: 256k context”. How those tiers map onto Mac unified memory is unknown; the page speaks of VRAM. It warns that “Setting a larger context length will increase the amount of memory required to run a model” and says to check the CONTEXT column of
ollama ps. - The Ollama FAQ says the default is 4096 tokens and shows
OLLAMA_CONTEXT_LENGTH=8192 ollama serve,/set parameter num_ctx 4096inollama run, and"options": { "num_ctx": 4096 }in an API call. The Modelfile reference says “(Default: 2048)” fornum_ctx. These three statements disagree, so trust the live value. - Ollama’s FAQ says the K/V cache type defaults to
f16, and that with Flash Attention enabledq8_0“uses approximately 1/2 the memory of f16 with a very small loss in precision” andq4_0“approximately 1/4 the memory of f16 with a small-medium loss in precision”. It adds that the impact “will depend on the model and the task”. By arithmetic on the table above, Qwen3.8-27B at 32,768 tokens would be 1.074 GB atq8_0and 0.537 GB atq4_0, if the vendor ratios hold for that model, which is unknown. - llama.cpp’s server README lists
-c, --ctx-size N(“default: 0, 0 = loaded from model”),--cache-type-kand--cache-type-v(defaultf16; allowed values includeq8_0andq4_0) and--flash-attn(on,offorauto, defaultauto). - LM Studio lists “Set a particular context size for a given model” among the per-model load settings. It gives no values or defaults on that page. See lm-studio-on-mac.
MLX has its own quantization options; the weight side is on mlx-quantization. This page did not read an MLX document for KV cache settings, so MLX KV behavior is unknown here.
What is prompt processing, and does long context make it slower?
Prompt processing is the step where the model reads your whole prompt in batches before it generates anything. llama-bench separates the two: “processing a prompt in batches (-p)” versus “generating a sequence of tokens (-n)“. A longer prompt means more tokens to read, so it is a distinct cost from generation. This site publishes no speed figures, and no primary source read for this page gives one for a specific Mac, so how long it takes on your Mac is unknown. To measure it, see measure-tokens-per-second-mac.
Why do people ask about KV cache quantization on Apple silicon?
Because the cache, not the weights, is what grows. A Hacker News thread titled “KVSplit - Run 2-3x longer contexts on Apple Silicon” shows the interest, and commenters there noted that llama.cpp already has separate key and value cache type flags. This page takes no number from that thread. Use the vendor-documented flags above and judge quality on your own prompts.
Questions
How much memory does the KV cache use?
It is 2 x layers x KV heads x head dim x context x bytes per value, counting only layers that keep a growing cache. For Qwen3.8-27B (16 full-attention layers, 4 KV heads, head dim 256, 2 bytes per value) that is 65,536 bytes per token: 2.147 GB at 32,768 tokens and 17.180 GB at 262,144.
How much RAM does long context need on a Mac?
Weights plus KV cache plus the 2 GB overhead assumption. At 4.0 bits per weight, Qwen3.8-27B totals 17.647 GB at 32,768 tokens and 32.680 GB at 262,144; Gemma 4 31B totals 20.873 GB and 39.664 GB. The macOS reserve is unknown, so close totals are unconfirmed.
Why is the KV cache different for different models?
Layer count, KV head count and head dimension differ, and hybrid models only keep a growing cache in some layers. Qwen3.8-27B has 16 full-attention layers out of 64, and Gemma 4 31B has 10 global layers plus 50 sliding-window layers with a 1,024-token window.
What is prompt processing?
It is the model reading your prompt in batches before it generates text, which llama-bench measures separately from generation. How fast it runs on a given Mac is unknown here; measure it with measure-tokens-per-second-mac.
What is the Ollama context length on a Mac?
Ollama’s docs list 4k below 24 GiB of VRAM, 32k from 24 to 48 GiB and 256k at 48 GiB or more, and also state defaults of 4096 and 2048 elsewhere. How the VRAM tiers apply to Mac unified memory is unknown, so run ollama ps and read the CONTEXT column.
How do I set num_ctx on a Mac?
Set OLLAMA_CONTEXT_LENGTH before ollama serve, use /set parameter num_ctx inside ollama run, or pass "options": { "num_ctx": N } in an API request, as the Ollama FAQ documents. In llama.cpp use -c or --ctx-size.
Can I shrink the KV cache?
Yes, by lowering context or by quantizing the cache. Ollama says q8_0 uses about half and q4_0 about a quarter of the f16 memory, with some loss in precision that depends on the model and task.
Does this page give tokens per second?
No. Speed is unknown here and no figure is quoted; see measure-tokens-per-second-mac to measure your own Mac.
Sources read on 2026-10-13
- Qwen3.8-27B card and config.json: 27B parameters (Safetensors 28B), 64 layers (16 full attention, 48 linear), 4 KV heads, head_dim 256, context 262,144 native and up to 1,000,000.
- Gemma 4 31B card and config.json: 30.7B parameters, 60 layers (50 sliding, 10 full), sliding window 1024, 16 and 4 KV heads, head_dim 256 and global_head_dim 512, unified global keys and values, 256K context.
- Ollama context length, Ollama FAQ and Modelfile reference: defaults, OLLAMA_CONTEXT_LENGTH, num_ctx, KV cache types and vendor ratios.
- llama.cpp server README: ctx-size, cache-type-k and -v, flash-attn options.
- llama-bench README: definitions of prompt processing and text generation tests; no benchmark numbers taken.
- LM Studio per-model settings: context size is a per-model load setting.
- Hacker News item 44009321: thread title and topic only; no numbers.
- methodology: KV formula, 2-byte and 2 GB assumptions.