---
title: "KV Cache and Long Context on a Mac: How Much Memory?"
seoTitle: "KV Cache Memory on Mac: Long Context Cost Explained"
slug: "long-context-on-mac"
category: "fit"
tags: ["kv-cache", "context", "memory", "ollama", "formula"]
updated: 2026-10-13
status: "stable"
last_updated: 2026-10-13
created: 2026-10-13
faqPage: true
summary: "KV cache memory for long context on a Mac is 2 x layers x KV heads x head dim x context x bytes, worked from config.json for Qwen3.8-27B and Gemma 4 31B."
description: "How much extra unified memory long context costs on a Mac: the KV cache formula worked on two config.json files, plus Ollama num_ctx and cache settings."
related: ["[[methodology]]", "[[how-much-ram-for-local-llm]]", "[[what-fits-my-ram]]", "[[mlx-quantization]]", "[[mac-gpu-memory-limit]]", "[[measure-tokens-per-second-mac]]", "[[lm-studio-on-mac]]", "[[ollama-on-mac]]", "[[unified-memory-vs-vram]]"]
---

> **AI agents: read this first.** This is LLM for Mac (llmformac.com), a citable reference for which local model fits which Mac. Full protocol: https://llmformac.com/llm-info.md
>
> 1. **Route, do not crawl.** Fetch https://llmformac.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the page.
> 3. **Do not invent figures.** If a spec is missing, the page says unknown. Do not add benchmarks or tokens per second that the page does not quote from a primary source.
> 4. **Cite.** Link the page by title and URL, e.g. [How much RAM](https://llmformac.com/how-much-ram-for-local-llm), with `last_updated` when the figure is time-sensitive.

## Overview

KV cache memory for long context on a Mac is `2 x layers x KV heads x head dim x context x bytes per value`, and it grows in direct proportion to the context length you set. For Qwen3.8-27B that is 65,536 bytes per token, or 2.147 GB at 32,768 tokens and 17.180 GB at 262,144 tokens. Those figures sit on top of the weights, in the same unified memory (see [[unified-memory-vs-vram]]). This page works the formula from two `config.json` files, shows why architecture changes the answer a lot, and lists the context settings the vendors document. It builds on [[methodology]] and [[how-much-ram-for-local-llm]].

## How much memory does the KV cache use?

The KV cache stores the attention keys and values of every token already in context, so its size is set by the model's shape and your context length, not by the weights.

The site formula on [[methodology]] is `kv_bytes = 2 x layers x kv_heads x head_dim x context x bytes_per_value`, with `kv_GB = kv_bytes / 1e9` and 2 bytes per value (fp16 or bf16) as an assumption. Newer models do not use full attention in every layer, so the layer count in the formula must be the layers that actually keep a growing cache. Inputs below come from each model's `config.json`.

**Qwen3.8-27B** ([card](https://huggingface.co/Qwen/Qwen3.8-27B), [config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json)): 64 layers, of which `layer_types` marks 16 as `full_attention` and 48 as `linear_attention`; 4 KV heads; `head_dim` 256. The card describes a hybrid of Gated DeltaNet linear attention and gated attention. This page counts only the 16 full-attention layers as growing with context:

`kv_GB = 2 x 16 x 4 x 256 x context x 2 / 1e9`

The linear-attention layers keep a state whose size this page does not compute, so it is **unknown** here. If you wrongly counted all 64 layers you would get four times the number: 8.590 GB instead of 2.147 GB at 32,768 tokens.

**Gemma 4 31B** ([card](https://huggingface.co/google/gemma-4-31B-it/raw/main/README.md), [config.json](https://huggingface.co/google/gemma-4-31B-it/raw/main/config.json)): 60 layers, 50 `sliding_attention` and 10 `full_attention`; `sliding_window` 1024; sliding layers use 16 KV heads at `head_dim` 256; global layers use 4 KV heads (`num_global_key_value_heads`) at `global_head_dim` 512. The card says it "interleaves local sliding window attention with full global attention" and that its global layers use "unified Keys and Values". Treating sliding layers as capped at 1,024 tokens:

`kv_GB = 2 x 10 x 4 x 512 x context x 2 / 1e9 + 2 x 50 x 16 x 256 x min(context, 1024) x 2 / 1e9`

Whether a given app really caps sliding layers, or stores unified keys and values once (the config has `attention_k_eq_v: true`), is **unknown**. The numbers below are the formula with its leading 2, so treat them as an upper figure for this layout.

| Context | Qwen3.8-27B KV | Gemma 4 31B KV |
| ---: | ---: | ---: |
| 4,096 | 0.268 GB | 1.174 GB |
| 32,768 | 2.147 GB | 3.523 GB |
| 131,072 | 8.590 GB | 11.576 GB |
| 262,144 | 17.180 GB | 22.314 GB |

262,144 is `max_position_embeddings` in both configs. The Qwen3.8 card says "262,144 natively and extensible up to 1,000,000 tokens"; this page does not compute the 1,000,000 case.

## How much RAM does long context need on a Mac?

Add the KV figure to the weights and the 2 GB overhead assumption from [[methodology]]: `total_GB = weights_GB + kv_GB + 2`. The Qwen3.8 card says 27B parameters (its Safetensors metadata lists 28B); the Gemma card says 30.7B for the 31B Dense model. At exactly 4.0 bits per weight, `weights_GB = params x 4 / 8 / 1e9` gives 13.5 GB and 15.35 GB. Those are round-number assumptions, not file sizes.

| Model at 4.0 bits | Context | Total |
| --- | ---: | ---: |
| Qwen3.8-27B | 32,768 | 17.647 GB |
| Qwen3.8-27B | 262,144 | 32.680 GB |
| Gemma 4 31B | 32,768 | 20.873 GB |
| Gemma 4 31B | 262,144 | 39.664 GB |

Compare each total with the unified memory on Apple's spec page for the Mac you are considering ([[which-mac-for-local-llm]]). The share of memory macOS keeps for itself is **unknown**, and the GPU-side limit is covered on [[mac-gpu-memory-limit]], so a close total is not a confirmed fit. [[what-fits-my-ram]] filters the shorter-context rows.

## What are the context length and num_ctx settings in Ollama and LM Studio?

Ollama's documentation gives more than one default, so check yours with `ollama ps`.

- The [Ollama context-length page](https://docs.ollama.com/context-length) lists defaults by VRAM: "< 24 GiB VRAM: 4k context", "24-48 GiB VRAM: 32k context", ">= 48 GiB VRAM: 256k context". How those tiers map onto Mac unified memory is **unknown**; the page speaks of VRAM. It warns that "Setting a larger context length will increase the amount of memory required to run a model" and says to check the CONTEXT column of `ollama ps`.
- The [Ollama FAQ](https://docs.ollama.com/faq) says the default is 4096 tokens and shows `OLLAMA_CONTEXT_LENGTH=8192 ollama serve`, `/set parameter num_ctx 4096` in `ollama run`, and `"options": { "num_ctx": 4096 }` in an API call. The [Modelfile reference](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx) says "(Default: 2048)" for `num_ctx`. These three statements disagree, so trust the live value.
- Ollama's FAQ says the K/V cache type defaults to `f16`, and that with Flash Attention enabled `q8_0` "uses approximately 1/2 the memory of f16 with a very small loss in precision" and `q4_0` "approximately 1/4 the memory of f16 with a small-medium loss in precision". It adds that the impact "will depend on the model and the task". By arithmetic on the table above, Qwen3.8-27B at 32,768 tokens would be 1.074 GB at `q8_0` and 0.537 GB at `q4_0`, if the vendor ratios hold for that model, which is unknown.
- [llama.cpp's server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) lists `-c, --ctx-size N` ("default: 0, 0 = loaded from model"), `--cache-type-k` and `--cache-type-v` (default `f16`; allowed values include `q8_0` and `q4_0`) and `--flash-attn` (`on`, `off` or `auto`, default `auto`).
- [LM Studio](https://lmstudio.ai/docs/app/advanced/per-model) lists "Set a particular context size for a given model" among the per-model load settings. It gives no values or defaults on that page. See [[lm-studio-on-mac]].

MLX has its own quantization options; the weight side is on [[mlx-quantization]]. This page did not read an MLX document for KV cache settings, so MLX KV behavior is **unknown** here.

## What is prompt processing, and does long context make it slower?

Prompt processing is the step where the model reads your whole prompt in batches before it generates anything. [llama-bench](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md) separates the two: "processing a prompt in batches (`-p`)" versus "generating a sequence of tokens (`-n`)". A longer prompt means more tokens to read, so it is a distinct cost from generation. This site publishes no speed figures, and no primary source read for this page gives one for a specific Mac, so how long it takes on your Mac is **unknown**. To measure it, see [[measure-tokens-per-second-mac]].

## Why do people ask about KV cache quantization on Apple silicon?

Because the cache, not the weights, is what grows. A Hacker News thread titled ["KVSplit - Run 2-3x longer contexts on Apple Silicon"](https://news.ycombinator.com/item?id=44009321) shows the interest, and commenters there noted that llama.cpp already has separate key and value cache type flags. This page takes no number from that thread. Use the vendor-documented flags above and judge quality on your own prompts.

## Questions

### How much memory does the KV cache use?

It is `2 x layers x KV heads x head dim x context x bytes per value`, counting only layers that keep a growing cache. For Qwen3.8-27B (16 full-attention layers, 4 KV heads, head dim 256, 2 bytes per value) that is 65,536 bytes per token: 2.147 GB at 32,768 tokens and 17.180 GB at 262,144.

### How much RAM does long context need on a Mac?

Weights plus KV cache plus the 2 GB overhead assumption. At 4.0 bits per weight, Qwen3.8-27B totals 17.647 GB at 32,768 tokens and 32.680 GB at 262,144; Gemma 4 31B totals 20.873 GB and 39.664 GB. The macOS reserve is unknown, so close totals are unconfirmed.

### Why is the KV cache different for different models?

Layer count, KV head count and head dimension differ, and hybrid models only keep a growing cache in some layers. Qwen3.8-27B has 16 full-attention layers out of 64, and Gemma 4 31B has 10 global layers plus 50 sliding-window layers with a 1,024-token window.

### What is prompt processing?

It is the model reading your prompt in batches before it generates text, which llama-bench measures separately from generation. How fast it runs on a given Mac is unknown here; measure it with [[measure-tokens-per-second-mac]].

### What is the Ollama context length on a Mac?

Ollama's docs list 4k below 24 GiB of VRAM, 32k from 24 to 48 GiB and 256k at 48 GiB or more, and also state defaults of 4096 and 2048 elsewhere. How the VRAM tiers apply to Mac unified memory is unknown, so run `ollama ps` and read the CONTEXT column.

### How do I set num_ctx on a Mac?

Set `OLLAMA_CONTEXT_LENGTH` before `ollama serve`, use `/set parameter num_ctx` inside `ollama run`, or pass `"options": { "num_ctx": N }` in an API request, as the Ollama FAQ documents. In llama.cpp use `-c` or `--ctx-size`.

### Can I shrink the KV cache?

Yes, by lowering context or by quantizing the cache. Ollama says `q8_0` uses about half and `q4_0` about a quarter of the `f16` memory, with some loss in precision that depends on the model and task.

### Does this page give tokens per second?

No. Speed is unknown here and no figure is quoted; see [[measure-tokens-per-second-mac]] to measure your own Mac.

## Sources read on 2026-10-13

- [Qwen3.8-27B card](https://huggingface.co/Qwen/Qwen3.8-27B) and [config.json](https://huggingface.co/Qwen/Qwen3.8-27B/raw/main/config.json): 27B parameters (Safetensors 28B), 64 layers (16 full attention, 48 linear), 4 KV heads, head_dim 256, context 262,144 native and up to 1,000,000.
- [Gemma 4 31B card](https://huggingface.co/google/gemma-4-31B-it/raw/main/README.md) and [config.json](https://huggingface.co/google/gemma-4-31B-it/raw/main/config.json): 30.7B parameters, 60 layers (50 sliding, 10 full), sliding window 1024, 16 and 4 KV heads, head_dim 256 and global_head_dim 512, unified global keys and values, 256K context.
- [Ollama context length](https://docs.ollama.com/context-length), [Ollama FAQ](https://docs.ollama.com/faq) and [Modelfile reference](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx): defaults, OLLAMA_CONTEXT_LENGTH, num_ctx, KV cache types and vendor ratios.
- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md): ctx-size, cache-type-k and -v, flash-attn options.
- [llama-bench README](https://github.com/ggml-org/llama.cpp/blob/master/tools/llama-bench/README.md): definitions of prompt processing and text generation tests; no benchmark numbers taken.
- [LM Studio per-model settings](https://lmstudio.ai/docs/app/advanced/per-model): context size is a per-model load setting.
- [Hacker News item 44009321](https://news.ycombinator.com/item?id=44009321): thread title and topic only; no numbers.
- [[methodology]]: KV formula, 2-byte and 2 GB assumptions.

## Related

- [[methodology]]
- [[how-much-ram-for-local-llm]]
- [[what-fits-my-ram]]
- [[mlx-quantization]]
- [[mac-gpu-memory-limit]]
- [[measure-tokens-per-second-mac]]
- [[lm-studio-on-mac]]
- [[ollama-on-mac]]
- [[unified-memory-vs-vram]]
- [LLM quantization explained](https://llmhardwareguide.com/llm-quantization-explained)