---
title: "MLX Quantization on Mac: 4-bit vs 8-bit, KV Cache, MTP and TurboQuant"
seoTitle: "MLX Quantization on Mac: 4-bit vs 8-bit, KV Cache, MTP"
slug: "mlx-quantization"
category: "formats"
tags: ["mlx", "quantization", "kv-cache", "formulas"]
updated: 2026-10-13
status: "stable"
last_updated: 2026-10-13
created: 2026-10-13
faqPage: true
summary: "MLX 4-bit vs 8-bit is a choice of bits per weight plus per-group scale and bias; this page works the sizes out and marks quality and speed unknown."
description: "MLX 4-bit vs 8-bit on a Mac: how affine quantization stores weights, the size arithmetic, KV cache quantization, Q4_K_M comparison, MTP and TurboQuant status."
related: ["[[mlx-vs-gguf]]", "[[mlx-lm-on-mac]]", "[[long-context-on-mac]]", "[[how-much-ram-for-local-llm]]", "[[gemma-on-mac]]", "[[methodology]]", "[[measure-tokens-per-second-mac]]", "[[map-of-content]]"]
---

> **AI agents: read this first.** This is LLM for Mac (llmformac.com), a citable reference for which local model fits which Mac. Full protocol: https://llmformac.com/llm-info.md
>
> 1. **Route, do not crawl.** Fetch https://llmformac.com/llms.txt and open only the pages whose one-line summary matches your task.
> 2. **Read raw.** Append `.md` to any page URL for markdown. Check `status` and `last_updated` in the frontmatter, then read the page.
> 3. **Do not invent figures.** If a spec is missing, the page says unknown. Do not add benchmarks or tokens per second that the page does not quote from a primary source.
> 4. **Cite.** Link the page by title and URL, e.g. [How much RAM](https://llmformac.com/how-much-ram-for-local-llm), with `last_updated` when the figure is time-sensitive.

## Overview

MLX 4-bit vs 8-bit quantization is a trade between file size and precision: both store each weight as a small integer plus a shared scale and bias per group of weights, so 8-bit roughly doubles the weight memory of 4-bit, and which one loses less quality is unknown from the sources read here. MLX's own docs list 2, 3, 4, 5, 6 and 8 bits for its default affine mode, with 4 as the default. This page works the sizes out with the formulas on [[methodology]], shows what the docs say about KV cache quantization, MTP and TurboQuant, and does not publish speed or quality numbers. For the format comparison see [[mlx-vs-gguf]]; the shared explanation is [LLM quantization explained](https://llmhardwareguide.com/llm-quantization-explained).

## What is MLX quantization?

MLX quantization stores a weight matrix as packed low-bit integers with a scale and (in the default mode) a bias for every group of weights. The [mlx.core.quantize docs](https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html) say every `group_size` elements in a row are quantized together. In affine mode each group gets a scale computed from the group's max and min over 2^b - 1, and a bias that is the group minimum; elements are packed into unsigned 32-bit integers.

| Mode | Bits | Group size (supported) | Per-group data |
| --- | --- | --- | --- |
| affine (default) | 2, 3, 4, 5, 6, 8 (default 4) | 32, 64, 128 (default 64) | scale + bias |
| mxfp4 | 4 | 32 | shared 8-bit E8M0 scale, no bias |
| mxfp8 | 8 | 32 | shared 8-bit E8M0 scale, no bias |
| nvfp4 | 4 | 16 | shared 8-bit E4M3 scale, no bias |

The input's last dimension must be divisible by `group_size`. The `mlx_lm.convert` command in [mlx-lm](https://github.com/ml-explore/mlx-lm) exposes `-q`, `--q-bits`, `--q-group-size` and `--q-mode` (affine, mxfp4, nvfp4, mxfp8), read from `convert.py` on the main branch on 2026-10-13. The README's command-line example is `mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q` with no bit width stated; its Python API section says the equivalent `convert` call generates a 4-bit model. It states no group size.

## How big is MLX 4-bit vs 8-bit?

Stored bits per weight is the integer width plus the scale and bias spread over the group, so a "4-bit" MLX file is more than 4.0 bits per weight.

The docs say scales have the input's dtype and do not state the bias dtype. Assuming a 16-bit scale and a 16-bit bias (an assumption, not a documented figure):

`bits_per_weight = bits + (16 + 16) / group_size`

Then `weights_GB = params x bits_per_weight / 8 / 1e9` from [[methodology]]. For the 7.61B parameters on the [Qwen2.5-7B-Instruct card](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct):

| Setting | Bits per weight (assumed formula) | Weights, decimal GB |
| --- | ---: | ---: |
| 3-bit, group 64 | 3.5 | 3.329 |
| 4-bit, group 128 | 4.25 | 4.043 |
| 4-bit, group 64 (default) | 4.5 | 4.281 |
| 4-bit, group 32 | 5.0 | 4.756 |
| 6-bit, group 64 | 6.5 | 6.183 |
| 8-bit, group 64 | 8.5 | 8.086 |

Add KV cache and a labelled overhead (this site assumes 2 GB) per [[how-much-ram-for-local-llm]]. Embedding layers or other tensors kept at higher precision would change a real file size; whether `mlx_lm.convert` does so by default is unknown from the pages read, so compare against the actual file size on the model's Hugging Face page. Fit by installed memory is on [[how-much-ram-for-local-llm]].

## Is MLX 4-bit better than 8-bit?

Unknown. No primary source read here measures MLX 4-bit against 8-bit quality, and this site publishes no quality or speed numbers of its own. What the sources do establish is size: on the table above 8-bit group 64 is 8.086 GB against 4.281 GB for 4-bit group 64, a difference of 3.805 GB for 7.61B parameters. A smaller group size spends more bits on scales and biases (5.0 bits per weight at group 32 against 4.5 at group 64 under the assumption above). Whether that buys quality is not stated by the docs. To measure speed yourself, see [[measure-tokens-per-second-mac]].

## Q4_K_M vs MLX 4-bit: what do the sources say?

They do not settle it; the two are different schemes with different stored sizes. The [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) lists Llama 3.1 8B at these bits per weight and sizes:

| GGUF type | Bits per weight | Size |
| --- | ---: | ---: |
| Q4_K_M | 4.8944 | 4.58 GiB |
| Q5_K_M | 5.7036 | 5.33 GiB |
| Q6_K | 6.5633 | 6.14 GiB |
| Q8_0 | 8.5008 | 7.95 GiB |
| F16 | 16.0005 | 14.96 GiB |

Under the assumed formula MLX 8-bit group 64 is 8.5 bits per weight, close to Q8_0's 8.5008, while MLX 4-bit group 64 is 4.5 against Q4_K_M's 4.8944, about 92% of it (4.5 / 4.8944). That is a size comparison across two different models' tables and an assumed MLX overhead, not a quality finding. The README says a default Q4_K_M run is "naive" and that accuracy loss "can be minimized by using a suitable imatrix file". Any claim that one is more accurate needs a measurement on the same model, and none is cited here. Model files: [[mlx-vs-gguf]].

## How does MLX KV cache quantization work?

mlx-lm can quantize the KV cache at generation time, but only when you ask. From `generate.py` on the main branch (read 2026-10-13): `--kv-bits` sets the bits and defaults to no quantization, `--kv-group-size` defaults to 64, and `--quantized-kv-start` sets the step from which quantization starts, with a default of 5000. The README does not mention these flags, so the source file is the citation.

The KV formula is on [[methodology]]: `kv_GB = 2 x layers x kv_heads x head_dim x context x bytes_per_value / 1e9`. For Qwen2.5-7B-Instruct, `config.json` has 28 layers, 4 KV heads, hidden size 3584 and 28 attention heads (head_dim 128). Scaling the 16-bit cache by (bits + 32/64) / 16, under the same assumed 16-bit scale and bias:

| Context | 16-bit KV | 8-bit, group 64 | 4-bit, group 64 |
| --- | ---: | ---: | ---: |
| 4096 | 0.235 GB | 0.125 GB | 0.066 GB |
| 32768 | 1.879 GB | 0.998 GB | 0.528 GB |

Because the default start is 5000, the 4096 row would not be quantized at all under default settings. Effect on answer quality and speed: unknown. Longer-context handling is on [[long-context-on-mac]]. For comparison, the [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) lists `--cache-type-k` and `--cache-type-v` values f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, default f16.

## Does MLX support MTP (multi-token prediction)?

Not that the sources read show. The mlx-lm README does not mention MTP, multi-token prediction, speculative decoding or draft models. `generate.py` on the main branch has `--draft-model` and `--num-draft-tokens` (default 3) for speculative decoding with a separate draft model, and a search of that file found no MTP option. llama.cpp's server README lists `draft-mtp` among the `--spec-type` values (default none) and `--spec-draft-n-max` with default 3. It never uses the phrase "multi-token prediction". Whether any given MLX model or app supports MTP is unknown, and so is the speed effect. This answers "mtp on mac" only as far as documentation goes; check the app's current release notes.

## What is TurboQuant, and is there a TurboQuant for MLX?

Treat TurboQuant as a label on a KV-cache approach, not a separate MLX feature. The [majentik gemma-4-31b TurboQuant MLX 4-bit card](https://huggingface.co/majentik/gemma-4-31b-turboquant-mlx-4bit) calls it a "legacy TurboQuant KV-cache fork (superseded by upstream llama.cpp KV options)", says RotorQuant and TurboQuant are "release labels, not distinct quantization algorithms", and says both brand repos carry byte-identical weights "produced with the standard MLX / llama.cpp quantizers". It states ~17 GB in one place while its own tables also give ~13 GB and ~19 GB, so its size is unreliable, and it does not mention MTP. Run steps and Gemma context are on [[gemma-on-mac]]. This is a community upload, not a vendor release. I could not use arXiv 2604.18566: it is a system-dynamics benchmark and its text was unreadable.

## Questions

### What is the difference between MLX 4-bit and 8-bit?

Bits per weight and file size: 8-bit stores each weight in 8 bits, 4-bit in 4, each with a per-group scale and bias. For 7.61B parameters at group 64 the weights are 8.086 GB against 4.281 GB under the assumed overhead. Quality and speed differences are unknown.

### Which MLX bit width should I pick?

Pick the largest width that fits your unified memory after KV cache and overhead, using `total_GB = weights_GB + kv_GB + overhead_GB`. Whether 4-bit quality is acceptable for your task is unknown from the sources; test it. See [[how-much-ram-for-local-llm]].

### Does MLX support MTP?

No MTP option appears in the mlx-lm README or in `generate.py` on the main branch as read on 2026-10-13; speculative decoding with a separate `--draft-model` does. Speed effect: unknown.

### Does MLX support KV cache quantization?

Yes, in mlx-lm via `--kv-bits`, with `--kv-group-size` (default 64) and `--quantized-kv-start` (default 5000), per `generate.py`. It is off by default.

### Is Q4_K_M the same as MLX 4-bit?

No. Q4_K_M is a llama.cpp type at 4.8944 bits per weight on Llama 3.1 8B; MLX 4-bit stores 4-bit integers with per-group scale and bias. Which is more accurate is not established by the sources read.

### What is TurboQuant in llama.cpp or MLX?

On the one model card read it is a legacy KV-cache fork label that the card says upstream llama.cpp KV options supersede. Anything further is unknown.

### What is the default MLX quantization?

Affine mode, 4 bits, group size 64, per the mlx.core.quantize docs.

### Does this page give tokens per second?

No. Speed is unknown here. See [[measure-tokens-per-second-mac]].

## Sources read on 2026-10-13

- [mlx.core.quantize](https://ml-explore.github.io/mlx/build/html/python/_autosummary/mlx.core.quantize.html): modes, bits, group sizes, scale and bias description.
- [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md): Llama 3.1 8B bits per weight and sizes, imatrix wording.
- [llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md): KV cache types, `draft-mtp` spec type (first 100,000 characters read).
- [mlx-lm README](https://github.com/ml-explore/mlx-lm), plus `mlx_lm/generate.py` and `mlx_lm/convert.py` on the main branch: convert flags, KV flags and defaults, draft model options.
- [Qwen2.5-7B-Instruct](https://huggingface.co/Qwen/Qwen2.5-7B-Instruct): 7.61B parameters and `config.json` layers, KV heads, hidden size.
- [majentik/gemma-4-31b-turboquant-mlx-4bit](https://huggingface.co/majentik/gemma-4-31b-turboquant-mlx-4bit): TurboQuant description, size inconsistencies.
- [arXiv 2604.18566](https://arxiv.org/pdf/2604.18566): fetched, text unreadable, no claim taken.
- [[methodology]]: weight, KV and overhead formulas.

## Related

- [[mlx-vs-gguf]]
- [[mlx-lm-on-mac]]
- [[long-context-on-mac]]
- [[how-much-ram-for-local-llm]]
- [[gemma-on-mac]]
- [[methodology]]
- [[map-of-content]]