Overview
MLX 4-bit vs 8-bit quantization is a trade between file size and precision: both store each weight as a small integer plus a shared scale and bias per group of weights, so 8-bit roughly doubles the weight memory of 4-bit, and which one loses less quality is unknown from the sources read here. MLX’s own docs list 2, 3, 4, 5, 6 and 8 bits for its default affine mode, with 4 as the default. This page works the sizes out with the formulas on methodology, shows what the docs say about KV cache quantization, MTP and TurboQuant, and does not publish speed or quality numbers. For the format comparison see mlx-vs-gguf; the shared explanation is LLM quantization explained.
What is MLX quantization?
MLX quantization stores a weight matrix as packed low-bit integers with a scale and (in the default mode) a bias for every group of weights. The mlx.core.quantize docs say every group_size elements in a row are quantized together. In affine mode each group gets a scale computed from the group’s max and min over 2^b - 1, and a bias that is the group minimum; elements are packed into unsigned 32-bit integers.
| Mode | Bits | Group size (supported) | Per-group data |
|---|---|---|---|
| affine (default) | 2, 3, 4, 5, 6, 8 (default 4) | 32, 64, 128 (default 64) | scale + bias |
| mxfp4 | 4 | 32 | shared 8-bit E8M0 scale, no bias |
| mxfp8 | 8 | 32 | shared 8-bit E8M0 scale, no bias |
| nvfp4 | 4 | 16 | shared 8-bit E4M3 scale, no bias |
The input’s last dimension must be divisible by group_size. The mlx_lm.convert command in mlx-lm exposes -q, --q-bits, --q-group-size and --q-mode (affine, mxfp4, nvfp4, mxfp8), read from convert.py on the main branch on 2026-10-13. The README’s command-line example is mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q with no bit width stated; its Python API section says the equivalent convert call generates a 4-bit model. It states no group size.
How big is MLX 4-bit vs 8-bit?
Stored bits per weight is the integer width plus the scale and bias spread over the group, so a “4-bit” MLX file is more than 4.0 bits per weight.
The docs say scales have the input’s dtype and do not state the bias dtype. Assuming a 16-bit scale and a 16-bit bias (an assumption, not a documented figure):
bits_per_weight = bits + (16 + 16) / group_size
Then weights_GB = params x bits_per_weight / 8 / 1e9 from methodology. For the 7.61B parameters on the Qwen2.5-7B-Instruct card:
| Setting | Bits per weight (assumed formula) | Weights, decimal GB |
|---|---|---|
| 3-bit, group 64 | 3.5 | 3.329 |
| 4-bit, group 128 | 4.25 | 4.043 |
| 4-bit, group 64 (default) | 4.5 | 4.281 |
| 4-bit, group 32 | 5.0 | 4.756 |
| 6-bit, group 64 | 6.5 | 6.183 |
| 8-bit, group 64 | 8.5 | 8.086 |
Add KV cache and a labelled overhead (this site assumes 2 GB) per how-much-ram-for-local-llm. Embedding layers or other tensors kept at higher precision would change a real file size; whether mlx_lm.convert does so by default is unknown from the pages read, so compare against the actual file size on the model’s Hugging Face page. Fit by installed memory is on how-much-ram-for-local-llm.
Is MLX 4-bit better than 8-bit?
Unknown. No primary source read here measures MLX 4-bit against 8-bit quality, and this site publishes no quality or speed numbers of its own. What the sources do establish is size: on the table above 8-bit group 64 is 8.086 GB against 4.281 GB for 4-bit group 64, a difference of 3.805 GB for 7.61B parameters. A smaller group size spends more bits on scales and biases (5.0 bits per weight at group 32 against 4.5 at group 64 under the assumption above). Whether that buys quality is not stated by the docs. To measure speed yourself, see measure-tokens-per-second-mac.
Q4_K_M vs MLX 4-bit: what do the sources say?
They do not settle it; the two are different schemes with different stored sizes. The llama.cpp quantize README lists Llama 3.1 8B at these bits per weight and sizes:
| GGUF type | Bits per weight | Size |
|---|---|---|
| Q4_K_M | 4.8944 | 4.58 GiB |
| Q5_K_M | 5.7036 | 5.33 GiB |
| Q6_K | 6.5633 | 6.14 GiB |
| Q8_0 | 8.5008 | 7.95 GiB |
| F16 | 16.0005 | 14.96 GiB |
Under the assumed formula MLX 8-bit group 64 is 8.5 bits per weight, close to Q8_0’s 8.5008, while MLX 4-bit group 64 is 4.5 against Q4_K_M’s 4.8944, about 92% of it (4.5 / 4.8944). That is a size comparison across two different models’ tables and an assumed MLX overhead, not a quality finding. The README says a default Q4_K_M run is “naive” and that accuracy loss “can be minimized by using a suitable imatrix file”. Any claim that one is more accurate needs a measurement on the same model, and none is cited here. Model files: mlx-vs-gguf.
How does MLX KV cache quantization work?
mlx-lm can quantize the KV cache at generation time, but only when you ask. From generate.py on the main branch (read 2026-10-13): --kv-bits sets the bits and defaults to no quantization, --kv-group-size defaults to 64, and --quantized-kv-start sets the step from which quantization starts, with a default of 5000. The README does not mention these flags, so the source file is the citation.
The KV formula is on methodology: kv_GB = 2 x layers x kv_heads x head_dim x context x bytes_per_value / 1e9. For Qwen2.5-7B-Instruct, config.json has 28 layers, 4 KV heads, hidden size 3584 and 28 attention heads (head_dim 128). Scaling the 16-bit cache by (bits + 32/64) / 16, under the same assumed 16-bit scale and bias:
| Context | 16-bit KV | 8-bit, group 64 | 4-bit, group 64 |
|---|---|---|---|
| 4096 | 0.235 GB | 0.125 GB | 0.066 GB |
| 32768 | 1.879 GB | 0.998 GB | 0.528 GB |
Because the default start is 5000, the 4096 row would not be quantized at all under default settings. Effect on answer quality and speed: unknown. Longer-context handling is on long-context-on-mac. For comparison, the llama.cpp server README lists --cache-type-k and --cache-type-v values f32, f16, bf16, q8_0, q4_0, q4_1, iq4_nl, q5_0 and q5_1, default f16.
Does MLX support MTP (multi-token prediction)?
Not that the sources read show. The mlx-lm README does not mention MTP, multi-token prediction, speculative decoding or draft models. generate.py on the main branch has --draft-model and --num-draft-tokens (default 3) for speculative decoding with a separate draft model, and a search of that file found no MTP option. llama.cpp’s server README lists draft-mtp among the --spec-type values (default none) and --spec-draft-n-max with default 3. It never uses the phrase “multi-token prediction”. Whether any given MLX model or app supports MTP is unknown, and so is the speed effect. This answers “mtp on mac” only as far as documentation goes; check the app’s current release notes.
What is TurboQuant, and is there a TurboQuant for MLX?
Treat TurboQuant as a label on a KV-cache approach, not a separate MLX feature. The majentik gemma-4-31b TurboQuant MLX 4-bit card calls it a “legacy TurboQuant KV-cache fork (superseded by upstream llama.cpp KV options)”, says RotorQuant and TurboQuant are “release labels, not distinct quantization algorithms”, and says both brand repos carry byte-identical weights “produced with the standard MLX / llama.cpp quantizers”. It states ~17 GB in one place while its own tables also give ~13 GB and ~19 GB, so its size is unreliable, and it does not mention MTP. Run steps and Gemma context are on gemma-on-mac. This is a community upload, not a vendor release. I could not use arXiv 2604.18566: it is a system-dynamics benchmark and its text was unreadable.
Questions
What is the difference between MLX 4-bit and 8-bit?
Bits per weight and file size: 8-bit stores each weight in 8 bits, 4-bit in 4, each with a per-group scale and bias. For 7.61B parameters at group 64 the weights are 8.086 GB against 4.281 GB under the assumed overhead. Quality and speed differences are unknown.
Which MLX bit width should I pick?
Pick the largest width that fits your unified memory after KV cache and overhead, using total_GB = weights_GB + kv_GB + overhead_GB. Whether 4-bit quality is acceptable for your task is unknown from the sources; test it. See how-much-ram-for-local-llm.
Does MLX support MTP?
No MTP option appears in the mlx-lm README or in generate.py on the main branch as read on 2026-10-13; speculative decoding with a separate --draft-model does. Speed effect: unknown.
Does MLX support KV cache quantization?
Yes, in mlx-lm via --kv-bits, with --kv-group-size (default 64) and --quantized-kv-start (default 5000), per generate.py. It is off by default.
Is Q4_K_M the same as MLX 4-bit?
No. Q4_K_M is a llama.cpp type at 4.8944 bits per weight on Llama 3.1 8B; MLX 4-bit stores 4-bit integers with per-group scale and bias. Which is more accurate is not established by the sources read.
What is TurboQuant in llama.cpp or MLX?
On the one model card read it is a legacy KV-cache fork label that the card says upstream llama.cpp KV options supersede. Anything further is unknown.
What is the default MLX quantization?
Affine mode, 4 bits, group size 64, per the mlx.core.quantize docs.
Does this page give tokens per second?
No. Speed is unknown here. See measure-tokens-per-second-mac.
Sources read on 2026-10-13
- mlx.core.quantize: modes, bits, group sizes, scale and bias description.
- llama.cpp quantize README: Llama 3.1 8B bits per weight and sizes, imatrix wording.
- llama.cpp server README: KV cache types,
draft-mtpspec type (first 100,000 characters read). - mlx-lm README, plus
mlx_lm/generate.pyandmlx_lm/convert.pyon the main branch: convert flags, KV flags and defaults, draft model options. - Qwen2.5-7B-Instruct: 7.61B parameters and
config.jsonlayers, KV heads, hidden size. - majentik/gemma-4-31b-turboquant-mlx-4bit: TurboQuant description, size inconsistencies.
- arXiv 2604.18566: fetched, text unreadable, no claim taken.
- methodology: weight, KV and overhead formulas.