Overview
mlx-lm is a Python package, built on Apple’s MLX array framework, that runs, quantizes and serves language models on a Mac from the terminal, installed with pip install mlx-lm. The mlx-lm README gives the commands below; this page adds a runtimes section and three open issues, each cited by number. Sources were read on 2026-10-13. This site publishes no tokens-per-second figures, so speed is unknown here; measure your own Mac with measure-tokens-per-second-mac. Format trade-offs are on mlx-vs-gguf.
Search demand, from the Ahrefs US table pulled 2026-10-12: “mlx-lm” 400 (KD 56), “mlx lm” 200 (KD 0), “omlx” 3,300 (KD 7), “mlx-vlm” 350, “vllm-mlx” 200.
How do you install mlx-lm on a Mac?
Run one pip command. The README lists two install routes:
pip install mlx-lm- or
conda install -c conda-forge mlx-lm
MLX itself installs with pip install mlx, and the MLX README describes it as “An array framework for Apple silicon”. The mlx-lm README states a macOS requirement only for one feature: the large-model note (“This requires macOS 15.0 or higher to work”). It gives no Python version, so that is unknown here. For the M5 Neural Accelerator gains, Apple’s M5 article says “MLX requires macOS 26.2 or later”.
How do you run an MLX model on a Mac?
Use mlx_lm.generate for one prompt or mlx_lm.chat for a chat loop.
- Generate:
mlx_lm.generate --prompt "How tall is Mt Everest?" - Chat:
mlx_lm.chat - Pick a model with
--model. The README’s default ismlx-community/Llama-3.2-3B-Instruct-4bit. The README’smlx_lm.generateexample says it downloads the model from the Hugging Face Hub; caching behaviour is unknown from the README. - List options with
mlx_lm.generate -h.
To make your own quantized copy, the README gives mlx_lm.convert --model mistralai/Mistral-7B-Instruct-v0.3 -q, with --upload-repo to push the result to Hugging Face. What the bit widths mean is on mlx-quantization.
How do you start the mlx_lm.server OpenAI-style server?
Run mlx_lm.server --model <repo or path>. The server doc gives this example and says the server starts on port 8080 of localhost:
mlx_lm.server --model mlx-community/Mistral-7B-Instruct-v0.3-4bit
It documents two endpoints: /v1/chat/completions and /v1/models. A test call from the doc:
curl localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"messages": [{"role": "user", "content": "Say this is a test!"}], "temperature": 0.7}'
Two cautions from the same page. It says the server “is not recommended for production as it only implements basic security checks”. And with --kv-bits set, requests are handled one at a time, because quantized KV cache does not support batching. The README does not explain --kv-bits; for context memory see long-context-on-mac. The doc does not document --host or --port flags, so changing the port or exposing the server to the network is unknown from this source.
Where do MLX community models come from?
Mostly from the MLX Community organization on Hugging Face, which the READMEs use for every example model (for instance mlx-community/Qwen2-VL-2B-Instruct-4bit in mlx-vlm). The org page as fetched shows no description or model count, so how many models it holds is unknown. Check a repo’s card for its quantization and license before downloading.
Memory for an MLX model follows the same formula as any other: weights_GB = params x bits / 8 / 1e9, plus KV cache and a labelled overhead, from methodology. For Qwen2.5-7B at 4.0 bits and context 4096 that is 3.805 + 0.235 + 2 = 6.040 GB (worked on how-much-ram-for-local-llm). For measured MLX memory, Apple’s M5 article lists these figures from its own runs (vendor figures, MacBook Pro with 24GB):
| Model (MLX) | Memory (GB), Apple |
|---|---|
| Qwen3-8B-MLX-4bit | 5.61 |
| Qwen3-14B-MLX-4bit | 9.16 |
| gpt-oss-20b-MXFP4-Q4 | 12.08 |
| Qwen3-30B-A3B-MLX-4bit | 17.31 |
| Qwen3-8B-MLX-bf16 | 17.46 |
The README says models that are large relative to RAM can be slow, and that mlx-lm prints a warning in that case. It says to raise the wired limit with sudo sysctl iogpu.wired_limit_mb=N, with N above the model size in MB and below the machine’s memory. The safe value for a given Mac is unknown here; see mac-gpu-memory-limit.
Does MLX use the Neural Engine?
No source read says so. The MLX README names only the CPU and the GPU as supported devices and does not mention the Neural Engine. The M5 “Neural Accelerators” Apple describes are in the GPU: Apple’s article says they “provide dedicated matrix-multiplication operations” and that MLX uses Metal 4 TensorOps for them. Those are not the Apple Neural Engine. See m5-neural-accelerators.
What runtimes are built on mlx-lm?
| Runtime | What the primary source says | Status here |
|---|---|---|
| oMLX | ”LLM inference, optimized for your Mac”; native Swift/SwiftUI menu bar app, web admin UI at /admin, Apache-2.0, macOS 15.0+, Python 3.11-3.13. Concurrent requests go through mlx-lm’s BatchGenerator; started from vllm-mlx v0.1.0. | Verified |
| mlx-vlm | ”a package for inference and fine-tuning of Vision Language Models”; pip install -U mlx-vlm; mlx_vlm.server; MIT. | Verified |
| vllm-mlx | Named as oMLX’s origin. No repo read for this page. | Unverified |
| vmlx, Rapid MLX, Laya MLX | Names seen in research only. No primary source found. | Unverified |
oMLX versus LM Studio: LM Studio is a desktop app. Its system requirements list Apple silicon and macOS 14.0 or newer, and lm-studio-on-mac covers its MLX support. oMLX is an inference server with a menu bar app and a web admin UI. No head-to-head comparison from a primary source exists, so which is better for you is unknown. Other apps are on local-llm-apps-for-mac, and Ollama’s MLX path is ollama-mlx.
Can MLX run across several Macs?
Yes, with mx.distributed, which the mlx-lm README lists as a feature without setup commands. The MLX distributed docs list backends MPI, ring (TCP sockets), JACCL (RDMA over Thunderbolt) and NCCL (CUDA), with mlx.launch as the launcher, for example mlx.launch --hosts ip1,ip2,ip3,ip4 my_script.py. JACCL needs Thunderbolt 5 Macs in a fully connected mesh, RDMA enabled from macOS Recovery, and “Starting from macOS 26.2” per the docs. Clusters are on mac-llm-cluster.
What are the known mlx_lm.server memory and hang issues?
Three public issues, status as read on 2026-10-13:
| Issue | Reported | Status |
|---|---|---|
| mlx-lm #1757 | Server never clears MLX’s buffer pool on model swap, so it retains every model it has loaded. The reporter says “This is not a leak”, because mx.clear_cache() reclaims it. mlx-lm 0.31.3, macOS 26.5.2, M3 Ultra 96 GiB. | Closed by PR #1837 |
| mlx-lm #1807 | Memory grows past the prompt-cache cap in a long-running server on a hybrid-attention model; reporter saw a kill at about 30 GB with a roughly 16 GB model and 8 GB cache cap. M1 Ultra 128 GB. | Open, no maintainer reply shown |
| mlx #3830 | Distributed pipeline generation deadlocks under MLX_METAL_FAST_SYNCH=1. A maintainer wrote “At the moment METAL_FAST_SYNCH is not reliable and we don’t see a way to fix it.” | Closed |
The first is a cache-retention bug, not a leak by the reporter’s own account; the second is a reported growth bug awaiting a fix. If your server is memory-bound, restarting it between model swaps is a workaround implied by the reports, not a documented fix.
Questions
How do I install mlx-lm?
Run pip install mlx-lm, or conda install -c conda-forge mlx-lm. Both are in the mlx-lm README. The README gives no Python version, which is unknown here.
How do I run MLX models on a Mac?
Run mlx_lm.generate --prompt "..." for one answer or mlx_lm.chat for a chat, and choose a model with --model. The README’s default is mlx-community/Llama-3.2-3B-Instruct-4bit. The README says its example downloads the model from the Hugging Face Hub.
Is mlx_lm.server an OpenAI-compatible server?
It serves /v1/chat/completions and /v1/models on localhost port 8080 by default, per its SERVER.md. The doc warns it is not recommended for production because it only implements basic security checks. Full OpenAI API coverage beyond those endpoints is unknown.
Where do I find MLX community models?
On the MLX Community organization on Hugging Face, huggingface.co/mlx-community, which the mlx-lm and mlx-vlm READMEs use for their examples. The number of models is unknown from the page as fetched.
Does MLX use the Neural Engine?
No source read says it does. The MLX README lists the CPU and the GPU as supported devices. The M5 Neural Accelerators Apple describes sit in the GPU and are used through Metal 4. See m5-neural-accelerators.
Is oMLX better than LM Studio?
Unknown. oMLX is an Apache-2.0 inference server with a menu bar app and a web admin UI. LM Studio is a desktop app requiring macOS 14.0 or newer. No primary source compares them, and this site gives no speed figures.
Does mlx_lm.server have a memory leak?
Issue #1757 reported memory retained after model swaps, which the reporter called not a leak; it was closed by PR #1837. Issue #1807 reports memory growth past the prompt-cache cap on a long-running server and is open. Details are in the table above.
How fast is mlx-lm?
Unknown for your Mac. This site publishes no tokens-per-second figures. Apple’s M5 article reports its own benchmarks as vendor claims, for example 19-27% better generation than M4 on its test setup. Measure yours with measure-tokens-per-second-mac.
Can MLX do distributed inference?
Yes, through mx.distributed with ring, MPI, JACCL or NCCL backends, launched with mlx.launch. JACCL needs Thunderbolt 5 Macs and macOS 26.2 or later per the MLX docs.
Sources read on 2026-10-13
- mlx-lm README: install, quick start, convert command, wired-limit sysctl, macOS 15.0 note, distributed feature line.
- mlx_lm SERVER.md: launch command, port 8080, endpoints,
--kv-bits, production warning. - MLX README:
pip install mlx, CPU and GPU devices, shared memory, no Neural Engine mention. - Apple Machine Learning Research, Exploring LLMs with MLX and the Neural Accelerators in the M5 GPU: macOS 26.2 footnote, memory table, Neural Accelerator description. Performance numbers are Apple’s own claims.
- mlx-lm issue 1757, mlx-lm issue 1807, mlx issue 3830: reports, versions, status.
- MLX distributed docs: backends,
mlx.launch, JACCL requirements. - oMLX and mlx-vlm: descriptions, install, licenses, requirements.
- MLX Community on Hugging Face: page shows no description or count.
- LM Studio system requirements: Apple silicon, macOS 14.0 or newer.
- Ahrefs US keyword table (pulled 2026-10-12): search volumes quoted in the Overview.
- methodology: weight, KV and overhead formulas.