
Tuning Local LLM Inference on Apple Silicon: How oMLX Settings Interact
A local inference server on a Mac can have more than fifty settings. It is tempting to read them as a list of independent switches. They are not. Almost every setting that matters is a claim on the same pool of unified memory, or a choice about how to spend it, and the interesting behavior comes from how they combine.
This article is a simplified map of those interactions for oMLX, an MLX-based inference server for Apple Silicon. The examples come from a 32 GB machine serving models from 1.7B to 35B parameters, where memory is tight enough that every interaction shows up.
One pool, five claims
On Apple Silicon the CPU and GPU share one memory pool, so there is no separate VRAM to fill first. Everything below competes for the same 32 GB, along with macOS itself:
- Model weights. Fixed by the model and its quantization. A 4B model at 4-bit is roughly 2 GB.
- KV cache. The attention state for every token in the context. It grows with context length, and with each concurrent request.
- Prefix caches. Saved state that lets the server skip re-reading a prompt it has already seen. This can live in RAM, on SSD, or both.
- Runtime and transient buffers. The Python and MLX process, plus working memory during generation.
- Speculative decoding state, when a draft model or prediction head is enabled.
Every setting in the sections below either sizes one of these claims, caps the total, or decides how requests are scheduled against them.
Layer 1: the memory guard sets the ceiling
The memory guard is the outermost layer. You give it a ceiling (memory_guard_custom_ceiling_gb when the tier is custom) and two thresholds:
| Setting | Typical value | What happens when crossed |
|---|---|---|
soft_threshold |
0.85 of ceiling | New requests pause and idle models are evicted, least recently used first |
hard_threshold |
0.95 of ceiling | In-flight requests are aborted |
prefill_safe_zone_ratio |
0.80 to 0.95 of ceiling | Prefill chunks are sized so their predicted peak stays under the ceiling |
The thresholds are fractions of the ceiling, so the ceiling is the setting that moves all of them. With a 30 GB ceiling on a 32 GB machine, soft eviction starts at 25.5 GB, hard aborts at 28.5 GB, and macOS has about 2 GB to live in. That is a thin margin for a machine that also runs a browser and an editor. A 24 GB ceiling gives up 6 GB of model capacity and buys a calmer system.
The gap between the soft and hard thresholds is deliberate. macOS compresses memory under pressure, which makes usage oscillate. A gap of at least 10% keeps that oscillation from tripping the abort path.
Layer 2: per-model settings size the footprint
Each model can override the global defaults. Four settings do most of the work.
max_context_window is the largest prompt the server will accept for that model. It is also the main lever on KV cache size, because the cache scales with the context it may need to hold. A small model that only ever sees 8K-token prompts gains nothing from a 262K window except a larger worst case.
turboquant_kv_bits compresses the KV cache. Lower bit depths save more memory and cost more precision:
| Bits | Approximate KV compression |
|---|---|
| 2.0 | 3.5x |
| 2.5 | 2.8x |
| 3.0 | 2.3x |
| 4.0 | 1.8x |
hot_cache_max_size caps the RAM-resident prefix cache. ssd_cache_max_size caps the disk-resident one. The disk cache costs no RAM, only storage, which is why it can be set large (tens of gigabytes) while the hot cache stays small.
initial_cache_blocks pre-allocates cache blocks at startup and needs a restart to change. More blocks mean less allocation overhead during inference and more memory claimed up front.
Layer 3: scheduling decides who gets served
The remaining settings do not change how much memory is claimed so much as how the work is ordered.
max_concurrent_requests is the multiplier hiding in plain sight. Every concurrent request brings its own KV cache, so the context window, the quantization level and the concurrency limit multiply together. Doubling concurrency has the same memory effect as doubling context.
chunked_prefill (off by default) interleaves long prompt processing with decode steps. It reduces time-to-first-token for requests that arrive while a long prompt is being processed, at the cost of some per-step overhead. For a single user it changes little. For several agents sharing a server it matters.
burst_decode_mode controls how many decode steps run before the server yields to its event loop:
| Mode | Max steps | Time budget | Effect |
|---|---|---|---|
off |
1 | none | One token per handoff, smoothest sharing |
light |
64 | 50 ms | Conservative |
balanced |
64 | 100 ms | Default |
aggressive |
64 | 200 ms | Fastest stream, larger chunks, more latency variance for others |
Burst decode is a trade between one user’s streaming speed and everyone else’s responsiveness. It matters most when concurrency is above one.
Where the settings collide
Individually these are easy to reason about. The surprises come from the combinations.
Context, concurrency and quantization multiply. Worst-case KV memory is roughly context length times concurrent requests, divided by the compression factor. Lowering any of the three has the same kind of effect. Quantizing from 4 bits to 2.5 bits buys about 1.5x. Cutting a window from 262K to 32K buys 8x. Reach for context first.
Speculative decoding paths are mutually exclusive, and one conflicts with concurrency. oMLX supports several ways to decode faster: DFlash (a block-diffusion draft model), native multi-token prediction (MTP), a VLM assistant drafter and SpecPrefill for prompts. The server enforces the rules when it loads a model’s settings:
- DFlash and MTP cannot both be on, because each wants to own the decode loop.
- MTP and TurboQuant cannot both be on, because TurboQuant patches the attention path MTP depends on.
- DFlash runs requests one at a time, which directly conflicts with a concurrency limit above one.
- DFlash and TurboQuant, and SpecPrefill and TurboQuant, coexist fine.
So “fastest single-stream decode” and “serve four agents at once” are different configurations, not different points on one dial.
Speed features can undo caching. Prefix caches pay off when many requests share a long prompt, as agent workloads do. A decode accelerator that disables or invalidates that cache can be slower end to end, even if its tokens per second are higher. This is worth testing on your own workload instead of trusting a headline speedup.
The ceiling changes what the other limits mean. A model whose caches fit comfortably under a 30 GB ceiling may trigger constant eviction under 24 GB. The per-model limits are only meaningful relative to the guard.
A lesson in measuring instead of reading config
I started this exercise because a 4B model was using far more memory than its size explained. The server log reports an incremental cost of 4 to 7 MB per token, which weights alone cannot account for. Adding up the configured budgets on that model gave a worst case of roughly 23 to 27 GB:
| Claim | Configured budget (worst case) |
|---|---|
| Weights | about 2 GB |
| Runtime | about 0.5 GB |
| KV cache at a 262K window | about 8 to 12 GB (estimate) |
| DFlash prefix cache budget | 8 GB |
| Hot cache | up to 4 GB |
| Transient | 1 to 2 GB |
Then I checked the actual configuration. DFlash was disabled on every model, even though each carried an 8 GB budget for its cache. A budget on a disabled feature is probably not consuming anything, so the total above overstates real use, and I cannot attribute the observed cost to that cache with confidence.
That finding is the main point of this article. A configured budget is an upper bound, not a measurement. It tells you what the system is permitted to use. What it does use only shows up in the log line the memory enforcer prints, which reports current usage, per-token cost and the ceiling. Tune from that line.
A tuning order that works
- Set the ceiling first, leaving enough for macOS and your other applications. On 32 GB, that means 24 to 26 GB, not 30.
- Size each model’s context window to the work it does. Small models used for short tasks rarely need more than 32K.
- Pick a KV quantization level and check output quality on your own prompts before going below 3 bits.
- Set concurrency from the memory that remains, not from the number of clients you have.
- Choose one speculative decoding path per model, and only if you serve one stream at a time.
- Cap the RAM prefix cache modestly and let the SSD cache absorb the rest.
- Measure with the enforcer log under a realistic load, then change one setting at a time.
The starting values I am testing next on the 32 GB machine are a 24 GB ceiling, a 32K window for small models and a 2 GB hot cache. They are hypotheses until the log confirms them.
Takeaways
- Nearly every oMLX setting is a claim on, or a rule about, one shared memory pool.
- The memory guard ceiling is the setting that scales all the others, so set it first.
- Context length, concurrency and quantization multiply. Reduce context before touching bit depth.
- Speculative decoding options are mutually exclusive, and some conflict with concurrency and caching.
- Budgets in a config file are upper bounds. Trust the runtime’s own measurements.
Get the next one in your inbox
Enjoyed this? I write about local AI, self-hosting, and cloud cost ninjutsu. Join the Ninja Dispatch for occasional, high-signal notes. No fluff.

