oMLX
AIAn open-source LLM inference server for Apple Silicon with continuous batching and tiered KV caching, managed from the macOS menu bar.
oMLX
An open-source LLM inference server for Apple Silicon with continuous batching and tiered KV caching, managed from the macOS menu bar.
About oMLX
oMLX is an MLX-based local inference server for Apple Silicon Macs that pairs continuous batching with a tiered KV cache, keeping past context in a hot in-memory tier and a cold SSD tier so it stays reusable across requests even when the context changes mid-conversation. That persistence is what makes local models practical for real coding work with clients like Claude Code, OpenCode, Codex or Copilot, which otherwise pay a full prefill cost on every turn. It ships as a native macOS app with a menu bar controller, an in-app auto-updater and a CLI shim at ~/.omlx/bin/omlx that Apple Shortcuts and terminal commands can drive, plus Homebrew and from-source install paths. The server auto-discovers text LLMs, vision-language models, OCR models, embeddings and rerankers from subdirectories of a model folder and exposes them at an OpenAI-compatible endpoint on localhost:8000, with a built-in chat UI and an admin dashboard for real-time monitoring, model management, benchmarking and per-model settings in eight languages, fully vendored for offline use. Native Metal kernels give large speedups on some model families — the fused DSA prefill for GLM 5.2 is roughly 30x faster than the generic fallback — and source builds add experimental multi-Mac inference that splits one model across unequal-memory Macs over Thunderbolt or Ring RDMA.
Key Features
Use Cases
Frequently asked questions about oMLX
What is oMLX?
An open-source LLM inference server for Apple Silicon with continuous batching and tiered KV caching, managed from the macOS menu bar.
How does oMLX work?
oMLX works by combining Tiered KV Caching: Persists past context across a hot in-memory tier and a cold SSD tier, so cached context stays reusable across requests even when the conversation context changes mid-session., Continuous Batching: Serves concurrent requests through a batched scheduler rather than one-at-a-time, keeping throughput up when several clients or agent loops hit the server together., Menu Bar Management: Controls the server, pinned models, on-demand model swapping and context limits from a native macOS menu bar app with in-app auto-update., Native Metal Custom Kernels: Ships precompiled kernels in the official DMG that give large speedups on affected model families — roughly 30x faster fused DSA prefill for GLM 5.2 (845 vs ~29 tok/s measured on an M3 Ultra) with lower memory use., OpenAI-Compatible Endpoint: Exposes every discovered model at http://localhost:8000/v1 so existing OpenAI clients, coding agents and SDKs connect without modification. to help users with Local Coding Agents: Back Claude Code, OpenCode, Codex or Copilot with an on-device model where cached context makes repeated agent turns fast enough to be usable., Private Inference: Keep prompts, code and documents entirely on the Mac with no cloud provider in the path and no per-token billing., Serving a Team from One Mac: Run the OpenAI-compatible endpoint on a high-memory Mac so other machines on the network can use larger models than they could host themselves., Model Benchmarking: Compare throughput and per-model settings across quantizations and families from the built-in benchmark tools in the admin dashboard., Multi-Modal Local Pipelines: Serve embeddings, rerankers and OCR alongside chat models from a single endpoint to build local RAG without extra infrastructure..
What are the main features of oMLX?
Key features include Tiered KV Caching: Persists past context across a hot in-memory tier and a cold SSD tier, so cached context stays reusable across requests even when the conversation context changes mid-session., Continuous Batching: Serves concurrent requests through a batched scheduler rather than one-at-a-time, keeping throughput up when several clients or agent loops hit the server together., Menu Bar Management: Controls the server, pinned models, on-demand model swapping and context limits from a native macOS menu bar app with in-app auto-update., Native Metal Custom Kernels: Ships precompiled kernels in the official DMG that give large speedups on affected model families — roughly 30x faster fused DSA prefill for GLM 5.2 (845 vs ~29 tok/s measured on an M3 Ultra) with lower memory use., OpenAI-Compatible Endpoint: Exposes every discovered model at http://localhost:8000/v1 so existing OpenAI clients, coding agents and SDKs connect without modification..
Who is oMLX for?
oMLX is useful for anyone interested in Local Coding Agents: Back Claude Code, OpenCode, Codex or Copilot with an on-device model where cached context makes repeated agent turns fast enough to be usable., Private Inference: Keep prompts, code and documents entirely on the Mac with no cloud provider in the path and no per-token billing., Serving a Team from One Mac: Run the OpenAI-compatible endpoint on a high-memory Mac so other machines on the network can use larger models than they could host themselves., Model Benchmarking: Compare throughput and per-model settings across quantizations and families from the built-in benchmark tools in the admin dashboard., Multi-Modal Local Pipelines: Serve embeddings, rerankers and OCR alongside chat models from a single endpoint to build local RAG without extra infrastructure..
How much does oMLX cost?
oMLX is free to use.
How do I get started with oMLX?
Visit https://github.com/jundot/omlx to sign up and explore oMLX.
Explore more AI Ai Tools tools
Browse all Ai Tools tools →Compare oMLX: vs Agents Never Sleep · vs Port Radar for macOS · vs SubtitleGenerator · vs Zero