oMLX
AIAn open-source native macOS LLM inference server built on MLX whose paged SSD KV cache drops coding-agent time-to-first-token from 30-90s to under 5s.
oMLX
An open-source native macOS LLM inference server built on MLX whose paged SSD KV cache drops coding-agent time-to-first-token from 30-90s to under 5s.
About oMLX
oMLX is an Apache 2.0 licensed inference server for Apple Silicon Macs that targets the specific way coding agents use a local model. Agents invalidate the KV cache dozens of times per session, so oMLX persists every cache block to SSD in safetensors format using a two-tier scheme — hot blocks stay in RAM while cold blocks move to disk under an LRU policy — meaning a previously seen prefix is restored in milliseconds instead of recomputed, even across server restarts. On top of that it adds continuous batching through mlx-lm's BatchGenerator, reaching up to 4.14x generation speedup at 8x concurrency, and can serve LLM, VLM, embedding, and reranker models simultaneously with LRU eviction when memory runs low. The server exposes both OpenAI-compatible endpoints and a native Anthropic /v1/messages endpoint, so Claude Code, OpenClaw, and Cursor connect as drop-in clients, and it supports JSON, Qwen, Gemma, GLM, and MiniMax tool-calling formats plus MCP tool integration. Everything is managed from a signed, notarized native macOS menu bar app — not Electron — with a web dashboard for model downloads, chat, and real-time metrics. It requires macOS 15 or later on Apple Silicon.
Screenshots
Key Features
Use Cases
Frequently asked questions about oMLX
What is oMLX?
An open-source native macOS LLM inference server built on MLX whose paged SSD KV cache drops coding-agent time-to-first-token from 30-90s to under 5s.
How does oMLX work?
oMLX works by combining Paged SSD KV Caching: Cache blocks persist to disk in safetensors format with hot blocks in RAM and cold blocks on SSD, so previously seen prefixes are restored in milliseconds and survive server restarts., Sub-5s Agent TTFT: Cuts time-to-first-token for coding agents from 30-90 seconds down to under 5 seconds from the second turn onward., Continuous Batching: Handles concurrent requests through mlx-lm's BatchGenerator, measured at up to 4.14x generation speedup at 8x concurrency., OpenAI and Anthropic Drop-In API: Serves both OpenAI-compatible endpoints and a native Anthropic /v1/messages endpoint so Claude Code, OpenClaw, and Cursor connect without adapters., Multi-Model Serving: Loads LLM, VLM, embedding, and reranker models at the same time with LRU eviction when memory is constrained. to help users with Local Coding Agents: Run Claude Code or OpenClaw entirely against a local model without the 90-second waits that make local inference impractical for agents., Private Codebase Work: Keep proprietary source on-device by pointing an OpenAI-compatible IDE assistant at a local endpoint., Offline Development: Continue agent-assisted coding without network access or per-token API costs., Model Benchmarking: Compare Qwen3.5-122B, Qwen3-Coder-Next, MiniMax-M2.5, and GLM-5 throughput on the same Apple Silicon hardware., Multi-Client Serving: Serve several concurrent agent sessions from one Mac using continuous batching rather than queuing behind a single request..
What are the main features of oMLX?
Key features include Paged SSD KV Caching: Cache blocks persist to disk in safetensors format with hot blocks in RAM and cold blocks on SSD, so previously seen prefixes are restored in milliseconds and survive server restarts., Sub-5s Agent TTFT: Cuts time-to-first-token for coding agents from 30-90 seconds down to under 5 seconds from the second turn onward., Continuous Batching: Handles concurrent requests through mlx-lm's BatchGenerator, measured at up to 4.14x generation speedup at 8x concurrency., OpenAI and Anthropic Drop-In API: Serves both OpenAI-compatible endpoints and a native Anthropic /v1/messages endpoint so Claude Code, OpenClaw, and Cursor connect without adapters., Multi-Model Serving: Loads LLM, VLM, embedding, and reranker models at the same time with LRU eviction when memory is constrained..
Who is oMLX for?
oMLX is useful for anyone interested in Local Coding Agents: Run Claude Code or OpenClaw entirely against a local model without the 90-second waits that make local inference impractical for agents., Private Codebase Work: Keep proprietary source on-device by pointing an OpenAI-compatible IDE assistant at a local endpoint., Offline Development: Continue agent-assisted coding without network access or per-token API costs., Model Benchmarking: Compare Qwen3.5-122B, Qwen3-Coder-Next, MiniMax-M2.5, and GLM-5 throughput on the same Apple Silicon hardware., Multi-Client Serving: Serve several concurrent agent sessions from one Mac using continuous batching rather than queuing behind a single request..
How much does oMLX cost?
oMLX is free to use.
How do I get started with oMLX?
Visit https://omlx.ai/ to sign up and explore oMLX.
Explore more AI Ai Services tools
Browse all Ai Services tools →Browse by use case: Code Generation
Compare oMLX: vs Olostep · vs FreeLLMAPI · vs PageIndex · vs Speko