LocalAI
A local inference server exposing OpenAI-compatible APIs for language, image, audio, and other model workloads.
Why consider LocalAI?
LocalAI is an open-source, MIT-licensed AI engine that lets you run LLMs, vision, voice, image, and video models on any hardware — including CPU-only machines — with a drop-in OpenAI-compatible REST API. It uses a composable backend architecture where engines like llama.cpp, vLLM, whisper.cpp, and Stable Diffusion are pulled on demand as separate images, so you only install what you actually use.
Learn LocalAI by building
Practical setup notes, real use cases, and copy-ready examples in one focused guide.
In this guide12 sections
LocalAI: Run Any AI Model Locally Without a GPU
LocalAI is the open-source AI engine that lets you run LLMs, vision, voice, image, and video models on any hardware — no GPU required. Created by Ettore Di Giacinto, it exposes a drop-in OpenAI-compatible REST API so that existing AI clients work against a local server with nothing more than a URL change.
Repository: github.com/mudler/LocalAI
Website: localai.io
License: MIT
What Makes LocalAI Different
Most local-AI projects bundle a fixed set of dependencies into one massive download. LocalAI takes the opposite approach: a small core, not a bundle. Each backend wraps a best-in-class engine (llama.cpp, vLLM, whisper.cpp, Stable Diffusion, MLX…) in its own image, pulled only when a model needs it. You install nothing you don't use.
Key design principles, as stated in the project README:
- Composable by design — backends are separate and pulled on demand, so you install only what your model needs
- Drop-in API compatibility — OpenAI, Anthropic, and ElevenLabs APIs across every backend
- Any model, any modality — LLMs, vision, voice, image, and video behind one API
- Any hardware — NVIDIA, AMD, Intel, Apple Silicon, Vulkan, or CPU-only
- Multi-user ready — API key auth, user quotas, role-based access
- Built-in AI agents — autonomous agents with tool use, RAG, MCP, and skills
- Privacy-first — your data never leaves your infrastructure
As of August 2026 LocalAI has over 48,000 GitHub stars, ships 73 backends, and provides 1,255 one-click installable models.
Quickstart: Run LocalAI in Docker
The fastest path to a running server is Docker. LocalAI publishes tagged images for every hardware tier.
CPU-only (no GPU needed)
docker run -ti --name local-ai -p 8080:8080 localai/localai:latestNVIDIA GPU (CUDA)
# CUDA 13docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-13# CUDA 12docker run -ti --name local-ai -p 8080:8080 --gpus all localai/localai:latest-gpu-nvidia-cuda-12AMD GPU (ROCm)
docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/localai:latest-gpu-hipblasIntel GPU (oneAPI)
docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/localai:latest-gpu-intelTo restart an existing container you've already created, use docker start -i local-ai.
Once running, the API is available at http://localhost:8080 and is fully compatible with the OpenAI client libraries — just point them at your local address.
Loading Models
LocalAI can load models from multiple sources. The CLI makes this effortless:
# From the model gallery (see available models at https://models.localai.io)local-ai run llama-3.2-1b-instruct:q4_k_m# From Huggingfacelocal-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf# From the Ollama OCI registrylocal-ai run ollama://gemma:2b# From a YAML configlocal-ai run https://gist.githubusercontent.com/.../phi-2.yaml# From a standard OCI registry (e.g., Docker Hub)local-ai run oci://localai/phi-2:latestLocalAI automatically detects your GPU capabilities and downloads the appropriate backend. You never need to manually pick a llama.cpp build for your hardware tier.
Using the Terminal Agent
Once a model is running you can open a chat session directly from the terminal. The built-in agent answers questions, reads your files, and runs commands — asking for approval before anything that changes state:
# Terminal 1 — start the model serverlocal-ai run llama-3.2-1b-instruct:q4_k_m# Terminal 2 — open an interactive chat sessionlocal-ai chat --model llama-3.2-1b-instruct:q4_k_mInside the session, /models lists installed models and /model <name> switches between them.
Multi-Modal Capabilities
Behind that single API endpoint, LocalAI covers every major AI modality:
| Capability | What it does | Example backends |
|---|---|---|
| Reason | Language models, tool calling, structured output | llama.cpp · vLLM · MLX |
| Listen | Realtime voice, transcription, diarization | parakeet.cpp · whisper.cpp |
| Speak | Speech synthesis and voice cloning | moss-tts.cpp · piper |
| See | Vision, detection, recognition, depth, 3D | rf-detr.cpp · depth-anything.cpp |
| Create | Images, video, music and sound | diffusers · ace-step |
| Act | Agents, MCP, skills, RAG, interactive tools | built-in agent runtime |
A single session can combine all of these: transcribe speech, identify the speaker, call a tool, and answer out loud — over a single API, on a single machine.
parakeet.cpp — A Standout First-Party Engine
The LocalAI team writes their own engines when existing options are too slow, too heavy, or closed. One example is parakeet.cpp — an NVIDIA NeMo Parakeet port to C++ and ggml:
- 27× faster than whisper.cpp on CPU
- WER 0 parity with NeMo (transcript is bit-exact with NeMo's output)
- Covers 40+ locales with the multilingual streaming model
- Ships as a binary + GGUF file, verified in CI against the reference implementation
The team has written 19 such engines from scratch because the alternatives arrived as 9 GB Python installs, were closed-source, or simply did not exist.
Architecture: Composable Core
The website describes the runtime concisely: "One binary with an OpenAI-compatible API in front of it. Point an existing client at it and the calls keep working, except now the model is on your machine."
The swappable engine layer means one model can run on llama.cpp while the next loads on vLLM, SGLang, or MLX — and the client never notices. Switching is one line in the model's config file.
For distributed workloads, LocalAI supports smart routing, VRAM-aware placement, prefix-cache affinity, failover, P2P, NATS, and federation. You add hardware; the cluster figures out what to do with it.
Resources
Related tools
More options with a similar category or technology profile.
marimo
A reactive Python notebook that is reproducible, git-friendly, and executable as a script.
JupyterLab
The next-generation web-based user interface for Project Jupyter computational notebooks.
Trafilatura
Python package and command-line tool designed to gather text and metadata on the Web.
ScrapeGraphAI
Python scraper library that uses LLMs and direct graph logic to extract website data.