llama.cpp
C/C++ engine that runs LLMs and vision models locally on CPUs and GPUs, with a CLI and an OpenAI-compatible server.
Quantised inference on Apple silicon, NVIDIA, AMD, Vulkan and CPU, with an OpenAI-compatible API server.
llama.cpp is a plain C/C++ inference engine for running LLMs and vision language models on your own hardware. It ships a command-line chat tool and a server with an OpenAI-compatible API. It suits developers who want local models without a heavy Python stack.
What it is
You download a model from Hugging Face by name and run it with llama cli, or serve it with llama serve to get an OpenAI-compatible API and a built-in web UI. Quantisation from 1.5-bit to 8-bit cuts memory use, and CPU+GPU hybrid inference lets models larger than your VRAM run partly on the GPU.
Who it's for
It suits developers, hobbyists and teams who want to run open models on a laptop, workstation or server and build apps on a local API. It does not ship a model: quality and speed depend on the model and hardware you pick.
Key features
Plain C/C++ with no dependencies
The README describes a plain implementation without dependencies.
Wide backend support
Metal, CUDA, HIP, Vulkan, SYCL, OpenCL, CANN, WebGPU and more cover many devices.
Integer quantisation
1.5-bit to 8-bit quantisation reduces memory use and speeds inference.
CPU and GPU hybrid
Partially accelerate models larger than the total VRAM.
OpenAI-compatible server
llama serve exposes an API and a built-in web UI.
GBNF grammars
Constrain model output with grammar files documented in the repo.
Up and running in 3 steps
Our honest verdict
What's great
- Apple silicon is described as a first-class citizen, optimised through ARM NEON, Accelerate and Metal.
- Supports NVIDIA, AMD, Intel, Snapdragon and Huawei Ascend hardware through separate backends.
- One command downloads and runs a model from Hugging Face.
- Vision language models and an OpenAI-compatible server mean existing client code can point at it.
Where it falls short
- Larger models need substantial RAM or VRAM, and speed on CPU alone will be modest; you must choose a model and quantisation level that fits.
- Building for a specific GPU backend means following the build guide for CUDA, HIP, Vulkan or others, which takes real technical skill.
- It is an engine and command-line toolset rather than a full chat product; the README shows a built-in web UI against llama serve.
- Model licences are separate from the MIT code, so you must check each model you download.
Pricing & license
llama.cpp is MIT-licensed and free to run. Its README shows no paid or hosted plan, so the only cost is the hardware and electricity you run it on.
Open remote AI-training roles
768 open remote AI-training roles, updated November 14, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
These reviewed AI tools do the same job instantly — from our scored directory.
Use it, or have us set it up
FAQ
Do I need a GPU?
No. The README lists CPU instruction support such as AVX and AMX, and GPU backends are optional. A GPU helps with larger models.
Does it provide an OpenAI-compatible API?
Yes. The README says llama serve launches an OpenAI-compatible API server.
Can it run models bigger than my GPU memory?
The README describes CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM.
