inference
Serve LLMs, speech, and multimodal models via a unified API.
Run LLMs, speech, and multimodal models on cloud, on-prem, or your laptop using a unified inference API.
Xorbits Inference lets you deploy language, speech recognition, and multimodal models using a single command. It suits developers and data scientists who need to serve open-source AI models with an OpenAI-compatible RESTful API.
What it is
It manages model serving across heterogeneous hardware, supporting distributed clusters, automated request batching, and integration with third-party tools like LangChain and Dify. It provides multiple engines including GGML, vLLM, and TensorRT.
Who it's for
It suits engineers building local or distributed AI applications who want to swap out model backends without rewriting code. It does not suit users looking for a traditional out-of-the-box video editing application without code interaction.
Key features
Multi-engine serving
Supports vLLM, GGML, TensorRT, and custom backends through a unified API.
OpenAI-compatible API
Provides RESTful endpoints including Function Calling for easy drop-in replacement.
Heterogeneous hardware
Intelligently utilizes both GPUs and CPUs to accelerate inference tasks.
Distributed deployment
Seamlessly distributes model inference across multiple devices or machines.
Up and running in 3 steps
Our honest verdict
What's great
- Drop-in replacement for OpenAI endpoints by changing a single line of code
- Built-in support for text-to-image, audio, and multimodal models
- Includes distributed cluster deployment and auto-batching capabilities
- Integrates natively with LangChain, LlamaIndex, and Dify
Where it falls short
- Requires technical skill to set up and manage model weights and hardware resources
- Running large language and multimodal models demands significant GPU/CPU capacity
- Self-hosting involves ongoing infrastructure maintenance and monitoring costs
Pricing & license
The software is free under its open-source license, and the only cost is running the infrastructure yourself unless you opt for enterprise offerings.
Open remote AI-training roles
766 open remote AI-training roles, updated November 14, 2026; most pay $20–$175 an hour. Zekai lists them free; the application goes to the employer’s platform.
Alternatives
Use it, or have us set it up
FAQ
Can I use this as a drop-in replacement for OpenAI?
Yes, it provides an OpenAI-compatible RESTful API that includes Function Calling.
What hardware platforms are supported?
It supports heterogeneous hardware including GPUs and CPUs across cloud, on-prem, and laptops.
Does it support multimodal models?
Yes, it natively serves language, speech recognition, text-to-image, and multimodal models.
