AI article
How to serve an OpenAI-compatible local API with llama.cpp
Community description: Install llama.cpp, serve a Hugging Face GGUF model with one command, and point the OpenAI SDK at it by changing only the base URL. Includes JSON schema output and Docker.
Dev.to | Oct 25, 2026 | Deduped
Automated excerpt
You want to test it against a free local model without rewriting anything. llama.cpp is a dependency-free C/C++ engine for running LLMs and vision language models, and its server exposes OpenAI-compatible chat completions, responses and embeddings routes. This guide installs llama.cpp, starts a server with a model pulled straight from Hugging Face, and calls it with curl and the OpenAI Python library. By default the model id is the path to the model file; set a friendlier one with --alias, for example --alias gpt-4o-mini.
Selected automatically from source text; not independently written or fact-checked. Read the original for full context.