How llama.cpp is put together
llama.cpp implements LLM and vision-language model inference in plain C/C++ on top of the ggml tensor library, targeting a wide range of hardware through pluggable backends (CUDA, Metal, Vulkan, SYCL, HIP, OpenCL, RPC, and more). The repo is layered into: the ggml compute engine, hardware backends, the core llama inference library, a shared 'common' utility layer (argument parsing, sampling, grammar/chat parsing, downloads, HF cache), user-facing tools (cli, server, multimodal mtmd), Python model-conversion pipelines (convert_*.py + conversion/ + gguf-py), and a SvelteKit-based web UI (tools/ui). Data flows from Hugging Face or local GGUF model files through loaders into ggml compute graphs dispatched across CPU/GPU backends, while llama-server exposes an OpenAI-compatible HTTP API built on cpp-httplib. Build orchestration is CMake-first with Docker images per backend and CI workflows for server/Docker/winget.
Languages
Frameworks
Infrastructure
Major components
ggml compute engine
Underlying tensor math/graph execution library (vendored submodule per README/build docs) providing quantized ops and scheduling.
Hardware backends
Pluggable acceleration layers for CUDA/HIP/MUSA, Metal, Vulkan, SYCL, OpenCL, BLAS/BLIS, CANN, zDNN, ZenDNN, WebGPU, RPC, enabling CPU+GPU hybrid inference.
common support library
Shared utilities: argument parsing (arg.cpp), sampling, speculative decoding, GBNF/json-schema grammar parsing (json-schema-to-grammar.cpp, peg-parser.cpp), chat handling, Unicode, n-gram caches, and model downloading (download.cpp, hf-cache.cpp).
llama-server tool
OpenAI-compatible HTTP REST API server plus built-in web UI, built on the cpp-httplib single-header server (tools/server).
CLI and other tools
End-user tools such as llama cli (tools/cli), completion, and RPC, including multimodal (mtmd) support for image/audio input using stb and miniaudio decoders.
Model conversion pipeline
Python scripts (convert_hf_to_gguf.py, convert_lora_to_gguf.py, etc.) plus conversion/ package of per-architecture ModelBase subclasses that emit GGUF files.
gguf-py library
Standalone pip-installable Python package for reading/writing GGUF, including tensor_mapping.py mapping hundreds of upstream tensor names to GGUF tensors and CLI utilities (dump, metadata editing, GUI editor).
web UI (tools/ui)
SvelteKit 5 + TypeScript PWA client with Tailwind, Storybook, Playwright e2e tests, markdown/KaTeX rendering (remark/rehype), and IndexedDB persistence via Dexie.
A quieter week focused on version bumps to 0.3.0/0.22.0, crash fixes, and build-system improvements like ccache storage on Hugging Face.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quieter week focused on version bumps to 0.3.0/0.22.0, crash fixes, and build-system improvements like ccache storage on Hugging Face.
Week of 2026-08-17
A very busy week (149 commits) adding DeepSeek V4 fixes, GLM-4.5-Air MTP support, tabbed chat navigation in the UI, and a release-preparation script.
Week of 2026-08-10
New flagship models landed (Kimi-K3, MiniMax), the server got a redesigned scheduling thread model, and recurrent-state rollback arrived in ggml.
Week of 2026-08-03
Server security and usability took center stage with Docker-isolated tools and a working-directory picker, plus CUDA kernel fusion and a 3x faster test suite.
Week of 2026-07-27
A short week dominated by DeepSeek V4 groundwork: its MTP draft mechanism, the DSpark sidecar format, hyper-connections, and assorted GPU backend fixes.
Over the past five weeks llama.cpp saw a steady stream of new model support (DeepSeek V4 with its DSpark sidecar and MTP draft models, Kimi-K3, MiniMax, GLM-4.5-Air MTP), alongside a major version bump to 0.3.0 and a wave of performance work fusing GPU operations on CUDA, SYCL, Metal, and OpenCL. The built-in web UI gained tabbed chat navigation, filesystem mentions, and supply-chain protections, while the server added Docker-based tool isolation and a smarter model router. Infrastructure also matured, with ccache caching moved to Hugging Face buckets and a dedicated release script introduced.
Week by week
2026-08-24A quieter week focused on version bumps to 0.3.0/0.22.0, crash fixes, and build-system improvements like ccache storage on Hugging Face.latest6 changes
Chore
Version bump to 0.3.0
The main project was bumped to version 0.3.0 (with the underlying ggml library bumped to 0.22.0), marking a new release milestone.
Refactor
Faster CI builds via cloud ccache
Continuous integration now stores compiler caches in Hugging Face buckets so rebuilds can be faster, starting with a CUDA Ubuntu test job.
Fix
Metal out-of-memory crash fix
Apple-GPU users no longer crash when memory runs out during buffer allocation; it now fails gracefully instead.
Feature
Faster Metal shader compilation
Metal GPU code was split into per-operation files that compile in parallel, speeding up builds and enabling per-device tuning of flash attention.
Feature
DeepSeek V4 tensor splitting
Added support for running DeepSeek V4 split across multiple GPUs at the tensor level.
Fix
Grammar hyphen parsing fix
Structured-output grammars now correctly treat an escaped hyphen inside character classes as a literal dash rather than a range.
2026-08-17A very busy week (149 commits) adding DeepSeek V4 fixes, GLM-4.5-Air MTP support, tabbed chat navigation in the UI, and a release-preparation script.6 changes
Feature
GLM-4.5-Air draft-model support
The server can now use GLM-4.5-Air's MTP (multi-token prediction) module to draft several tokens at once for faster generation.
Feature
Tabbed chat conversations
The web UI now lets you open multiple chats as tabs and switch between them with keyboard shortcuts.
Fix
DeepSeek V4 multi-sequence fix
Fixed a bug where DeepSeek V4 could roll back state incorrectly when handling multiple conversation sequences.
Chore
Release tooling
A new release.sh script automates preparing releases, and the project was bumped to version 0.2.0 as part of the process.
Refactor
Faster concatenation operation
The core concat operation now copies whole rows at once instead of element-by-element, making it noticeably faster.
Feature
dots3-note vision+audio support
Added support for the dots3-note architecture's vision and audio inputs, plus WebP image loading via ffmpeg.
2026-08-10New flagship models landed (Kimi-K3, MiniMax), the server got a redesigned scheduling thread model, and recurrent-state rollback arrived in ggml.6 changes
Feature
Kimi-K3 text model support
Users can now run the Kimi-K3 text model locally in llama.cpp.
Feature
MiniMax model family support
Added support for both MiniMaxText01 and MiniMaxM1 architectures.
Refactor
Server queue redesign
The server's request-scheduling thread model was reworked to yield more gracefully between queued requests.
Feature
Recurrent state rollback in ggml
The core library can now roll back internal state for recurrent models, enabling reliable branching of conversations.
Fix
Template speedup
Fixed a quadratic slowdown in chat-template string gathering that could make long prompts slow to process.
Feature
MTP assistant models from a folder
You can now point --models-dir at a folder and have multi-token-prediction assistant models load automatically.
2026-08-03Server security and usability took center stage with Docker-isolated tools and a working-directory picker, plus CUDA kernel fusion and a 3x faster test suite.6 changes
Feature
Docker-isolated tool execution
Server-side tools can now run inside isolated Docker containers with their own working directory, keeping them sandboxed away from the host.
Feature
Working directory picker in UI
The web UI lets you choose which working directory a tool may access, and only offers one when a tool actually needs it.
Refactor
CUDA kernel fusion
On NVIDIA GPUs, normalization, multiplication, and rotary-position operations are now fused into single kernels for speed.
Chore
3x faster server tests
The server test suite was optimized to run three times faster, shortening developer feedback loops.
Feature
Router LRU scheduler
The multi-model router now evicts the least-recently-used models when memory is needed, while never evicting models currently in use.
Fix
Supply-chain protection for npm
The UI's package manager now enforces a minimum release age for dependencies, protecting against freshly published malicious packages.
2026-07-27A short week dominated by DeepSeek V4 groundwork: its MTP draft mechanism, the DSpark sidecar format, hyper-connections, and assorted GPU backend fixes.6 changes
Feature
DeepSeek V4 MTP + DSpark
Initial support for DeepSeek V4's multi-token-prediction drafting and its companion DSpark sidecar file format landed.
Feature
DSpark sidecar conversion option
Model conversion can now produce a separate DSpark GGUF file alongside the main model weights.
Feature
DeepSeek V4 hyper-connections on Metal
Apple GPUs can now run DeepSeek V4's hyper-connection layers natively.
Feature
Qwen3 specialized parser
Chat responses from Qwen3 models are now parsed by a dedicated parser for more accurate tool-call and formatting handling.
Feature
Metal F16 binary operations
Metal gained half-precision support for basic math operations, plus a SILU backward pass for training-style workloads.
Fix
iGPU classification fixes
Integrated GPUs are now correctly classified on SYCL and OpenCL backends, avoiding misapplied optimizations.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required