vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
How vllm is put together
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs, exposed via the `vllm` CLI (`vllm.entrypoints.cli.main:main`) and an OpenAI-compatible serving stack. The codebase is polyglot: a large Python core (`vllm/`) built on PyTorch, a Rust workspace (`rust/`) providing the HTTP server (axum/hyper), tokenizer, parser, chat templating, and bench crates, plus CUDA/C++ kernels under `csrc/` wired through CMake. Behavior is driven by a hierarchy of typed `@config` dataclasses rooted at `VllmConfig` (cache, attention, compilation, parallelism, LoRA, speculative decoding, etc.) that flow into the V1 engine core, workers, and pluggable attention backends. Multi-hardware support (CUDA, ROCm, CPU/Xeon, Arm Neoverse, Intel Gaudi/HPU, TPU, XPU, s390x/ppc64le) is handled through per-platform Docker images, requirements files, and Buildkite CI pipelines including a manual performance-benchmark suite.
Languages
Frameworks
Infrastructure
Major components
vllm.config
Typed, validated configuration dataclasses (VllmConfig, CacheConfig, AttentionConfig, CompilationConfig, ParallelConfig, etc.) that parameterize every part of the engine.
vllm.compilation
Wraps torch.compile/Dynamo (guard dropping, bytecode hooks, AOT compile, NVTX tracing) to compile model forward passes and manage CUDA graph modes.
rust workspace (rust/src/*)
Rust crates for the HTTP server (axum), chat rendering, tokenizer, output text processing, reasoning/tool parsers, metrics, engine-core-client, mock-engine, and benchmarking.
csrc/ custom ops
C++/CUDA kernels and torch bindings (custom all-reduce, cumem allocator, flashkda/qutlass registration, fs_io) compiled via CMake.
vllm.entrypoints.cli
CLI entry points including `vllm serve`, `vllm bench latency/throughput/serve` subcommands.
.buildkite/performance-benchmarks
Manually triggered ~1hr benchmark suite (latency, throughput, fixed-QPS serving) configured via tests/*.json across B200/A100/H100/CPU/Gaudi/Arm hardware.
tools/
Build helpers (deepgemm, flashinfer, gdrcol, nixl installers), pre-commit validators, and the Recipes-to-vllm-config converter.
docker/
Per-hardware container image definitions (CUDA, ROCm incl. gfx1250, CPU, TPU, XPU, s390x, ppc64le) with bake files and version pinning.
A quieter week focused on bug fixes across DeepSeek V4, LoRA, and multimodal models, plus notable kernel performance wins for Qwen3.6 and batch-invariant matmul.
3 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quieter week focused on bug fixes across DeepSeek V4, LoRA, and multimodal models, plus notable kernel performance wins for Qwen3.6 and batch-invariant matmul.
Week of 2026-08-17
The busiest week of the period (293 commits), headlined by a standardized KV-cache layout refactor, Mamba prefix-caching speedups, and broad multimodal processing simplifications.
Week of 2026-08-10
DeepSeek V4 sparse-attention work dominated, with end-to-end fixes and kernel optimizations, while CI test suites were sharded into parallel jobs to cut wait times.
Over the past three weeks, vLLM's development has centered on polishing the DeepSeek V4 and Kimi K3 model stacks, with a steady stream of performance tuning for sparse attention and MoE kernels alongside dozens of targeted bug fixes. A major KV-cache layout refactor landed mid-August, unlocking better prefix caching and throughput gains for Mamba-style models, while speculative decoding (MTP/DSpark) received both new fused-kernel speedups and reliability fixes. The team also continued expanding hardware support (ROCm, XPU) and hardening CI infrastructure through test sharding to keep long evaluation suites fast.
Week by week
2026-08-24A quieter week focused on bug fixes across DeepSeek V4, LoRA, and multimodal models, plus notable kernel performance wins for Qwen3.6 and batch-invariant matmul.latest6 changes
Refactor
Fused QK-norm + partial MRoPE + gate kernel for Qwen3.6
Attention pre-processing steps were combined into one faster kernel for Qwen3.6 models.
Refactor
Tuned configs for batch-invariant persistent matmul
Per-GPU-architecture tuning delivered roughly 3x faster decode kernels on RTX 4090D and H20 cards.
Feature
FlashInfer expert backend option for DeepSeek-V4
Users can now opt in to an alternative mixture-of-experts execution path for DeepSeek V4.
Fix
Security fix for audio file size limits
The maximum audio clip size setting is now enforced on all audio input paths, closing a potential bypass.
Refactor
Sleep/wake-up functionality refactor
The code that lets the server free GPU memory when idle was reorganized for maintainability.
Chore
Removed ten deprecated model architectures
Old, no-longer-supported model definitions were cleaned out of the codebase.
2026-08-17The busiest week of the period (293 commits), headlined by a standardized KV-cache layout refactor, Mamba prefix-caching speedups, and broad multimodal processing simplifications.6 changes
Refactor
Standardized KV cache layout
The way models store their memory cache was unified across the codebase, simplifying future work.
Refactor
Mamba prefix caching with internal prefill checkpoints
New checkpointing during prompt processing improved time-to-first-token by 9-25% for Mamba-based models.
Feature
Qwen3-Omni DSpark support
Qwen's newest all-in-one audio/vision/text model now works with vLLM's speculative decoding mode.
Feature
Sparse checkpoint updates via native weight loaders
Reinforcement-learning training loops can now update only changed weights instead of reloading everything.
Fix
KV connector NIXL producer support
Fixed the external KV-cache sharing connector so it works correctly when serving as a data producer.
Fix
Restored native Hunyuan V1/VL implementations
A removal of Hunyuan model code was rolled back after issues surfaced.
2026-08-10DeepSeek V4 sparse-attention work dominated, with end-to-end fixes and kernel optimizations, while CI test suites were sharded into parallel jobs to cut wait times.6 changes
Fix
DeepSeek V4 sparse MLA working end-to-end
The efficient sparse attention mode for DeepSeek V4 now works reliably for plain decoding, MTP, and DSpark modes.
Refactor
Optimized top-k metadata kernels for DeepSeek V4 prefill
Faster selection kernels boosted prompt-processing throughput for DeepSeek V4.
Refactor
Fused MTP trailing all-reduce with local-argmax drafts
Speculative decoding got faster by combining communication steps and computing draft tokens locally.
Feature
Model Runner V2 supports attention-free models
The next-generation execution engine can now run models that don't use standard attention layers.
Chore
CI job sharding rollout
Large evaluation suites (quantization, MoE, multimodal, pooling) were split into parallel shards targeting under 30 minutes each.
Fix
Anthropic API returns proper 4xx client errors
Bad requests to the /v1/messages endpoint now get clear client-error responses instead of server errors.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
POST /score, /v1/score, /rerank, /v1/rerank, /v2/rerank, /pooling, /classify, /generative_scoring, /tokenize, /detokenize (both the Python FastAPI stack and the Rust axum stack)
An attacker with network access uses paid GPU inference for free and submits arbitrary text for embedding or scoring on a server the owner believed was token protected.
Fix: Extend GUARDED_PREFIX in authenticate.py and GUARDED_PREFIXES in rust/src/server/src/middleware/auth.rs to cover every model serving route (for example add "/score", "/rerank", "/pooling", "/classify", "/generative_scoring", "/tokenize", "/detokenize"), or mount these routers under an already guarded prefix such as /v1.
POST /scale_elastic_ep, POST /fault_tolerance/apply, POST /abort_requests (token_in_token_out variant), POST /is_scaling_elastic_ep, GET /fault_tolerance/status
Any network reachable client can drain, rescale, or disrupt the serving fleet and abort other users requests without possessing an API key.
Fix: Place these operational routers behind the authenticated prefix set (add their paths to GUARDED_PREFIX and GUARDED_PREFIXES) or gate registration on an explicit operator flag the way the profiler and LoRA routers are gated.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required