huggingface/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
How transformers is put together
huggingface/transformers is the model-definition framework for state-of-the-art ML models across text, vision, audio, video, and multimodal tasks, supporting both inference and training. It is a large pure-Python library (src/transformers) whose core abstractions are PreTrainedModel/PreTrainedConfig/tokenizers/processors, with an 'auto' layer that maps model types to implementations and a Pipeline API for high-level inference. Model definitions are authored as 'modular_*' source files and mechanically converted into per-model modeling/configuration/tokenization modules under src/transformers/models/. The repo also contains extensive developer tooling: ~70 checker/utility scripts in utils/, a programmatically generated CircleCI config (.circleci/create_circleci_config.py), Dockerfiles for many hardware targets, and a v2 benchmarking framework.
Languages
Frameworks
Infrastructure
Major components
Core modeling layer (src/transformers/modeling_utils.py, modeling_layers.py, cache_utils.py, modeling_rope_utils.py, masking_utils.py)
Defines the base PreTrainedModel machinery, attention-mask utilities, KV-cache handling, RoPE, and shared layer primitives all model implementations build on.
Configuration system (configuration_utils.py, _typing.py, fusion_mapping.py)
Provides PreTrainedConfig serialization/deserialization and typed config dataclasses that every architecture's configuration_*.py derives from.
Tokenization & preprocessing (tokenization_utils_base.py, tokenization_python.py, convert_slow_tokenizer.py, processing_utils.py, feature_extraction_utils.py, image_processing_*.py)
Implements fast/slow tokenizer base classes, processor pipelines combining text/image/audio preprocessors, and GGUF tokenizer conversion.
Auto-mapping & dynamic module loading (models/auto/*, dynamic_module_utils.py, conversion_mapping.py)
Maps model type strings to config/model/tokenizer classes and supports loading code dynamically (including remote/trusted-code paths) at runtime.
Modular conversion pipeline (utils/modular_model_converter.py, utils/create_dependency_mapping.py, utils/check_modular_conversion.py, utils/modular_integrations.py)
Topologically sorts modular_*.py sources by inter-model dependencies and converts them into the generated per-model files, verifying they stay in sync.
Repo consistency checkers (utils/check_*.py with CHECKER_CONFIG dicts, utils/checkers.py, utils/checkers-requirements.txt)
A pluggable suite of style/consistency checkers (config attributes, docstrings, auto mappings sorted alphabetically, dummies, imports, copies) run via make style / make fix-repo, some inside a sandboxed normalize container.
CI orchestration (.circleci/create_circleci_config.py, parse_test_outputs.py, utils/tests_fetcher.py, utils/notification_service.py)
Programmatically builds the CircleCI workflow: per-job Docker images, pytest sharding with random-order buckets, flaky-test rerun patterns, artifact/report collection, and Slack notifications.
Benchmarking framework (benchmark_v2/)
Runs performance benchmarks with a framework of config/runner/data-class/hardware-metrics modules covering NVIDIA (nvidia-ml-py) and AMD (amdsmi) GPUs.
A quieter week focused on correctness fixes, including a big memory optimization for MoE load-balancing and a batch of small regression fixes.
5 weeksWeek of 2026-08-24
Week of 2026-08-24latest
A quieter week focused on correctness fixes, including a big memory optimization for MoE load-balancing and a batch of small regression fixes.
Week of 2026-08-17
A very busy week dominated by repairing stale integration-test values across many GPU types, plus two notable model additions.
Week of 2026-08-10
Whisper got a cluster of speculative-decoding and timestamp fixes while a wave of A10G GPU test failures was cleaned up and two new models arrived.
Week of 2026-08-03
The big story was a refactor moving all linear-attention models onto native kernels, plus the launch of an automated code-review bot that posts inline PR comments.
Week of 2026-07-27
Foundational cleanups dominated: rotary embedding modules were simplified everywhere, T5 gained modern attention backends, and old PyTorch support was dropped.
Over the past five weeks the Transformers library saw a steady stream of bug fixes and test repairs alongside several bigger pushes: new model support (ESMC/ESMFold2, CohereCompass, Granite-swa variants), a major refactor of linear-attention models onto native kernels, and expanded hardware support for XPU and MLU. A large share of effort went into stabilizing integration tests across GPU types and fixing regressions in generation, caching, and multimodal pipelines. Tooling also improved, with an automated 'mlinter' review bot posting inline PR comments and CI dashboards getting more robust.
Week by week
2026-08-24A quieter week focused on correctness fixes, including a big memory optimization for MoE load-balancing and a batch of small regression fixes.latest6 changes
Refactor
MoE load-balancing loss memory cut by ~99.7%
The load-balancing loss is now computed per layer instead of materializing giant one-hot tensors, saving enormous memory at 128k-token scales.
Fix
Fixed tied-word-embedding config regression
Some vision-language model configs no longer lose their tie_word_embeddings setting when loaded, restoring earlier behavior.
Fix
Recurrent FLA fallback repaired
The fallback path for gated-delta linear attention models was fixed so they run correctly when native kernels are unavailable.
Fix
Export crash on kernel-decorated functions fixed
Model export no longer crashes when it encounters functions decorated with kernel annotations.
Fix
LoRA training dtype mismatch resolved
Grouped matrix-multiply fallbacks used in LoRA training on Mamba-style models no longer hit dtype mismatches.
Chore
Integration tests unblocked
Several broken integration tests (serge, xcodec2) were repaired after an earlier change had regressed them.
2026-08-17A very busy week dominated by repairing stale integration-test values across many GPU types, plus two notable model additions.6 changes
Feature
ESMC and ESMFold2 protein models ported
Two protein-folding language models were brought into Transformers, expanding scientific-model coverage.
Feature
Tensor-parallel inference and training API
A new dtensor-based tensor-parallelism API landed, letting large models be split across GPUs for both inference and training.
Feature
Per-layer cache configuration
Users can now configure caching and attention-mask selection per layer, though this feature was briefly reverted and reworked within the week.
Chore
Sweep of integration-test value updates
Dozens of pull requests refreshed expected test outputs for CUDA sm_80/sm_86 GPUs across Gemma4, LLaVA, InternVL, Qwen Omni, Mistral3, EsmFold2, and more.
Fix
Whisper speculative decoding fixed
Suppressed tokens are now correctly preserved during speculative decoding, fixing broken Whisper transcriptions.
Chore
Smarter reviewer assignment
The repo now assigns a reviewer even when a code owner has left, and routes review requests by model modality.
2026-08-10Whisper got a cluster of speculative-decoding and timestamp fixes while a wave of A10G GPU test failures was cleaned up and two new models arrived.6 changes
Fix
Whisper speculative decoding overhauled
Three separate bugs — an error, cache corruption, and a speed regression — were fixed in Whisper's fast decoding path.
Feature
New model added (step 3.7)
Another new architecture joined the library as part of an ongoing model-integration series.
Feature
CohereCompass modeling added
Support for the CohereCompass model was introduced with accompanying documentation.
Fix
Emu3 image generation restored
Emu3's image generation, broken since an earlier change, works again — described cheerfully as 'the black Labrador is back'.
Chore
A10G GPU test cleanup
Expected logits and test settings were updated for A10G GPUs across OLMo, OLMoE, Gemma, AXK1, DeepSeekV2, and others to stop out-of-memory and stale-value failures.
Fix
Gemma sliding-window config bug fixed
Gemma's sliding_window setting no longer gets halved every time a config is saved and reloaded.
2026-08-03The big story was a refactor moving all linear-attention models onto native kernels, plus the launch of an automated code-review bot that posts inline PR comments.6 changes
Refactor
Linear-attention models refactored onto native kernels
All linear-attention models were restructured to use optimized native kernels with a reliable fallback, improving speed and consistency.
Chore
MLinter bot posts inline PR reviews
An automated linter now leaves its findings directly as inline comments on pull requests, with deduplication and artifact handling built up over the week.
Fix
Read-only filesystem download fix
Cached files no longer silently return stale data when running on read-only filesystems, with matching fixes applied to kernel downloads.
Fix
Multi-head latent attention (MLA) fixed
The MLA attention mechanism used by some large models was corrected after being broken.
Feature
Mistral tekken tokenizer support
AutoTokenizer can now load Mistral's native tekken tokenizer format directly.
Fix
Memory savings from static cache change
Generation no longer stores the static cache as a model attribute, reducing memory usage.
2026-07-27Foundational cleanups dominated: rotary embedding modules were simplified everywhere, T5 gained modern attention backends, and old PyTorch support was dropped.6 changes
Refactor
Rotary embedding modules simplified
All RoPE (rotary position embedding) modules across the library were consolidated into simpler, shared implementations.
Feature
SDPA enabled for T5 family
T5 and its relatives can now use faster, memory-efficient attention backends instead of only the legacy implementation.
Chore
PyTorch 2.4 support dropped
The library raised its minimum supported PyTorch version, retiring compatibility code for 2.4.
Feature
Granite-swa and Granitemoe-swa models added
Two sliding-window variants of IBM's Granite models gained first-class support.
Fix
CodeLlama tokenizer whitespace fix
The CodeLlama tokenizer no longer drops leading whitespace when converting tokens back to text.
Docs
Docs housekeeping sweep
Spelling and grammar were corrected across English documentation and model files in three batches, plus new guides for chat templates, MTP, and MPS graph caching.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
35 files reviewed; 1 of 1 claimed findings confirmed after verification.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required