openai/whisper
Robust Speech Recognition via Large-Scale Weak Supervision
How whisper is put together
Whisper is OpenAI's general-purpose speech recognition library: a Transformer sequence-to-sequence model implemented in Python/PyTorch that performs multilingual transcription, speech translation, and language identification. The architecture is a classic ML inference pipeline: audio files are loaded and resampled to 16kHz mono (whisper/audio.py via ffmpeg), converted to log-Mel spectrograms, then decoded autoregressively over sliding 30-second windows by the model defined in whisper/model.py using beam/greedy search logic in whisper/decoding.py. A tokenizer layer (whisper/tokenizer.py) built on tiktoken handles multilingual text tokens and special task tokens, while whisper/transcribe.py orchestrates the full pipeline and exposes both a Python API and a 'whisper' command-line entry point. There is no server, datastore, or network service component; it is an offline inference library plus packaged model/tokenizer assets.
Languages
Frameworks
Infrastructure
Major components
whisper.transcribe
High-level transcription pipeline and CLI: loads models, windows audio into 30-second segments, runs decode passes, applies word-level timing, and exposes output formats.
whisper.model (Whisper)
Defines the Transformer encoder-decoder architecture, model dimensions for each size variant, and load_model() which downloads/fetches pretrained weights.
whisper.decoding
Implements greedy and beam-search decoding, DecodingOptions/DecodingResult, temperature fallback, and language detection over mel spectrograms.
whisper.tokenizer
Wraps tiktoken BPE tokenizers (English GPT-2 and multilingual vocabularies), defines special tokens, task/language tokens, and the supported language list.
whisper.audio
Audio ingestion: loads/resamples arbitrary media to 16kHz float32 PCM via ffmpeg, pads/trims to 30-second frames, and computes log-Mel spectrograms using bundled filter banks.
whisper.timing
Word-level timestamp extraction via cross-attention alignment (DTW) between decoder attention and the medians-filtered spectrogram.
whisper.triton_ops
Optional Triton GPU kernel optimizations (e.g., fused QKV matmul) used on Linux x86_64 when triton is available.
whisper.normalizers
Text normalization utilities (basic punctuation/casing rules and English-specific normalizations) primarily for evaluation/WER computation.
Whisper got a reliability fix for its search mode and a new JSON-lines output option.
This week's work on the Whisper repository focused on two practical improvements: making transcription more reliable when using beam search, and giving users a new, machine-friendly way to get their results. Together these changes improve both accuracy and how easily Whisper's output can be used by other tools.
Fix
Fixed a slowdown in beam search
Corrected an issue where a faster built-in method for processing audio was being skipped during beam search, causing transcriptions to run slower than they should.
Feature
Added JSONL output format
Users can now save transcripts as one small JSON record per line, which makes it much easier to feed results into other programs and tools.
Dependencies and code review
Dependency advisories
Security Watch
No known advisories across 0 scanned dependencies.
No known advisories in the scanned dependencies.
Code review
No concrete code-level issues confirmed.
Get this report every week for your repos.
GitZoid learns each repo, reports what changed, and flags what needs attention. One flat price for the whole team.
$19 a month, flat · First 10 outputs free · No card required