rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4

Distil-Whisper large-v3 — PyTorch Q4 (NF4) Quantized (GPU - bitsandbytes)

This model is a quantized/optimized version of the baseline Distil-Whisper (distil-large-v3) model. It is part of a benchmarked suite of quantized models evaluated on local hardware.

This model is part of a suite of optimized/quantized versions of the base model. Other variants in this direction:

• PyTorch Q4 (GPU Quantized - NF4): rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4 (Current)
• CTranslate2 INT8 (CPU Quantized): rudrakshrakeshzodage/distil-whisper-large-v3-ct2-int8

Base model: distil-whisper/distil-large-v3

---

📊 Quantization & Performance Benchmark Results

Below is the comparative performance table generated empirically using 8 CPU threads / cores:

Backend	Precision	Device	Model Size (GB)	Mean Latency (s)	Throughput (Words/s)	RTF	WER (%)	Peak RAM (GB)	Peak VRAM (GB)
PYTORCH	FP32	CUDA	3.024	0.427	44.81	0.064	5.29%	3.62	2.98
PYTORCH	FP16	CUDA	1.512	0.189	100.25	0.028	5.29%	2.12	1.46
PYTORCH	INT8	CUDA	0.756	0.293	64.00	0.042	5.29%	1.92	0.85
PYTORCH	Q4	CUDA	0.378	0.377	50.63	0.056	5.29%	1.93	0.60
GGML	FP16	CPU	1.409	17.269	1.14	2.633	5.29%	4.14	0.00
GGML	INT8	CPU	1.409	9.271	2.18	1.441	5.29%	1.50	0.00
GGML	Q5	CPU	1.409	16.735	1.18	2.566	5.29%	2.28	0.00

💾 Model Size Reduction Comparison

📈 Transcription Speed & Latency Comparison

🚀 Words Transcribed per Second (Throughput)

---

🚀 Usage

import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, BitsAndBytesConfig

model_id = "rudrakshrakeshzodage/distil-whisper-large-v3-pytorch-q4"

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id,
    quantization_config=quant_config,
    device_map="cuda"
)
processor = AutoProcessor.from_pretrained(model_id)

---

📜 Citation & Credits

Quantization research and benchmarking by RudrakshRakeshZodage.