Update README.md

README.md

@@ -1,83 +1,269 @@
 ---
-library_name: transformers
 license: apache-2.0
+language:
+- nl
 base_model: openai/whisper-small
 tags:
-- generated_from_trainer
+- automatic-speech-recognition
+- whisper
+- dutch
+- speech
+- audio
+- synthetic-data
+- asr
+- hf-asr-leaderboard
+datasets:
+- mozilla-foundation/common_voice_17_0
+- yuriyvnv/synthetic_transcript_nl
 model-index:
 - name: whisper-small-mixed-cv-nl
-  results: []
+  results:
+  - task:
+      type: automatic-speech-recognition
+      name: Automatic Speech Recognition
+    dataset:
+      name: Common Voice 17.0 (Dutch)
+      type: mozilla-foundation/common_voice_17_0
+      config: nl
+      split: test
+    metrics:
+    - type: wer
+      value: 10.86
+      name: Test WER
+  - task:
+      type: automatic-speech-recognition
+      name: Automatic Speech Recognition
+    dataset:
+      name: Multilingual LibriSpeech (Dutch)
+      type: facebook/multilingual_librispeech
+      config: dutch
+      split: test
+    metrics:
+    - type: wer
+      value: 30.04
+      name: Test WER (MLS)
+pipeline_tag: automatic-speech-recognition
+library_name: transformers
 ---
 
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment. -->
-
-# whisper-small-mixed-cv-nl
-
-This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) on an unknown dataset.
-It achieves the following results on the evaluation set:
-- Loss: 0.1552
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- learning_rate: 1e-05
-- train_batch_size: 256
-- eval_batch_size: 8
-- seed: 42
-- optimizer: Use OptimizerNames.ADAMW_TORCH_FUSED with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
-- lr_scheduler_type: linear
-- lr_scheduler_warmup_ratio: 0.1
-- num_epochs: 5
-
-### Training results
-
-| Training Loss | Epoch  | Step | Validation Loss |
-|:-------------:|:------:|:----:|:---------------:|
-| 0.2429        | 0.1961 | 50   | 0.2343          |
-| 0.1317        | 0.3922 | 100  | 0.1946          |
-| 0.1159        | 0.5882 | 150  | 0.1804          |
-| 0.0959        | 0.7843 | 200  | 0.1690          |
-| 0.0927        | 0.9804 | 250  | 0.1625          |
-| 0.0626        | 1.1765 | 300  | 0.1591          |
-| 0.0623        | 1.3725 | 350  | 0.1560          |
-| 0.061         | 1.5686 | 400  | 0.1544          |
-| 0.0568        | 1.7647 | 450  | 0.1519          |
-| 0.0587        | 1.9608 | 500  | 0.1484          |
-| 0.0373        | 2.1569 | 550  | 0.1499          |
-| 0.0338        | 2.3529 | 600  | 0.1504          |
-| 0.036         | 2.5490 | 650  | 0.1502          |
-| 0.0368        | 2.7451 | 700  | 0.1499          |
-| 0.0354        | 2.9412 | 750  | 0.1484          |
-| 0.0253        | 3.1373 | 800  | 0.1509          |
-| 0.0247        | 3.3333 | 850  | 0.1515          |
-| 0.0234        | 3.5294 | 900  | 0.1529          |
-| 0.0247        | 3.7255 | 950  | 0.1516          |
-| 0.0237        | 3.9216 | 1000 | 0.1522          |
-| 0.0189        | 4.1176 | 1050 | 0.1538          |
-| 0.0191        | 4.3137 | 1100 | 0.1549          |
-| 0.0183        | 4.5098 | 1150 | 0.1549          |
-| 0.0194        | 4.7059 | 1200 | 0.1554          |
-| 0.0198        | 4.9020 | 1250 | 0.1552          |
-
-
-### Framework versions
-
-- Transformers 4.50.2
-- Pytorch 2.5.1+cu124
-- Datasets 3.6.0
-- Tokenizers 0.21.2
+# Whisper-Small Dutch - Mixed Synthetic Data (Mid-High Quality Filtered)
+
+This model is a fine-tuned version of [openai/whisper-small](https://huggingface.co/openai/whisper-small) for Dutch automatic speech recognition (ASR). It was trained on Common Voice 17.0 Dutch combined with **WAVe-filtered synthetic speech data** (quality threshold q ≥ 0.5).
+
+## Introduction
+
+### How the Data Was Created
+
+The training data combines real speech from Common Voice 17.0 with synthetic speech generated through a two-stage pipeline:
+
+1. **Transcript Generation**: We used GPT-4o-mini to generate Dutch transcripts that match the word count distribution observed in Common Voice, ensuring realistic utterance lengths and diverse linguistic content.
+
+2. **Speech Synthesis**: Each transcript was converted to audio using OpenAI's TTS-1 model with 9 different voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer), producing 34,898 synthetic samples.
+
+3. **Quality Filtering with WAVe**: Raw synthetic speech often contains defects such as mispronunciations, omitted words, or prosodic anomalies. To address this, we applied **WAVe (Word-Aligned Verification)**, a model that assesses audio-text alignment at the word level rather than the sentence level. WAVe uses multi-head attention to align each word to its corresponding audio frames and assigns per-word confidence scores via a GLU-based scorer. Samples scoring below the threshold (q < 0.5) were removed, retaining 30,182 high-quality synthetic samples.
+
+### How the Model Was Created
+
+The model was fine-tuned from `openai/whisper-small` using the Hugging Face Transformers library with the following approach:
+
+1. **Mixed Training**: Combined 34,952 real speech samples from Common Voice 17.0 Dutch with 30,182 WAVe-filtered synthetic samples (65,134 total).
+
+2. **Optimization**: Trained for 5 epochs with a learning rate of 1e-5, global batch size of 256, and BF16 precision on an NVIDIA H200 GPU.
+
+3. **Checkpoint Selection**: The best checkpoint was selected based on validation loss, occurring at step 500 with a validation loss of 0.1484.
+
+This approach achieves the **best Test WER (10.86%)** among all Whisper-Small Dutch configurations while maintaining strong cross-domain generalization.
+
+## Model Details
+
+| Property | Value |
+|----------|-------|
+| **Base Model** | openai/whisper-small |
+| **Language** | Dutch (nl) |
+| **Task** | Automatic Speech Recognition (transcribe) |
+| **Parameters** | 244M |
+| **Training Data** | Common Voice 17.0 + Mid-High Quality Synthetic (q ≥ 0.5) |
+| **Total Training Samples** | 65,134 |
+| **Sampling Rate** | 16kHz |
+
+## Evaluation Results
+
+### This Model (whisper-small-mixed-cv-nl)
+
+| Metric | Value |
+|--------|-------|
+| **Validation Loss** | 0.1484 |
+| **Validation WER** | 8.73% |
+| **Test WER (Common Voice)** | 10.86% |
+| **Test WER (MLS)** | 30.04% |
+| **Best Checkpoint** | Step 500 |
+| **Max Training Steps** | 1,270 |
+
+### Comparison with Other Training Configurations (Whisper-Small Dutch)
+
+| Training Data | Max Steps | Val Loss | Val WER | Test WER (CV) | Test WER (MLS) |
+|---------------|-----------|----------|---------|---------------|----------------|
+| Common Voice Only | 680 | 0.1491 | 8.73% | 11.13% | 30.71% |
+| High-Quality Filtered + CV | 890 | 0.1493 | 8.76% | 11.00% | 29.91% |
+| **Mid-High Quality Filtered + CV** | **1,270** | **0.1484** | **8.73%** | **10.86%** | **30.04%** |
+| All Synthetic + CV (Unfiltered) | 1,365 | 0.1484 | 8.64% | 10.91% | 30.06% |
+
+### Key Performance Highlights
+
+- **Best Test WER** (10.86%) on Common Voice among all Whisper-Small Dutch configurations
+- **2.4% relative improvement** on Common Voice test set vs baseline (10.86% vs 11.13%)
+- **2.2% relative improvement** on MLS benchmark vs baseline (30.04% vs 30.71%)
+- **7% fewer training steps** than unfiltered synthetic data while achieving better in-domain performance
+
+## Training Data
+
+### Dataset Composition
+
+| Source | Samples | Description |
+|--------|---------|-------------|
+| [Common Voice 17.0 Dutch](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0) | 34,952 | Real speech from Mozilla's crowdsourced dataset |
+| [Synthetic Transcript NL](https://huggingface.co/datasets/yuriyvnv/synthetic_transcript_nl) (q ≥ 0.5) | 30,182 | WAVe-filtered TTS audio from GPT-4o-mini transcripts |
+| **Total** | **65,134** | |
+
+### Synthetic Data Generation Pipeline
+
+The synthetic dataset ([yuriyvnv/synthetic_transcript_nl](https://huggingface.co/datasets/yuriyvnv/synthetic_transcript_nl)) was generated using:
+
+1. **Transcript Generation**: GPT-4o-mini, matching Common Voice word count distribution
+2. **Speech Synthesis**: OpenAI TTS-1 model with 9 voice variants (alloy, ash, coral, echo, fable, nova, onyx, sage, shimmer)
+3. **Quality Filtering**: WAVe model filtering at threshold q ≥ 0.5
+
+### WAVe Quality Distribution (Dutch Synthetic Data)
+
+| Quality Level | Samples | Percentage | Used in This Model |
+|--------------|---------|------------|-------------------|
+| High (q ≥ 0.8) | 10,555 | 30.2% | ✓ |
+| Medium (0.5 ≤ q < 0.8) | 19,627 | 56.2% | ✓ |
+| Low (q < 0.5) | 4,716 | 13.5% | ✗ |
+
+## Training Procedure
+
+### Hyperparameters
+
+| Parameter | Value |
+|-----------|-------|
+| Learning Rate | 1e-5 |
+| Batch Size (Global) | 256 |
+| Warmup Steps | 200 |
+| Max Epochs | 5 |
+| Precision | BF16 |
+| Optimizer | AdamW (fused) |
+| Eval Steps | 50 |
+| Metric for Best Model | eval_loss |
+
+### Training Infrastructure
+
+- **GPU**: NVIDIA H200 (140GB VRAM)
+- **Operating System**: Ubuntu 22.04
+- **Framework**: Hugging Face Transformers
+
+### Training Curve
+
+```
+Step  100: val_loss = 0.1946
+Step  250: val_loss = 0.1625
+Step  400: val_loss = 0.1544
+Step  500: val_loss = 0.1484 ← Best checkpoint
+Step  750: val_loss = 0.1484
+Step 1000: val_loss = 0.1522
+Step 1250: val_loss = 0.1552
+```
+
+## Usage
+
+### Transcription Pipeline
+
+```python
+from transformers import pipeline
+
+transcriber = pipeline(
+    "automatic-speech-recognition",
+    model="yuriyvnv/whisper-small-mixed-cv-nl",
+    device="cuda"
+)
+
+result = transcriber("path/to/dutch_audio.wav")
+print(result["text"])
+```
+
+### Direct Model Usage
+
+```python
+from transformers import WhisperProcessor, WhisperForConditionalGeneration
+import librosa
+
+processor = WhisperProcessor.from_pretrained("yuriyvnv/whisper-small-mixed-cv-nl")
+model = WhisperForConditionalGeneration.from_pretrained("yuriyvnv/whisper-small-mixed-cv-nl")
+model.to("cuda")
+
+audio, sr = librosa.load("path/to/dutch_audio.wav", sr=16000)
+input_features = processor(audio, sampling_rate=16000, return_tensors="pt").input_features.to("cuda")
+
+predicted_ids = model.generate(input_features)
+transcription = processor.batch_decode(predicted_ids, skip_special_tokens=True)[0]
+print(transcription)
+```
+
+### Specifying Language
+
+```python
+model.generation_config.language = "nl"
+model.generation_config.task = "transcribe"
+```
+
+## Methodology
+
+This model leverages **WAVe (Word-Aligned Verification)**, a word-level quality assessment method for filtering synthetic speech data. Unlike sentence-level filtering approaches, WAVe:
+
+- Aligns each word to its corresponding audio frames using multi-head attention
+- Assigns per-word confidence scores via a GLU-based scorer
+- Detects localized synthesis errors (mispronunciations, omitted words, prosodic anomalies)
+- Achieves **6.5% improvement** over sentence-level filtering methods
+
+For full methodology details, see the references below.
+
+## When to Use This Model
+
+This model is ideal when:
+- **Best in-domain accuracy is required**: Achieves 10.86% Test WER (best among Small Dutch models)
+- **Balanced performance**: Good tradeoff between in-domain and cross-domain generalization
+- **Moderate compute budget**: 7% fewer steps than unfiltered approach
+
+Consider alternatives based on your needs:
+- [whisper-small-high-mixed-nl](https://huggingface.co/yuriyvnv/whisper-small-high-mixed-nl): 35% fewer steps, slight accuracy tradeoff
+- [whisper-small-cv-fully-synthetic-nl](https://huggingface.co/yuriyvnv/whisper-small-cv-fully-synthetic-nl): Maximum data, similar performance
+
+## Limitations
+
+- **Domain specificity**: Optimized for general Dutch; may underperform on technical domains
+- **Acoustic conditions**: Trained on clean speech; noise robustness not guaranteed
+- **Dialect coverage**: Performance may vary across Dutch regional variants
+
+## Citation
+
+```bibtex
+@article{perezhohin2024enhancing,
+  title={Enhancing Automatic Speech Recognition: Effects of Semantic Audio Filtering on Models Performance},
+  author={Perezhohin, Yuriy and Santos, Tiago and Costa, Victor and Peres, Fernando and Castelli, Mauro},
+  journal={IEEE Access},
+  year={2024},
+  publisher={IEEE}
+}
+```
+
+## References
+
+- **Base Model**: [openai/whisper-small](https://huggingface.co/openai/whisper-small)
+- **Training Data (Real)**: [mozilla-foundation/common_voice_17_0](https://huggingface.co/datasets/mozilla-foundation/common_voice_17_0)
+- **Training Data (Synthetic)**: [yuriyvnv/synthetic_transcript_nl](https://huggingface.co/datasets/yuriyvnv/synthetic_transcript_nl)
+- **Whisper Paper**: [Robust Speech Recognition via Large-Scale Weak Supervision](https://arxiv.org/abs/2212.04356)
+- **IEEE Access Paper**: [Enhancing ASR with Semantic Audio Filtering](https://ieeexplore.ieee.org/document/10720758)
+
+## License
+
+Apache 2.0