burakaydinofficial commited on
Commit
07d2604
·
verified ·
1 Parent(s): c00b5e9

Card polish: bf16 training precision, hedged cost curve, consistency

Browse files
Files changed (1) hide show
  1. README.md +8 -8
README.md CHANGED
@@ -44,17 +44,17 @@ memory-constrained deployments where the cache cut matters most.
44
  | ka | 111.8 / 75.8 | +12.74 ✱ — floor |
45
 
46
  Absolute quality is base-tier-typical (much lower than `small` — that is the base model, not
47
- MLA). Encoder frozen both arms; 15,000 steps; warmup+cosine; fp16.
48
-
49
 
50
  **Matched control now published** — verify the conversion cost yourself: [`burakaydinofficial/whisper-base-cv11`](https://huggingface.co/burakaydinofficial/whisper-base-cv11) (trained identically, minus the MHA→MLA conversion). Evaluate both with `scripts/validate.py`.
51
 
52
  ## Limitations
53
- - **Costs are recovery-mix-specific:** do NOT compare a language's conversion cost across the cv11 and 24-language lines — different recovery sets move each language's low-rank fit (e.g. fr +0.32 here vs +1.22 in the 24-lang model). Not irreproducibility.
54
  - **What the 62.5% is (cache scope):** it is the *decode self-attention* KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream *total* decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
55
- Same as the flagship: transformers==4.46.x + trust_remote_code required; not loadable in
56
- whisper.cpp/faster-whisper/CT2; coverage = these 11 languages (unseen scripts degrade);
57
- ka reported as the labeled model-class floor; read-speech domain.
 
58
 
59
  ## Acoustic conditions of the evaluation
60
  Evaluated on crowdsourced **consumer-microphone recordings with real environmental noise** —
@@ -63,6 +63,6 @@ The numbers above already include that heterogeneity. Not yet benchmarked: far-f
63
  (8 kHz), overlapping speech; an SNR-ladder robustness section will be added when measured.
64
 
65
  ## FLEURS (out-of-domain) — this model, WER% (CER% for th/zh)
66
- ABSOLUTE FLEURS numbers on a different read-speech corpus. Do NOT compare these per-language to the CommonVoice table above — the corpora differ in transcript normalization/orthography and speaker pools (e.g. Welsh reads easier on FLEURS, Georgian worse), so cross-corpus swings are corpus artifacts, not model behavior. The clean out-of-domain claim is the PAIRED MLA-vs-control cost (verified small on the small tier). Per-language absolute: en 16.1 · de 25.8 · es 15.3 · fr 30.4 · ru 30.5 · tr 36.0 · cy 82.0 · ar 55.4 · th 34.7 · zh-CN 23.5 · ka 100.2.
67
- (Full JSON: the release repo's eval artifacts / project docs/results/2026-07-06-fleurs-family/.)
68
 
 
44
  | ka | 111.8 / 75.8 | +12.74 ✱ — floor |
45
 
46
  Absolute quality is base-tier-typical (much lower than `small` — that is the base model, not
47
+ MLA). Encoder frozen both arms; 15,000 steps; warmup+cosine; bf16 (weights released as fp16).
 
48
 
49
  **Matched control now published** — verify the conversion cost yourself: [`burakaydinofficial/whisper-base-cv11`](https://huggingface.co/burakaydinofficial/whisper-base-cv11) (trained identically, minus the MHA→MLA conversion). Evaluate both with `scripts/validate.py`.
50
 
51
  ## Limitations
52
+ - **Costs are recovery-mix-specific:** do NOT compare a language's conversion cost across the cv11 and 24-language lines — different recovery sets move each language's low-rank fit (French in the small model: +0.32 cv11 vs +1.22 24-lang). Not irreproducibility.
53
  - **What the 62.5% is (cache scope):** it is the *decode self-attention* KV-cache — the part that grows with output length and concurrency. The (larger, encoder-length ~1500-frame) cross-attention/encoder memory is NOT compressed, so single-stream *total* decode-memory savings are modest; the 62.5% cut compounds at output-length × batch concurrency, which is where it pays off.
54
+ - **Requires `trust_remote_code=True` and `transformers==4.46.x`** (custom MLA attention modules); not loadable in whisper.cpp / faster-whisper / CTranslate2.
55
+ - **Language coverage is the 11 recovery languages** unseen scripts degrade.
56
+ - Georgian (ka) reported as the labeled model-class floor; not usable at this tier.
57
+ - **Read-speech domain**, greedy-decode evals — beam-5 adds ~1-2 WER on both arms without changing the conversion cost (measured on the small tier).
58
 
59
  ## Acoustic conditions of the evaluation
60
  Evaluated on crowdsourced **consumer-microphone recordings with real environmental noise** —
 
63
  (8 kHz), overlapping speech; an SNR-ladder robustness section will be added when measured.
64
 
65
  ## FLEURS (out-of-domain) — this model, WER% (CER% for th/zh)
66
+ ABSOLUTE FLEURS numbers on a different read-speech corpus. Do NOT compare these per-language to the CommonVoice table above — the corpora differ in transcript normalization/orthography and speaker pools (e.g. Welsh reads easier on FLEURS, Georgian worse), so cross-corpus swings are corpus artifacts, not model behavior. The clean out-of-domain claim is the PAIRED MLA-vs-control cost (verified to be small on the small tier). Per-language absolute: en 16.1 · de 25.8 · es 15.3 · fr 30.4 · ru 30.5 · tr 36.0 · cy 82.0 · ar 55.4 · th 34.7 · zh-CN 23.5 · ka 100.2.
67
+ (Full JSON: the release repo's eval artifacts.)
68