Sauti TTS — Dia-1.6B Swahili (full SFT)

Open Swahili text-to-speech from Msingi-AI: a full supervised fine-tune of nari-labs/Dia-1.6B on a 500-hour Swahili speech corpus pooled from 15 openly-licensed public datasets.

This is a single-voice model — it takes text and speaks it in one learned voice. It does not clone voices and needs no reference audio.

It handles code-switching, English embedded in Swahili sentences, which is how a great deal of Swahili is actually spoken and where TTS models commonly break down: code-switch CER 0.050 against a multilingual ASR judge.

	Plain Swahili	Code-switched
CER	0.011	0.050
WER	0.054	0.286

Plain Swahili is scored by a Swahili-tuned ASR judge, code-switch by a multilingual one — for a reason. Full tables in Evaluation.

Read this before quoting the numbers. The figures above come from a curated 48-sentence set, generated with ASR-gated retries (up to 4 attempts). On a harder 500-sentence benchmark of unseen news text, plain-Swahili CER is 0.021 with retries and 0.060 single-shot. Retries matter a lot — see Release benchmark for the full picture and for why you should use the retry loop in production.

---

⚠️ Read this first: the [sw] language tag requires a source patch

Dia's tokenizer maps a language tag to a byte via a LANG2BYTE table that does not contain Swahili upstream. This model was trained with sw mapped to byte 8. If you load it with an unpatched checkout, [sw] is fed as literal ASCII bytes and inference silently disagrees with training — output degrades to babble or wrong-language prosody.

Patch both dia/model.py and the training file before use:

LANG2BYTE = {"sw": 8, ...}   # add "sw": 8 as the first entry

Our patcher does this idempotently and fails loudly if upstream layout changes: hpc/dia/patch_dia_fork.py in the sauti-tts-v2 repo.

Text must be formatted exactly as trained — bare text with the tag, no [S1] dialogue tags:

text = f"[sw]{normalized_swahili_text}"     # ✅ matches training
text = f"[S1] {swahili_text}"               # ❌ never used in training

---

Usage

Requires the stlohrey/dia-finetuning fork (the upstream dia package lacks the multilingual tag path), patched as above.

Setup

# 1. environment (torch 2.5.1+cu121 is the combination we validated)
pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
git clone https://github.com/stlohrey/dia-finetuning && pip install -e dia-finetuning
pip install descript-audio-codec soundfile

# 2. patch LANG2BYTE for Swahili (REQUIRED — see warning above)
python sauti-tts-v2/hpc/dia/patch_dia_fork.py --dia-dir dia-finetuning \
    --epochs 1 --eval-step 1 --save-step 1

# 3. weights
hf download msingiai/dia --local-dir ./dia-sw

import dac, torch
from dia.config import DiaConfig
from dia.layers import DiaModel
from dia.model import Dia

device = torch.device("cuda")
cfg = DiaConfig.load("config.json")
model = DiaModel(cfg)
model.load_state_dict(torch.load("model.pth", map_location="cpu"))

# DiaModel builds layers in config.training.dtype (bfloat16) but this
# checkpoint is fp32 — cast to float or attention crashes on mixed dtypes.
model = model.float().to(device).eval()

engine = Dia(cfg, device)
engine.model = model
engine.dac_model = dac.DAC.load(dac.utils.download()).to(device)

with torch.inference_mode():
    wav = engine.generate(
        text="[sw]Habari za asubuhi, karibu katika matangazo yetu ya leo.",
        temperature=1.3,     # our eval default; see note below
    )
# -> float32 numpy array, 44.1 kHz mono

Post-processing (recommended)

Raw takes contain occasional sample-level transients. Our eval harness applies declick() plus a 12 ms raised-cosine fade at both ends (scripts/declick.py, scripts/synth_finetuned_dia.py). After that repair, every delivered clip starts and ends at exactly zero and near-silence tick levels are flat across epochs — measured, not assumed (output/dia_samples/measure_artifacts.py).

Do not skip the fade: it is what removes edge clicks.

Temperature and ASR-gated retry

Generation is sampling-sensitive — failures show up as instant-EOS (empty audio) or babble, and re-rolling fixes them. For batch work our harness retries with a temperature ladder [1.3, 1.0, 1.2, 0.9, ...] until a check-ASR (openai/whisper-small) agrees with the input at CER ≤ 0.30, max 4 attempts. For low-latency serving, use a single attempt at temperature=1.3 and handle empty output by retrying.

---

Evaluation

48-sentence held-out set, stratified across general / named-entities / numbers-dates / code-switch. Two ASR judges: zero-shot openai/whisper-large-v3 and Swahili-tuned Jacaranda-Health/ASR-STT.

Released checkpoint (epoch 10):

Scope	whisper-large-v3 WER / CER	Jacaranda ASR WER / CER
overall_plain_sw (headline)	0.243 / 0.050	0.054 / 0.011
overall (incl. code-switch)	0.254 / 0.050	0.111 / 0.030
general	0.215 / 0.043	0.081 / 0.018
named_entities	0.249 / 0.040	0.056 / 0.010
numbers_dates	0.267 / 0.067	0.025 / 0.006
code_switch	0.286 / 0.050	0.281 / 0.088

Release benchmark: 500 unseen sentences, single-shot vs retried

The table above is our curated 48-clip set with ASR-gated retries. Because that is both small and curated, we also ran a 500-sentence benchmark on Swahili news text (MasakhaNEWS) that neither this model nor its training corpus has seen — each sentence checked against the actual training text and dropped on exact, near-duplicate or shared-5-gram match.

We report it both ways, because the difference is large and you should plan for it:

Scope (Swahili judge, WER / CER)	Single-shot	ASR-gated (≤4 attempts)
overall_plain_sw	0.154 / 0.060	0.095 / 0.021
general	0.081 / 0.015	0.083 / 0.015
named_entities	0.285 / 0.133	0.115 / 0.025
numbers_dates	0.097 / 0.033	0.088 / 0.025
code_switch	0.112 / 0.036	0.112 / 0.036

What this means in practice:

1. Use the retry loop in production. It cuts plain-Swahili CER by ~65% (0.060 → 0.021). Generation is sampling-sensitive; a failed take is usually fixed by re-rolling, and one check-ASR pass is far cheaper than shipping bad audio.
2. Named entities are the weak spot. Single-shot they degrade 5× (CER 0.133 vs 0.025 gated) — proper nouns are rare tokens and destabilise sampling. If your text is name-heavy (news, directories, announcements), retries are not optional.
3. Code-switch is stable either way — identical with and without retries, so its quality comes from the model rather than from re-rolling.
4. Numbers on this set are higher than on the curated 48-clip set (0.021 vs 0.011 gated). News prose is harder and carries source typos and quote-splitting artefacts that count against the model. Use the 48-clip figure only for comparison against our other models on that same set; use this table to predict real-world behaviour.

Read code-switch with the multilingual judge only

Jacaranda-Health/ASR-STT is Swahili-only and cannot transcribe English words, so it scores code-switch backwards. It rated epoch 9 (0.108) worse than epoch 4 (0.101); the multilingual whisper-large-v3 rated epoch 9 better (0.067 vs 0.074) — and the native listener agreed with the multilingual judge. For any code-switch decision, use whisper-large-v3.

Epoch sweep (all 10 epochs)

Plain CER improves monotonically while code-switch does not — which is why the released checkpoint is epoch 10, not the best-plain-CER epoch 9.

Epoch	Plain CER (SW judge)	Code-switch CER (multilingual)
1	0.460	0.126
2	0.022	0.093
3	0.021	0.068
4	0.015	0.074
5	0.013	0.051
6	0.010	0.051
7	0.011	0.063
8	0.012	0.048
9	0.008	0.067
10 (released)	0.011	0.050

Epochs 8, 9 and 10 were auditioned by a native Swahili listener, who approved 9 and 10. Epoch 10 was released for its better code-switch score at statistically indistinguishable plain-Swahili quality.

UTMOS was deliberately not run: it is English-trained and unreliable for Swahili.

How it compares in our internal bake-off

Same 48-clip set, same judges, so these are directly comparable. The other rows are internal reference points and are not part of this release.

Model	Plain SW CER	Code-switch CER (multilingual)	Voice cloning
VoxCPM2 full SFT (not released)	0.007	0.047	✅ zero-shot
this model (Dia full SFT e10)	0.011	0.050	❌ single voice
VoxCPM2 LoRA, 300 steps (not released)	0.012	—	✅
Chatterbox LoRA e30 (not released)	0.022	—	prompt-based
CosyVoice3 e1 (not released)	0.029	—	✅

We publish the comparison rather than only our best number: on plain Swahili a VoxCPM2 fine-tune scores better than this model. This model is competitive on plain Swahili, stronger on code-switching, and needs no reference audio to manage — which is why it is the one we release.

---

Training

Base model	nari-labs/Dia-1.6B
Method	Full supervised fine-tune (all parameters, fp32 + autocast, AdamW8bit)
Epochs	10 (3 825 steps/epoch/rank), ~2 h/epoch, ~21 h total
Hardware	4 × A100-64GB (CINECA Leonardo)
Optimiser	lr 1e-5, 500 warmup steps, grad-clip 1.0
Batching	batch size 2 × grad-accum 4 × 4 GPUs = effective batch 32
Audio	44.1 kHz output (DAC codec)

Effective batch and learning rate deliberately match our VoxCPM2 full SFT (1 × 8 × 4 = 32 at lr 1e-5) so the two runs are comparable.

Training ran in two segments: epochs 1–4, then a warm-start resume for epochs 5–10 after a Leonardo filesystem incident killed the first job mid-epoch-5. The resume restores weights only — optimiser state is not persisted by the trainer, so the LR schedule restarted with a fresh 500-step warmup at epoch 5.

Data

Same pooled corpus as our VoxCPM2 model (sw_voxcpm_corpus_v1: WAXAL swa_tts, OpenSLR-25 Swahili, AfriVoice Swahili subsets, FLEURS), but heavily filtered by Dia's architecture:

	Clips	Hours
Full corpus	99 495	500.00
Usable by Dia (≤ 17.5 s)	30 692	126.11
Dropped (too long)	68 803	373.89

Dia's config.json sets data.audio_length = 1536 DAC frames ≈ 17.9 s; longer clips would be truncated mid-utterance with the full transcript still attached, destroying text/audio alignment. This model therefore saw only 25 % of the corpus that trained VoxCPM2 — re-segmenting the long clips is the single biggest known lever for improving it.

---

Limitations

• Trained on 126 h, not 500 h (see above). The most promising future work.
• Single voice, no cloning. If you need a specific speaker, use VoxCPM2.
• Plain Swahili is behind VoxCPM2 (0.011 vs 0.007 CER).
• Short-form only. Eval is single sentences (~4–9 s); long-form and multi-sentence synthesis are unvalidated.
• Sampling-sensitive — budget for retries. Single-shot plain-Swahili CER is 0.060 vs 0.021 with ASR-gated retries. Named entities are worst affected (0.133 vs 0.025). Occasional takes come back as instant-EOS or babble and are fixed by re-rolling.
• Interior transient sharpness rises with training (median max sample-jump 0.307 at epoch 2 → 0.415 at epoch 10). A jump-threshold detector counts these as "clicks", but it cannot distinguish a click from a plosive, and listening did not confirm degradation. Treat raw click counts as a prompt to listen, not as evidence.
• No formal listening study. Quality confirmed by one native listener, not a MOS panel.

Responsible use

Carried over from the Dia-1.6B disclaimer, and it applies here too:

• Do not use this model to impersonate a real person without their explicit consent.
• Do not use it to generate deceptive or misleading content (fake news, fraudulent audio, misrepresentation).
• Do not use it for illegal or harmful purposes.

By using this model you accept responsibility for upholding the relevant legal and ethical standards in your jurisdiction. The voice in this model is learned from a pooled multi-speaker corpus and is not intended to represent any identifiable individual.

Licence and attribution

This model is released under CC-BY-4.0. You may use it commercially, modify it, and redistribute it, provided you give attribution.

CC-BY-4.0 is chosen because the training data carries CC-BY attribution requirements which must be passed on; the base model is Apache-2.0, which is compatible.

Base model

Model	Licence
nari-labs/Dia-1.6B	Apache-2.0

Training data

All sources are openly licensed. Attribution below satisfies CC-BY-4.0; please carry it forward if you redistribute derivatives.

Source	Dataset	Licence
WAXAL swa_tts	google/WaxalNLP	CC-BY-4.0
FLEURS Swahili (KE)	google/fleurs	CC-BY-4.0
AfriVoice Swahili (agriculture, education, financial, government, health)	DigitalUmuganda/Afrivoice_Swahili	CC-BY-4.0
Swahili Speech 400h	badrex/swahili-speech-400hr	CC-BY-4.0
YodaLingua Swahili	Thomcles/YodaLingua-Swahili	CC-BY-4.0
Kiswahili TTS	Bateesa/kiswahili-tts-dataset	CC-BY-4.0
Swahili TTS	jacksonwambali/swahili-tts-dataset	CC-BY-4.0
Kenyan Swahili (non-standard)	cdli/kenyan_swahili_nonstandard_speech_v1.0	CC-BY-4.0
Swahili words parallel	michsethowusu/swahili-words-speech-text-parallel	CC-BY-4.0 (audio originally published by the International Bible Association)
STEM Swahili speech	stem-content-ai-project/swahili-speech	MIT
OpenSLR-25 (ALFFA)	openslr.org/25	MIT
Common Voice 17 Swahili	mozilla-foundation/common_voice_17_0	CC0-1.0 at time of corpus build; Mozilla moved Common Voice to the Mozilla Data Collective in Oct 2025 — check current terms for your use case

Our thanks to every dataset author above. Swahili speech technology exists because people chose to release this data openly.

Files

File	Purpose
model.pth	fine-tuned Dia-1.6B weights, fp32 (6.0 GB)
config.json	Dia model config (must match the training config)
samples/	48 eval-set generations from this checkpoint
eval/results.json	full per-clip eval output
eval/results.md	eval summary tables

Optimiser/scheduler state is intentionally excluded (not needed for inference).