wav2vec2-bert-base-s-emotion-russian-resd

Speech emotion recognition for Russian over seven classes: anger, disgust, enthusiasm, fear, happiness, neutral, sadness.

Audio tower jonatasgrosman/wav2vec2-large-xlsr-53-russian (WavLM/wav2vec2 fine-tuned for Russian ASR) paired with the text encoder DeepPavlov/rubert-base-cased; the model reads the waveform and a transcript together. Trained on Aniemore/resd.

• Parameters: 493M · weights: 1887 MiB (fp32)
• Input: 16 kHz mono waveform and the utterance transcript
• Quantized builds: wav2vec2-bert-base-s-emotion-russian-resd-quantized — INT8, FP8 and INT4, up to 5.8x smaller on disk at the same score

Results

Test set	n	UA	WA	macro-F1
RESD test	280	0.7141	0.7179	0.7182
Dusha podcast test	12079	0.3396	0.1504	0.1217
CAMEO test	5187	0.2528	0.2871	0.2487
RESD test, transcript blanked	280	0.6982	0.7036	0.7017

The last row is the audio branch alone. Production rarely has a gold transcript, and the gap between 0.718 with one and 0.702 without is what a transcript is worth here.

Usage

import torch, librosa
from transformers import AutoConfig, AutoTokenizer, AutoModel

repo = "Aniemore/wav2vec2-bert-base-s-emotion-russian-resd"
cfg = AutoConfig.from_pretrained(repo, trust_remote_code=True)
model = AutoModel.from_pretrained(repo, trust_remote_code=True).eval()
tok = AutoTokenizer.from_pretrained(repo)

# Resample to 16 kHz. RESD ships at 44.1 kHz, and 44.1 kHz audio
# labelled as 16 kHz is stretched 2.8x in time — the model answers,
# it just answers about other audio.
wav, _ = librosa.load("clip.wav", sr=16000, mono=True)
wav = torch.tensor((wav - wav.mean()) / (wav.std() + 1e-7))[None]
t = tok("расшифровка реплики", return_tensors="pt")
with torch.no_grad():
    logits = model(input_ids=t['input_ids'], input_values=wav,
                   text_attention_mask=t['attention_mask'],
                   audio_attention_mask=torch.ones_like(wav).long()).logits

probs = logits.softmax(-1)[0]
print({model.config.id2label[i]: round(p.item(), 3) for i, p in enumerate(probs)})

Loading the audio without librosa

# torchaudio
import torchaudio
wav, sr = torchaudio.load("clip.wav")
wav = torchaudio.functional.resample(wav, sr, 16000).mean(0).numpy()

# torchcodec, the newer decoder
from torchcodec.decoders import AudioDecoder
wav = AudioDecoder("clip.wav", sample_rate=16000).get_all_samples().data.mean(0).numpy()

# straight from the dataset — `datasets` resamples on the column, so
# the mixed 16/44.1 kHz in RESD is handled for you
from datasets import load_dataset, Audio
ds = load_dataset("Aniemore/resd", split="test")
ds = ds.cast_column("speech", Audio(sampling_rate=16000))
wav = ds[0]["speech"]["array"]

Evaluation protocol

Audio resampled to 16 kHz mono, clips capped at 12 s, normalized per utterance, padding masked. UA is macro-averaged recall, WA is accuracy, F1 is macro-averaged. All three test sets went through the same harness, so the rows are comparable to each other.

The RESD split matches fold 1 of EmoBox bit for bit. The top entry there is WavLM-large at WA 56.47 / UA 55.87 / F1 55.82. These numbers are higher, but the training protocol differs — EmoBox freezes the encoder and trains a probe, this is a full fine-tune — so read it as a different recipe on the same split, not as a like-for-like win.

Limitations

RESD is acted and class-balanced. Real speech is neither. The Dusha podcast row above is the honest signal for spontaneous audio, and it is far below the RESD row. Spontaneous Russian is roughly 93% neutral, and a model tuned on balanced acted data over-predicts the emotional classes on it. Measure on your own material, and calibrate the neutral logit if you deploy.

Seven classes only, Russian only, single-speaker clips.

Citation

@misc{aniemore,
  author = {Lubenets, Ilya and Davidchuk, Nikita and Amentes, Aleksandr},
  title  = {Aniemore: an open library for emotion recognition in Russian speech},
  url    = {https://github.com/Aniemore/Aniemore},
  year   = {2023}
}

License

MIT.