GigaChat Audio MLX — Q8/BF16

English · Документация на русском

Compact mixed Q8/BF16 GigaChat Audio model for Apple Silicon. It uses less disk and memory and runs faster than the BF16 MLX model while keeping the audio encoder and projector in BF16.

This is an independent port, not an official release by the upstream authors.

GitHub · PyPI · All models · Full results · Upstream model · Paper

Models

gigachat-audio-mlx — BF16 · best measured quality
gigachat-audio-mlx-q8-bf16 — Q8/BF16 · smaller and faster ← this model

Model	Choose it for	Model size	Measured process memory	Processing 5 min 17 sec*
BF16	Best measured answer quality	22.54 GB	22.84 GiB	10.41 s
Q8/BF16 · this model	Less disk and memory, higher speed	13.75 GB	14.80 GiB	6.90 s

_{* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.}

Choose Q8/BF16 when size, memory, and speed matter; choose BF16 for the best measured answer quality. Both variants use the same gigachat-audio-mlx runtime.

39% smaller, 35% less memory, and 34% less processing time than MLX BF16 — with factual errors in 3 more cases in our blinded review.

This is a compact 13.75 GB mixed Q8/BF16 derivative of the GigaChat Audio MLX BF16 artifact. Choose it when storage, memory, and speed matter; choose BF16 for the best measured answer quality.

What do I need to run it?

Model	Model size	Free disk to set aside	Measured whole-process memory	Processing 5 min 17 sec*
Q8/BF16	13.75 GB	about 16 GB	14.80 GiB	6.90 s · 46.0× realtime

_{* Time to process a 5 min 17 sec audio file on Apple M4 Pro. Lower is better.}

The free-disk figure is a rounded practical reserve; the CLI dry run reports the exact value. Memory is measured use, not a minimum-memory guarantee. This release is verified only on a 48 GB Mac; 16/24/32 GB support is not yet claimed.

Why MLX matters on this Mac

Same Apple M4 Pro / 48 GB. Original is a stopped memory-safety smoke, not a completed speed benchmark; MLX values are medians of five clean runs.

Runtime	Audio	Outcome	Whole-process footprint
Original PyTorch BF16 on MPS	9.84 s	No answer after 277 s; stopped	at least 51.92 GiB
MLX BF16	317.27 s	Completed in 10.41 s	22.84 GiB
MLX Q8/BF16	317.27 s	Completed in 6.90 s	14.80 GiB

The pinned upstream Transformers path is demonstrated for CUDA. On MPS it exceeded this Mac's safe memory budget and nearly filled system swap during the short smoke. The 5-minute Original run was not started, so no speedup ratio is claimed. MLX is what makes the model usable on the tested Mac.

What changes from MLX BF16

Lower is better in every row.

Metric	MLX BF16	MLX Q8/BF16	Change
Model size, decimal GB	22.54	13.75	−39.0%
Whole-process memory, 5-minute profile	22.84 GiB	14.80 GiB	−35.2%
End-to-end time, 5-minute profile	10.41 s	6.90 s	−33.7%

Model size is the file size, not RAM. The 5-minute profile contains 317.27 seconds of audio and reports the median of five clean processes on the verified M4 Pro / 48 GB machine.

Measured quality difference

In an independent blinded review of 64 paired timed-description and summary items:

Model	Responses with a factual error, lower is better
MLX BF16	26/64 (40.625%)
MLX Q8/BF16	29/64 (45.3125%)

That is 3 additional reviewed cases with a factual error (+4.6875 percentage points). Most automatic ASR, translation, emotion, and temporal-localization comparisons were close.

Quantization

decoder default: weight-only affine Q8, group size 64;
decoder BF16 whole-layer overrides: 0, 17, 21, and 23;
Conformer, audio subsampler, and audio projector: BF16;
embeddings, LM head, router/gates, norms, and incompatible tensors follow the audited tensor manifest;
no activation quantization, custom Metal kernels, PyTorch fallback, or remote executable code.

Field	Value
Runtime payload	13,748,019,336 bytes
MLX BF16 runtime payload	22,539,048,633 bytes
Disk reduction	39.0036%
Source revision	`bf73d03a43bdf5118f5a4dbdc24ba6f56ac31cfb`
Verified target	Apple M4 Pro / 48 GB / macOS 15.7.7
Weight files commit	`03a82c331193b3ba388b01abdc0076da0a19d59f`
Manifest SHA-256	`656daf991a515961498426a14df8d75b8d730c33c14048ad0fd45bace5e1c3c3`

Download and run

brew install uv
uv tool install gigachat-audio-mlx
gigachat-audio-mlx doctor

gigachat-audio-mlx download --variant q8-bf16 --dry-run

Review the download plan before adding --yes. The main package quick start uses BF16; select q8-bf16 when you prefer the smaller, faster option.

One reviewed difference

Prompt paired with English audio:

Translate the speech into Russian. Return only the translation.

The source discusses Deity Yoga in Tibetan meditation. BF16 rendered the key concept as «йога Визуализации», while Q8/BF16 changed it to «дзогчен». This is a semantic substitution, not a harmless wording difference.

Resource evidence

On the verified M4 Pro / 48 GB machine, Q8/BF16 reduced whole-process memory by 35.2% and end-to-end time by 33.7% in the 5-minute profile. This release makes no 90-minute support claim. The post-publication Original PyTorch/MPS check did not complete a 9.84-second smoke: after 277 seconds its process footprint had reached at least 51.92 GiB and system swap was nearly full. It was stopped before the 5-minute run, so no Original Mac latency is inferred.

Limitations and responsible use

All BF16 limitations apply. Quantization can change factual details, names, numbers, translations, timestamps, and summaries, so important outputs require verification against the original audio. Other chips, memory sizes, and macOS versions are not supported claims.

Do not use this model as the sole basis for medical, legal, employment, safety, or surveillance decisions. Users are responsible for consent, privacy, copyright, and dataset terms for audio they process.