javid48/readout-guidance-audio-tfg

Readout Guidance on TFG's audio benchmark (declipping / inpainting)

Readout head, diagnostics and full sweep for training-free guidance of audio declipping and audio inpainting, steering the frozen teticio/audio-diffusion-256 mel-spectrogram diffusion model under the TFG (Ye et al. 2024) protocol.

This is a negative / boundary result, and is published as such.

Summary

A y-agnostic multi-scale head (929,795 params) reads frozen UNet activations and predicts E[x₀ | xₜ]; guidance is -‖mask ⊙ x̂₀ − y‖₂. Both tasks are masked linear inverse problems.

Guidance works (FAD 2.43 → 0.25), but the readout head does not beat guiding with the denoiser's own x̂₀:

task	method	recover ↑	FAD ↓
declipping	unguided	−53.61	2.428
declipping	eps (denoiser x̂₀)	−9.85	0.252
declipping	readout head	−11.45	0.268
inpainting	unguided	−53.20	2.491
inpainting	eps (denoiser x̂₀)	−4.04	0.185
inpainting	readout head	−8.74	0.248

(N = 256, guidance scale 4, η = 1, 100 DDIM steps.)

Why

For a linear inverse problem the guidance term depends only on x̂₀, which the denoiser already computes for free. Measured on held-out mels (x0_compare.json, MSE against true x₀):

t	10	300	700	800	900	990
head	0.00143	0.02535	0.06441	0.08274	0.09742	0.11102
denoiser	0.00108	0.01540	0.05308	0.07274	0.13251	0.48791

The denoiser wins below t ≈ 850; the head wins above it, by 77% at t = 990, because the ε-parameterized x̂₀ divides by a vanishing √ᾱ while the head regresses x₀ directly.

Restricting the head to that high-noise band (--head_t_lo 800 --head_t_hi 900) recovers most of the gap on declipping — recover −10.51 → −8.19, FAD 0.438 → 0.283 at N = 128, reaching parity with eps — but hurts on inpainting, where more of the signal is pinned by the measurement.

Conclusion: readout guidance pays off when the sampling-time target is not the posterior mean (a class label, a molecular property, a style). On a linear inverse problem the required quantity is the posterior mean, so the head competes with its own backbone and at best ties.

Contents

head/            trained x_0 readout head (929,795 params, val MSE 0.0336)
runs/            N=256 runs: config + metrics (none / eps / readout, both tasks)
sweep/           N=128 sweep: guidance scale, t-schedule, iter_steps, x0_scale
x0_compare.json  per-t MSE of head vs denoiser x_0 -- the diagnostic above
*_table.txt      rendered result tables
code/            training, sampling, evaluation and diagnostic scripts
RESULTS.md       full write-up, including TFG-repo bugs found

Requires the TFG codebase on PYTHONPATH. Three fixes are needed before their audio tasks run at all (scipy private-symbol import in torch_sqrtm.py, a best-of-N block in pipeline.py that only type-checks for images, and a mask mismatch in audio_inpainting.py) — all documented in RESULTS.md.