Meta's own benchmark is where the Demucs SDR numbers come from

The system

Demucs is a music source-separation model that splits a mixed recording into stems such as drums, bass and vocals. Its fourth generation, Hybrid Transformer Demucs (HT Demucs), was documented in “Hybrid Transformers for Music Source Separation” by Simon Rouard, Francisco Massa and Alexandre Défossez, submitted to arXiv on 15 November 2022. The project repository, published under the facebookresearch organisation, describes it as replacing the innermost layers of the earlier Hybrid Demucs architecture with a cross-domain Transformer encoder that applies self-attention within, and cross-attention across, the waveform and spectrogram branches.

What the documents establish

The paper reports that HT Demucs, trained only on the MUSDB dataset, “performs poorly,” but that adding 800 extra training songs produces a 0.45 dB improvement in signal-to-distortion ratio (SDR) over the equivalently trained Hybrid Demucs baseline. The repository's own README states the released model “achieves a SDR of 9.00 dB on the MUSDB HQ test set,” and that using sparse attention kernels to extend the model's receptive field, plus per-source fine-tuning, reaches “state-of-the-art 9.20 dB of SDR.” Both figures are measured on the MUSDB HQ benchmark set using the extra 800-song training data described in the paper, not on a held-out set independent of the authors.

Craft and rights

For an engineer choosing a stem-separation tool, the SDR figures are genuinely useful as a documented, reproducible starting point — the code and the described MUSDB HQ evaluation are public, so the number can, in principle, be checked by someone else running the same test. But it is still the authors' self-reported benchmark on their own chosen dataset and metric, not an independently audited or third-party-run comparison across competing separation tools. Framing 9.20 dB as an absolute, universal quality score for Demucs, rather than a specific score on a specific test set as measured by the people who built the model, overstates what the source documents claim.

Outcomes and open questions

The repository itself now carries a maintenance notice that its original author is “no longer working at Meta” and that the repository is unmaintained, with an independent fork continuing bug fixes only, so future SDR improvements or corrections are unlikely to appear in the cited repository going forward. Readers should watch for whether later, independently run comparisons (for example, community leaderboards using the same MUSDB HQ set) reproduce the paper's reported numbers, and treat any single dB figure as tied to one specific dataset and training configuration.

• Was a cited Demucs SDR figure measured on MUSDB HQ with the extra training data, or on the smaller original MUSDB set where the paper reports much weaker results?
• Has an independent party reproduced the reported 9.00 or 9.20 dB SDR figures on the same test set?
• Does a comparison between Demucs and a competing separation tool use the same evaluation metric and dataset for both?

The paper and repository are unusually transparent about their own method, but transparency about the benchmark is not the same as independent verification of the result, and the two should not be conflated when the numbers get repeated elsewhere.