A benchmark pairs AI tracks with the music that seeded them

The system

FakeMusicCaps is a dataset built by Luca Comanducci, Paolo Bestagini and Stefano Tubaro, described in an arXiv paper first posted 16 September 2024 and revised 25 September 2024. It exists to give the audio-forensics community a shared testbed for detecting and attributing text-to-music (TTM) output, built by taking the captions from Google's MusicCaps dataset and re-generating audio from them through five named TTM systems: MusicGen, MusicLDM, AudioLDM2, Stable Audio Open and Mustango.

What the documents establish

Because every synthetic track traces back to a real MusicCaps caption and its original human-made reference, the dataset lets a detector be tested against a known generator identity rather than an anonymous fake. The full paper reports closed-set classification results, where a detector is trained and tested on the same five generators plus genuine audio: balanced accuracy of 0.90 for an M5 classifier, 0.88 for RawNet2, and 1.00 for a ResNet18-plus-spectrogram model. Open-set results, testing a detector against generators it had not trained on, were markedly weaker: a threshold approach reached 0.85 balanced accuracy for the best model, while a support-vector-machine approach fell to 0.48, close to chance for a multi-way task.

Craft and rights

The gap between closed-set and open-set numbers is the practical lesson for anyone relying on AI-detection tools to police originality claims or platform content policies: a detector that looks near-perfect against generators it was trained on can perform close to chance against a system it has not seen, and new TTM releases arrive faster than any single benchmark can catalog them. Nothing in this paper's tested conditions involves adversarial editing, re-recording, or mixing of detected tracks, so a claim that any of these classifiers would hold up against a track that had been remixed or partially re-recorded goes beyond what was tested here.

Outcomes and open questions

The authors frame FakeMusicCaps as extensible, meaning future TTM systems can be added using the same generation procedure, but the benchmark as published covers only the five listed generators. Detection accuracy under real-world conditions, adversarially edited audio, re-encoded or partially human-modified tracks, or generators released after this paper, is not measured in the reported experiments and remains an open question for anyone building on this benchmark.

• Was a detector's reported accuracy measured against generators it had already seen, or genuinely unseen ones?
• Would this detector's numbers hold up against a track that has been edited, remixed or re-recorded afterward?
• Has a newer generator been added to this benchmark since the version I am citing?

FakeMusicCaps documents a real, sizeable gap between how well detection works inside a known set of generators and how much of that accuracy survives against an unseen one, a distinction worth checking before trusting any single detector's headline number.