MusicCaps captions came from ten musicians, not the crowd

The system

Evaluating whether a text-to-music model actually follows a prompt requires paired examples of real audio and the words a person would use to describe it, and Google built MusicCaps to supply exactly that. The dataset was released alongside the MusicLM paper, submitted 26 January 2023, which introduces MusicLM as a model for generating music from text and MusicCaps as the evaluation set built to test it.

What the documents establish

The paper states MusicCaps "includes 5.5k music clips from AudioSet, each paired with corresponding text descriptions in English, written by ten professional musicians." For each ten-second clip, the dataset provides a free-text caption averaging four sentences and a list of music aspects — covering genre, mood, tempo, singer voices, instrumentation and rhythm among others — averaging eleven aspects per clip. The paper positions MusicCaps as a complement to the existing AudioCaps dataset: both draw clips from AudioSet, but where AudioCaps includes non-music sound, MusicCaps "focuses exclusively on music," drawing from both the training and evaluation splits of AudioSet and including a genre-balanced 1,000-example subset. Google's own dataset card, published under its organisation account and retrieved 16 September 2026, confirms 5,521 examples in this same caption-plus-aspect-list structure and states the release licence as CC BY-SA 4.0.

Craft and rights

Two rights layers sit inside MusicCaps. The audio is not redistributed as files; since clips are drawn from AudioSet, obtaining them means fetching the referenced YouTube videos directly, subject to each video's own availability and licence terms. What Google licenses at CC BY-SA 4.0, per its dataset card, is the caption and aspect-list text the musicians wrote — a licence on the description, not a clearance for the recordings. Separately, MusicCaps is an evaluation set of 5,521 clips built to test MusicLM, not the model's training corpus, which the same paper describes as a far larger, unnamed proprietary collection of five million clips. A claim that a system was "trained on MusicCaps" should be read with that mismatch in mind.

Outcomes and open questions

Because MusicCaps points to AudioSet's YouTube references rather than shipping self-contained audio, the corpus a researcher can actually reconstruct shrinks over time as source videos are removed or made private — a reproducibility problem neither document quantifies. The captions also reflect the vocabulary and judgment of ten specific professional musicians rather than a broad listener population, which matters for anyone treating MusicCaps as a general model of how people describe music rather than as the benchmark it was built to be.

• What share of the original 5,521 referenced clips remains retrievable from YouTube today, and does that affect benchmark comparability?
• Does a paper citing "trained on MusicCaps" mean fine-tuning on the evaluation set or something looser, like tuning prompts against it?
• How much would captions differ if written by a larger and more demographically varied group of musicians than the original ten?

MusicCaps is a carefully scoped evaluation instrument; its value depends on keeping that scope in view rather than treating it as a stand-in for training data at large.