AudioLM turned audio continuation into next-token prediction

The system

AudioLM is a research framework from Google Research, described in a paper submitted 7 September 2022 and summarised the following month in a research blog post. It treats audio generation as a language-modelling problem: an input recording is mapped to a sequence of discrete tokens, and a Transformer predicts continuations of that sequence the way a text model predicts the next word. The paper demonstrates the approach on two tasks — continuing a short speech prompt, and continuing a short piano recording — without ever training on a transcript, a musical score or any other symbolic annotation.

What the documents establish

The paper's central technical claim is a hybrid tokenisation scheme. It combines two representations it did not invent from scratch: semantic tokens, drawn from the intermediate layers of a pretrained w2v-BERT model, which the authors say capture long-term structure such as phonetics in speech or local melody in music; and acoustic tokens, produced by a SoundStream neural codec, which capture the fine detail needed for high-fidelity synthesis. Used alone, semantic tokens reconstruct poorly and acoustic tokens alone lose long-range coherence; AudioLM's contribution is combining both in one autoregressive model. The paper states the speech experiments draw on 60,000 hours of the Libri-Light corpus, while the piano experiments retrain every component on an internal dataset of 40,000 hours of piano recordings ranging from beginner exercises to advanced repertoire. The blog post adds the researchers' names and restates the two-token design without adding new technical detail.

Craft and rights

AudioLM is a research architecture, not a product a musician can install; the paper frames the piano result as evidence the framework generalises beyond speech, not as a finished composing tool. What it establishes for later systems is a template: MusicLM and SingSong, published months afterward, both reuse its tokenisation approach for text-to-music and vocal-to-accompaniment generation. The rights question the paper raises directly, and unusually explicitly, concerns the speech side: the authors trained a classifier that detects AudioLM-generated speech with very high accuracy, specifically because a system that continues a voice convincingly creates a documented impersonation risk. No equivalent safeguard is offered for the piano results, an asymmetry worth noting rather than resolving.

Outcomes and open questions

Because AudioLM was never released as a public model or weights, independent replication depends on later systems that built on its method rather than on the original codebase. The paper does not disclose the provenance of the 40,000-hour piano dataset beyond calling it internal, leaving open whether it drew on licensed, purchased or otherwise cleared recordings. Readers should watch whether descendant systems close that gap.

• Does a tokenisation method built for speech carry assumptions that distort music, or vice versa?
• When a paper discloses a detection safeguard for one modality, why not for another it also generates?
• What was the licensing basis for training data described only as an internal dataset?

Read on its own terms, AudioLM is a method paper about representation, not a music product; its lasting influence is visible in the systems that cited and extended it within the same year.