Riffusion turned music generation into an image problem

The system

Riffusion was a hobby project by Seth Forsgren and Hayk Martiros that treated music generation as an image-generation problem. Their own writeup, preserved via the Wayback Machine snapshot of riffusion.com/about from December 2022, and the openly published code repository, describe fine-tuning a Stable Diffusion v1.5 checkpoint, with no architectural modification, on images of audio spectrograms paired with text captions. The live riffusion.com domain has since become a different product, Flow Music, so the original method is documented only in the archived page and the repository.

What the documents establish

The writeup states the model outputs a spectrogram image, which is then converted back to audio through the Griffin-Lim algorithm, because the spectrogram captures amplitude but not the phase information needed for an exact audio reconstruction, and the frequency axis uses the perceptual Mel scale. The authors describe chaining outputs into longer pieces by interpolating between prompts and seeds in the diffusion model's latent space rather than concatenating separately generated clips, which they say produces smoother transitions than splicing raw audio. The GitHub repository confirms the code is released under the MIT licence and states plainly that the project is no longer actively maintained.

Craft and rights

Riffusion's method is notable for what it does not require: no new audio-specific architecture, only a fine-tune of an existing image model and a signal-processing conversion step, which is why the authors could publish both the method and the code openly under a permissive licence. That openness is itself the rights-relevant fact for a producer evaluating it today: because the checkpoint and code are MIT-licensed, reuse terms are unusually clear compared with closed commercial systems in this lane, though the writeup does not state what audio was used to build the spectrogram training images beyond describing them as paired with text captions, leaving that provenance question open.

Outcomes and open questions

Because the original riffusion.com now hosts an unrelated product, and the GitHub repository is archived and unmaintained, this entry describes a closed chapter rather than a live tool; a reader who wants to try the method should expect to run the published code rather than visit the current site. What became of the original authors' company and its later products is outside what these two 2022 documents establish.

• Does an image-generation shortcut like spectrogram fine-tuning introduce artifacts a waveform-native model would not?
• What does an MIT-licensed research checkpoint actually clear you to do with its outputs?
• When a project's original domain now hosts something else, where should you look for the original documentation?

Riffusion's contribution was a method, not a maintained product, and its own repository now says so directly.