A hobby fine-tune turned an image model into a music one

Riffusion repurposed Stable Diffusion to paint spectrograms, showing image diffusion could generate usable audio.

Historical event

December 15, 2022

Site publication

September 18, 2026

What happened

In mid-December 2022, two engineers, Seth Forsgren and Hayk Martiros, published a hobby project called Riffusion. Their method took Stable Diffusion, an image-generating model, and fine-tuned it to produce pictures of audio spectrograms, a plot of frequency against time, from a text prompt such as 'funk bassline with a jazzy saxophone solo'. The image was then converted back into a short audio clip. The Internet Archive's earliest capture of the site is dated 15 December 2022, consistent with a fast, unannounced release rather than a staged launch.

What the documents say

The creators' own description says this is the ordinary Stable Diffusion v1.5 model, 'fine-tuned on images of spectrograms paired with text' with no architectural changes, so the underlying image model's ordinary sampling techniques, such as img2img and prompt interpolation, carried over unchanged into an audio context. Because the fine-tuned model produces only the amplitude of a spectrogram and not its phase, audio is reconstructed using the Griffin-Lim algorithm, an approximation rather than an exact inverse transform - a real quality ceiling worth knowing before comparing this to a phase-accurate audio model. The code repository, released under the MIT licence, adds that the project is 'no longer actively maintained', and hosts the trained weights on Hugging Face separately from the code.

Why it matters for makers

The mechanism worth naming is domain transfer: an image model repurposed for audio by treating a spectrogram as a picture, rather than a model built for audio from the ground up. That choice explains both Riffusion's strengths, inheriting every image-diffusion trick for free, including seed interpolation for smooth transitions between clips, and its ceiling, since Griffin-Lim phase reconstruction produces the watery artefact audible in its samples. For a producer, the significance is less the audio quality than the proof of concept: a fine-tune built over a short window showed that diffusion, not only autoregressive transformers, could generate usable musical audio, and later purpose-built audio diffusion models had to be measured against this starting point.

What to check before you use it

Riffusion's repository states plainly the project is unmaintained, so anyone finding the code today should treat it as a historical artefact: check the repository's licence before reusing the code commercially, and check separately what licence the Hugging Face weights carry, since a permissive code licence does not automatically extend to weights trained from a base model with its own terms. This is an editorial caution, not a claim the repository makes about downstream use.

• Does a generator convert between representations, such as spectrogram to audio, in a way that loses information, and does that show up as an audible artefact?
• Is the code licence the same as the model-weights licence, or do the two need checking separately?
• Is an unmaintained research release being relied on as if it were a supported product?

Riffusion is a useful marker because it was small: a fine-tune, not a foundation model, built from a technique not designed for sound. The gap between that hobby result and the licensed commercial audio models that followed within a year measures how quickly the field moved.

Sources & reading trail

Creators' own account of Riffusion's method (fine-tuning Stable Diffusion v1.5 on spectrogram images), Griffin-Lim audio reconstruction, and earliest public availability, archived 16 December 2022.

Source published: Not established · Retrieved: 16 September 2026

Confirms the MIT code licence, that trained weights are hosted separately on Hugging Face, and the project's unmaintained status.

Source published: Not established · Retrieved: 16 September 2026

Papers, terms and official documents establish the record; the maker reading and the checks are Signal to Song editorial analysis. This retrospective draft does not imply the site published on the event date.

Continue reading

• NSynth shipped as an open dataset and model, not just a demo
• Stable Audio switched from a licensed library to Creative Commons
• Suno's free and paid tiers have always owned output differently
• Browse the complete the archive

Published September 18, 2026, not on the date of the event described.