WaveNet generated raw audio, and then music, sample by sample

What happened

On 8 September 2016, DeepMind announced WaveNet, a neural network that generates audio one sample at a time. The company's own blog post describes it as 'a deep generative model of raw audio waveforms' and reports that, for text-to-speech, it 'reduces the gap with human performance by over 50%' compared with existing systems. A detailed technical write-up followed as an arXiv paper submitted four days later, on 12 September 2016.

What the documents say

The paper's abstract states the model is 'fully probabilistic and autoregressive, with the predictive distribution for each audio sample conditioned on all previous ones,' meaning every sample of the output waveform is generated in sequence, each informed by everything generated before it. This differs from earlier speech synthesis methods, which stitched together pre-recorded fragments; WaveNet instead builds the waveform itself. The abstract reports that human listeners rated the resulting speech 'significantly more natural sounding than the best parametric and concatenative systems' in both English and Mandarin, and that one trained model could represent many speakers. Separately, the blog post states the DeepMind team trained the same architecture on a dataset of classical piano recordings because, in the post's own words, 'we thought it would also be fun to try to generate music,' and that the model then generated new piano passages without being given a score, describing the results as 'fascinating samples.' The paper's abstract independently confirms that training on music yields 'novel and often highly realistic musical fragments.'

Why it matters for makers

The mechanism, generation at the level of the raw waveform rather than at the level of notes, symbols or pre-recorded units, is why WaveNet is treated as a starting point for the generative audio tools that followed it. A system working directly on the waveform is not constrained to notes a scoring system can represent; timbral detail, room tone, breath and attack sit inside that sample stream. That is why such models are computationally heavy: modelling every sample is expensive because it is unconstrained by musical structure.

What to check before you use it

A maker evaluating any later tool that claims a WaveNet-style or waveform-level architecture can ask what it was trained on, since the 2016 documents are explicit that the piano results came from a specific genre of training material, while its speech results came from production text-to-speech data, not music. Results in one domain do not automatically transfer to another. This is an editorial caution about generalising from a demonstration to a general capability claim.

• Was this tool's demonstrated quality measured on the same type of material I intend to generate?
• Is the model generating at the waveform level, or at a higher, more constrained representation, and does that change what artefacts to expect?
• If a listening comparison is cited as evidence of quality, was it a controlled listening test or an informal claim?

WaveNet's own documents describe a specific, narrow technical claim, sample-by-sample generation improves naturalness, tested rigorously for speech and demonstrated informally for music. The distinction between tested and demonstrated is worth preserving whenever a later tool cites WaveNet as its ancestor.

Continue in Listening & research, or NSynth shipped as an open dataset and model, not just a demo.