A neural codec's tokens explain a generator's sonic signature

What happened

On 24 October 2022, Meta researchers Alexandre Défossez, Jade Copet, Gabriel Synnaeve and Yossi Adi posted 'High Fidelity Neural Audio Compression' to arXiv, describing the codec later released as EnCodec. The paper presents a streaming encoder-decoder that compresses audio into a quantised stream of discrete codes rather than a waveform, trained end to end, and reports it judged superior to baseline codecs in subjective MUSHRA listening tests across speech, noisy speech and music. Code and pretrained models were published in the encodec repository.

What the documents say

The repository describes the architecture as a convolutional encoder feeding a residual vector quantiser, which produces several parallel streams, or codebooks, of discrete tokens, followed by a matching decoder; at 6 kbps the documentation states eight codebooks are used. The AudioCraft announcement, dated 2 August 2023, explains why this compression choice mattered beyond file size: it let Meta 'train autoregressive language models over these discrete audio tokens to generate new tokens and new sounds and music', with the resulting sequence decoded back to audio through EnCodec's own decoder.

Why it matters for makers

The mechanism explains a real, audible fact: when a generator like MusicGen makes music, it is not predicting a waveform, it is predicting the discrete codes a neural codec would have assigned to one, then handing those codes to that codec's decoder to become sound. Any characteristic timbral signature, compression artefact, or limitation of the specific codec and bitrate a model was trained against is therefore inherited by every track that model produces, in the same way a familiar MP3 encoder leaves its own signature on compressed audio.

What to check before you use it

If a generated track has a persistent sonic quality, a slight softness, a particular treatment of transients, it is worth asking whether that traces to the underlying neural codec rather than to the model's creative choices, since the codec is fixed at training time and reused for every output. This is an editorial inference beyond what either document claims directly: the papers describe the mechanism, not any specific model's audible signature, which would need its own listening comparison.

• Does the generator you are using state which neural codec, and at what bitrate, its tokens are decoded through?
• Is a recurring sonic quality in the output more likely a codec artefact than a deliberate creative parameter?
• Would a higher-bitrate or different codec, if the tool offers one, measurably change the result?

Naming the codec names the mechanism: a generator built on token prediction sounds the way its codec's decoder renders those tokens. That is a specific, checkable claim the source papers support, not a general statement about AI music sounding a particular way.

Continue in Listening & research, or Meta open-sourced MusicGen with a licensed dataset and an NC licence.