A shared dataset made separation scores comparable, not perceptual

What happened

On 17 December 2017, Zafar Rafii, Antoine Liutkus, Fabian-Robert Stöter, Stylianos Ioannis Mimilakis and Rachel Bittner published MUSDB18, a dataset of 150 full songs with isolated drum, bass, vocal and 'other' stems, on Zenodo. The dataset, split into 100 training and 50 test tracks, became the shared benchmark for music source separation research described on the SigSep project page. It underpinned the 2018 Signal Separation Evaluation Campaign, SiSEC, whose paper states the campaign itself dates to 2008 and this was its first release built for training data-hungry systems, not only for testing them.

What the documents say

The SiSEC paper explains the standard scoring tools, the BSS Eval metrics, assess separation through 'Source to Distortion, to Artefact, to Interference ratios', SDR, SAR and SIR, computed after optimally aligning an estimated stem to the true one through a matching filter. A higher SDR indicates less distortion energy relative to the target signal, not a direct perceptual rating. The paper states this explicitly: 'BSS Eval scores are in fine relative to squared-error criteria', and reports that earlier 'perceptual studies showed' a method preferred by human listeners could still score lower on SAR than a method they liked less, because the metric optimises for a mathematical fit, not a listening preference.

Why it matters for makers

The mechanism is signal-level error measured in decibels against a known ground truth, which is exactly why it is useful for comparing separation systems on the same held-out songs: no listening panel is required, and the number is reproducible. But a separation plug-in advertising a leaderboard SDR score is quoting a distortion-energy measurement on MUSDB18's specific genres and mix styles, not a promise about how clean the vocal stem will sound on your particular mix, especially one outside the pop, rock and electronic material the dataset weights toward.

What to check before you use it

When a separation tool cites an SDR figure, check whether it was measured on MUSDB18 or a different, possibly easier, test set, since scores are not comparable across datasets. Listen to the stem on your own material rather than trusting a decibel figure alone; this is an editorial recommendation the papers themselves support, given their own finding that SDR and human preference can diverge. Note MUSDB18's licence permits non-commercial, educational use, which matters if a tool's benchmark claim rests on training against it.

• Was the cited SDR figure measured on MUSDB18, and is that comparable to the material you actually work with?
• Does a higher SDR score match what you hear when you listen to the separated stem yourself?
• If a tool trained on MUSDB18, does its licence, non-commercial by default, affect how the resulting model may be used?

MUSDB18 and SDR gave source separation a shared, reproducible yardstick, which most audio tasks still lack. The campaign's own paper is the clearest source for the limits of that yardstick: a decibel gain is not the same claim as a listener's preference.

Continue in Listening & research, or Two decades of shared MIR evaluation still exclude generation.