A broadcast standard specifies how to blind-test audio

What happened

The International Telecommunication Union’s broadcasting standards arm maintains two recommendations for subjective listening tests, both periodically revised: BS.1116, for detecting small impairments, most recently approved in February 2015, and BS.1534, the “MUlti Stimulus test with Hidden Reference and Anchor” method – MUSHRA – for intermediate audio quality, most recently approved in October 2015. MUSHRA exists specifically because BS.1116’s method, the recommendation itself notes, “is not suitable for assessing systems with intermediate audio quality,” which is closer to where a comparison between a generated track and a human recording usually sits than the near-transparent differences BS.1116 targets.

What the documents say

The MUSHRA recommendation specifies a structure a home listening test typically skips: every trial includes the unprocessed original as a hidden, unlabelled reference alongside the test items, plus at least one mandatory “hidden anchor” of known, deliberately degraded quality, so a listener’s ratings can be checked against a known floor and ceiling rather than trusted at face value. It also sets a post-screening rule: an assessor should be excluded from the results if they rate the hidden reference below 90 out of 100 more than 15 percent of the time, since that suggests they cannot reliably identify the original when it is disguised among the test items. The companion BS.1116 recommendation confirms the two methods are deliberately scoped for different jobs, not competing versions of the same test.

Why it matters for makers

The mechanism the standard protects against is expectation bias: a listener who knows which track is “the AI one” rates it differently than the same audio heard blind, and a listener never shown the real original has no anchor for how far “close to the original” actually stretches. A producer comparing a generated stem or a mastering assistant’s output against a human reference is running exactly this kind of test, informally, usually without hiding which is which. The music information retrieval research community publishes its own reproducibility resources separately from these broadcast standards.

What to check before you use it

This is an editorial adaptation of a broadcast-engineering standard for a small studio, not a substitute for a properly resourced listening test. Include the unprocessed original in the comparison set, unlabelled, alongside whatever is being evaluated. Add at least one deliberately degraded version as a low anchor, so “good” and “bad” have fixed points on the scale rather than only relative ones. Ask listeners to identify the hidden original among the set before trusting their other ratings, since MUSHRA’s own screening rule treats failure to do so as grounds to discard that listener’s data.

• Is the unprocessed original included in the comparison, hidden and unlabelled?
• Is there a known, deliberately degraded anchor to calibrate what “poor” means on the scale?
• Can each listener correctly pick out the hidden original before their other ratings are trusted?

None of this requires broadcast-lab equipment, only the discipline of the method: hide identities, include a known reference and a known anchor, and screen out listeners who cannot tell the difference before drawing a conclusion from the ones who can.

Continue in Listening & research, or Two decades of shared MIR evaluation still exclude generation.