Two decades of shared MIR evaluation still exclude generation

What happened

The Music Information Retrieval Evaluation eXchange, MIREX, is a community evaluation campaign run alongside the annual conference of the International Society for Music Information Retrieval, ISMIR. The MIREX wiki, as retrieved on 16 September 2026, lists results archives back to 2005, and its 2026 page sets out established tasks including audio chord estimation, beat tracking, key detection and lyrics-to-audio alignment, run through a shared submission server against published metrics. Task captains collect entries, run them against common data, and publish a results page every year.

What the documents say

ISMIR describes itself as a non-profit advancing 'computational tools for processing, searching, organizing, and accessing music-related data', and runs the peer-reviewed conference where MIREX results are presented. The MIREX page describes how a task works: participants read a 'Participant Agreement', submit a system with an extended abstract, and a task captain reports results on a public wiki page, a structure unchanged in outline since the earliest listed years. For 2026, the same page adds new tasks named 'AI-Generated Music Detection' and 'Music Audio Generation', but under a section labelled 'What's new' that calls for 'novel evaluation pipelines' because, in the document's own words, there is 'a growing need for new evaluation methodologies' for generative systems.

Why it matters for makers

The mechanism that makes MIREX useful for tasks like separation or transcription is comparability: every submitted system is scored against the same held-out data with the same metric, so a claimed improvement is a claim other researchers can check. That comparability took two decades to build for analysis tasks. The same document shows generation evaluation is still being called for rather than settled, meaning a headline claim that one music generator 'beats' another lacks the shared, adversarial-tested benchmark that transcription or key-detection claims can now be checked against.

What to check before you use it

When a tool markets itself using an MIR benchmark, check whether the benchmark measures the specific task you care about, such as separation quality, and not a different task the marketing implies. Where a generation claim cites no shared, community-run evaluation, read that as this record does, editorially: the absence reflects a real gap the field's own organisers describe as unresolved, not an oversight in a single company's reporting.

• Does the cited MIREX or ISMIR result actually cover the task the marketing claim is about?
• Is the comparison run on shared, held-out data with a published metric, or on the vendor's own chosen examples?
• For a generation claim specifically, does any community evaluation exist yet, or is this still, per MIREX's own 2026 call, an open problem?

Two decades of shared evaluation gave analysis tasks a common yardstick. Generation is still waiting for one, and the organisers running MIREX say so themselves rather than claiming otherwise.

Continue in Listening & research, or A shared dataset made separation scores comparable, not perceptual.