Two decades of shared MIR evaluation still exclude generation

MIREX's own 2026 call for proposals says generative-model evaluation is a gap, not a solved comparison.

Site publication

September 18, 2026

What happened

The Music Information Retrieval Evaluation eXchange, MIREX, is a community evaluation campaign run alongside the annual conference of the International Society for Music Information Retrieval, ISMIR. The MIREX wiki, as retrieved on 16 September 2026, lists results archives back to 2005, and its 2026 page sets out established tasks including audio chord estimation, beat tracking, key detection and lyrics-to-audio alignment, run through a shared submission server against published metrics. Task captains collect entries, run them against common data, and publish a results page every year.

What the documents say

ISMIR describes itself as a non-profit advancing 'computational tools for processing, searching, organizing, and accessing music-related data', and runs the peer-reviewed conference where MIREX results are presented. The MIREX page describes how a task works: participants read a 'Participant Agreement', submit a system with an extended abstract, and a task captain reports results on a public wiki page, a structure unchanged in outline since the earliest listed years. For 2026, the same page adds new tasks named 'AI-Generated Music Detection' and 'Music Audio Generation', but under a section labelled 'What's new' that calls for 'novel evaluation pipelines' because, in the document's own words, there is 'a growing need for new evaluation methodologies' for generative systems.

Why it matters for makers

The mechanism that makes MIREX useful for tasks like separation or transcription is comparability: every submitted system is scored against the same held-out data with the same metric, so a claimed improvement is a claim other researchers can check. That comparability took two decades to build for analysis tasks. The same document shows generation evaluation is still being called for rather than settled, meaning a headline claim that one music generator 'beats' another lacks the shared, adversarial-tested benchmark that transcription or key-detection claims can now be checked against.

What to check before you use it

When a tool markets itself using an MIR benchmark, check whether the benchmark measures the specific task you care about, such as separation quality, and not a different task the marketing implies. Where a generation claim cites no shared, community-run evaluation, read that as this record does, editorially: the absence reflects a real gap the field's own organisers describe as unresolved, not an oversight in a single company's reporting.

• Does the cited MIREX or ISMIR result actually cover the task the marketing claim is about?
• Is the comparison run on shared, held-out data with a published metric, or on the vendor's own chosen examples?
• For a generation claim specifically, does any community evaluation exist yet, or is this still, per MIREX's own 2026 call, an open problem?

Two decades of shared evaluation gave analysis tasks a common yardstick. Generation is still waiting for one, and the organisers running MIREX say so themselves rather than claiming otherwise.

Sources & reading trail

Lists MIREX results archives since 2005, describes the task-captain process, and calls for new generative-model evaluation pipelines for 2026.

Source published: Not established · Retrieved: 16 September 2026

Describes ISMIR's mission and its role hosting the conference where MIREX results are presented.

Source published: Not established · Retrieved: 16 September 2026

Papers, terms and official documents establish the record; the maker reading and the checks are Signal to Song editorial analysis. This retrospective draft does not imply the site published on the event date.

Continue reading

• A shared dataset made separation scores comparable, not perceptual
• A generation metric adapted from images admits a moderate correlation
• Open source made stem separation common but not a rights grant
• Browse the complete the archive

Published September 18, 2026, not on the date of the event described.