midiviz

Harmony analysis of MIDI: the theory, the algorithms, and what other apps do

Background for #041 Harmony and harmony.js. The task: as a MIDI file loads, find its keys and its chords, name them (symbols such as G7/B) and explain them (Roman numerals such as V65, tonic / predominant / dominant, cadences), then show it in time while it plays.

1. What an analyst does

Chords. A chord is named by its root, its quality and its bass. The qualities that cover most common-practice music are the four triads (major, minor, diminished, augmented) and five sevenths (dominant, major, minor, half-diminished, fully diminished). In the Kostka–Payne textbook corpus the major triad is 43.6 % of the labels, the dominant seventh 21.9 %, the minor triad 19.4 %, the diminished seventh 4.4 %, the half-diminished seventh 3.7 % and the diminished triad 1.8 % (Pardo & Birmingham 2002, table 2). In the Beethoven sonatas of BPS-FH: major 4,720 segments, dominant seventh 3,156, minor 2,076, diminished seventh 665, diminished 370, minor seventh 184, half-diminished 144, augmented sixth 97, augmented 35, major seventh 31 (our count of the dataset). Pop and jazz add sus2/sus4, sixths, add9, ninths and “power chords” (root and fifth).

Inversion. Root position has the root in the bass; first inversion the third, second the fifth, third (sevenths only) the seventh. Roman numerals mark it with figured-bass numbers: triads I, I6, I64; sevenths V7, V65, V43, V42. Chord symbols use a slash: G7/B.

Notes that are not in the chord. Most of the notes in real music are chord tones, but melodies move through passing notes (stepwise between two chord tones), neighbour notes (step away and back), suspensions (held over from the previous chord, resolving down), appoggiaturas (accented, leapt to), anticipations and pedal points (a held bass under changing chords). Analysts discount them, and judge them by rhythm: a non-chord tone is usually short and weak, a chord change usually lands on a strong beat. The harmonic rhythm (how often the chord changes) is usually steady within a passage.

Keys. A key is a tonic and a mode. Music modulates: a sonata’s second theme is in the dominant (major) or the relative major (minor), and development sections travel. Brief tonicizations, a V7/V on the way to V, don’t count as modulations. Most analysts label the local key per passage.

Roman numerals. The chord’s root as a scale degree of the local key, upper case for major / augmented, lower for minor / diminished, ° diminished, ø half-diminished, + augmented. In minor, the raised leading tone makes V major and vii° diminished. Chromatic chords have their own names: applied (secondary) dominants V/x and leading-tone chords vii°7/x; mixture from the parallel mode (iv, ♭VI, ♭VII in major); the Neapolitan N6 (major on ♭2, usually in first inversion); augmented sixths (Italian, French, German) on ♭6 resolving to V.

Functions. Riemann’s three: tonic (I, vi, iii), predominant / subdominant (IV, ii, and vi going to them), dominant (V, vii°). Phrases go T → PD → D → T. I64 before V is the cadential six-four, a dominant.

Cadences. Perfect authentic (V–I, both in root position, on a strong beat; strictly also the tonic in the melody), imperfect authentic (one of them inverted or no tonic on top), half (phrase ends on V), deceptive (V–vi; V–VI in minor) and plagal (IV–I).

Spelling. In C major the chord on the raised fourth degree is F♯dim7, not G♭dim7, because it leads to G. Analysts spell chords from their function, so a program has to name keys and chords on the line of fifths, not on the twelve pitch classes.

Sources: Kostka, Payne & Almén, Tonal Harmony; Open Music Theory, Roman numerals; Masada & Bunescu 2018, appendix A (chord types) and B (figuration).

2. Published algorithms

Key finding

Chord labelling and segmentation

Datasets

KP corpus (46 textbook excerpts), BaCh (60 Bach chorales), TAVERN (27 sets of variations), BPS-FH (first movements of Beethoven’s 32 sonatas, Chen & Su 2018; github, GPL-3.0), When in Rome and the DCML corpora (about 1,700 works between them), POP909(-CL) for pop.

3. How other apps do it

app what it does how (as far as documented)
Logic Pro 12 Chord ID: drag an audio or MIDI region onto the Chord track, or Chords → Analyze Chords; writes a chord group. The control bar also names the chord of the selected or playing notes Not documented. Users report it works “with varying degrees of success depending on how complicated the source MIDI is” (pianoandsynth.com)
Cubase Project → Chord Track → Create Chord Symbols from selected MIDI, or drag a part onto the chord track; Chord Assistant proposes progressions One symbol per change of the sounding set; users say it “always transfers the chord (or cluster) to the Chord Symbol”, so passing notes make chords (Steinberg forum)
Studio One Drag a MIDI or audio clip to the Chord Track; chords are detected and can drive other tracks Not documented
Melodyne (in Studio One and stand-alone) Chord Track and chord recognition from audio Polyphonic pitch detection, then chord recognition
Dorico Write → Generate Chord Symbols from Selection; the dialog lets you set how much surrounding music to take into account Per rhythmic position, with a context setting (Steinberg manual)
MuseScore 4 No built-in recognition (“there’s no automatic way”, MuseScore forum, 2024); plug-ins such as Chord Identifier (Pop & Jazz) and ChordID write chord symbols from the notes, and m13253’s automatic chord identifier also colours notes by their role (root, third, seventh) Vertical sonorities per onset or beat
Scaler 3 / Scaler Detector Detect chords and key from MIDI (and audio); lists detected chords and the scales that fit “Detecting MIDI in Scaler 3 does not take into account chord duration or timing. It simply presents a list of detected chords” (Scaler forum)
Ableton Live 12 No chord recognition built in; MIDI Monitor shows chord names; Live 12 extensions and Max for Live devices (e.g. Kord Detect, HyperChord) name chords per bar Third-party: per-bar or per-onset matching, Roman numerals from the Set’s key
Hooktheory / Hookpad Writes everything as Roman numerals in a key chosen by the user Input is already chords
tonal.js Chord.detect ["E", "G#", "B", "C#"] → ["E6", "C#m7/E"] Pitch-class set lookup in a chord dictionary, the first note taken as the bass; no time
music21 chordify() slices at every onset (“salami slicing”), roman.romanNumeralFromChord(chord, key); analyze('key') with the profiles above, windowed analysis Vertical, no segmentation beyond the slices
MIDI plug-ins (e.g. Dusk Audio Chord Analyzer) Name the chord being held: root, quality, slash bass, inversion Instant lookup of the held notes

Two families, then. Most DAWs and plug-ins name what sounds at an instant (or in a bar) by looking it up: fast, live, and wrong whenever a melody note is not in the chord or the chord is broken over time. The research systems segment and label together over the whole piece, with metre, bass, figuration and key. #041 does the second for files and the first for a MIDI keyboard, and its Instant lane shows the first beside the second so the difference can be seen.

4. What harmony.js does

  1. Slices. Partition points at note onsets (chords struck together count once). In each slice a note weighs its duration in beats, × (1 + 0.6 × (accent − 0.3)) for a strong beat, × 0.55 for the part only the sustain pedal holds, decaying with its age so a long-held note does not dominate later chords. Notes that look like non-chord tones are marked (see 4a).
  2. Bass. A slice’s lowest note votes for the bass only if it is within 9 semitones of the lowest note in the surrounding two beats, so the upper notes of a broken chord don’t. A chord’s bass is its lowest chord tone sounding for at least a fifth of a beat.
  3. Keys, pass 1. Beat histograms smoothed by a two-sided exponential window (6.7 beats), correlated with a key profile (Aarden–Essen by default), Viterbi over the 24 keys with a cost per modulation plus per step on the circle of fifths.
  4. Chords. A semi-Markov Viterbi over partition points (a semi-CRF with hand-set weights): a segment’s score is its length in beats × the fit of its best labels, where the fit is Pardo & Birmingham’s in − out − missing (normalised; fifths may be left out, sevenths not; a marked non-chord tone that is not in the label is partly or wholly excused) plus the root sounding, the bass on the root or a chord tone, a quality prior, and the chord being diatonic and usual in the local key. Each change costs a constant, more off the beat, less on a strong beat; root motion down a fifth and a dominant resolving earn a bonus. Up to 18 beats per segment; repeated labels merge. The weights were tuned by coordinate ascent on the odd-numbered BPS sonatas (--tune), the even ones kept for testing.
  5. Diminished sevenths and augmented triads are symmetric: the root is the one that resolves up by step (or a fourth for augmented), else the leading tone of the key, else the bass.
  6. Keys, pass 2. A second Viterbi over the chords: a key gains when a chord is diatonic in it, more for its tonic, its dominant, and most for a dominant that resolves to the tonic; applied dominants that resolve inside the key are not counted against it. This moved BPS key accuracy from 65 to 75 %.
  7. Roman numerals, functions, cadences, spelling from the local key: applied chords, mixture, Neapolitan, German / Italian sixths, the cadential six-four; PAC / IAC / HC / DC / PC; names on the line of fifths from the key (the file’s key signature decides F♯ vs G♭ major; local keys are spelled near the home key, so A♭ minor in an A♭ major piece), applied chords spelled from their target.
  8. Alternatives and confidence per chord (the next best labels for the same span), shown on hover.

For a MIDI keyboard, nameChord names the held notes on their own, the way the instant tools above do.

4a. Non-chord tones: passing notes, suspensions, appoggiaturas

markOrnaments() marks candidates before the chord search, from the notes alone, after Masada & Bunescu’s figuration rules (appendix B) with an appoggiatura rule added:

kind rule
passing / neighbour at most ¾ of a beat, approached by step from a note that just ended and left by step to a note that just starts, on a weaker beat than the note before
suspension (held) a note held while a new chord arrives on a beat at least as strong as its resolution, dissonant with a voice that holds on (over the bass a 2nd / 9th, 4th or 7th; as the bass a 2nd or 7th under an upper voice), then stepping down (or up a semitone) to a note consonant with the bass. Only the part after the new chord is ornamental; the preparation is a chord tone
suspension (restruck) the same pitch just ended as the note starts, dissonant, resolving down by step
appoggiatura / accented passing struck dissonant on a stronger beat than its resolution, resolving by step; “accented passing” when it was also approached by step

A dissonance that would make a seventh chord with the other sounding notes is not a suspension (it is the chord’s seventh). Each suspension gets its figure (4–3, 9–8, 7–6, 2–3 in the bass).

Marking a note does not remove it. In the segment score a marked note counts as “out” only by (1 − excuse) when it is not a tone of the label being scored, and fully “in” when it is (excuse: passing / neighbour 0.7, suspension 1, appoggiatura 0.5). So a marked note that is really a chord tone costs nothing, and the search still decides. (Giving the resolution point a higher chord-change cost as well made things worse on both corpora, so it was left out.)

How often the marks are right: on BPS-FH 56–64 % of marked suspensions and appoggiaturas are outside the analyst’s chord, against 18 % of all notes; on the When in Rome Bach chorales 83 % of marked suspensions are. Of the suspensions in the analysts’ labels (a note held or restruck into a chord it is not in, resolving down by step to a tone of that chord), 77 % are marked in the chorales and 48 % in the Beethoven.

What it changes (the “under suspensions” column is the share of time correct inside those suspensions):

corpus   root quality inversion Roman numeral root under suspensions root + quality under suspensions
BPS-FH even sonatas (held out) without 79.5 74.3 69.6 45.5 48.1 44.2
  with 80.3 75.7 70.3 46.3 54.6 50.4
When in Rome, 371 Bach chorales without 77.6 72.7 73.0 37.2 73.3 61.8
  with 77.8 74.0 74.2 37.8 75.2 66.5

The chorales were never used for tuning (harmony/test/wir_export.py exports When in Rome’s analyses with music21’s scores; bps.js --corpus wir/chorales scores them). Their low key score (59 %) is mostly cadences in the relative or subdominant key that the analysts label as tonicizations inside the home key.

How well it does

node harmony/test/bps.js on BPS-FH (all 32 movements, the same CSR measure as the table above, 1/8-beat frames; augmented sixths left out of root and quality):

version key root quality inversion Roman numeral
templates + segmentation, key pass 1 65.4 75.4 67.5 66.2 32.8
+ key pass 2 (from the chords) 75.0 75.4 67.5 66.2 37.5
+ bass from the lowest nearby note, strong-beat changes 76.2 76.8 69.7 69.7 44.9
weights tuned on the odd sonatas, all 32 76.5 80.6 76.2 71.2 48.1
the same, even sonatas only (held out) 75.4 80.4 75.7 70.5 45.6
+ suspensions and appoggiaturas (4a), all 32 76.8 80.7 76.6 71.4 48.6
the same, even sonatas only (held out) 76.5 80.3 75.7 70.3 46.3

These are not directly comparable with the neural models’ rows (they report the BPS test split, and were trained on the rest of the corpus and more), but the scale is the same: 2–4 points behind the trained models on root, between them on quality (75.7 vs 74.1–79.0), 3–6 behind on inversion, 7–10 behind on key, and the full Roman numeral at their level (45.6 held out vs 45.4–49.1). Most remaining key errors are the subdominant or the relative key: a passage in the dominant that the analyst hears as still in the tonic, or the reverse. Quality errors are mostly a passing seventh made part of the chord or left out, which human analysts also argue about.

It is fast enough to run on load: 0.4 s for Bach’s BWV 846 (3:46, 1,284 notes), 1 s for Chopin’s Polonaise Op. 53 (5:52, 6,052 notes), in a worker.

5. Limits and ideas