# 01 — Audio → MIDI / MusicXML transcription for the browser: models, real-time options, runtimes

Research brief, 17 Oct 2026. Scope: what can turn audio (a file or a live microphone) into notes
inside a midiviz page, how good each option is, how close to real time it gets, and what it
takes to run it client-side. Companion to `fingering/research/02_algorithms.md` and
`strings/research/02_algorithms.md` (the consumers of the notes).

Tags:
- [V] = verified this session in a fetched source (paper abstract or HTML, model card, repo
  README, npm registry); [V, abstract] = only the abstract was read.
- [V, via X] = the detail comes from source X, not from the original.
- [T] = background knowledge, not checked this session.
- [?] = uncertain or not found.

Contents
1. Short answer
2. Taxonomy
3. General-purpose / multi-instrument models (Basic Pitch, MT3 family, MuScriptor)
4. Piano models (offline)
5. Streaming and low-latency piano
6. Monophonic pitch (voice, violin, winds)
7. Source separation as a front end
8. Performance MIDI → score (MusicXML)
9. MuScriptor vs Transkun
10. Browser runtimes: onnxruntime-web, Transformers.js, tfjs, ggml
11. Commercial reference points
12. Recommended plan for midiviz
13. References

---------------------------------------------------------------------------------------------

## 1. Short answer

- **Easiest today:** Spotify's Basic Pitch (`@spotify/basic-pitch`, tfjs, <17K parameters [V, via
  NeuralNote README]). Any instrument, pitch bends, tiny, Apache-2.0 [T]. Weak on dense piano and
  full mixes, no instrument labels, near-real-time at best (windowed) [T].
- **Best open multi-instrument:** MuScriptor (Kyutai + Mirelo, July 2026). 100M / 300M / 1.3B
  decoder-only transformer, Multi-F1 48.2 vs 21.9 for YourMT3+ on the authors' test set [V].
  Weights **CC BY-NC 4.0**, no velocity, 5 s chunks, autoregressive [V]. No browser build yet.
- **Best piano:** Transkun v2 (semi-CRF + non-hierarchical transformer, ISMIR 2024), claims state
  of the art on every MAESTRO subtask, MIT, outputs velocity and pedals [V]. Non-autoregressive,
  so it exports to ONNX far more easily than any seq2seq model [T].
- **Live piano:** nothing off the shelf runs under ~100 ms. Research systems: Kwon et al. 2024
  (128–320 ms), Mobile-AMT (Yamaha), Hu et al. ISMIR 2025 (strictly causal, targets <30 ms,
  code released) [V, abstract].
- **Live monophonic:** SwiftF0 (96K params, ~42× faster than CREPE on CPU) and PESTO
  (self-supervised, streaming) are small enough for an AudioWorklet [V, abstract].
- **Score:** transcription gives performance MIDI. MusicXML needs a second stage (beats,
  quantization, voices, hands, spelling): MIDI2ScoreTransformer, PM2S, ScoreTranscriber, or the
  heuristics in MuseScore (`webmscore.html` in this repo already imports MIDI and exports
  MusicXML).
- **Runtime:** Transformers.js is a layer on top of onnxruntime-web, not an alternative to it
  (v4.3.1 depends on `onnxruntime-web` 1.31.0-dev [V, npm]). For custom audio models, use
  onnxruntime-web directly with the WebGPU EP and a WASM fallback.

## 2. Taxonomy

| Family | Output stage | Examples | Browser cost |
|---|---|---|---|
| Frame / onset roll + post-processing | per-frame pitch activations, thresholds | Onsets and Frames, Basic Pitch, Kong et al., hFT-Transformer, Mobile-AMT | one forward pass per window, cheap to stream |
| Structured decoding | intervals decoded by dynamic programming | Transkun (semi-CRF) | one forward pass + Viterbi in JS |
| Seq2seq token decoding | MIDI-like tokens, autoregressive | MT3, MR-MT3, YourMT3+, Aria-AMT, MuScriptor | hundreds of decoder steps per segment, needs a KV cache |
| Monophonic F0 | one pitch per frame | CREPE, SPICE, PESTO, SwiftF0, pYIN | tiny |
| Audio → score end-to-end | notation tokens (**Kern, MusicXML-like) | piano-a2s, Rubato | seq2seq cost, research code |

## 3. General-purpose / multi-instrument models

### Basic Pitch (Spotify, ICASSP 2022)
- Lightweight CNN over a harmonic-stacked CQT, three outputs (onset, note, contour) with pitch
  bends [T]. Under 17K parameters [V, via NeuralNote README].
- `@spotify/basic-pitch` on npm runs on tfjs; latest 1.0.1, last modified Aug 2025 [V, npm].
  The Python package ships TF, CoreML, TFLite and ONNX models [V, via docsearch summary].
- Processes overlapping ~2 s windows [T], so live use means a sliding window with about a second
  of delay. No instrument identity, velocity is a rough amplitude estimate [T].
- NeuralNote v1 ran it with RTNeural (CNN) and ONNX Runtime (CQT + harmonic stacking) [V].
  NeuralNote v2 dropped it for MuScriptor [V].

### MT3 family
- MT3 (Google, ICLR 2022): T5 encoder-decoder, spectrogram → event tokens, multi-instrument [V,
  via MuScriptor wiki].
- MR-MT3 (2024): memory retention and token shuffling against instrument leakage [V, abstract].
- YourMT3+ (QMUL, 2024): PerceiverTF encoder with spectral cross-attention, mixture of experts,
  multi-channel decoder, cross-dataset stem augmentation, singing tokens [V]. Was the open state
  of the art before MuScriptor.

### MuScriptor (Kyutai + Mirelo, with IRCAM; released 9 July 2026)
- Decoder-only transformer; the mel spectrogram of a 5 s segment (16 kHz mono, n_fft 2048, hop
  160 → 100 frames/s, 512 mel bins) is projected and used as a prefix; MT3-like tokens over 36
  instrument groups (`MT3_FULL_PLUS`); greedy decoding with optional classifier-free guidance [V].
- Sizes: small ≈100M (dim 768, 14 layers), medium ≈300M (1024, 24), large ≈1.3B (1536, 48) [V].
- Training: 1.45M synthetic MIDI files rendered with 250+ soundfonts, then 170K real recordings
  (11,000+ hours), then GRPO reinforcement learning on 300 verified tracks [V, via wiki].
- Results on the authors' 372-track test set: large Multi-F1 48.2 vs YourMT3+ 21.9 (onset F1
  32.5 for YourMT3+) [V]. Scaling study (trained on real data only): small onset 51.2 / frame
  67.2 / offset 38.7 / multi 38.2; medium 52.4 / 68.0 / 40.3 / 39.7 [V].
- Limitations from the model card: **no velocity**, cannot represent two simultaneous notes of the
  same pitch and instrument, drums onset-only, weaker offsets [V]. Instrument conditioning (tell it
  which groups are present) improves results and keeps instruments consistent across chunks [V].
- Licence: code MIT, weights CC BY-NC 4.0 plus conditions (user must hold the rights to the
  music transcribed), gated on Hugging Face [V].
- Native speed (`muscriptor.cpp` on ggml, Apple M1 Pro, real-time factor, >1 is faster than real
  time) [V]:

  | Size | CPU | Metal | F16 weights | KV cache |
  |---|---|---|---|---|
  | small | 1.94× | 3.48× | 209 MB | ~218 MB |
  | medium | 0.69× | 1.56× | 618 MB | ~499 MB |
  | large | — | 0.48× | 2.74 GB | ~1.5 GB |

  Medium decode step: 13.1 ms CPU, 7.3 ms Metal; prefill of the 5 s prefix 1.71 s CPU, 0.16 s
  Metal [V]. GGUF files: `DamRsn/muscriptor-gguf` [V].
- Streaming: the Python API yields NoteStart/NoteEnd events as decoded [V], but the unit of work is
  a 5 s chunk, so live latency is at least the chunk length plus decode time [T].

## 4. Piano models (offline)

| Model | Year | Notes | Licence |
|---|---|---|---|
| Onsets and Frames | 2018 | in `@magenta/music` (tfjs), runs in a browser today; older accuracy [T] | Apache-2.0 [T] |
| Kong et al. (ByteDance) high-resolution | 2021 | regression onsets/offsets, pedals; community ONNX ports [T] | MIT [T] |
| Transkun v1 (semi-CRF) | NeurIPS 2021 | intervals scored and decoded by Viterbi, then velocity and refined onset/offset [V] | MIT [V] |
| Transkun v2 | ISMIR 2024 | encoder-only non-hierarchical transformer on a low-time-resolution feature map, inner-product interval scoring; "state of the art across all subtasks" on MAESTRO v3 [V, abstract] | MIT [V] |
| hFT-Transformer (Sony) | 2023 | hierarchical frequency-time transformer [T] | [?] |
| Aria-AMT (EleutherAI) | 2024 | seq2seq, built for robustness on real recordings; transcribed the Aria-MIDI dataset (1.19M files, ~100K hours) [V] | [?] |
| Fernandez, "onsets and velocities" | 2023 | ~3.1M params, onset F1 96.78 on MAESTRO, real-time on commodity hardware [V, abstract]; its latency is 4–9 s per Hu et al. [V] | [?] |

Transkun's pip checkpoint is trained **without** extending notes by the sustain pedal and with
augmentation; papers usually report the pedal-extended convention, so compare like with like [V].

## 5. Streaming and low-latency piano

Latency budget from Hu et al.: digital instruments want ~10 ms, networked ensemble playing
tolerates ~30 ms; earlier online systems sit at 128–320 ms [V].

- **Kwon, Jeong & Nam (2024):** acoustic CNN with FiLM, pitch-wise LSTMs, multi-state softmax
  (onset / sustain / re-onset / offset / off) with an autoregressive connection; 128–320 ms [V, via
  Hu et al.].
- **Mobile-AMT (Kusaka & Maezawa):** lightweight convolutions, augmentation for unseen recording
  conditions; online state of the art, reported 174 ms, but Hu et al. find an oversight that makes
  it 10 s [V, via Hu et al.].
- **Hu, Peter, Schlüter & Widmer (ISMIR 2025):** remove all non-causal processing from
  Mobile-AMT, share computation, vary model size, study STFT / label encoding / causal
  post-processing. Accuracy drops with strict causality and there is a preprocessing-latency vs
  accuracy trade-off. Released as a baseline [V].
- **Niikura et al. (2025):** convolutional encoder plus an autoregressive onset decoder and a
  separate offset decoder with sustain-pedal validation; streaming, comparable to or better than
  offline state of the art on MAESTRO at lower compute [V, abstract].
- **Score following:** pairing real-time transcription with symbol-level tracking (2025) [V,
  abstract], relevant if a page should follow a known score rather than transcribe blind.

## 6. Monophonic pitch

- **SwiftF0 (2025):** STFT + 2D CNN, 95,842 params, 91.80 % harmonic mean at 10 dB SNR, ~42× faster
  than CREPE on CPU [V, abstract]. ONNX available [T].
- **PESTO (2023, TISMIR 2025):** self-supervised, Siamese, VQT frames, transposition-equivariant
  objective, built for streaming [V, abstract].
- CREPE / CREPE-tiny and SPICE run on tfjs (ml5 wraps CREPE) [T]. DSP without a model: YIN /
  pYIN (`pitchy`, `pitchfinder`), aubio compiled to WASM, Essentia.js [T].
- Direct use here: microphone → F0 → notes → the #033 fingering engine.

## 7. Source separation as a front end

- HT-Demucs (4 or 6 stems, incl. piano and guitar) has ONNX exports (`demucs-onnx`) [V].
- BS-RoFormer-SW 6-stem has an ONNX build packaged for onnxruntime-web on WebGPU
  (`elicwhite/bs-roformer-sw-6stem-onnx`) [V].
- Separate, then run a piano or monophonic model on one stem: the non-NC alternative to
  MuScriptor for full mixes.

## 8. Performance MIDI → score

- **MIDI2ScoreTransformer** (Beyer & Dai, ISMIR 2024): seq2seq piano performance MIDI → score
  with compound tokens, 3.5× shorter sequences than earlier tokenizations [V, abstract].
- **ScoreTranscriber:** one transformer for quantized rhythm, performance errors and hands, a
  second for MusicXML [V].
- **PM2S** (Liu et al.): CRNN for beats, quantization, hand part, key and time signature [T].
- **Transformer rhythm quantization with beat annotations** (2026) [V, abstract]; needs a beat
  tracker (e.g. Beat This! [T]).
- **End-to-end audio → score:** piano-a2s (Zeng et al., IJCAI 2024; hierarchical bar/note decoder,
  synthetic pre-training then human recordings, **Kern output) [V]; Rubato (2026) [V, via Dual
  Evaluation]; training on fully synthesized data (2026) [V, abstract].
- **Evaluation:** "A Dual Evaluation for Music Transcription" (Aug 2026) separates notation
  similarity from playback similarity, finds CLEWS the playback metric that best matches 100+
  listeners, and finds that across 24 pipelines (8 audio→MIDI × 3 MIDI→score) the MIDI→score stage
  decides which objective a pipeline favours [V, abstract].
- **In this repo:** `webmscore.html` (libmscore WASM) imports MIDI and exports MusicXML, a working
  baseline before any ML.

## 9. MuScriptor vs Transkun

| | MuScriptor | Transkun v2 |
|---|---|---|
| Task | any instruments, full mixes, 36 instrument groups | solo piano only |
| Architecture | decoder-only transformer, audio prefix, autoregressive tokens | encoder-only transformer, interval scores, semi-CRF Viterbi |
| Output | onset, offset, pitch, instrument; **no velocity**, no pedal; drums onset-only | onset, offset, pitch, **velocity**, **sustain/soft pedal** |
| Piano accuracy | not reported on MAESTRO in the model card [?] | state of the art on MAESTRO v3 (authors) |
| Inference | 5 s chunks, hundreds of sequential decode steps, KV cache | one parallel forward pass per segment + DP decode |
| Size | 100M – 1.3B | much smaller [?]; exact count not checked |
| Speed | small 1.9× RT on M1 Pro CPU natively | not measured here [?] |
| Licence | weights CC BY-NC 4.0 (+ rights condition) | MIT |
| Browser route | port `muscriptor.cpp`/ggml to WASM (+ ggml WebGPU), or ONNX export with a custom decode loop | ONNX export, onnxruntime-web, Viterbi in JS |

For midiviz, velocity and pedal drive the synth, the key animation and the hammers, so Transkun
is the better fit for piano material. MuScriptor is the choice for anything that is not solo
piano, if the non-commercial licence is acceptable. A combined pipeline is possible: MuScriptor
for instrument assignment, Transkun for the piano part.

## 10. Browser runtimes

### onnxruntime-web
- Latest stable on npm: 1.30.0; releases roughly monthly through 2026 (1.24.2 Feb … 1.29.0 Aug)
  [V, npm].
- Execution providers: `wasm` (CPU; every ONNX operator), `webgpu`, `webgl`, `webnn`; the GPU EPs
  support a subset of operators [V, ORT docs]. Import `onnxruntime-web/webgpu` for WebGPU [V].
- The WebGPU EP has been rebuilt in C++ (shared with native ORT), which Transformers.js v4 adopted
  and tested across ~200 architectures [V, via Transformers.js v4 blog]. WebGL is the legacy path
  [T].
- WASM multithreading needs `SharedArrayBuffer`, i.e. a cross-origin-isolated page (COOP/COEP
  headers) [T]. `serve.sh` does not send them, so WASM would run single-threaded here.
- WebGPU reached about 85 % of global users by March 2026 [V, via Transformers.js docs]; Firefox
  and older Chromium may need a flag, Safari support depends on version [V].

### Transformers.js
- v4 released 9 Feb 2026; current 4.3.1, depending on `onnxruntime-web` 1.31.0-dev and
  `onnxruntime-node` 1.30.0 [V, blog + npm].
- What it adds on top of ORT: Hugging Face Hub loading and caching, tokenizers (now the
  standalone `@huggingface/tokenizers`), feature extractors (e.g. Whisper log-mel), task
  pipelines, generation loops with KV cache, quantized dtypes, `ModelRegistry` for download sizes
  and cache state [V].
- v4 changes: C++ WebGPU runtime, also in Node, Bun and Deno; exports rebuilt with ORT contrib ops
  (`GroupQueryAttention`, `MatMulNBits`, `MultiHeadAttention`, `QMoE`), ~4× faster BERT
  embeddings; esbuild, `transformers.web.js` 53 % smaller [V].
- It has no music-transcription pipeline [?, not found]. It helps only if a transcription model is
  exported in one of its supported architectures. A MuScriptor port would reuse its decoder
  machinery (KV cache, contrib attention ops) but needs custom glue for the mel prefix.
- For Basic Pitch, Transkun, Kong, SwiftF0, PESTO, Demucs: use onnxruntime-web directly.

### TensorFlow.js
- Still what `@spotify/basic-pitch` and `@magenta/music` use. Its development has slowed relative
  to ORT [T]. Switching Basic Pitch to its ONNX model would put every model on one runtime.

### ggml / llama.cpp style
- `muscriptor.cpp` runs on ggml with CPU, Metal and Vulkan backends [V]. ggml has WebGPU and WASM
  builds [T]; compiling `muscriptor.cpp` with Emscripten is the most direct path to MuScriptor in a
  browser, unverified.

## 11. Commercial reference points

Klangio Transcription Studio, Songscription (reportedly built on Beyer & Dai's model [V, via
EverybodyWiki]), AnthemScore, ScoreCloud, Melodyne, Ivory, song2notation. Pianolyze transcribes
piano entirely in the browser (MP3/WAV/FLAC/M4A → piano roll, sheet view, MIDI) [V, search
snippet]; its engine was not checked [?].

## 12. Recommended plan for midiviz

1. **Baseline page:** Basic Pitch (tfjs, or its ONNX model on onnxruntime-web) for files and a
   sliding-window microphone mode; output through the existing player and `webmscore` for
   MusicXML. Add SwiftF0 or PESTO in an AudioWorklet for live monophonic input.
2. **Piano quality:** export Transkun v2 to ONNX, run on onnxruntime-web WebGPU with WASM
   fallback, Viterbi decode in JS (or a worker). Compare against Basic Pitch on MAESTRO clips with
   `mir_eval`-style onset / offset / velocity F1, the way #031 scored against RoboPianist.
3. **Live piano:** port the Hu et al. 2025 causal baseline (or Kwon 2024) and measure
   mic → note latency end to end in an AudioWorklet.
4. **Full mixes:** MuScriptor-small via an Emscripten build of `muscriptor.cpp` if the NC licence
   is acceptable; otherwise BS-RoFormer separation + per-stem models.
5. **Score:** start with MuseScore's MIDI import; try MIDI2ScoreTransformer later and judge it with
   both notation and playback similarity, as the Dual Evaluation paper recommends.

Open checks before shipping: licences of hFT-Transformer, Aria-AMT, SwiftF0, PESTO and Kong
checkpoints; Transkun parameter count and speed; whether MuScriptor reports MAESTRO numbers in
its paper.

## 13. References

- Basic Pitch: https://github.com/spotify/basic-pitch-ts
- NeuralNote: https://github.com/DamRsn/NeuralNote
- MuScriptor: https://github.com/muscriptor/muscriptor ; https://huggingface.co/MuScriptor/muscriptor-small ;
  paper arXiv 2607.08168 ; GGUF https://huggingface.co/DamRsn/muscriptor-gguf ;
  performance https://github.com/DamRsn/muscriptor.cpp/blob/main/docs/PERFORMANCE.md
- YourMT3+: https://arxiv.org/abs/2407.04822 ; MR-MT3: https://arxiv.org/abs/2403.10024 ;
  MT3: https://arxiv.org/abs/2111.03017
- Transkun: https://github.com/Yujia-Yan/Transkun ; v2 paper https://arxiv.org/abs/2404.09466
- Aria-AMT: https://github.com/EleutherAI/aria-amt ; Aria-MIDI https://arxiv.org/abs/2504.15071
- Fernandez 2023: https://arxiv.org/abs/2303.04485
- Kwon, Jeong & Nam 2024: https://arxiv.org/abs/2404.06818
- Hu et al. 2025: https://arxiv.org/abs/2509.07586
- Niikura et al. 2025: https://arxiv.org/abs/2503.01362
- Score following 2025: https://arxiv.org/abs/2505.05078
- SwiftF0: https://arxiv.org/abs/2508.18440 ; PESTO: https://arxiv.org/abs/2508.01488
- BS-RoFormer ONNX: https://huggingface.co/elicwhite/bs-roformer-sw-6stem-onnx ; demucs-onnx: https://pypi.org/project/demucs-onnx
- MIDI2ScoreTransformer: https://arxiv.org/abs/2410.00210 ; ScoreTranscriber: https://github.com/NZX23/ScoreTranscriber
- Rhythm quantization 2026: https://arxiv.org/abs/2604.22290
- piano-a2s: https://arxiv.org/abs/2405.13527 ; synthetic A2S: https://arxiv.org/abs/2609.00640
- Dual Evaluation: https://arxiv.org/abs/2608.04511
- onnxruntime-web: https://onnxruntime.ai/docs/tutorials/web/
- Transformers.js v4: https://huggingface.co/blog/transformersjs-v4