midiviz

#031 · RoboPianist vs #029

RoboPianist (Zakka et al., CoRL 2023) trains two simulated Shadow Dexterous Hands to play the piano in MuJoCo with deep RL. #029 picks fingers with rules and plans the motion. This experiment runs both on the same physics and scores them the same way.

What is compared

  RoboPianist Ours (#029 + this folder)
Fingering human labels from PIG / Rousseau, used as a reward during training chosen by fingering/fingering.js (with Shadow Hand spans, see below)
Motion learned policy, 20 Hz, 38 joint targets fingering/motion.js fingertip plan → damped-least-squares IK at 100 Hz → 38 joint targets
Per song 5 M training steps (hours on a GPU) nothing: fingering 20–200 ms, then plays at real time
New MIDI needs a new policy any file
Physics MuJoCo, 200 Hz, Shadow Hands the same scene, loaded from their browser demo

The RL side is RoboPianist’s own browser demo data: the 20 Hz actions its authors recorded from their trained policies, replayed open loop through the same MuJoCo 2.3.1 wasm build.

Results (node robopianist/eval.js --human)

RoboPianist’s metric (wrappers/evaluation.py): a 20 Hz piano roll, the keys pressed after each 50 ms step, and precision, recall and F1 averaged over the frames.

Clip (the two RoboPianist ships MIDI for) RL policy F1 Ours F1 Ours with the human fingering
Twinkle Twinkle (Rousseau, 7.9 s) 89.1 86.5 71.2
Chopin Nocturne Op. 9/2 (Rousseau, 26 s) 84.0 57.1 58.4

Our precision is as high as the RL policy’s (92–99 %): we rarely press a wrong key. Our recall is lower: we miss notes. The shortfall comes from the robot, not from the fingering:

The RL policy learned to work around these limits for each song. We only model them, in the fingering span table and the IK.

Our fingering agrees with the human labels on hand 94–100 % and on hand plus finger about 48 % of the time. On this hand, though, the human fingering plays worse than ours on Twinkle and about the same on the Nocturne.

Pieces with no policy, for scale: Für Elise (full, 165 s) scores F1 60.6 and Bach’s C major Prelude 63.1. RoboPianist reports an average F1 of 0.79 on its Etude-12 set, but each of those songs was trained separately.

How our side drives the hands (sim.js)

  1. Fingering: fingerPiece runs with SHADOW_FINGERING, a span table measured with this IK (calibration on white keys) and a high cost for thumb crossings.
  2. Timing: notes are re-timed to the metric’s 50 ms frames (framedNotes), so a re-struck key gets the one-frame gap the score requires. Held keys that the hand cannot keep are let go (releaseOutOfReach).
  3. Fingertip plan: buildMotion (#029). Here glide is on and the approach is longer, because the robot arm is slower than a pianist’s.
  4. IK: on a second, kinematics-only MuJoCo data, a damped-least-squares solve over 19 actuated variables per hand:
    • targets are the five fingertips plus each last knuckle, which keeps fingers curved rather than clawed;
    • the Jacobian comes from grouped finite differences (8 kinematics() calls per iteration for both hands);
    • joint limits are handled with an active set.
  5. Servos: each servo is close to first order, so its target is q + τ·q̇.
  6. Closed loop: a finger presses only once its real fingertip is within 8 mm of the key. Otherwise it hovers, so it does not land on a neighbour on the way.

Files