Trackform — Audio-to-MIDI & ML Evaluation
Can AI ease live preparation while preserving the artist’s musical expression?
Trackform in use
Recorded in the updated macOS app on 25 October 2026: a 20-second bass recording is converted, inspected, edited, and previewed. Silent demo; file selection and processing waits are shortened.
Research progress
Completed work and planned validation, shown separately.
- Completed
Music workflow
Transcription, separation, piano-roll editing, audition, and export.
- Completed
Held-out evaluation
Slakh test 20 rendered mixes and MDB 23 real recordings; per-song and per-class scoring.
- Completed
Fine-tuning & decision
Full and frozen-block adaptation compared; rejected after real-recording regression.
- Not started
Generalization & usability
Secure real-recording training data, test rehearsal, and measure MIDI correction effort.
ADTOF · HT-Demucs stems · 5-class onset F1 · per-song mean · 50 ms tolerance
| Evaluation | Pretrained A0 | Frozen A4f |
|---|---|---|
| Slakh validation · 16 songs | 0.695 | 0.786 |
| MDB real recordings · 23 songs | 0.859 | 0.816 |
Decision: reject fine-tuning; retain pretrained weights. The paired per-song MDB F1 difference is −0.043, with a 95% bootstrap interval of [−0.066, −0.022]. A4f, selected on validation F1, freezes the convolutional and batch-normalization block.
Training used 40 Slakh train songs, with 16 separate validation songs. Real-recording rehearsal was not run because of data-access constraints. The reversal is consistent with domain shift or forgetting, but does not establish either cause.
Experiment report ↗ · Validation data ↗ · Real-recording data ↗
Comparison conditions and class-level changes
Full fine-tuning A1: Slakh 0.760, MDB 0.816. The A1 MDB evaluation was an additional reference run outside the single-candidate selection rule. A4f MDB class F1: kick 0.902 → 0.932, cymbal 0.680 → 0.723, hi-hat 0.810 → 0.747, tom 0.586 → 0.528.
Training settings and freezing implementation ↗Earlier experiment: seven-song synthetic LOSO
Seven folds trained the earlier classifier on six songs, excluding each evaluation song. Mean F1: mixture 0.384, Demucs stem 0.752, reference stem 0.763. The model and dataset differ from the current ADTOF experiment, so these scores are not a direct before/after comparison.
Recorded LOSO results ↗ · Evaluation configuration ↗My contribution
- Wonder
- Can AI ease live preparation while preserving the artist’s musical expression?
- Build
- Integrated specialist transcription models and an editable MIDI workspace; implemented fine-tuning comparisons and reproducible scoring across rendered and real recordings with AI assistance.
External tools & AI
AI-assisted development using existing Transkun, Basic Pitch, ADTOF, and Demucs models. Training and evaluation code, settings, and results are linked for inspection.
- Find out
- ADTOF fine-tuning improved rendered validation F1 (0.695 → 0.786) but reduced real-recording F1 (0.859 → 0.816). The pretrained model was retained.
Process & photos
Trackform turns recorded music into editable MIDI for performance preparation. It combines specialist transcription models, Demucs stem separation, a piano-roll editor, audition, and MIDI export. The Next.js interface and FastAPI processing pipeline also support a local Tauri desktop application.
The ML question is whether adapting a pretrained model to the actual processing path improves transcription on unfamiliar recordings. Piano uses Transkun, bass uses Basic Pitch with hybrid post-processing, and drums use ADTOF on HT-Demucs stems. These are existing models integrated into an assistive editing workflow; generated notes remain open to correction.
ADTOF fine-tuning targeted separation artifacts and weak drum classes. Slakh supplies paired rendered audio and MIDI labels, making supervised adaptation to separated stems possible. The reduced experiment used 40 Slakh train songs for training and a separate 16 for validation. MDB Drums, 23 real recordings, was reserved for test evaluation. The original plan included real-recording training and rehearsal data; ADTOF data required access approval and the ENST download was too slow, so those conditions were not run.
The experiment compared pretrained weights (A0), full fine-tuning (A1), and a frozen convolutional/batch-normalization block with trainable GRU and output layers (A4f). The initial learning rate was 1e-4, one fifth of the original setting. A smaller update and frozen feature extractor were conservative adaptation choices, not a demonstrated optimum from a learning-rate sweep. The implementation uses Adam, binary cross-entropy, seed 0, validation-loss early stopping (patience 3), and learning-rate reduction. Candidate selection then uses validation F1.
A4f led rendered validation: F1 rose from 0.695 to 0.786. On MDB, it fell from 0.859 to 0.816; the paired difference was −0.043 with a 95% bootstrap interval of [−0.066, −0.022]. Hi-hat and tom performance declined despite gains for kick and cymbal. The candidate was rejected and pretrained ADTOF remained the default. A1 also scored 0.816 on MDB, but that additional test was recorded outside the planned single-candidate selection rule.
This reversal is consistent with domain shift and loss of useful pretrained behavior, but it does not isolate the cause. A proposed follow-up would compare rendered-only training, real-recording training, and mixed training with rehearsal, each with frozen and unfrozen feature blocks. Data access, matched training budgets, multiple seeds, real-recording validation, and a fresh untouched final test set would be needed. Per-class errors and paired intervals would determine adoption; repeated inspection of MDB should not become test-set tuning.
This is AI-assisted development with documented model integration, fine-tuning code, and evaluation records. The research contribution presented here is the question, experimental comparison, and evidence-based adoption decision. The project does not claim a new foundation model or proven workflow benefit. MIDI correction effort and rehearsal preparation time still require user evaluation.
