Audio-to-MIDI System (Ongoing Research)
How can AI reduce the repetitive work of preparing a live session while leaving musical decisions with the artist?
A web-based mini DAW prototype that turns recordings into editable performance-session drafts with Trackform transcription, stem separation, and a browser piano roll.
Analyse BPM, key, form, and lyric timing; generate instrument-specific MIDI and session elements; then let the musician inspect, edit, audition, and export the result.
The current prototype supports piano, general audio, bass, and drums, with Demucs stems and a shared Next.js/FastAPI pipeline that can be packaged for desktop.
Compare one to three test songs against manual reference sessions and test click, cue, lyric, and playback workflows in a rehearsal-style environment.
Preparing a live performance from an existing recording involves more than transcription: musicians must identify BPM, key, song form, lyric timing, and the parts needed for playback, then build click and cue tracks. These steps often require both musical judgement and specialist DAW skills, creating a high technical burden for young musicians and solo creators.
This project explores an assistive workflow in which AI creates a first draft and the musician remains responsible for listening, correcting, arranging, and deciding what reaches the stage. The longer-term prototype is designed to analyse an uploaded song, propose MIDI and performance-session elements, and provide a structured editor for review rather than treating AI output as a finished result.
The current Trackform prototype supports specialist paths for piano, general audio, bass, and drums, plus Demucs stem separation. Piano mode combines Transkun transcription with an audio-guided sustain repair pass. Generated MIDI opens in a browser piano roll where notes can be moved, resized, auditioned, and exported; the shared Next.js and FastAPI pipeline can also be packaged as a Tauri desktop application.
The next phase tests one to three songs against manually prepared reference sessions and explores BPM, key, song-form, lyric-cue, click-track, and cue-track workflows. Playback in a speaker and rehearsal-style environment will be used to identify failures and iterate on the editor. The goal is a small, demonstrable performance-session prototype—not a replacement for a full DAW—whose value can be judged by how much repetitive preparation it removes while keeping creative decisions visible.
