44 points hidude562 3 days ago 10 comments

I released dettato, my open weight audio to midi transcription model. It beats muscriptor 1.3b on every set and is roughly 60x smaller, without training on copyrighted recordings. Feel free to ask questions

gregzem 2 days ago | parent

Congrats with SOTA model. How good is it in recognition of the instruments which are overlapping by frequencies? Does is provide note offset and velocity in the transcribed MIDI? Thanks!

oever 53 minutes ago | parent

Audio-to-MIDI sounds very ambitious. How do you verify that the midi fits the audio? Does it result in useful beat detection? E.g. I'd very much like to have a way to map audio offset in a recording to labels in sheet music. But even beat detection in fairly standard jazz recordings fails often.

Does the model know when a piece (whole piece, not a section) is 3/4 or 4/4 th and can it give the onsets of all measures? Or all the midi notes just based on time and not organized rhythmically?

peaseagee 48 minutes ago | parent

Congrats on training on open and public domain recordings. Thanks for proving this kind of work can be done in a more ethical way.

piltdownman 36 minutes ago | parent

TBF it's by far the best corpus for this use case anyway.

Classical Music as an academic concern is very pedantic in the best way, and has the lions share of the research hours and funding compared to any other genre. As a result the best 1:1 approximations of multi-instrument Audio into multi-channel MIDI files is done simply by OCR of the established Gesamtausgabe in each case. Computer based scoring and playback has been solid since Finale/Sibelius/Score came on the market, and democratized at a wide consumer level somewhere between Amiga mod-trackers and Macintosh Quicktime 2.0.

JodieBenitez 48 minutes ago | parent

I need this in Bitwig !

mycall 46 minutes ago | parent

I'm still waiting for someone to convert beatboxing to midi in realtime. That would be a game changer for that art.

maxgraf96 45 minutes ago | parent

You should check out https://vochlea.com/ :)

chahmed312 38 minutes ago | parent

Is it fully open source?

pikann22 28 minutes ago | parent

crazy how a 20m param model actually beats the 1,3b muscriptor. awesome work!

jsphweid 25 minutes ago | parent

I've been out of the loop since Magenta's MT3 model. This feels a lot better and significantly smaller. Very impressive, dude. And thank you!