Marie Chupeau and the human side of Newzik’s artificial intelligence
On the Scoring Notes podcast, David MacDonald and Philip Rothman talk with Marie Chupeau, the co-founder of Magic LEMP, about Maestria, the artificial intelligence technology that allows Newzik to not just display a score, but also to understand it musically, resulting in a LiveScore that can be played directly in Newzik or exported to other software for further use. Listen now:
When music reading on screens — rather than paper — first became viable, the first inclination of many users and app developers was to recreate the familiar experience of reading music from paper. There are immediate benefits and drawbacks to this approach, which I have written about previously on Scoring Notes. But as our devices have become more ubiquitous, more powerful, and more connected, developers have started to reimagine what a music reading experience might include in ways that are beyond the paper model.
Newzik is one of the score readers that has been at the forefront of this digital re-imagining by including network sync and collaboration features very early on, implementing the best MusicXML support in its category, and a functional web app, allowing cross-platform support with Windows, Mac, and even Chromebooks, in addition to iPad. Their latest addition, an AI-based optical music recognition (OMR) system called Maestria was first released in beta earlier this year, and is now ready to exit beta and enter day-to-day use.
At the fundamental level, Maestria is a technology that allows Newzik to not just display a score, but also to understand it musically. The resulting score is what Newzik calls a LiveScore, which includes the ability to view the score as with any other, and also the ability to play it back with MIDI sounds, mix levels of each instrument staff, and even export MusicXML and MIDI to edit in another application.
LiveScore conversion
Any PDF score can be instantly converted into a LiveScore: this includes any file you can import from a cloud service, the Files app, or even paper documents created using the iPad’s camera and Newzik’s built-in scanner feature. Long-time Newzik users who have already developed a library in the app will even be able to convert their existing documents. (It is worth noting here that scores purchased through Newzik’s publisher partners, such as Universal Edition, are not available for conversion.)
To create a LiveScore, simply tap the LiveScore button either on Newzik for Web or the iPad, and the process will begin. The conversion process happens on Newzik’s cloud servers, so you’re free to leave the web page or quit the app while this is happening. In my experience, this took anywhere from a few minutes to over an hour for longer scores. I would recommend starting the conversion and setting Newzik aside for a bit while the servers crunch the data.
There is a LiveScores shortcut on the app home screen and in the navigation menu on the web, so you can quickly see what has been converted and what is still in the queue.
LiveScore playback and features
Once the conversion is completed, opening the score will have all the same options and tools that it had before, plus a handful of others. First, and perhaps most notably, is that the score can now be played back. Newzik adds an attached MIDI file to the score. Previously, MIDI, audio, and video, could be added to a score, but it was a manual process and not as precisely synchronized to the score during playback. With LiveScores, Newzik will show a playback indicator similar to those seen in digital audio and notation software. If you have already added media to a score, you can control which are played back.
The details of the playback and controls differ somewhat between web and iPad. The two sets of settings that are common to both are around the instrument sound and the volume levels. In either platform, you’ll be able to solo or mute any instrument staff, as well as turn them up or down in the mix. I can imagine this being particularly useful for working with student or amateur musicians who might need help focusing on their part in practice.
You can also change the playback sound. While Newzik does a pretty good job of interpreting the names of instruments based on the labels at the first system, it occasionally falls back to its default of “Acoustic Grand Piano”. It’s easy to change the instrument from a pull-down list of General MIDI instruments, though these seem to be much more organized (even searchable!) on the app, and not organized on the web.
I also found that the order of MIDI tracks was sometimes different between the web and the app. On the web, the score order of my woodwind quintet—flute, oboe, clarinet, horn, bassoon—was reflected in the MIDI track order; on the app, the bassoon was listed at the top (as “part 4”?). Additionally, while it is easy to change the instrument, this change doesn’t sync between platforms. The same is true of tempo, which I found can vary between the two platforms as well.
The quality of the sound is practical for learning and proofreading, but it’s not going to win any audio awards. You’ll only have access to General MIDI instruments (even including my favorite synth name: “Goblins”). These are either synths provided by Newzik in the app, or the operating system’s synths on the web.
Transposing scores can often provide a challenge to OMR engines, and Maestria is no exception. If you are scanning a transposed score, you’ll need to manually set the transposition for each instrument. Unfortunately for iPad-only or iPad-first users, this feature is only available on the web. In the Audio & MIDI panel along the right, you can set any semitone adjustment. That’s perfectly simple for clarinet in B-flat (-2 semitones), or even horn in F (-7 semitones), but things might get a bit tricky for larger transpositions if you’re not quick in figuring out how many semitones are in a major thirteenth to make your baritone saxophone parts play back correctly (-21 semitones!).
Once you’ve completed the conversion, adjusted the instruments, and set and transpositions, playing back the score is pretty simple using the media player interface at the bottom of the score view. If you have any other media set up, you’ll need to mute it, otherwise you’ll hear both the MIDI that Maestria simultaneously with any separate audio or MIDI that you’ve uploaded. I wish this was easier for larger scores. There’s no way to mute all the tracks of a given MIDI file in the web version of Newzik; you can only toggle individual tracks (staves in a score).
If the media player is active and LiveScore view is toggled on, you can see a green highlight of the current measure, and a playback line following the specific notes that are sounding. Each of these can be toggled on and off independently if you find that annoying. One thing that I found frustrating was that it’s nearly impossible to play back a score from a specific point on the web, though this is a planned feature. Currently you can play back from a specific point in the score by long-pressing a measure in the app. Otherwise, can scrub through the player at the bottom of the staff.
Conversion results
Maestria’s OMR engine is … fine. It’s not the same quality that you would get from a dedicated OMR application. John Hinchey recently did an excellent round-up of some leading OMR products for Scoring Notes. Maestria does a very good job with simpler scores, but it doesn’t take much to trip it up either.
I tested scores with a variety of staves, durations, and levels of complexity. My best results were from a set of hymn settings that I use in my first-year music theory classes. My worst results were from a somewhat gnarly saxophone quartet I wrote in 2009. You can see some of both within the three movements Germaine Tailleferre’s String Quartet.
In short, simple textures are relatively reliable. I found that Maestria was occasionally tripped up by ties in the Tailleferre quartet, and occasionally whiffed on quarter-note triplets (presumably the lack of beam or bracket in this edition makes them tricky). Sometimes the triplets were ignored and the rhythm was adjusted elsewhere to fit within the measure. Other times, the triplet was rendered as a trill. And a few times it was interpreted correctly.
On the hymns, things went very well until two voices on the same staff had a slight rhythmic divergence—for example, a suspension in an inner voice. This often threw off the rhythm for the whole staff. Recognition was slightly better with a score that I exported directly from Dorico Pro 3.5 than with the same score printed and scanned using Newzik’s camera-scanning feature.
If you are accustomed to using OMR to scan a score into notation software for editing, these kinds of errors would clearly be unacceptable. But it’s worth considering that the goals for music recognition in Newzik might be a bit different, and that “good-enough” can be measured differently for each user and use-case. For example, even with all these errors, you can still listen to the first two movements of the Tailleferre quartet here and get a reasonable sense of what it “sounds like”. Maestria’s LiveScores might be accurate enough to convey some broad sound notions that could be useful to some users.
Lastly, there are some symbols that Maestria simply doesn’t recognize. Newzik, to its credit, provides an extensive list of all the score elements that are recognized by their engine. Many of the omissions from this list are quite understandable, such as nonstandard key signatures. But some are quite limiting. In both my saxophone quartet and the Tailleferre quartet, there are lots of 16-denominated meters (the last movement of Tailleferre is mostly in 6/16). These are admittedly less common that 2-, 4-, and 8-denominated meters, but you wouldn’t be able to represent Bach’s Well-Tempered Clavier without them. The same is true for double-whole/breve note values, which Maestria doesn’t recognize, even though it includes 4/2 and 6/2 time.
Round-tripping other software
If you want to take advantage of Maestria’s AI-based music recognition in another application, you can export either a MusicXML or a MIDI file from your LiveScore. From there, you can open your MusicXML file in any score editor or DAW and make changes.
On web version of Newzik, you’ll download them like any other files, but on the iPad, you can use the Share Sheet to send these files directly into another app. For example, you could send a project to StaffPad to take advantage of the high-end sample libraries, or you could send it to Dorico to create a beautiful new edition (or more likely, to correct recognition errors).
If you do make changes to the score in another application, you can bring MusicXML, PDF, MIDI, or audio back into Newzik (again, using the Share Sheet). A cool Newzik feature is that you can keep related files all together, attached to the same item in your library. For example, you might have a score, parts, MIDI, and audio of the same piece all in one spot. It’s worth noting that this sort of workflow is only possible on the iPad, as Newzik for Web doesn’t handle MusicXML files. You can’t even view MusicXML files you’ve previously uploaded in the app!
One downside to this kind of round-trip in the Newzik iPad app is that it doesn’t seem to be possible to “re-attach” a corrected MusicXML or MIDI file as a LiveScore, so you won’t get to see it highlight during playback. Newzik does generate a new MIDI file directly from an imported MusicXML, so if you have corrections to make, you can do them in just one place, and if you’re more comfortable in notation than in a MIDI sequencer, this might be a better option. MIDI files created by Maestria’s OMR have a LiveScore icon next to them, while imported files do not.
Pricing and conclusions (for now)
Regular Newzik users can convert up to 10 pages per month to LiveScores. For $29.99/year, Newzik Pro subscribers can convert up to 100 pages per month. I’m not sure this is a feature that is worth the subscription on its own, but for those who are already Pro users, I can definitely see this becoming part of some workflows.
Developing new apps and features to cover music creation and notation is an immense challenge. It involves making assumptions about what people will need and how and why. So in considering these apps, I often ask myself who is this for? What percentage of possible uses will this be able to cover? If you’re working with clean scans of modern editions of music that is within Maestria’s expectations, LiveScores could be a major improvement for you, especially if you regularly work with student or amateur musicians who may not all read music notation at the same level. However, I don’t think this technology is ready for me just yet.
In spite of that, one benefit to the running an AI application in the cloud is that the team at Newzik can continue to rapidly iterate on the Maestria OMR engine, since it doesn’t change the application on your iPad to do so. And the nature of AI means that the more it gets used, the more it can learn from its mistakes, and the better it gets. Maestria learns based on the feedback users give. Though I imagine most users won’t spend time on this, Newzik tries to make it as easy as possibly by prominently placing rating and feedback links in the score viewer. And Newzik’s Aurélia Azoulay told me that one of their next priorities will be to address some of divergences I’ve pointed out between the web and app versions, so that they are more functionally aligned.
Maestria is the result of years of research and development from the Newzik team, and it’s incredible to have a score reader app that promises to actually read scores. Having said that, if you’re not currently a Newzik user and are interested in an iPad app that can interpret scores, I think you should keep looking. Newzik Maestria and LiveScores are ambitious and promising, but for now we’ll have to wait a just bit longer.
Heinz Roggenkemper
It is great to get a detailed account of new OMR system!
The Hymn Tunes really shows how well an OMR system handles voices. It may look simpler than it is.
I believe that the Tailleferre string quartett would be challenging for any OMR system (the staff lines do not look clean, and that usually causes problems). Do you know an OMR system that handles this score well?
The statement that Maestria’s LiveScore enables Newszik ‘to not just display a score, but also to understand it musically’ was slightly surprising. What does it do that goes substantially beyond what SharpEye was doing 20 years ago?
The most intriguing feature of of LiveScore could be its ability to learn from feedback. Have you seen evidence of this already, and if not, do you plan to follow up on that at a later point in time?
David MacDonald
Good question. None of them gets it totally right, and they all get a slightly different subset of things right. The big concern for me was that Newzik didn’t even try to read 6/16 (there is a list of supported meters, and this isn’t included). The result is that it really depends on which elements you prioritize. For the cost of Newzik though, I’d start there.
I can’t speak to the improvements over time yet. At the moment, the feedback mechanism built into the app is pretty rudimentary, so it will probably be a little while longer.