StaffPad “captures” the imagination — and your piano performance

Listen to the podcast episode

On the Scoring Notes podcast, David MacDonald and Philip Rothman talk with David William Hearn, the co-founder and lead designer of StaffPad, about everything from product updates to what it was like making a live demo for Apple in 40 minutes, along with the implications of AI, industry changes, the Muse Group acquisition, and much more. Listen now:

Scoring Notes

StaffPad updates go beyond just keeping the lights on

Play Episode Pause Episode

Mute/Unmute Episode Rewind 10 Seconds 1x Fast Forward 30 seconds

00:00 / 1:10:54

Amazon Apple Podcasts Google Podcasts Pandora PocketCasts Podchaser RSS Spotify Stitcher TuneIn

RSS Feed

Share

Link

Embed

StaffPad, the tablet-based scoring app for iPad and touch-based Windows devices, released an “Autumn 2023 Update” to coincide with the beginning of the season. Headlining this update is a new feature called Piano Capture, StaffPad’s term for audio recognition and transcription technology, which is designed to transcribe the audio directly from piano performances recorded into StaffPad, and, using the device’s microphones and on-device machine learning, create music notation in near real-time.

Alongside Piano Capture is a related feature called MIDI Capture, which will be more familiar to users of traditional software in that it records MIDI input from any USB- or Bluetooth-connected MIDI device and transcribes in into instrument staves in a StaffPad score.

In addition, this update introduces a new staff type — the Video Staff — which can be added to the score and manipulated using many of StaffPad’s familiar audio editing and effects tools.

Rounding out this update are the addition of the Muse Sounds to the StaffPad store (a free download); an automatic harmonizer, and numerous other improvements.

Piano Capture

First teased way back in April 2021 in coordination with the announcement of Apple’s new M1-powered iPads, and announced upon Muse Group’s acquisition of StaffPad a month later, polyphonic transcription of an acoustic instrument source such as a piano is known as Piano Capture in StaffPad. The lofty goal of translating sound into readable notated music is done with the help of an artificial intelligence engine, and spent more than two years in development from its announcement to eventual release.

Crucially, according to StaffPad’s co-founder David William Hearn, all of the processing is done on the device, so the audio remains private, and you don’t need to be online for it to work.

David told Scoring Notes that the confluence of the Apple silicon-powered iPad and the advancements in AI contributed to the creation of Piano Capture:

“There’s some really interesting opportunities that have come through research fields and in AI. We knew we were going to add audio; the question naturally came to us and said, well, wouldn’t it be cool if you could see this audio in notation, so if you did have a situation like you’re playing a piano, you’re recording it, can you skip the transcription step? It’s way quicker for me to play the piano than it is to write out the notes… That was the basic premise. We did a lot of research and we did a lot of proof-of-concepts. And then we had a model that was able to transcribe polyphonic audio, which is really quite hard to do, but it needed a lot of power.

Apple came to us and were working on a new chip — so it was the launch of the iPad Pro, but it was also the launch of Apple Silicon — which as we know is now ubiquitous in every device. But they said to us, ‘We really want to showcase what the Apple Silicon chip can do. But, it’s hard to to find apps that use it. Have you got anything that you’re doing that might be able to showcase this power?’ We said, ‘Well, yeah, we’re actually hitting this issue at the moment because we’ve got this pretty tricky machine learning problem.’ And they said, ‘Great, let’s try it on this new device.’

And it smashed it. It was just like, OK, this becomes now a viable thing to do on-device.”

Piano Capture, as the name implies, works only with piano, or a keyboard with a good acoustic piano sound, because it’s trained to recognize the specific audio profile of that instrument. David told Scoring Notes:

“It’s the one that just made sense from a composer’s perspective to focus on the piano. The way these models work is that you have to gather lots and lots of data. Part of the challenge is, is how you do that in an ethical way and in the right way. We ended up kind of building special systems that they could actually compose their own music — hours and hours and hours of music that made sense.

And then it would recognize — because it had written what it was now trying to recognize — it can compare the two and work out where it was going wrong and correct itself and everything. So that training loop is really complicated in itself. And that’s just a little peeling back of the onion, the level of complexity that goes into these things.

If we wanted to do a model that would recognize guitar, we would need to repeat the process to collect all the data required for guitar and customize the models for that; ditto with harp or flute or anything. It’s very specific to the sonic signature of the instrument.”

StaffPad generates a digital click which is ignored by the device when transcribing the audio, but it allows the player to perform without needing to wear headphones.

MIDI Capture

For all of those other instruments — or if you don’t have a audible piano readily available, but you do have a MIDI keyboard — you can use StaffPad’s MIDI Capture feature to get your musical thoughts into the app. If you have thought of StaffPad as “the handwriting app”, the introduction of Piano Capture and MIDI Capture bring it much more into the realm of fully-featured scoring software, where for many people playing into the score is a common, if not primary method of composing.

Any MIDI-capable device should work — Scoring Notes’ David MacDonald specifically asked David William Hearn if his EWI-playing student could use it, to which DWH replied, “It should — it’s pretty agnostic, so give it a go!” StaffPad particularly highlights drum kit input, where, according to StaffPad, “you can play the included drum kit sounds with a “GM” mapping style, and it will automatically assign the notes to the appropriate note-head styles, voice layers, etc.”

Video Staff

We’ve known for a while that video support was inevitable for StaffPad; indeed, scoring to picture has been touted as one of the app’s primary use cases. In the app’s July 2022 update, a “Time Staff” was introduced with support for a variety of frame rates, SMPTE timecode, tempo mapping, and time markers, for easy synchronization with hit points.

Now, it’s finally possible to add video directly to StaffPad via a Video Staff, which is found within the Utility category. Yes, the video is a staff; remember, the key is in the name of the app: StaffPad!

Once you’ve got your video in, thumbnails appear, similar to a video track in a DAW, and you can show or hide the video preview window; an iPad-only feature leverages the familiar pinch-to-zoom gesture to resize; Windows users will have to settle for the more desktop-link dragging of the right-corner.

You can adjust the audio of the video, including volume, pan, and reverb; even audio effects are possible.

Muse Sounds

Although StaffPad and MuseScore are very different products in many respects, one practical result of the Muse Group acquisition of StaffPad is the sharing of common technologies and resources. One such manifestation is the inclusion of the Muse Sounds in StaffPad — the same instrument library that is now standard with MuseScore.

First introduced with the MuseScore 4 release, Muse Sounds were actually recorded by David William Hearn over the past several years and, interestingly, originally intended as a paid product, before Muse Group decided to make them available for free. The entire symphonic collection is:

• Muse Brass
• Muse Choir
• Muse Harp
• Muse Keys
• Muse Percussion
• Muse Strings
• Muse Woodwinds

You do need to download them one at a time in StaffPad, but once installed, you can easily use StaffPad’s Swap feature to swap out any relevant instrument for a Muse instrument.

These are the same sounds as found in the Muse Hub, so if you are already using those sounds, you’ll be well-acquainted with them.

In our review of MuseScore 4, David MacDonald said, “These sampled instruments sound great, especially for the cost. For users who are used to working with thousands of dollars worth of sample libraries and massage MIDI data in a DAW, these samples may not be sufficient. However, for users like me who usually take whatever we can get with a minimal amount of fussing in a notation application, Muse Sounds represents a major leap forward for free. I think it’s particularly good in chamber settings where synthesized instruments often fall short.”

The rest

Among the other features in this StaffPad update is a vocoder-style harmonizer which analyzes the chord staff or harmonic content of the score and automatically adds harmony to the primary line. Created somewhat as a by-product of StaffPad’s existing abilities to pitch- and time-shift the music, David William Hearn told us, “Combine the two and you get a kind of a fun Jacob Collier-style vibe!”

• For the more serious-minded, here’s a run-through of other improvements:
• Better MIDI import engine
• Better responsiveness of the Welcome page on Windows
• Re-ordering of the Staff Category alphabetically, in response to user feedback
• Aforementioned new “Digital” metronome click, designed to be used with Piano Capture
• New Dynamic Mastering slider, replacing the earlier compression slider
• Other bug fixes

This StaffPad update is free for existing users and you’ll likely already have it if you have automatic updates enabled.

A new one-time purchase of StaffPad on iPad or Windows (must be purchased separately) is $90; in-app purchases are available. (The StaffPad Reader, the synchronized digital music stand designed to be used with StaffPad, is free on iPad and Windows; a $13 purchase unlocks “Pro” features for session leaders and directors.)