Available today exclusively from the Windows Store is StaffPad, a truly impressive new music handwriting app for pen-and-touch based Windows 8 tablets like the Surface Pro. Composer, orchestrator and programmer David William Hearn is the creator of StaffPad, and he and I spoke several weeks ago in advance of its release.
Many of his comments appear in the extensive review of StaffPad on this blog, but I thought readers would be interested in more of his remarks, particularly regarding why he was inspired to create the app. Our interview follows.
Q: Tell us why you decided to create StaffPad.
I didn’t go to a music college. When I was at school, they made us write out Bach chorales on pen and paper. The students who didn’t play piano didn’t know if they were writing anything that made any sense, and it was something that everybody ground to a halt on. It seemed much harder than it really was, maybe because the explanations weren’t quite there, but also because it seemed irrelevant. What I mean is, what’s on paper doesn’t really mean anything until it’s played by a human being or interpreted in some way.
That was the moment where I started thinking about the art of music notation, which is ancient, and how it’s almost irrelevant in today’s technological world. But then when I started working in London, I was working on pop records. Everything was on the computer. I worked in sequencers and had to buy Sibelius because I was doing string arrangements for pop records. But my focus was on sequencers, and it seemed that the two lived in very different worlds. You were expected to write into a sequencer and then lay it out and finish the job in Sibelius, if it was ever going to be played by real musicians.
That killed productivity and inspiration for me. As much as I contribute to it with Cinematic Strings, I really dislike the act of getting up in the morning and switching on the computer, opening up Cubase, having some iLok error to deal with, creating MIDI tracks, loading in samples, buying new sounds… it’s not a musical process, it’s a technological process.
As I moved out of the pop world and into the film world, I was really lucky to be mentored by Nicholas Dodd. He’s probably one of the top orchestrators in the world. That’s where I really learned to write for orchestra.
Nick worked on pen and paper, and he’s incredibly prolific — it’s amazing. He gets up at 5 am, and by the time I stir at 9 o’clock, he’s already written a film cue that’s six minutes long. It’s because he thinks about nothing but the music — about the dots, structure, melody, harmony, voices and form. That’s what he concerns himself with. He doesn’t concern himself with legato transitions, bumpy mod wheel data, or anything like that.
The actual moment of ‘let’s do this’ came whilst I was sat in Air Studios with Nicholas Dodd. There’d been some crisis on the film session we were at, and whilst we were having lunch chatting about how we could solve those kinds of issues, I floated the idea that I’d been having about making a notation app that’s designed around pen. Nicholas lit up in a way that you rarely see, and flooded the room with enthusiasm. That was the moment that I realised that, if it was done right, the app could be useful to true professionals – but also be simple enough for the casual musician, teacher, student to enjoy.
As I got working with more orchestrators at the top of their game, working on pen and paper, that really struck me as the difference between firing up Cubase and writing a piece of music. That was the crux of where I thought, let’s see if we can re-boot that approach and not just try to replicate pen and paper on a screen, but try to evolve it and bring it into the next generation. So, you could, conceivably, not be fluent with notation but still feel really comfortable with it. Notation is not all that different from MIDI, really — they’re both grid-based systems, just different ways of expressing the data. I think notation is superior, but it’s being lost because of workflow and ease of use.
The idea, in the end, is to get it to a point where it forms a part of a digital music stand system. I’m investigating cloud playback so that, when you press play, you can really get those fancy sample libraries. Again, not giving it to the user to worry about, but you can’t do that on a tablet device. You’d have to do that in the cloud and stream it down.
So this is really only step one of trying to reboot the composing experience.
Q: What do you mean by “reboot the composing experience?”
I do a lot of MIDI programming for digital orchestras, so I understand that world quite clearly. I was doing a lecture at BFI with Nick about how I demo his scores. Towards the end of the lecture, I played one of my own compositions that I’ve recorded with a live orchestra. Afterwards, someone came up to me, all enthusiastic, and was like, ‘Oh, that piece at the end! What string library was that?’ Arrgh! I thought he might say, ‘I liked it!’ Can you imagine, a room full of 150 composers listening to the music, and all they’re thinking is, ‘how do you program that?’ And so the balance has gotten a bit out of hand, I think.
The tragedy is, if Nick wasn’t already successful — if he were starting today with the skill set that he has — he would not be able to find much work, because he doesn’t have the technological skills. While today’s composers need to know the technology, first and foremost they need to know about the music. If we can make technology good enough to allow composers to compose, then we’ve come a good way toward achieving the goal of this product.
I hope that, in some way, I can make notation seem a bit more relevant for younger people. I often get messages from young composers asking what kind of gear they should buy, and I think, shouldn’t you be more concerned with being a good composer and not buying the right kind of gear?
Q: Well, forgive me, but we need to talk gear for a moment. What kind of gear does StaffPad run on?
StaffPad is designed for the Surface Pro 3, and, crucially, it comes with an active digitizer: a real pen. It’s not a capacitive stylus, and it has really good palm rejection.
We have support from Microsoft. Obviously the were surprised that someone spent three years making this app for Windows 8 — it’s all because of the pen. The best experience on the app is had on any of the Surface Pro 3s. It will work on the earlier Surface Pros, but the Surface Pro 3 has the bigger screen and better experience.
One of the reasons I didn’t make StaffPad for the iPad in the end, was, although it makes business sense since everyone has an iPad, the design concessions were just too great. For example, you’d have to have a drawing mode, a navigation mode, or split the screen, which means that you wouldn’t have much room for your music; you don’t get the palm rejection. It makes the market much smaller, but it’s the only way I could design it how I really saw it to be.
Q: Okay, so how did the StaffPad project begin?
I didn’t come from a musical family; I came from a tech family. My brother is a senior engineer for Google and my father worked for a technology company that consulted for television stations. All the computers we had were shared, so I never got much time on them. But it meant that I picked up some of this stuff as I was growing up, and it seemed quite natural to me. I know C#, which is the language that StaffPad is written in, and C++, which is what the sampler and reverb is written in, for high-performance reasons.
At first, I did all of the design and built what looked like a functioning app, but without the code behind it to make all the features necessary. Basically, it was an elaborate wireframe, or prototype. When I started looking around for freelance developers that I could hire to help me build this vision, I thought, well, I should first look to see if anyone’s done it already. At that time, nobody had, at all.
But there was a page on Google Code that Matthew Tesch had started. It was basically an outline of something he wanted to do, but hadn’t started yet. We had a few differences in the way we thought it should be approached. For instance, I didn’t think you should do either constant recognition — which is what NotateMe tries to do — or recognition after you’ve written your whole score. I thought the recognition should happen one bar at a time; that will keep it fast and predictable. If it gets something wrong, you don’t mind correcting one bar at a time, but you don’t want to go through a whole score and try and fix all of the mistakes.
It’s the way we write, naturally. We don’t do what Sibelius does, where it fills in rests. So I can write that don’t really make sense in the time signature. That’s actually quite good; it means if I want to change something I wrote later, I can continue to make changes. It’s really intuitive.
Q: Tell us about working with Matt.
Matt’s been brilliant. He’s the coding genius that makes it all happen. The recognition is all Matt’s work. He works for Carnegie Mellon University programming search and rescue robots! We’ve been working on this together for the past three years.
Q: How did development progress?
I had to do a lot of work making sure that the app doesn’t slow down. One of my biggest bugbears with Sibelius and Finale is that, on larger scores, scrolling and zooming suffers. So it occurred to me that we could use technology that Google Maps uses, where the map is divided into tiles, and you render the tiles that you need. As a result, we have scores that can be any size, and the scrolling and zooming will stay 60 frames per second.
I was also convinced that OCR was not the way to make the app. Although that gives you some benefit with things like correctional strokes, it’s more efficient and accurate to take the position and temporal information from the pen, and then use musical context to decide what the music is trying to be. That way, you can do things that would totally confuse OCR. Because we know the order of the strokes and where they are in relation to the notes, we can say, OK, that’s a natural, that’s a sharp. It also means we can take training data from other people and build on it.
The key difference between this and Sibelius is that it doesn’t try and auto-complete things for you. If you change the time signature, it won’t re-flow the music. That’s not to say we won’t have that eventually. But when you’re working on paper, you may be writing along, and then, in your head, you may have done a time signature change, and you just want to erase a note to fix it. That’s really easy to do in StaffPad. The assumption is that, musically, the user knows what he or she is doing.
Q: Speaking of the user, how does the app work, from the user experience?
The idea from day one was not to try and compete with Sibelius and Finale, which are amazing at what they do. We wanted to do something different, to try and make it not just for composers, orchestrators and arrangers, but also just for musicians. I actually think that’s one of the least-represented demographics: your cello teacher, your flute teacher, musicians in general, who don’t have the time to learn, or the money to buy high-end packages like Sibelius or Finale. But they might just want to scribble something down during their lessons. If you know how to write music, you’re already using the app; there’s no learning curve. There are a few things which are discoverable in the app; once you get to be more of a power user you’ll find these features, and, hopefully, love them.
As we went through, I initially thought we were going to keep things very simple and just say that this is a very basic front end for Sibelius or Finale. But as you can see, the output is pretty good on its own. It does a lot of collision avoidance; you can print out individual parts and it will do some layout for you; you can change the paper size. I wouldn’t necessarily put this output down in front of the LSO, but for the majority of cases, it should be just fine.
The recognition is pretty robust, but if StaffPad doesn’t recognize something in the bar, the bar will turn orange, which means it’s a pending bar. It will ask you what each unrecognized stroke is. You can report it, ignore it, or tell it what it is, and the app will learn your style. Train it well! This, conceptually, is key. Sometimes you might be writing graphic notation, and not have StaffPad try and apply recognition to it. This is really the first time you can write completely what you want and be experimental. That’s where this could really get handy: when you’re not trying to write conventional notation. It will still print; it will still do the parts extraction. It’s all being synced up to the cloud; it’s saving versions of your score all the time.
All of the individual elements are erasable. We can go in and erase the individual accidentals. If you erase the beams, it will turn into flags; you can even erase the flags and turn them into quarter notes. If you want to erase the entire thing, just erase the notehead. We tried to build in as much intuitiveness into each part of the process.
Q: What about playback, printing, and sharing withe other people or other software, like Finale, Sibelius, and the upcoming Steinberg app?
I’ve programmed an entire library straight into the app, so you have the full orchestra as well as a few other things. It’s quite detailed with multiple dynamic layers and round-robins. I have a lot of sampling experience from Cinematic Strings. The Expression layer is basically a MIDI CC-1 overlay.
I’ve recorded a lot of sounds, licensed from Sonic Implants, or SONiVOX, as they’re now known. So these are Boston Pops samples, completely re-programmed from scratch to fit nicely with notation. The sampler is built from scratch, as well, with its own playback rules. It is, however, tablet-based, so it’s meant to be very functional, so you don’t hit play and expect to hear back a real orchestra.
Our internal format is based on MusicXML with extensions, so it exports really clean XML. For me, StaffPad was designed from day one to be a front end, to start in this and finish in Sibelius.
I’ll be first in line to buy the Steinberg product when it’s ready. I have so much respect for those guys and what they’ve done in the past. I know what they do in the future will be off the charts. But their focus is, unashamedly, smart auto-layout with huge amounts of pro-level tweaking power. Our focus is the composer, the musician, who doesn’t even want to see a menu and just wants to get scribbling.
Q: Are you using StaffPad to write your own music?
I’m using StaffPad to write my own music now. It’s quite good fun to see it come together!