A look at using NotateMe with built-in PhotoScore technology

More than 20 years ago I had completed writing out the score and parts to an orchestra score by hand. Afterwards, I recall thinking that there had to be a “word processor for music” to make the process easier. My journey eventually led to my local music store in Buffalo, where I handed the store’s employee two crisp $100 bills. In return, I received two dimes and a box containing the academic version of Finale 2.0 (I was a student at the time).

Using Finale at that time might best be described as two parts elation and one part frustration. To enter music into the computer and not only see it, but hear it played back truly seemed miraculous. Yet wrangling the score into an acceptable state required patience, persistence, and frequent consultations of the multi-volume reference guide, not to mention frequent calls to the friendly customer representatives at Coda Music Technology (I can still recall my product serial number from memory).

Still, I was hooked, and never looked back. In time, Finale improved, Sibelius came on the scene, and, for now at least, the two programs dominate the desktop notation market. Other products have, or have had a loyal following: Notion was the first to tout the quality of its playback samples as a selling point; users of Score or LilyPond believe, with some justification, that those products produce superior notated output; MuseScore has the advantages of being free and open-source; Noteflight is the dominant web-based product and is wholly owned by Hal Leonard Corporation. There have been others, like Igor Engraver (one of the first with a linked parts feature), Encore (still supported), Graphire Music Press, Mosaic, and a few more.

Some people care only about the visual aspect of music notation software, but many others use the software primarily for its instant aural feedback and playback features. For most users, seeing and hearing one’s music are intertwined. In order to keep atop the market, Finale and Sibelius have needed to produce results with that in mind.

The reason for this long preamble, with apologies, is twofold: 1. There are parallels between the early days of desktop notation software and these current early days of tablet notation apps, and; 2. Beyond sight and sound, there’s a third sense involved when working with music, which we’ve been missing for a while: touch.

“You can touch your music”

During the 2007 unveiling of the original iPhone, Steve Jobs said, “You can just touch your music — it’s so cool.” On its face, this was nothing new; after all, for centuries people had been touching their music, whether it meant bringing a dog-eared book of Bach chorales to rehearsal, thumbing through a stack of well-worn LPs, or practicing scales repeatedly on a badly tuned upright piano. And of course, while writing one’s music out by hand can surely be laborious, it can also be a uniquely personal and satisfying process.

In the digital age, though, the tactile aspect to creating music was disappearing. For those creating notated music on a computer, the keyboard, mouse and screen placed both a physical and, perhaps, an emotional barrier between the user and the music. For listeners, the CD and the early days of the iPods exuded much in the way of wonder, but little physical understanding of how the digital bits of 0s and 1s got translated into sound.

Jobs wasn’t referring to notation in his now-famous presentation, but he had identified a fundamental connection among seeing, hearing, and touching music.

It was this connection that a group called ThinkMusic Technology tapped into in January 2013. In demoing a tablet music handwriting app that soon proved to be vaporware, they had nevertheless served an important function: publicly demonstrating the interest and excitement that such a product would generate.

Martin Dawe, the CEO of Neuratron, told me, “we in fact began work on NotateMe way before the ThinkMusic announcement. It was Microsoft’s release of Windows 8 to developers in summer 2012 (and announcement of the Surface tablet) that made me realize that touchscreen devices powerful enough to perform complex tasks such as handwritten recognition were just around the corner. I began work developing a proof-of-concept on Windows almost immediately, using our existing technology.”

In a way, though, ThinkMusic performed valuable market research, by validating that there was great interest in such a product — if it could come to market. They also eventually experienced what developers like Martin have known for a long time: it’s hard to develop music software.

Elation

We’re all guilty of grumbling about why or how our favorite program doesn’t work the way we think it should. But even simple music is devilishly complicated when you break it down into its component elements.

I kept that in mind as I took another look at NotateMe, the music handwriting app for Apple’s iOS and Android tablets and smartphones. First released in the fall of 2013, the app is not yet one year old, but like Finale and Sibelius in the heyday of the desktop era, it may very well be the start of a new paradigm in digital music notation. The app has been on a rapid development pace, with many features added in May of this year for NotateMe 2.0; the current version is 2.6.

With NotateMe, you write out your music on a virtual staff that appears on your device’s screen. The app does its best to instantly convert your scrawls to legible music notation, and it intelligently adapts its interpretation over time to learn your personal handwriting style and produce more accurate results.

The “kicker,” though, as Guy Fieri might say, is the new PhotoScore add-in, first made available about a month ago. This makes use of the camera on the phone or tablet, taking a snapshot of sheet music and converting it into readable music for use within the app.

Neuratron is no stranger to this concept, of course. They make the PhotoScore desktop scanning software, a lite version of which is bundled with Sibelius. Using a fixed scanner and desktop apps is one thing; doing it with a device held in the palm of your hand is another entirely.

My first test was a couple of pages from a beautiful 1960 edition of George Gershwin’s songs. In 1960, of course, the mere concept of taking a photo of anything and instantly seeing it was itself a novelty.

Using the NotateMe version of PhotoScore is simple. You select PhotoScore from the app’s dropdown menu and take a photo. Neuratron recommends taking pictures by a window in daylight if possible; which I did, although that’s not always practical. Processing each page takes between 15 and 30 seconds.

I was curious to see how NotateMe performed “out of the box” without any user intervention other than processing the score as a NotateMe document. While some of the notational elements definitely need improvement, the playback result is gloriously George (up-tempo, to be sure, but I could have easily changed the metronome marking):

This is unquestionably the “elation” part of the equation. Can you imagine, when this book was published in 1960 — or even just a few years ago — taking a picture of sheet music and having your camera instantly play the music back to you? It’s otherworldly, and a remarkable achievement.

(I should add, for the video above, I transferred the unmodified file to my iPad using NotateMe’s built-in Dropbox integration, so that the music could be seen more easily.)

In fact, Neuratron reports that the most common use of the new PhotoScore feature so far is helping people practice; notation details and text recognition are less important to these users, as they just want to hear the music played back. Unsurprisingly, there is a great deal of interest in the educational market as well.

Neuratron also says that text recognition is a significant difference between the NotateMe version and the desktop version of PhotoScore. The desktop product licenses an industry-standard OCR engine and which also requires 300 dpi, and so they recommend the desktop product for professional use. I didn’t find NotateMe too bad, though, with text; the main wrinkle was squashing the second and third verses together (the company says improved handling of more than two verses is on their to-do list).