The story of Sibelius 6: part 4

Opinion

This is the fourth and final post in a series about the development of Sibelius 6. The previous parts are here, here and here.

In the previous posts I’ve explained how we went about tackling two of our threemajor goals for Sibelius 6. In this final part I want to talk about our third, and perhaps most important, goal:

Add some innovative new features that will appeal to users buying notation software for the first time, existing Sibelius users, and existing users of competing products alike.

One of the things that has really set Sibelius apart over the years is how we have attempted to break new ground with each new version, and Sibelius 6 is no exception. More after the jump.

I’ve already discussed several of the innovative new features in Sibelius 6 in previous posts in this series. In particular, Classroom Control and Versions are brand new features for educators that make Sibelius more powerful and more useful as a teaching and learning tool.

But the most obviously innovative feature in Sibelius 6 is, of course, Magnetic Layout.

Why collision avoidance?

There are centuries-old conventions for the placement of musical symbols in printed sheet music, all of which are designed to ensure that the music is presented as clearly and unambiguously as possible to performers. The general principle is that no two symbols should be allowed to collide with each other (though of course there are exceptions: musical notes naturally collide with the staff on which they are positioned, and symbols such as clefs, key signatures, time signatures and so on also naturally lie on the staff and therefore collide with it).

This general principle gives rise to hundreds of specific conventions concerning the placement of particular kinds of symbols: for example, lyric text should be positioned below the staff in a row across the width of the system, as close as possible vertically to the notes to which they belong; text dynamics and hairpin lines should be aligned at the same vertical position relative to the staff above, below any low notes, and grouped such that if they are within a certain horizontal distance, they should be aligned together, so that the musician’s eye can take them in at a single glance.

Other objects should be aligned down the score across multiple staves: for example, if several instruments all have a change of dynamic at the same rhythmic position, those dynamics should ideally be positioned at the same horizontal position on all staves, so that the conductor can take them all in at a single glance.

Music notation software in general goes some way towards following these conventions automatically, by providing sensible default positions for different types of object. This allows the user to create a score and add all the required symbols relatively quickly. However, 50% or more of the time spent creating a score involves the painstaking adjustment of individual objects in order to make the score as clear, pleasing to the eye and unambiguous as possible.

Existing solutions

Automatic collision avoidance has been attempted before in music notation software, but it’s never been attempted so comprehensively and ambitiously as in Sibelius 6.

For example, Finale includes two specific collision avoidance features: Engraver Slurs avoid collisions with notes at their end points and with notes, some articulations, and some accidentals underneath their arc; and note-attached Expressions, which include things like text dynamics, can move up and down to maintain a minimum distance from the note to which they are attached as the pitch of the note is raised or lowered, such that if the note is positioned on leger lines above or below the staff, the Expression will not collide with the note. However, no consideration is given to avoiding collisions between the Expression and objects of other kinds, so it is common for Expressions below the staff to collide with e.g. hairpin lines, rows of lyric text, and so on.

Default Expression placement relative to lyrics in Finale 2010
Default Expression placement relative to lyrics in Finale 2010

Notion includes simplistic collision avoidance for text dynamics and hairpin lines. When the user creates a dynamic, an extra “gutter” is created below the lowest note on the staff, and all dynamics are then positioned in this gutter, aligned in a row across the width of the system. No consideration is given to preventing horizontal collisions between text dynamics and hairpin lines. The result of creating this extra “gutter” is that dynamics can often appear much further away vertically from the note with which they are associated than they ideally would, because no dynamic can appear closer to a note than the pitch of the lowest note on the entire system. This creates a very “loose” vertical spacing that is very unlike professionally engraved sheet music.

The late-lamented Igor Engraver included both Finale-like slurs that could avoid collisions with notes under their arc, and a Notion-like “gutter” for text dynamics and hairpins. However, Igor’s approach to dynamics is inferior even to Notion’s, with the result that the gutter is always positioned a fixed distance below the staff; as soon as a note is sufficiently low that it collides with the gutter, the note collides with the dynamics. One nicety of Igor’s approach is that placing a text dynamic in the middle of a hairpin line splits the hairpin line into two separate hairpins, though this is inferior to the manner in which Magnetic Layout in Sibelius 6 truncates a hairpin by clipping or “whiting out” the middle of the hairpin line.

So when we were looking at how best to solve the problem of collision avoidance in Sibelius 6, we knew we wanted to clear the bar set by existing solutions very considerably.

How Magnetic Layout works

Our approach in designing Magnetic Layout to solve the collision avoidance problem was to codify the natural order of precedence for the placement of musical symbols relative to the staff into a rules-based system that defines an overall order of priority (such that higher-priority objects force lower-priority objects to move out of their way), planes of movement (e.g. up/down or left/right, or both), ranges of movement (i.e. how far in the allowed planes of movement the object is allowed to move), and whether objects should be grouped together with other identical or similar objects, horizontally or vertically, or (in some cases) both.

With knowledge of these rules, Magnetic Layout calculates the bounding boxes around every musical symbol, and then examining where those bounding boxes abut or overlap to determine whether a collision is allowed, and if not, which objects should be moved in order to resolve the collision, according to the set of rules defined. It operates on a whole system of music at once, i.e. examining all of the objects attached to all of the staves in a system, and resolves collisions by dynamically redrawing the objects at new positions in real time.

Diagnostic drawing options showing how objects in the score look to Magnetic Layout
Diagnostics showing the Magnetic Layout bounding boxes for objects in the score

It follows the principle that objects that should naturally lie closest to the staff are given the highest priorities, and those types of objects that can lie further away from the staff are then given progressively lower priorities. Objects of a higher priority force objects of a lower priority to move out of their way.

The planes of movement are determined for each type of object: very few objects are permitted to move left (because this would often mean that the object would appear to be associated with a note prior to the one with which it is actually associated), but many are allowed to move up and down. The ranges of movement are likewise typically limited, to prevent an object from being moved far enough that it appears to be associated with another note, or possibly even with a different staff.

Objects can also be part of a primary (typically horizontal, i.e. aligned across some of or all of the width of a staff) and a secondary (typically vertical, i.e. aligned at the same rhythmic position down some or all of the staves in a system) group. When an object is considered part of a group, if it moves, all other objects in the group must move together.

Unlike other programs, Magnetic Layout works with the available space rather than allowing staves to be moved further apart. This provides an end result that is most similar to professionally printed music produced by expert human engravers, who try to use the existing space most efficiently and keep everything as tight as possible. Occasionally this means that the algorithm is unable to resolve all collisions satisfactorily, so in such situations the colliding objects are coloured red in the score so that you may take action to resolve the collision yourself: typically all you have to do is choose Layout > Optimize Staff Spacing, which calculates the ideal distance between staves in order to allow all objects to appear at their default positions, and moves the staves apart towards those positions to resolve any remaining collisions.

The end result is that Magnetic Layout can reduce the total time it takes to produce a completed score by up to 50%. (And that’s not marketing bluster: we asked our beta testers to quantify the time saving, and nearly all of them told us that it cut their score preparation time in half!)

Keeping things snappy

One of the things that makes Magnetic Layout so slick is that it calculates everything in real time. But this presents some challenges, chief among them the inescapable fact that the more computation Sibelius has to do in order to draw the score, the slower it gets.

We were able to claw back some speed of operation by caching the results of Magnetic Layout page by page. So scrolling around the score is actually faster in Sibelius 6 than in previous versions of Sibelius: once Sibelius has drawn each page once, it doesn’t need to recalculate the positions of any objects until you do some editing. The cost of this is slightly higher memory usage so that the drawing instructions for each page can be kept in memory rather than being recalculated on every draw, but for today’s computers this overhead is pretty negligible.

Our developers also put a lot of time into speeding up other editing operations, so that even if Magnetic Layout caused the overall speed of editing to be reduced, we had already made a number of gains in performance that would offset the slowdown. As a result, operations like transposition, note spacing, inputting or changing the duration of notes at the end of the score, and so on, are considerably faster in Sibelius 6 than in previous versions: turn off Layout > Magnetic Layout to see this for yourself.

And, of course, our developers spent considerable time looking closely at the performance characteristics of drawing and editing the score using dedicated profiling tools, looking for performance hotspots that could be cleaned up. Even though the performance gains for each individual optimization may be modest, being able to take 5% off the speed of an operation, then another 5%, then another 5%, soon adds up.

The result is that even with all the extra work that Sibelius 6 is doing to provide real-time automatic collision avoidance, the program feels snappy and responsive, which is no mean feat!

But nobody’s perfect

As always, we didn’t get to do absolutely everything we wanted to do. When developing a complex suite of functionality like Magnetic Layout, you have to make trade-offs in order to get things finished and out of the door. A couple of notable omissions in Magnetic Layout in Sibelius 6 are articulations, and slurs.

A common complaint about Sibelius is that articulations like accents and tenutos on the first of a pair of tied notes can collide with the tie. We had hoped that we would be able to make articulations use Magnetic Layout, but we had to descope this functionality from the plan due to a lack of time.

More frustrating was that, despite all the many improvements to slurs in Sibelius 6 (discussed in part 3 of this series), we were not able to manage automatic collision avoidance for objects under slurs in this release. It became clear several months before release that in order to get the rest of the feature up to scratch in terms of quality and performance, we’d have to scale back our ambitions somewhat.

Nevertheless, we’re absolutely delighted with how Magnetic Layout turned out. We really do feel that it is the most revolutionary feature added to notation software since we added dynamic parts in Sibelius 4. Fortunately, the thousands of Sibelius users who have upgraded to Sibelius 6 already agree with us!

Leave a Comment

Your email address will not be published. Required fields are marked *