Making Music Interactive: Elaboration of the Feature Set in Wwise Part 3

Article by Louis-Xavier Buffoni

This Is The Third Part In The Series. If You Have Not Read The Second Part Start Here. The First Part Is Here.

Elements of a Connecting Approach

Description

The connecting approach consists of composing songs into small parts, and assembling or arranging these parts at run-time, according to game state. The end result should feel convincing, as if the song had been composed this way.

At the end of a song composed to accompany a combat sequence, we could compose two dramatically different segments depending on the outcome of the combat. These segments should have sufficient similarities with the main segment so that we can find ways to transition between them as if one was really building up into the other, as in standard recordings. The way they morph into one another is very specific to the song.

In order to provide tools for organizing this, we need the following:

- An object that will act as a building block;

- A way to associate pieces of music to game states;

- A way to define rules for transitioning from any block to any other block, similar to discrete game sync changes presented in the layering approach.

The Segment as a Building Block

We already have an object type that is very appropriate to be used as the building block of the connecting approach: the segment. Thanks to the pre-entry and post-exit, it can effectively be played next to another and sound good, when audio content is related. And it also has knowledge of meter information.

Music Switch Container

Now that we have building blocks, we need a way to organize them and assign them to game states. The Music Switch Container is the equivalent of the Actor-Mixer hierarchy's switch container, but adapted for interactive music. Like its counterpart, it selects the children that should be played according to the current state. Its children are music objects: segments, but also playlist containers, and even other switch containers. The main difference with the classic switch container is its ability to define music-related rules that are applied when states change.

Transition Matrix

The host may change a State Group's state to any other state, and therefore all children of a switch container must be able to modulate into any other. A matrix would thus be appropriate to represent rule assignations.

When a state change occurs, we know what segment is currently playing (we call it the "source"), and through the switch container's state-child association we determine the segment that we must transition to (which is called the "destination").

A rule is defined by a source, a destination, and a behavior. Source and destination may be any music object: if a playlist is specified as a source, then the rule will apply to any of its children. It is also important to understand that they are set beforehand; the rule is selected according to them, and not the other way around.

Figure now that we compose a song in C major, but with a section that modulates in D. We split this song in several segments and arrange them in a playlist, but one of them is in a different key. Now, this playlist is placed under a Switch container in order to be able to transition to another song. Chances are that the transition rule between the alien segment and that other song will be different than the rule used by all the other segments. Since switch containers may have any type of music object as children, including playlists, it must be possible to specify rules even for such exceptions.

Consequently, the size of the matrix would not be the number of direct children of the switch container, but the number of all segments inside its children. This number may be very large, but experience tells us that the same rule often applies for many segments in a given music design. The idea was to present the Transition Matrix as an exception list of rules, where the first rule is the most general and the last rule is the most specific (see figure below). For example, you specify a rule for the playlist mentioned above, and below you addan exception for the segment in D. If that segment was playing when State changed, that rule is selected instead of the general rule for the playlist.

Transition Rule Behaviors

Below the rules list (figure above), users define how the source and the destination segments should be synchronized. The sync point in the source is expressed in the form of a next boundary, where the boundary may be a beat, a bar, a cue. The sync point in the destination must be a cue. We also came up with "same time as playing segment".

Additionally, you may use cross-fading, and also decide if the post-exit and/or pre-entry regions should be played back.

I said earlier that rules do not force the destination. Well, if the destination that defines the rule is a playlist, there is no reason why we could not force the playlist item where it should start. This is the "Jump To Playlist" attribute, which applies only to Playlists.

Sometimes it is impossible to obtain a satisfying connection between two segments by simple rules like "stop here, start there". This is mostly true with segments that differ in music style, key, tempo or mood. In these cases, a piece of music (a segment) has to be composed specifically to connect them. In a transition behavior, you may also specify such a segment. It is called a "transition segment". Any segment in the interactive music hierarchy can act as a transition segment, in any transition rule.

Combining Both Approaches

The connecting approach blends very well with the paradigm of Wwise, with its hierarchical structure and overriding mechanisms. But its true power comes from the choice of the segment, originally intended as a container for layered tracks, as the building block of the connecting approach. It allows one to combine techniques from both approaches, and binds the whole feature set together.

Stingers

Another request of music designers is the ability to playback little motifs, or pieces of music, based on punctual events. They must be synchronized with the background music, and consistent with the musical context. Such motifs are commonly called Stingers (2).

The first thing to do is to define a new game sync, the Trigger. Music objects are registered to triggers, which are associated to stingers. When the game posts a trigger while a music object is playing, the appropriate stinger is played back.

Stingers could have been simple sounds, but it seemed more appropriate to use segments instead, for two reasons. First, because the segment naturally defines a point where it should be synchronized with music that is already playing back: the entry cue. Second, because they would inherit the benefits of the segment, like instrument-level variation.

Stingers depend on context. To control this, all music objects have a list of triggers to which they are registered. Each trigger specifies the segment that should play and the synchronization (next beat, bar, cue, and so on). Trigger-stinger associations are again based on hierarchy and overriding principles. Recall the playlist with the segment in D. A stinger could be associated to a trigger at the level of the playlist, but be overridden specifically for the alien segment. When it plays, another, more appropriate stinger is heard instead. What about muting the background music while a stinger is playing (this is sometimes referred to as a "breakdown stinger")? We realized that users could simply use ducking busses to obtain this effect. Whenever the stinger segment plays, the volume of music will go down. Et voilà.

CONTINUE READING ON PAGE 4 >>