Creating truly interactive music can be rather hard. One of the reasons is the nature of music. Music is rather complex, and for a player to feel that they are participating in the creation of music, often, musical qualities have to be compromised. Nevertheless, it’s interesting to explore, and below are a few examples of concepts I worked on for a recent AR book & game by Step-in-books called MUR.

 

About MUR

The starting point for MUR was a children's book about a bear that can’t sleep. We already had this clear idea that music should play a bigger role in MUR than it did in previous games made by Step-in-books. How we were going to achieve that was yet to be determined, but I knew that I wanted to use Wwise to ease the process and integration of sound and music. Apart from that, the game would be an AR experience for tablets and would use the Vuforia software.  

Here’s a short trailer showcasing the MUR experience:    

  

During the first couple of weeks of development, we created several mini-games and puzzles which were mostly music-based, yet out of context from the MUR universe. I sat down with in-house game designers and programmers, and we brainstormed concepts and ideas as well as how they could be integrated in Wwise. We ended up with around 10 mini-games and puzzles, selected those that we thought would work best, and began putting them into context with the MUR universe. 

Below I’ll share some of these concepts and the role of music and sound within.

  

Ambient Music

The first full level you enter is an introduction to the MUR universe. You can touch different elements and learn how to control the game, but there are no actual puzzles. I wanted the music to reflect that this level was about exploring the universe, so I decided that music here should be more ambient - almost sound design in nature - and non-repetitive, and not move in any particular direction. 

Out of the main theme, I created 4 different A-parts, 4 different B-parts, and a C part. Because I wanted the phrases to trigger with random delay times, I didn't use the music structuring in Wwise, but instead placed them as SFX sounds and used Sequence Containers and Random Containers.

This in a way creates infinite music, and random delay times would obviously work best with ambient music that is more rubato (without rhythm). This concept would have also been possible to implement with more layers at the same time, and we could technically change parameters on the fly to the intensity of the music when needed. But for this game, I kept it simple.

Developing Music

For the second level, we wanted the music to evolve as the player progresses through 4 stages. I didn't want it to just sound like a combination of musical layers, but rather music with transpositions that would really evolve.  So I composed 4 pieces of music in different yet related keys, that would be looped, and via transition parts, go to the next section. The objective was to give the player the sense of prompting the music to evolve as they click through the level. It had to feel snappy enough to be in line with the players’ movement within the game, so I created extra custom trigger points for the transition pieces.

 

Another element we added was to give every object/animal a sound effect and musical voice. For example, touching a squirrel sitting in a tree would make it climb up the tree while a major scale would play ascending on violin. When the squirrel would slide down the tree, the scale would play descending. Since some of the environment sounds also followed transpositions, they are easily controlled by following the same triggers as the music would follow.

Interactive Music

For the third level, we wanted the player to feel that they are shaping the music while solving different puzzles. The second puzzle in this level consists of putting 6 fireflies into their correct 6 alcoves. So I created 6 strong music layers for each fly. The layers would start playing, synchronized with the volume at 0 db, and when you place a firefly in its alcove, the volume gets turned up. But, of course, this would only happen as a confirmation of correct gameplay when the firefly is placed in the correct alcove. So, to make it more interactive, I created several layers for each instrument for when the firefly is placed in 'wrong' alcoves. I decided to use pitch shift that would modify depending on which 'wrong' alcove a firefly is placed in, to make it more interesting.

The result was that when a layer would play a wrong pitch and tempo, when the firefly was put in the correct alcove, the layer would play in sync with the other layers. The great thing about this was that it also reduced memory, since only one sound file would be needed. I did, however, have to run a stream of the sound file that was 'wrong' and could be manipulated, as well as the 'correct' layer when in sync with the other correct layers. To emphasize when a music layer would be correct, I ended up creating a couple of extra layers that also got turned up in volume when a firefly was put in the right alcove.

We played around with some other cool concepts, like having Unity receive the tempo from Wwise so the player has to touch objects specifically in relation to the beat/tempo. But, due to the age range for this experience, several puzzles had to be simplified or didn’t make it to the game.

 

Mixing for Tablets

The game will be played on a tablet and most likely be played without headphones, so getting MUR to sound good (and loud) on a small set of speakers was very important.

This also limited the experience of 3D sound. And because some of our objects needed very specific volumes, depending on where the player looks and the distance of the object, I used 1 RTPC bind to the built-in parameter Azimuth, instead of 3D sound on several environmental sounds.

For example, the bird whistling in the second level has volume changes, a low-pass filter, and intensity in repetition, depending on the angle of the camera:

Mixing through the tablet's speaker (with Remote Connection), while tweaking RTPCs and volumes, eased up the process a lot.

Another great thing which I used a lot in Wwise was to monitor the amount of voices that would be played simultaneously (on the tablet). For example, with the musical games it would be very bad if sounds would get cut off, as it would ruin the sync of those musical layers. I found that up to 30 musical layers can simultaneously play without any problems; but, in the end, I kept it below 20 as a precaution. We tested on many tablets and found that this was a good amount tablets could handle, without us worrying about sounds dropping out. Needless to say, I also set the priority of sound effects lower than music for most levels.