During MIGS17, Vincent Gagnon, Audio Director at Ubisoft, gave a presentation on Music Theory for Sound Design. The concept behind his talk revolved around the following idea, ‘What if we tune our game?’

We know that our world certainly sounds different to how it did a century ago. In his book The Soundscape: Our Sonic Environment and the Tuning of the World, R.M. Schafer, a Canadian composer and environmentalist explores the evolution of soundscapes through the ages and how our sonic environment influences our experience as we travel through different audio contexts. Our world would certainly sound very different in Jerusalem, London, or Paris, for example, and Gagnon suggests listening out for that in Assassin's Creed.

Gagnon had just finished reading Schafer’s book when he had started to work on Splinter Cell: Conviction. Inspired by ‘the tuning of our world’ idea, Gagnon considers how sound designers can apply this to game audio design to perhaps tune a game, and how by using music theory sound designers can become even more creative as they imagine and realize a game’s soundscape. He decided to examine how sound and music in combination can be used very specifically to influence the players' experience and stimulate more immersive gameplay. For example, how by tuning sounds to music we can create cohesion, and how by using juxtaposition we can create tension.

So, how do we tune a game?

Gagnon suggests that we can start by establishing a key and, perhaps, related keys. He explains how Beethoven’s fifth symphony, for example, is in C minor; however, while the first movement is in C minor, the second movement is in A♭ major, the third in C minor, and the last in C major with glimpses of C minor. So, just like with music, tuning our game will ‘revolve around this journey through tonality’.

For Splinter Cell: Conviction, Gagnon began by listening to all the music they had for the game. He noted that the key that was mostly used was D minor, and also realized that he could use the theme music of the game as inspiration for sound design elements and towards creating even more unity between the audio and the music.  

 

Watch Vincent Gagnon's MIGS17 presentation on the Audiokinetic Channel 

Theme Music

In Splinter Cell: Conviction, as the player walks through the amusement park, the various amusement park booths are actually playing variations of the game’s theme music. The player may or may not notice, but sonic unity is certainly achieved while the diverse arrangements correspond to the contrasting unique atmospheres and moods of the booths (9:12). 

Another element where sound design utilized the theme music was noticeably the system phone dials. By embracing the theme music and tuning the dial tones so they would actually play the theme music, the sound design intends to provoke a feeling of familiarity as the player identifies it (10:55). A final example on how sound design made use of the theme music, in this case to amplify the player’s excitement during a critical moment in the game, was by explicitly featuring it during the final mission. The main system music and all gameplay sounds are ducked, and the theme music kicks in just as the player enters the final scene in hopes of becoming a hero. This provokes an appropriate ‘feel good, we’re about to save the world’ anticipation (11:33).

 

 

Tuning Sound Effects

Sound effects across the entire game, such as the church bells or the sound effects that would accompany the projected text (text that would project in the player’s environment to provide them with instructions towards their mission), were tuned to the music. Even when there was an objective behind the sound design to create tension, the idea of tuning the sounds allowed for a more deliberate and refined implementation. The sound effect for the projected text for example were chords played in reverse, and while it was designed to alert and create some anxiety during the moments of these projections, the reversed chords were awkwardly clashing with the music at first. Therefore, and while the player may not notice this, the audio team created four different versions of this sound effect, tuned to the music in different parts of the game for a more seamless integration throughout. The four tuned chords would trigger depending on the music (13:36). 

 

Tuning Room Tones and Ambiances

Gagnon explains how room tones and ambiances tuned to the music can create the ‘Sharawadgi Effect’, an aesthetic effect that characterizes the feeling of plenitude that is sometimes created by the contemplation of a sound motif or a complex soundscape of inexplicable beauty.

Sharing an example from Splinter Cell: Conviction, he demonstrates the room tone for the museum’s environment. The museum room tone needed to represent the environment where a bad guy, who is an art collector, lives. This bad-guy art collector has refined taste, so also happened to have tuned his space to D minor. The room tone was looped, processed through an EQ using frequencies specific to D minor and boosting specific frequencies matching D minor’s base notes, D, F, and A, towards achieving the Sharawadgi Effect (14:50)! 

Another example Gagnon presented to demonstrate the connection that can be created between music and environment ambiances was how the sounds of the loon, which were tuned to the music, were also calling the chord changes in the music. This cohesion was created by syncing two segments of equal lengths from the music and ambiance. Using their louder or quieter moments, or their rhythmic elements for example, an interaction between the two was composed, creating a wider-ranging comprehensiveness to the environment (16:15).

 

Using a similar approach across the franchise

Moving on to Splinter Cell: Blacklist. This project was split amongst Ubisoft’s teams in Toronto, Shanghai, and Montreal. The Montreal audio team was assigned the portion of customizing unique game play experiences for the Mercenary, and the Spy.

Still in D minor and using the same theme music from Splinter Cell: Conviction, Gagnon explains how once again, they started with the music. The composers were asked to create two different arrangements, using different instruments that would define the distinct characters of the Mercenary and the Spy. The goal was to create an asymmetric relationship and a kind of duality between the two characters. The Mercenary needed to invoke a more ‘brutal, rough, and beefy’ mood, and the Spy, a more ‘sneaky, stealthy, and eerie one’ (20:50). 

The same goals were used for the unique sound effects created for the spy and the mercenary. A great example for tuning sound effects in this game is the radio noise added on the voices (note the radio noise, not the voices) of the three mercenaries. The radio noise was tuned to D minor, but since there were three different characters, they decided to give each one a note from the D minor key: D, F, A (26:27). The interesting thing about tuning sound effects is that while it seems like more work in the beginning, it becomes easier to do other things later on. Gagnon explains how they would tune in D, A, E, F; however, when they wanted to create tension, they could easily slip in an E♭, for instance, and suddenly it would sound stressful.

 

Sound should always be moving

Gagnon is interested in the idea that ‘sound should always be moving’. Room tones and ambiences especially, should not be repetitive, and when they vary and are in constant movement, the slight differences in their content make them less noticeable and ensure that listeners would not tire from hearing them.   

When creating sounds that are constantly moving, it is important that we don’t have any strange artifacts. Gagnon provides an example that uses the multi-band technique to create diversity in the sound effects for the grenades in Splinter Cell: Blacklist, and explains that the same technique can be applied to room tones and ambiences to achieve movement and diversity. Starting with five different grenade sounds, four bands were created from each (low, mid-low, mid-high, and high), and were combined randomly to create infinite variations for grenade sounds. Volume changes, filtering, and reverb were also applied on the bands. For example, the farther you were from the grenade, the more reverb there would be. Even when the differences were subtle, the desired outcome was achieved: the ear would not tire from hearing the same thing (27:18). 

 

Applying the multi-band technique to room tones

Let’s say we start with some room tones, chop them up, and create three bands for each: low, mid, and high. Gagnon explained that we could play them randomly taking a high, mid, and low from different files. It is important that all the files are different in length so that they do not crossfade at the same time. This will allow for crossfades to sound more natural or seamless and expel the potential to have any strange artifacts. For example, the highest band could be transitioning while the bass is still the same, or the bass could be transitioning while the mid is just about to change, and so on.

Gagnon’s example also had three EQs applied on each of the bands, which provided a total of 9 different files to work with. He suggests that we can perhaps add RTPC’s on EQs so the EQs would ‘move’ in volume or frequency. Perhaps the E in C# minor shifts to F, making it major, and during a moment of tension in the game, the room tone plays a part in signaling this. In his example, Gagnon also added some RTPCs on the LFOs that were affecting the EQs, so that the ‘movement’ is not too consistent. And there we have it, three bands being mixed with different EQs. (34:43)

Future Work

For his future work, Gagnon is interested in the movement of room tones and ambiences, in addition to creating ‘sense’ for their progressions, using sequences. He concludes by highlighting that we should start with a vision for the soundscape of a particular project, think about unifying the audio, and strategically use the tuning or the untuning of sounds. While this methodology might not work for all projects since they are all different of course, there’s certainly a lot to explore.