Hello to everyone out there in Wwise user land! My name is Paul Ruskay and I am the Audio Director at Studio X Labs in Vancouver, Canada. In this blog, I am going to explain how, in our upcoming Indie 3D Puzzle Platformer title: Lumote (PC, Switch, PS4, XBox One), I used the Wwise Audio Metering plugin to drive in-game VFX, such as particles, lighting, camera shaking and mesh warping. Due to Lumote being an indie title, I was given a great amount of time and freedom to experiment with the Metering plugin and to learn how to use it in the context of the game’s world and narrative. Since I was designing sound elements to specifically drive visual effects, I found many opportunities to create unique visual and auditory moments with this setup.

Lumote is a colourful and beautiful 3D-Puzzle-Platformer slated for release on February 20, 2020 on Steam . It will be available on Switch, XBox One and PS4 later in 2020.

Overview

The process of creating game audio typically has the sounds being created after an animator or VFX artist has completed their work. Our task, as sound designers, is to create audio that will match to these finalized animations and visual effects. For Lumote’s development, I used the Wwise Metering plugin to “harvest” the sound design’s waveforms, in order to generate a RTPC value string that was fed back into the game engine, to drive various visual effects and animation systems. With a team of four devs and no animators, we needed to adapt our approach and we soon realized that a lot of visual interest could be generated in a procedural manner, by having the audio drive many of the game’s VFX systems.  This approach to audio-driven VFX has led to a game that feels very organic and fluid. I have an extensive portfolio of game audio production and this approach of driving visual effects from waveforms was new to me and a lot of fun.

The topics for this Blog are as follows:

1. Description of Lumote
2. Development Team
3. Lumote’s Audio Design
4. Metering Plugin Setup
5. Examples from Lumote of audio driven VFX

About Lumote

Lumote is a game about two opposing powers. Blue power is controlled by Lumote and Red power is controlled by the Mastermote, the antagonist in our game. The player navigates Lumote through this 3D-Puzzle Platformer, in which the player has only two abilities: the ability to possess creatures and the ability to jump. Possession allows the Lumote to take control of the world's inhabitants, (the “Motes”) and to use their unique abilities to solve increasingly difficult puzzles. Depending on which energy controls a Mote, their behaviour will either help (with Blue Energy) or hinder (with Red Energy) Lumote’s goal of reaching the bottom of the world map and take control away from the Mastermote.

Lumote is a 3D-Puzzle Platformer that will take the player on a 6-hour journey to the bottom of the world map. It was developed in a custom game engine (rEngine), coded for this project by members of the Lumote Dev Team (Kyle Rocha and Aaron Arnt)

The world of Lumote is broken into three biomes, each comprising three towers that need to be converted from Red Energy to Blue Energy before Lumote can proceed to the next series of puzzles. Each tower represents a Mote type and a specific game mechanic, that you must master by completing the tower. There are five Mote types in the game (Small Dumbats, Large Dumbats, Hover Movers, Jellyfish and Divertors). Each Mote has its own special ability that Lumote must master in order to reach the end of the game.

 

Early concept art for Motes. Some great character design that works beautifully in the context of the puzzle design. 

 

Development Team:

From left to right:

Michelle Rocha , Puzzle Design & Code             

Kyle Rocha, Code & Art   

Aaron Arnt , Code & Rendering          

Paul Ruskay, Music, Sound Design & Wwise Integration                

We all worked remotely, so our development environment was Perforce, Slack, rEngine and Wwise. We kept meetings to a minimum and used Slack as the primary means of communication throughout the project. Everyone worked within their respective silo, while easily being able to Slack someone when needed. Looking back, it's a great way to make a game, which provided me with the time and creative freedom to experiment with audio-driven VFX.Luminawesome Games is an indie studio, located in Vancouver, Canada, and is made up of veterans from the video game industry. Every team member has worked on and shipped AAA titles, so there was a lot of development experience applied to Lumote. In October 2014, Kyle and Michelle were one of the winning teams in an Unreal Game Jam, with their mini game, Bump. They loved the concept and decided to develop it into a full game. Aaron joined in 2015 and it was decided then to move away from the Unreal Engine. Over the course of 2015, Kyle and Aaron coded a custom graphics engine, called rEngine, specifically for Lumote. I joined the project in March 2016 and was charge of music composition, sound design and Wwise integration.

Lumote’s Audio Design

The main goal of the sound design for this game was to create a world that would feel alive and expressive, whenever a player interacted with it. Lumote needed to sound and look organic, electronic and otherworldly. We were a small team with no animators, and so we needed to optimize our resources. With that in mind, we quickly realized that many of the Mote’s animations could be generated in a procedural manner, from their respective sound design. By using the Metering plugin, we were able to create an RTPC value string from the game object’s decibel meter, in its associated audio folder in Wwise. Kyle programmed the VFX in rEngine, which would take the RTPC output from Wwise and send it to lighting, particle emissions, camera shaking and mesh warping systems. So Lumote’s visual effects became literal translations of the waveforms being triggered from the game objects Wwise folder. That meant I would need to think about audio in a visual way. Below is a breakdown of what I came up with:

A constant pulsing sound applied to lighting would generate a strobe effect. The same sound applied to particles would create a constant stream. It was interesting to have to think about how the audio wave would translate visually as an animation, especially within the context of character responses.

 

 

The most time consuming task was developing the five varieties of Motes, as all five needed to communicate both their character State and their reactions to Lumote. The other challenging part of Lumote’s audio design was having to create two different sound sets for each energy type, per Mote (Red and Blue). At any given time in the game, the environment and the Motes could carry either Red Energy (Mastermote possessed) or Blue Energy (Lumote possessed) and as a result, each energy type needed to have a its own special audio treatment. This resulted in having two sound design versions for every game interaction, based on energy type. I had to effectively sound design the game twice to make all this work. Also, the Mote's sound design needed to blend harmonically with the music, as many of the Mote's voices were generated from samples of musical instruments (flutes, bells, whistles, etc). The musical style that best suited the environment was ambient, as I needed to create music that felt “underwater” and “otherworldly”. The ambient music would also not distract the player, who would need to concentrate to try and solve the spatial 3D puzzles. There are also no spoken words in Lumote’s character reactions, only vocalizations. This meant that no language localization recordings were needed, but it also meant the game’s narrative would be non-verbal and would have to be told through audio design and music alone. 

My inspiration would come from games like Inside, Limbo and Ori and The Blind Forest, that all have great “non-verbal” narratives that sustain interest, build tension and create convincing gameplay moments that communicate the emotional state of the main character.

Even though Lumote’s core gameplay loop of solving puzzles has a very simple sequence of audio Events, I discovered that there was an amazing amount of subtlety and variation that could be created within the game, for example, as Lumote possessed and changed a Mote’s energy State.  

Mote’s Core Audio Event Loop:

Idle – Hello – Goodbye (Blue or Red)

Idle – Hello – Possession Start – Energy Change - Move Start – Move End – Possession End – Idle Start

 

Lumote Core Audio Event Loop:

Walk

Jump

Double Jump

Land (RTPC driven based on falling distance)

Possession Start

Possession End

Mote Hello Reaction

 

Lumote has different reactions for each Mote type, based on the Energy State. It is apprehensive when greeting Red Motes and pleasantly cordial when meeting a Blue Mote.

 

Metering Plugin Setup and Wwise Setup

My goal was to create a Wwise project that would allow me to have maximum control over all game audio assets, with a minimal amount of programming support. To accomplish this, Kyle and I “paper-designed” the audio Event layer, based on game object groupings that shared common Events. This way we could have 80% of the audio Events being passed to Wwise before I even started the audio production. Our Switch categories and Common Events were:

With rEngine passing audio Events to Wwise, I set out to start building my folder structures and populating them with audio assets. By getting 80% of the Events passed from rEngine at the start of the sound development process, I was able to work independently and could create and test all of the audio elements with very little programming support. With a Dev team composed of only four people, it was obvious everyone needed to be as independent as possible.

 

Kyle created a series of OutputMeter presets that I could assign to game objects to drive their lighting, animation and particles. The OutputMeter_Voice was used 90% of the time, because that was the preset that linked the RTPC from Wwise to the Mote's visual systems. 

AkGameObject, AkSoundEngine.PostEvent:

This is how we passed the RTPC from Wwise to the game’s engine:

1. We set the Mote Type Switch for each character in the game.

2. We then set PostEvent with a common Event like Common_OnMoveBegin and Common_OnMoveEnd.

3. We then used AK::SoundEngine::Query::GetRTPCValue() to read back the metering parameters from Wwise, based on the audio folder’s decibel meter. 

4. We post-processed those metering parameters in the game to modify the amplitude and frequency of a Sine wave, which is what drives the actual lighting, mesh animations, camera shaking and emissive material parameters.

 

What I found working with Metering Plugin:

1. We used the Metering plugin to reinforce Motes States and created creature voices for a range of States (idle, surprised, stop, start, power change).

2. The Metering plugin worked great as long as there was no “hangover” signal from the game object’s previous audio Event. I needed to spend time working in the Event layer to make sure a game object’s sound design had completely stopped before the next SFX was triggered. Any audio overhang from a previous audio Event would takeover the Metering plugin and not pass the new audio RTPC to the game engine. I also found that using Pause Events on a folder would also takeover the Metering plugin. It was best to stop playing a folder completely before triggering the next audio Event, so the Metering plugin remained available for the new audio signal.

3. Adding verb FX on Mote subfolders allowed for both audio and visual blending between different Mote States. It helped blend the separate sound folders into a cohesive whole.

4. Linking Impacts to the Pitch RTPC (for Impacts) yielded really interesting visual results. The lower pitched versions of the SFX created slower and broader visual effects.

Conclusions

1. Lumote's sound design turned into an evolutionary process as I began to play around with the Metering plugin. When we hooked camera shaking to the audio of the Mastermote, it really brought the Tower Sequences to life.

2. Be open to crazy ideas and always pursue the most enjoyable and memorable experience for the player. Also, ask yourself: Is there too much sound in the game? I there enough “negative space” in the audio to allow the player to be comfortable in the game environment? 

3. "Paper design" the Event layer and Switch categories early.

5. Research the goals of creatives in other fields (coding, art, and design), and ask yourself how you can help the people working in those fields achieve their goals with audio?