On one side there’s sound for films- linear, on the other there’s sound for video games- interactive. In the middle, there is this gray area where immersive XR soundscapes can be created using both linear and interactive sound design approaches. Coming from the linear world, learning interactive audio has been a journey (and still is) but the ability to combine both to explore new ways of storytelling is extremely rewarding and makes me more passionate about my work every day. My name is David Grinbaum and I’m the Head of Post Production at Chapter Four, a NYC-based full service production company specializing in immersive and interactive content. I began my journey in the linear world before venturing into interactive audio.

From traditional film and television to 360 video production

A few years ago, I had the opportunity to learn game audio but I was resisting the idea at first. My goal was always to work on big blockbuster movies, so there was only one thing I wanted to do: post production sound for film and television. In 2015, José Frías, one of my friends and now business partners, told me that he had been working on a few 360 video productions with ambisonics and thought it would be interesting if we could work with those recordings in post production. After a bit of research, I found Two Big Ears (now Facebook360), and started exploring spatial audio. After talking to two other friends, Sarah Gibble-Laska and Karim Douaidy, we all decided to form Chapter Four and focus our work on spatial audio. We could tell that sound was the secret ingredient to providing viewers with truly immersive experiences: we had found our holy grail! The transition from standard film and television to 360 videos was fairly straight forward. It was still linear, and spatializing audio felt similar to working in surround, just more rewarding!  On our first year, we got to work on some really cool pieces, including The Circle of Life in 360, the opening number from The Lion King on Broadway, and The White Helmets 360, a documentary by Vice and Samsung taking place in Syria.

 

Discovering the world of game audio

Soon after, I was fortunate enough to meet the Dolby Atmos VR team and became a beta tester of their Dolby Atmos for VR suite. Thanks to them, I went to my first GDC in 2017 and discovered the world of video games and game audio. It was overwhelming! I met a few people from Audiokinetic as well as folks from Vibe Avenue who were kind and patient enough to walk me through the basic steps of what game audio was, and coming from the linear world, I found it fascinating. Instead of attaching an event to a timeline, I can attach sound and physical properties to an object to dictate how it is going to behave in a virtual world. Mind blown.

Back in NY, I started learning Wwise by going through the Wwise 101 certification. At GDC18, I went back to the Audiokinetic booth and saw that VR was more and more present. Late to the game (pun intended), I finally realized learning game audio was the next step in my career and that I had a lot to catch up on. Luckily, the game audio community is very welcoming, so I knew I would be able to find help if needed. I went back to NY and finished the Wwise 101 certification, and after a few months, I jumped into the Wwise 201 certification, and then finally the Wwise 301 certification.

I love sound. I’m passionate about creating soundscapes and bringing things  to life. I’m constantly amazed by what technology allows us to do and by what people keep creating. Wwise and interactive audio opened a whole new world of possibilities for me. A world of creating sounds for new mediums and experiences, a world of thinking differently about my role as a sound designer, but it certainly required a mind shift and commitment. I think one of my favorite things about interactive audio is how scientific it can be as much as it is creative. This may also be why many sound designers coming from the linear world hesitate to jump into game audio however. Interactive audio can certainly seem overwhelming, but it certainly inspired me with a lot of ideas for storytelling. To me, the discovery of interactive audio was the equivalent of a kid going to Disney World, but for a sound designer.

 

Stepping out of your comfort zone with small victories

For those of you who come from the film industry, I want you to know that transitioning to game audio is not as scary as it looks! Coming from an IT background, I had a slight advantage because object-oriented languages and coding were concepts that I was already familiar with (although not passionate about). That being said, you don’t actually need to know a lot about coding when starting out, as the Wwise integration already comes with pre-made functions that allow you to integrate sound into the game engine of your choice. While the learning curve can be fairly steep because there is pretty much nothing from the DAW world you can use in the interactive world (besides preparing stems), the reward is hugely gratifying! 

For me, this is the beginning of a new adventure. Interactive audio made me even more passionate about sound. Like a kid with new toys! Everytime I finally figure out what’s missing for the object to be able to emit sound, or for the engine to use the RTPC value, or for the soundbank to be recognized (all of which, I’m sure, sounds very easy to some of you), I hear the Zelda music for opening a chest! 

For those of you who come from the linear world, and would like to learn game audio but don’t know where to start, here are a few checkpoints to help you get your feet wet. This is by no means comprehensive, as I'm still discovering new tools and techniques myself:

• Jump into the Wwise 101 Certification. It will all sound weird at first but after your second run through their lessons, you’ll start understanding the logic behind it. By the end of it, you won’t know exactly how to implement your own sound objects in Unity or Unreal but you’ll have a better grasp of how engines and Wwise communicate.
• If you don’t know anything about programming, there are tons of free tutorials (and also paid ones) that you can explore to learn the basics of coding. The most useful programming language to start with is C#. Brackeys’ YouTube channel is a fantastic resource for game engine and programming tutorials. I recommend getting a bit familiar with C# and at least one engine (if you’ll be learning with Brackeys, then it’ll be Unity) so that you are able to understand more than just interactive sound.  An audio programmer's role might involve helping game designers with minor bugs so that the back and forth between the audio and game dev teams are reduced. 
• The Wwise 201 Certification for interactive music is not a mandatory course (at first) but it will eventually become important that you learn about the mechanics of game music. From implementing transitions to generating a random cue so that music is never repetitive, a lot of the tools you learn in this module can be used for other effects.
• The Wwise 301 Certification is probably the most important after you're done with the Wwise 101 Certification. You can try to figure out on your own how to integrate Wwise into Unity and maybe trigger your first event, but if you want to do it right and learn all the different ways available, this course really does an excellent job at teaching you the good habits. You’ll learn how to use pre-made functions available to you through the Wwise integration, but also learn to write in C#, which will give you a lot more flexibility.

 

Some high level differences between linear and interactive projects that you'll first notice 

If you’re coming from the film industry, you’ve been used to certain specs and timelines. The specs are standardized depending on the distribution medium (number of channels or loudness requirements, etc.) and you usually start working once the picture is locked (note the “usually”). You are not limited by how many sounds can be used (besides creative direction) nor limited in how much DSP you can put on your tracks (provided your computer can handle it). All that matters is that your audio file matches the length of the video, has the correct number of channels and is synced. There is no limit on the size of the file. As a matter of fact, the file size can be calculated in advance since you already know the sample rate, word length and number of channels. 

Things are slightly different in video games and interactive media. You will be dealing with restrictions on how much CPU can be allocated to process sound or how much space is left for the audio assets - your ‘audio budget’. When working in Wwise, you can choose to have some of the DSP happening in real-time (using more CPU) or bake it into audio files (using more storage). Footsteps are a good example of how you could lower the amount of storage it requires: you can use as little as two different footstep files per surface and then randomize how much pitch is applied every time they’re played to get an infinite variety of footsteps, always sounding different to our ears. Another big difference I found is how we think about panning. There are three different panning modes in Wwise: 2D, 3D user-defined and 3D game-defined. Without going into detail, you can either associate a sound event to an object (3D game-defined), or automate sounds in 2D or 3D around the user for visual objects that aren’t actual game objects, or music and narrative that don’t necessarily need to be panned in 3D.

 

Next steps in my journey

There is still so much to learn! Now that I realize how many possibilities there are in interactive sound and game engines in general, I want to explore as much as I can. One of the things I mentioned earlier is interactive storytelling. While this is a concept well known within the video game industry, we could use this form of storytelling without the gaming aspect, giving a choice to the user to explore a specific part of a story and creating their own experience. I’m very curious to see where this takes me in my career as I intend to keep working in the film industry as well as getting more projects in the interactive world (games and immersive media).