menu
 

Achieving Cinematic Scoring through Proactive Audio

게임 오디오 / 상호작용 음악

When was the last time you truly noticed the music playing in a game during an epic battle? Game scores just don't command the same sort of attention as, say, a memorable film score from the likes of John Williams. While he expertly accentuates a well-placed haymaker to the jaw of a fascist soldier, a game score quietly ducks out the background music, deemphasizing the music. This is in large part because a film composer watches a near-finished version of the film to match music to action whereas a game has unpredictable player agency to contend with. A player can choose to do none of the things the music would otherwise be trying to highlight. As game developers, how do we create scores for unpredictable player action while also reaching the level of immersion cinema has enjoyed for over a century?

To get to a place where games match music scores, we have to understand the two kinds of methods to match game music to action: reactive and proactive audio. Most game developers are familiar with the reactive form of game audio. This method uses Triggers, Switches, States, Events, and RTPCs to provide the audio engine with game state updates that allow the audio engine to adjust the sound mix. This method is put to great effect in almost all games. Such approaches have enabled games like Resident Evil to ratchet up the hallway music intensity when a specific puzzle key is found. The developers of Killer Instinct on the Xbox One were able to turn their combo system into a clever music sequencing system that integrates a character's leitmotif into the BGM with a properly executed Ultra Combo. Nintendo has used dynamic MIDI compositions to create music that ebbs and flows based on location or major character events. Super Mario Kart uses area triggers to shift music instrumentation based on where the player is in the level. Middle-Earth: Shadow of War features a system that plays musical stingers based on very specific actions to give it more of a custom scored soundtrack feel. All of this has been made possible with the more advanced systems that audio engines like Wwise offer.

These methods are proven to create really impactful game moments, but how do we bring our audio interactions to that cinematic level where the game music adaptively highlights player actions? The answer is to flip the approach on its head: instead of driving music from game action, you drive actions based on music. That’s proactive audio design. Traditional interactive game audio is reactive, as communication starts in the game engine and ends in the audio engine.

AudioKineticDiagrams_Traditional System Diagram

Reactive vs. Proactive Game Engines

A proactive audio system, then, is one where the audio engine sends information back to the game. Wwise support for proactive audio design is implemented through Marker, MIDI, and Music callbacks. With Marker callbacks, a cue can be embedded in the audio and cause Wwise to send a notification back to the game.

unnamed

Wwise’s system creates a great foundation for proactive audio design, but what if we were to turn that approach up to 11? What if we didn't just say “here is a point of interest in the music” but pushed it to the level of “here is an upcoming beat, with this much intensity, this many milliseconds in the future.” A game system armed with this information can make important-yet-simple logic decisions: a brawling system could delay a game’s haymaker punch animation by 40ms to land exactly on beat, gaining a perceptual boost from the synchronization with the inherent musical impact. The game could simultaneously queue a sound effect, or musical flourish, to land on that future sync point and use a standard reactive audio trigger to cue the audio system to layer the composition with more intense music. That new music can then send new signals to the game. This game loop can be built up and made as complex as the designer wants. What results is a system that allows the music to be more traditionally composed and a game that can adjust to work with the audio (and vice versa). It is a far more intelligent interactive audio architecture where the game systems and the audio engine can coordinate with each other. What’s more; this can be achieved with relatively minimal game logic overhead, and can compliment existing audio systems like Wwise.

How about an example? Let’s look a bit deeper at how proactive audio might cinematically enhance a standard brawling system for just about any action game. A cinematic brawling system is one where the background music ebbs and flows with the balance of power between hero and enemies; where swells of music coincide with powerful hits and devastating blows. The music reinforces the actions of the player while simultaneously providing the type of situational feedback that we've come to expect from game audio. Let's take a look at what achieving this effect looks like. In the diagrams that follow, our goal is to align the impact of the punch animation with a high point of music intensity.

unnamed (1)

With the standard reactive audio approach, the game's animation system has no knowledge of what's happening in the audio, so the animation simply plays immediately, resulting in no synchronization between music, sound effects and/or musical flourishes, and animation.

AudioKineticDiagrams_Proactive Slowmo

With the detailed timing information provided by a proactive audio system, the animation system can slow down an animation (or entire simulation), and its accompanying sound effect, such that a special moment aligns with a powerful moment in the music. In this case, an animation with a punch impact 300ms in is "stretched" by 150ms to match a musical impact point 450ms away.

AudioKineticDiagrams_Proactive Delay

Alternatively, the animation system can delay the start of an animation such that the animation will reach its "crescendo" at the same time as the music.

AudioKineticDiagrams_Proactive Hybrid

These solutions can be combined depending on the needs of the game to provide a flexible, responsive experience for the player.

The result, without any adverse change to the player, is a system that naturally aligns action with music and preemptively determines when to add a musical stinger that accentuates the player's actions.

But why would you go through the trouble of flipping audio design on its head? Proactive audio enables game systems to coordinate with the traditional [reactive] audio system. Game audio doesn’t have to be relegated to the background when you can match the unpredictable player behavior to the soundtrack. You can go as far as creating a dynamic music sequencer that is driven by, and synchronized with player action. This new approach allows game developers to close the gap between game soundtracks and the kind of cinematic scoring you see in huge blockbuster movies. There is no longer any reason games can't reach the same level of quality.

Music is hugely important to the game experience and the industry's ability to tell compelling stories. It’s easy to let music slip when graphic rendering techniques are pushed to sell ever more realistic visuals. As we hit the limits of huge graphical leaps, however, deeper integration of music with other systems will become ever more important. Proactive audio systems, paired with an excellent audio engine like Wwise and clever game design, can get us there today: dynamic musical scoring can be a reality.

Happy designing!

Sonic Bloom

Sonic Bloom

At Sonic Bloom, we've been developing a proactive audio system, called Koreographer, that can achieve our dreams of dynamic, cinema quality game scores. It provides tools to create proactive-style data and APIs for the other game systems to consume it. A game armed with Koreographer gains awareness of the internals of the music mix which helps the game anticipate musical moments. We built it to naturally pair with reactive audio engines like Wwise and the combination of the two enables some amazing things. It’s the product of 10 years of Sonic Bloom’s collective history in music game companies. We began to build Koreographer out of the desire to change the way music is experienced in games: we want Raiders of the Lost Ark style scoring in Uncharted and believe that proactive audio will make this possible.

댓글

댓글 달기

이메일 주소는 공개되지 않습니다.

다른 글

음악 디자인 & "히라에스 프로젝트(Project Hiraeth)": 내가 존재하지 않는 게임을 위해 1년을 보낸 이유

히라에스 프로젝트(Project Hiraeth)는 완전히 어뎁티브하고, 전문적으로 믹싱 되고, 라이브 오케스트라와 함께 녹음된 세계 최초의 비디오 게임 스코어 라이센스입니다....

15.4.2020 - 작성자: Seth Wright

합성 만으로 빗소리 만들기

몇 년 전 저는 원하는 모든 사운드를 합성할 수 있을까 하는 궁금증이 생겼습니다. 바람, 새의 노랫소리, 곤충 소리 등 다양한 자연 소리를 합성하기 시작했죠. 이런 작업에서는...

3.9.2020 - 작성자: 알렉산더 킬코(ALEKSANDR KHILKO)

Wwise 미디 기본 지식: 뉴 슈퍼 럭키스 테일(New Super Lucky Tale)의 폭스베리 타이머 음악적 미디 마법!

안녕하세요 멋진 Wwise 사용자분들 :) 게임 오디오 업계에서 살아남기 위한 필수적인 스킬은 바로 문제를 해결하는 능력입니다. 사용하는 도구의 크고 작은 모든 면을 아는 것은...

8.2.2021 - 작성자: 애론 브라운(AARON BROWN)

Wwise+GME 게임 음성 솔루션: 다양한 음성 플레이 대방출, 생생한 몰입감 선사

AppAnnie2021 모바일 게임 리포트는 강력한 소셜 인터랙션 속성을 가진 배틀 그라운드, 슈팅 및 온라인 MOBA가 플레이어들의 사랑을 많이 받았으며 게임 시간 증가를...

13.1.2022 - 작성자: Tencent Cloud

에이지 오브 엠파이어 IV의 음악

안녕하세요, 저는 린 가디너(Lin Gardiner)라고 합니다. Relic Entertainment(렐릭 엔터테인먼트)의 수석 오디오 디자이너이자 에이지 오브 엠파이어...

22.1.2025 - 작성자: 린 가디너(Lin Gardiner)

Wwise로 게임 마스터링하기 | 제 1부: 게임 오디오 마스터링 접근 방법

게임은 다양한 구성에서 뛰어난 청각적 경험을 제공해야 합니다. 이것은 헤드폰의 3D Audio가 될 수도 있고, 스테레오 TV 스피커 혹은 광대한 7.1.4 Atmos 시스템이 될...

19.3.2025 - 작성자: 로익 쿠티에(Loïc Couthier) & 단제리 스켐브리(Danjeli Schembri)

다른 글

음악 디자인 & "히라에스 프로젝트(Project Hiraeth)": 내가 존재하지 않는 게임을 위해 1년을 보낸 이유

히라에스 프로젝트(Project Hiraeth)는 완전히 어뎁티브하고, 전문적으로 믹싱 되고, 라이브 오케스트라와 함께 녹음된 세계 최초의 비디오 게임 스코어 라이센스입니다....

합성 만으로 빗소리 만들기

몇 년 전 저는 원하는 모든 사운드를 합성할 수 있을까 하는 궁금증이 생겼습니다. 바람, 새의 노랫소리, 곤충 소리 등 다양한 자연 소리를 합성하기 시작했죠. 이런 작업에서는...

Wwise 미디 기본 지식: 뉴 슈퍼 럭키스 테일(New Super Lucky Tale)의 폭스베리 타이머 음악적 미디 마법!

안녕하세요 멋진 Wwise 사용자분들 :) 게임 오디오 업계에서 살아남기 위한 필수적인 스킬은 바로 문제를 해결하는 능력입니다. 사용하는 도구의 크고 작은 모든 면을 아는 것은...