Uncharted Territory, Pun Intended

What Songs Tell Us About All Interactive Music

 "Interactive Songs are a Worthy and Unsolved Challenge."

--Albert Einstone, who is somebody I made up

 

Area Fat Man Thinks about Two Things

Thing One: As a songwriter, I think a lot about all the little bits that go into writing, performing, arranging, recording, and mixing a song. Words, chords, and lyrics. All the instrumental parts, all the little skills and considerations that need to be brought into play. The historic precedents, the "secret sauce" tricks that "make it a hit," the balance between what's familiar and what's novel. The lessons learned from all the Beatles books and "How to Write Songs" videos. Who is the audience? What story am I telling? Why is this going to mean something to somebody who hears it? It's a hugely complex web of thoughts, each string tugging at the other. 

At the end of the day, though, the whole process is usually driven by something pure and unnamed. But as a placeholder, let's call it "heart" or "Art." I think you know exactly what I'm talking about, even if you aren't a songwriter, because every art is like this. And truth be told, that is why we do this thing we do instead of some career that makes better business sense.

Thing Two: With all those considerations, how do we bring "Interactive" into the picture without drastically over-complicating the whole process, and without knocking the "heart/art" out of the deal? Because if there's no heart, let's just cave in, take that day job and be done with it.

So, yeah, I think about those two things—a lot. And I've rewritten this article at least three times as of this writing, maybe more by the time you read this. So let me share some of what my brain and heart have argued about.
 

We Have the Tools, We Have the Technology 

I've said it before, technology that allows you to "do things you've never dreamed of" is like a step-stool. It's cool and all, but I prefer technology that allows me to do things I did dream of, which I see as a window, a door, a wrecking ball that knocks down the walls. That's the point of this series ("What Are You Wanting to Do, Technology Aside?"). I want to do the thing that I am wanting to do. 

And thank goodness, we finally have Wwise, and we can look through that window, walk through that door, fly into that interactive sky with full confidence that we won't likely be stopped, at least in all the early stages. 

And that having been checked off, when we "get interactive" with a song (or instrumental music for that matter)...what are we wanting to do?

 

Songs and Interactivity: Two Legacies

The Good Legacy: On one hand, songs come from a tradition of being among the most interactive things in the world. They've only been "set in stone" as recordings for about a hundred years...I suppose at one point they were "set in stone" more literally than figuratively, and I'm sure songs go back much much farther than that.

Interactive songs? Historically, people working or playing together have always had occasion to sing together. I've heard tell that entire communities can break into song. A chanting monk of any tradition has to tune his voice to the other monks and change pitch when the others do. Bardic tunes and bar songs are meant to be sung along to, often with improvisation in the lyrics, timing, melody, and form. How long does it go on? Depends on how it's going over.

Even when we're talking about modern songs, certainly when the act of singing happens, there is interactivity, or at least reactivity. In the studio, the singer reacts to the instruments he hears in the headphones, and reacts to the producer's feedback. On stage, the singer reacts to what the audience "feels like"; the song might go out in a high- or low-energy way. A vamp might be extended, solos might go long, a repeat might feel redundant and get cut out, "rockstar" endings might happen. The audience might be asked to sing along or clap. There might be encores. 

The Bad Legacy is that there is a shortage of experiences that pass "the five-minute test." 

We've had computers and audio for a good while now, and in the many years it's been possible to create interactive and reactive music, we have not (yet!) developed a great track record for interactive music experiences, let alone interactive song experiences. Let's look at all the noble, beautiful experiments in games and apps based on music, and let's define "success" as "we can play it for five minutes and still want to play it some more." What we see is reminiscent of Gettysburg's battlefield. A very few survivors stand tall or kneeling amongst a sizable field of wounded and bleeding bodies. Among the successes, I'm going to count Parappa the Rapper, Guitar Hero, Rock Band, and, stretching the definition of "audio experience," the awesome interactive soundtracks to many games. As a bonus, I'll also call out the interactive(!) soundtrack to Disneyland's Space Mountain ride, which transparently responds to the speed and loading of each vehicle that goes down the track. I expect I have missed a handful, I hope I have missed a lot, but I believe that by comparison the successes are few and the wounded are many.

As to the dead and wounded, I don't want to get too far into opinions or pointing the finger; one might just search Google for "interactive audio experience" and see what proportion comes up that is worth that second five minutes.

And if you find yourself agreeing because, of course, songs, then let's play a cute mental trick and substitute "instrumental music" for songs. Oh, there are a lot of similarities here which apply to nearly every type of instrumental music, especially from before records and movie scores. Thank you, Songs, you have opened our eyes.

Instrumentals, Video Motivation and Justification

Leaving songs for a moment, and thinking about traditional instrumental scores: When scoring for video or media, or when editing video media, we often talk about "motivation" or "justification." One example would be if we edit film to a 120 BPM song so that scene changes happen every half second. We can say that the edits are "justified by" or "motivated by" the musical tempo. If we change music suddenly in response to a jump scare in a horror movie, that "cue was well motivated." Music that builds to a spooky climax can build expectation for what's happening in the plot, and that expectation can be filled (the monster jumps at you) or not (it turns out it was only the cat).

Similar phenomena occur in Rock Band, Parappa, Monkey Island, and other successfully scored games and experiences, even slot machines. First we score so that the atmosphere is right, then we score to the game design, or what I call "score to the score." We want the sound to hit the intent of the game, to inform us (directly or indirectly or playfully) when something good has happened, or something bad, when odds of "good" or "bad" have dropped or risen, or sometimes just send the message that the game is just messing with us when that's something the game designer wants or accepts. I like to exaggerate by encouraging people to design sound so that the players can play with their eyes shut and know what's going on.

In linear media, a sound can be justified by a visual, a visual can be justified by a sound. Add interactivity/gameplay to the "justification" mix, and you get a very interesting threesome.

The thing that seems to separate the successful audio interactions from the failures is that the music is not just sprinkled into an experience, nor is interactivity sprinkled over a piece of music. There is something about the movie or the game that seems to "know about" the sound, and something about the sound that "knows about" the game. 

And then I want to go further...how would "gameplay justification" work if the music were a song? How are the triggers of "good things" "bad things," and "odds going up or down" to be related to things like "audience energy" that historically have affected how a song has been performed? How do we make that connection, and does it bring us closer to solving the unsolved mystery of interactive songs?

AND I'm gonna blow your mind now, we tend to think about songs as having form (such as verse-chorus-verse-chorus-bridge-chorus) and we tend to think of movie scores as being atmospheric and free-form to fit the movie. But instrumental music historically has form, just as much as songs. If we can make songs work in an interactive context, will their form be preserved? Will that bring power to the interaction? And then, will we be able to use what we learn to bring the power of form back into games? And then will it leak back into MOVIES? Woah. Yeah, that just happened.

 

Area Fat Man Looks at Hopeful Future and Exciting Adventures

So, as I said, there are some fertile directions that seem to emerge from this line of thinking, and which I hope to explore in coming blogs:

1. Traditionally, songs are interactive. So, there is a precedent. Starting from those traditions, what part of that do we WANT to continue to accomplish, explore, or expand on, and how do we use the new tools to enable that?

2. Even if there aren't sooooo many songs used in games, song,s and instrumental music, how can we take those implementations and apply them to instrumental music in games/experiences?

3. Looking at the principles of motivation and justification in film scoring and successful game music, how do we relate those lessons to interactive songs? And what will that teach us in turn about how to do the instrumental stuff even better and in new ways?

When we first started doing audio on computers, tools were scarce and exotic, and people who were interested in it were few and far between. For a while, the best "integration software" was iMUSE, and there were only three people in the world who were capable of using it. Now we have a whole world of people with brilliant ideas, a fresh sense of what's possible, and Teh Internetz. Furthermore, we have Wwise and we have this blog space. So share your ideas and let us know what you think of all this. If we put our heads together, we can raise the bar, even when, once again, the bar lines are not set in stone.

--George

Bonus request for your comments: A song that gets played back differently every time you hear it may be a good idea on paper, and it may be pleasing under some conditions. Isn't there something kind of not-what-a-songwriter-or-listener wants about that? Let's look unflinchingly at the idea in a future blog, shall we, and see if we can determine when and how and if it is a good idea to do that. Feel free to pop your thoughts, in advance, into the comments.

 

Put that technology aside, George!

 

What Are You Wanting to Do...Technology Aside? Part 1