menu
 

PixelSound | An Audio Toy

互动音频 / 互动音乐

Whenever I start a new creative project, I always get a new toy - some cool plugin, or musical instrument, or spiffy microphone - something to just play around with for no other reason than to have fun. Making time for play encourages “childlike discovery” which is when good ideas happen.


"PixelSound"

"PixelSound" is an audio toy that makes a sound based on the color of the pixel your mouse is pointing at. It's a Unity/Wwise project (partly inspired by the excellent program PhotoSounder) that loads up a jpeg, then sets RTPCs to red, green, blue, and grayscale values including x/y screen position.

It's also a way to use Wwise as a sound design tool. Need to create some related variations for a random container? Set up a mix of assorted stems; let it run while recording the output; cut up the results and pick the ones you like. It's a common enough technique ...

Using the color values of pixels to set Wwise event parameters is just a way to generate pseudo-random variations. You can set up patterns to trigger specific effects, or load up a photo of your dog just to see what happens. This kind of "seemingly but not really completely by chance" algorithm can be considered "stochastic", a composition method famously utilized by Xenakis to produce interesting tonalities.

To play, fire up the Unity project, and click on the screen. Move the mouse around to hear how the event reacts to the colors and screen position. Click again to stop.

Download the project here.

You can drop your own .jpg files into the Assets/Pictures folder, then drag them into the "Picture" field of the GetPixelParameters script attached to the "pictureSprite" game object. This formats and resizes the picture to the screen, starts/stops the selected Wwise event, and sets the RTPCs.

The Wwise project contains some demonstration and test events that modify volume, pitch, trigger rates, EQ, and other parameters. They produce a variety of noises you can play with -- but of course, the whole point is to have fun messing around with your OWN sounds and pictures. There's also a Wwise Recorder instance on the Master Audio Bus, in case you want to record the output for later editing.

PixelSound

**Project update - "VPixelSound"**

VPixelSound is basically the same idea as PixelSound, except it uses video instead of jpegs. You drop in a .mov and select a position (center screen by default). RTPCs are set to red/green/blue/greyscale values as before, but I added some State processing as well, whereby color values set Wwise states on a ten step scale (i.e. blue_00, blue_01 ... blue_10).

Using this version, Wwise events are controlled by moving picture, rather than by moving mouse. The series of colors that generate the audio is also repeatable, in case you wanted to craft an event for a specific sequence of images. Multiple cursor positions and color averaging options are available as well.

Most videos will provide interesting input for sonic experimentation. I used QuickTime Screen Recording to create the included samples, but any capture method will do. Note: no soundtrack needed, and any video format Unity supports should work.

Remember, this project is an audio toy; the point is to try out your own videos and Wwise events to see what noises you can produce. Is it useful as a Serious Sound Design Tool? <shrug> All I know is it sure is fun to play with! (and isn't that an important aspect of any creative work?)

Video demo:


The "VPixelSound" project can be downloaded here.

I can possibly see this technique being useful in a game:

  • Modify the ambience based on the color of a procedurally generated sky
  • Change filter settings based on the greyscale value of a scene
  • Randomize UI/selection sounds based on screen position

Granted, it's a whimsical technique that could maybe support creative audio solutions. But mostly, it's just supposed to be fun to play around with, and experiment with the powerful features available to sound designers when using Wwise as a synthesizer.

What does blue sound like to you today?

Peter "pdx" Drescher

Sound Designer and Composer

Peter "pdx" Drescher

Sound Designer and Composer

Peter "pdx" Drescher produces audio content, implementation, and code for games and mobile devices. He is a talented sound designer, accomplished musician/composer, published author, experienced programmer, and skilled audio engineer. His ProTools project studio "Twittering Machine" is located in the SF Bay area.

www.twittering.com/

评论

留下回复

您的电子邮件地址将不会被公布。

更多文章

《Divinuet》的互动音乐系统 – 第 1 部分

早前我开发了自己的第一款游戏《inter-view》,一方面是希望由影视作曲慢慢转型做游戏作曲,另一方面是想向潜在客户证明自己通晓互动音乐创作到 Wwise...

6.1.2021 - 作者:梅根·卡恩斯 (Megan Carnes)

互动音乐:根据玩家在游戏中所做的选择播放不同的乐段

在 2018 年众筹成功之后,inXile Entertainment 开始全力制作《Wasteland 3》。尽管上一款游戏《Bard's Tale IV》的配乐赢得了一致好评,布莱恩•法戈...

1.12.2021 - 作者:亚历山大•布兰登 (Alexander Brandon)

zerocrossing 为 Wwise 开发的 SpectralMultiEffect 插件

SpectralMultiEffect 是一款为 Wwise...

6.1.2022 - 作者:哈维尔•阿西尼加斯 (Javier Arciniegas)

2021年国际互动音乐研讨会概要

2021 年 8 月 26 日,Audiokinetic...

6.7.2022 - 作者:Audiokinetic (音频动能)

音乐外包工作流程

在最近的一次Audiokinetic国际互动音乐研讨会上,理查德•勒德洛 (Richard Ludlow)、安德莉亚•张 (Andrea Chang)、钱斯•托马斯 (Chance Thomas)...

20.7.2022 - 作者:Audiokinetic (音频动能)

利用 Wwise 和 Unity 制作和演奏生成式音乐专辑

简介 今年早些时候,我发行了一张生成式氛围音乐专辑《Púrpura》。在本文中,我将探讨自己如何使用 Wwise 制作了这张专辑 – 从使用 Actor-Mixer Hierarchy...

2.7.2024 - 作者:Caio M. Jiacomini

更多文章

《Divinuet》的互动音乐系统 – 第 1 部分

早前我开发了自己的第一款游戏《inter-view》,一方面是希望由影视作曲慢慢转型做游戏作曲,另一方面是想向潜在客户证明自己通晓互动音乐创作到 Wwise...

互动音乐:根据玩家在游戏中所做的选择播放不同的乐段

在 2018 年众筹成功之后,inXile Entertainment 开始全力制作《Wasteland 3》。尽管上一款游戏《Bard's Tale IV》的配乐赢得了一致好评,布莱恩•法戈...

zerocrossing 为 Wwise 开发的 SpectralMultiEffect 插件

SpectralMultiEffect 是一款为 Wwise...