SpectralMultiEffect is a plug-in for Wwise. It was created with the intent of enhancing interactivity in games, and of providing a rich source of experimentation and audio transformation to sound designers. Two major features are provided: A variety of effects to modify the audio signal, and a system to enable access to information from the audio data itself for direct use in the game.

SpectralMultiEffect offers many ways to transform the signal through processing applied to its frequency spectrum. Up to three effects can be set up in series, and all available parameters can be controlled at runtime through the Wwise RTPC feature. This includes the ability to switch effects on or off, or to swap them at runtime. 

I have noticed sound designers are interested in finding ways of enhancing interactions between sound content and the game. For this plug-in project, I wanted to provide the ability to use data obtained from the audio signal to directly influence or control aspects of the game's behavior. During playback, time or frequency domain data obtained from the signal can be made available to the application for any desired use. This can be useful in establishing a close connection between the game and the audio content.

A set of callbacks is provided, which developers can use to obtain data from the audio in multiple formats in real-time. As plug-ins can be attached to channels or sounds directly, it is also possible to set up multiple streams of data, through separate instances of the plug-in, each with its own callbacks.

Whereas audio plug-ins for games are often created to solve a specific and well-known technical problem, SpectralMultiEffect can be thought of as a creative tool. I am hoping it will help encourage an exploratory approach to working with audio which can lead to finding interesting sounds, creating an engaging and interactive experience for your game.

The following features are available:

1. Eleven different effects applied in the frequency domain, including pitch changes, multi-band delay, and spectrum freezing, among others.

2. Obtaining information from time or frequency domain signal data, to be sent to the application, enabling the possibility of synchronization between audio and other game elements.

3. Audio visualization in the Wwise Authoring Tool while monitoring the application.

4. Presets to demonstrate possible sound transformations. These can be used as a starting point for your own.

Using SpectralMultiEffect

Audio Effects

Three slots are available for effects to be applied in series. They can be selected and switched on or off at any time during the plug-in's operation. Additionally, it is possible to perform pitch-shifting within a range of -/+ 3 octaves without affecting sound duration.

Effect selection area in the Wwise Authoring Tool

While the plug-in is active, incoming audio data is converted to the frequency domain. Processing is done at this level, before converting back to the time domain for playback. 

Once you have selected effects for use, you can control their parameters using the RTPC system. There are many available, but you only need to control those involved in the effects that you have enabled.

Obtaining audio signal data through plug-in callbacks.

User-defined callbacks are available to prepare and obtain data from the audio signal.

Options include peaks or RMS in the time domain, and frequency bins or bands in the frequency domain. Data can be sent to the user either before or after applying effects processing.

To obtain data from the audio signal for your application:

1. Create an instance of the effect in the Wwise project, assigning a unique name tag that will identify it to the application at runtime.

Adding a tag to identify a plug-in instance

Callback declarations in ZcSpectralMultiEffectFXCallback.h

2. Implement the Initialization, Termination, and Execute callbacks. These are declared in the file: ZcSpectralMultiEffectFXCallback.h. (This file will be copied to your Wwise include directory during the plug-in's installation, so you just need to add an #include directive to use it in your source code.)

The initialization callback is used to configure the preferred data format. This is done by filling in the passed in ZcSpectralMultiEffectDataFormat structure. 

During playback, the Execute callback will provide the application with data from the signal in the format that was requested. The identifying tag set by the user will also be received, to facilitate managing multiple plug-in instances of the plug-in.

The termination callback notifies that a particular plug-in instance's operation is ending.  Any clean-up work that you may need to do to match actions taken during initialization can be done here.

Authoring Tool Monitoring

When monitoring your application in the Wwise Authoring Tool, you can visualize signal data in the plug-in's Effect Editor window. This can be done with either time or frequency domain data, as can be seen in the screenshots below.

Presets

A few presets are included, which can be selected in the Authoring Tool, from the drop-down menu on the bottom right of the Effect Editor window. They show examples of settings that may be interesting to you. You can generate factory presets based on them, or create your own, and use them as you would for any other plug-in.

Supported Platforms

SpectralMultiEffect is initially available on mobile (Android / iOS) and desktop (Mac / Windows) platforms. I am considering expanding support, including game consoles. If you would like to see SpectralMultiEffect available on your project's platform please contact me through the website, and let me know. Based on interest from users I will consider platforms to be added soon.

Audio Samples

Here are three sets of samples demonstrating audio processing with SpectralMultiEffect.

Spoken Voice File

This file begins with an unprocessed voice, listen as it gradually transforms using different combinations of effects in SpectralMultiEffect.

Piano Sound Files

The first audio sample is the original unprocessed. Each following sample is processed in different ways with  SpectralMultiEffect.

Gong Sound Files

The first audio sample is the original unprocessed. Each following sample is processed in different ways with  SpectralMultiEffect.