Introduction
Angst: The VR Experience is a supplemental VR experience to Angst, a documentary on anxiety in students. Written and directed by Stephanie Riggs, it puts the audience in the position of a test taker undergoing a panic attack. It then leads the audience through a few exercises to calm down.
Concept
Because the experience was both linear and user-responsive in nature, I wanted the sound design to reflect this. I thought about how we could combine the best of both worlds to achieve this result. In Pro Tools, we could create a linear ambisonic track that played continuously in the background. In Unity, we could create the triggers that would deliver voiceover lines that were relevant to the user’s attention. The result would be an experience that is responsive to the user’s attention while also progressing the storyline at the pace set by the director.
Prototype
And so, that’s what I begin to build. In order to convey this idea to the director and producer, I mocked up a prototype in Unity to demonstrate the general concept. In the background played a music track, while each item in the room triggered a different voiceover. After receiving the green light from both the director and the producer, production began.
Ambisonics
We recorded the experience on set using the Zoom H3 microphone. It was then decoded from Ambisonics A to B in Reaper. While the Zoom mic served as the “boom” for the shot, we also ran three wireless lav mics for the teacher/therapist and for spot miking a few of the students.
Mixing
Because we needed to combine the ambisonic cinematic mix with a stereo headlocked voiceover and sound design, we settled on using Spatial Workstation by Facebook 360 (formerly Two Big Ears) in Pro Tools. This allowed us to pan and tilt the spot lav mics to match the baseline ambisonic track provided by the Zoom H3, as well as combine both diegetic and atmospheric sound design.
Sound Design
FB360 also allowed us to create a stereo headlocked track that would be embedded in their proprietary playback format (.tbe). Into this track we mixed the voice over and designed a soundscape that would mimic the physical response one hears during a panic attack. Fuzziness, sizzling, and deep thudding heartbeats were all combined to envelop the listener and drown out the diegetic sounds of the classroom.
Unity Integration
The beauty of using FB360 lays in their straightforward Unity integration. They provide an SDK that can be imported into Unity. A the time of the project, Unity did not support native video playback. Instead, Easy Movie Texture by Jae Yun Lee was used. A script was needed to synchronize video playback with the FB360 decoder SDK. Once this was accomplished, it was time to move onto the sound triggers.
Sound Triggers
Because the goal of the sound triggers was to tailor the playback of the experience to the attention of the user, I designed a series of sound grids that were triggered by the user’s gaze. Each trigger represented a physical object in the scene that the user could see and that would produce a voiceover reaction to (e.g. a microscope, a clock, the board, other students, etc.). These grids were set to activate and deactivate at preplanned intervals during the video playback, culminating in all of them being active during the height of the panic attack to simulate runaway thoughts.
Final Integration
Because the final mix combined so many different elements, I ended up performing the mix automation directly in Unity’s audio mixer so as not to overload the binaural output. The result of all this was to create an experience that pushed the boundaries of what was possible in mixed media while also educating people on how remain calm during periods of heightened anxiety.