Final Project: Interactive Audiovisual Interface

Our group began the final project with an idea to create an interactive music

experience. We played with different ideas on how to interact with the musical

environment including color and shape cues and DJ-ing with dance. Ultimately, we

decided to try to modify audio and visual output with both shadow movement and

pressure pads.

The basic idea was to project a video piece onto a two-way projection screen and

have people stand in front of the projector to create a shadow. That shadow would then

be read by a webcam behind the screen, and the shadow’s actions would be translated to

a change in the video output. Simultaneously, four pressure pads connected to a makey-

makey board will result in audio output change.

This system design catered to individual strengths of the members of the group.

Gus’ main focus was designing and writing a max patch to read visual input from the

webcam in order to modify video. He also mixed the audio for the piece. Logan focused

on the video art piece, designing and editing the visuals in final cut pro. Annie designed

and constructed the pressure pads.

As with any system, we ran into more than a few snags along the way. From a

half-baked idea to implement LEDs into the foot-pads, to last minute coding bugs, we all

ended up learning a great deal about each other’s specialty through team troubleshooting.

In creating the pressure pads, Annie spent a lot of time choosing materials to build with,

stumbling upon the frosted corrugated plastic as an alternative to plexiglass when the

plan was to light the pads internally. The next challenge was to ensure that the circuit

would remain broken until the pad was pressed, while having a reliable connection at that

point. In coding, Gus first had to find ways to practically realize our vision of sensing

shadow input via webcam. Next, he spent time researching and field testing the options

for sensing systems and taught himself various programming techniques needed to

finalize the max patch. Ultimately, the largest hurdles included incompatibility on his

computer (resulting in an untestable patch), an unanticipated bug in the final presentation

– We theorize that the jitter object was at fault in the end, failing to output. Logan spent

his time designing and editing the visual output feature of the instillation. Major

troubleshooting on his end included rendering video to be compatible with the max patch

as well as a last minute scramble to edit in white bars (for sensing) when the code failed

to overlay them.

In the end, our instillation was extremely successful in sensing pressure cues and

subsequently mixing audio output, and the video added to the overall aesthetic and

ambient quality of the experience despite not reacting appropriately to the shadow-cue.