Multimedia - Fall 2006

Paul Gazzillo



Sound Mirror (12/21/2006)

A multiple pitch detector that translates sound input into frequencies, discovers the fundamental pitches and their overtones, and finds the closest corresponding note on the piano. The sound mirror can record the notes it hears and play them back forwards or backwards with a synthesizer.
The frequencies are displayed as colored circles. Fundamental frequencies are the solid circles and their overtones are unfilled circles of the same color. The volume of each frequency is represented by the size of the circles. The fundamental frequencies are also displayed as highlighted keys on the piano. The closest key on the piano to the frequency is used. How close the frequency is to the piano note is represented by the red line in the box at the bottom.

Video Sliding Tiles (10/07/2006)

The sliding tiles puzzle with video.

Rescue (10/12/2006)

Become an heroic circle and rescue the triangles and squares.

Stormy Weather (10/06/2006)

Use a microphone or a keyboard to control the strength of a storm.

Video Filters (10/04/2006)

My first crack at playing with the pixels of captured video.

Arm (10/04/2006)

A simple simulation of an arm.

Player Piano (09/30/2006)

A player piano. See the code for routines to generate the frequencies for all the keys in a piano.

Song Game (09/28/2006)

Sing your way through obstacles. (Mouse only right now)

Simple Instrument (09/29/2006)

A basic musical instrument to learn the Ess library.

Pong (09/28/2006)

The classic game, play against the computer, or watch the computer play itself.

Rough Project Ideas

Counterpoint from Video

Type: Sound to Video

Description: Automatic music generation that is influenced by what is happening on camera, e.g. dark colors, soft music, bright colors, loud music, rapid movement active/fast/atonal music, no movement, calm music, blue/dark colors, minor music, bright/yellow colors, major music. Chord progressions could be automatic, influenced by camera, or influenced by specific hand movements.

Hand Movement Recognition

Type: Video to Video

Recognize the location of the hand and arm such that a single finger (or the point an open hand makes) is the location of the hand. This can be used to move objects around the screen, draw, or control sliders and other widgets.

A simpler possibility is to paint a colorful dot (or wear a glove with a dot on it), and track the dot to acheive above objectives in a simpler way.

First project ideas

Zeus's Apocalypse

Type: Video to Video

Description: Become Zeus and use lightning bolts to destroy things arbitrarily. Represented as a cloud moving across the sky, the player makes a throwing motion with their hand that controls where and how fast the lightning bolt moves (it's a god, he doesn't have to obey physics).

Details: Three things must be detected, whether the hand is open or closed and the velocity, direction and speed. These three parameters set the initial direction and speed of the lightning bolt. Instead of lightning bolts, this could be a wormwood star, rocks, etc. A thunder sound would accompany the bolt.

Sound Mirror

Type: Sound to Sound

Description: An echo is to sound as a mirror is to light...almost. A single mirror shows the image flipped on a vertical axis, and funhouse mirrors bend light to give interesting modifications of an image. Similarly, this project would echo back modified versions of sound, possibly by filtering it, but mostly by discovering the pitch and rhythm of the sound input and playing it back. The song gleaned from the sound in can be mirrored back after a given time in reverse, upside-down (melodically), or in other ways composers use to modify a melody. Perhaps the person's sound can be played back with the mirrored melody creating an augmented sound reality.

Ninja Game

Type: Sound to Video

Description: A simple side-scrolling fighting game. Arrow keys control the direction of the ninja, but voice pitch control the attacking moves and a soundtrack.

Details: There would be three pitchs, high, medium, and low. High is a kick, medium is a punch, and low is a low attack. An ascending pitch causes the ninja to jump (ascending pitch with a longer high pitch at the end is a jump kick), a descending pitch holding the low pitch causes the ninja to duck until the pitch ends. Other pitch patterns, oscillating for example, could control bomb-throwing or other actions.

Automatic Location Update

Type: Cell Phone with AR

Description: By pointing the cell phone camera in your location, your myspace/facebook/etc location is automatically updated.

Details: Need black/white patterns placed in home, office, locker, etc. When the camera is pointed at them, it can update your location on the internet automatically.

Project Projections

  1. Mirror that promotes interpersonal understanding by changing a person's facial features: skin color, size, cheekbones, eye shape, hair color, so that one can see what one looks like as another race or ethnicity.
  2. Some kind of air instrument that automatically harmonizes or plays counterpoint on top of what the user is playing.
  3. An instrument that plays a song reflecting the environment based on video inputs: color, size, roughness, smoothness; or human inputs: heartrate, movement, position.
  4. A device that measures a persons breathing rate and heart rate and projects synchronized animations of the heart and lungs on their chest.


Course page