Adam Rokhsar
2009
materials: computer, camera, microphone
software: Max/MSP/Jitter
I created the audio for this piece using computer vision techniques, and the video was made out of analysis of the audio signal. Rubbing my finger over the camera produced a pitch, and then the zero-crossing of the resulting waveform are displayed in the screen on the top half of the screen. The camera’s output is displayed on the bottom half along with the zero-crossings of the second audio channel.
by Adam Rokhsar
I used face detection, several photos and programming in Max/MSP to make it look like the computer is watching me.
by Adam Rokhsar
When I was a therapist, my supervisor taught me that engulfment was the experience of feeling absorbed or swallowed up by another person.
Here is an exploration of engulfment by a machine: watch as the computer finds my face using automatic face detection, and then recursively averages it until each pixel becomes entirely white.
As the amount of whiteness increases, the sampling rate and bit depth of the audio decreases– this means that fewer and fewer samples of sound are taken, and that each sample is increasingly constrained to fewer values.
The original audio is the sound of an empty room, and the air conditioning.
Computers “see” only with numbers.
This is one way the computer sees me:
video and processing by Adam Rokhsar
This piece uses pitch detection to control granular synthesis parameters, and Fast Fourier Transform to extract amplitude data, which is downsampled and “bit-crushed” before the voice is re-synthesized.
The video effect is based on edge detection, using the light from the monitor to create a kind of feedback loop, and motion detection.
mew
by Adam Rokhsar
watch it performed April 3rd at Penn State.
Mew is working mostly in realtime, using computer vision algorithms to track my face, which is processed separately from the rest of the image. Additionally motion tracking data is mapped to a hidden 3D mesh covering the screen, so that when I move the parts of screen that contain the most light appear to pop out along the z-axis.
I analyze the audio signal and extract its brightness level, which is then used to control video processing and cut timing. Some of the visual debris or “glitches” were made using a hex editor and changing the file by hand. In the future I plan on implementing algorithms to do this for me.
The song was made in Max/MSP, Reason, and Protools.
This is a quick demo of some preliminary tests of using face and motion tracking. Here the music controls scaling along the z-axis, and my face is being detected, removed, altered, and replaced back into the original video. Feedback and the operation joining the processed face with the original video are mostly responsible for effects. Last part is the best, I think.