GestureFX

Your webcam becomes an instrument: move your hand and it plays. Three coloured zones, each with its own sound and effect; the height of your hand sets the pitch. In the browser, nothing installed.

From gestures to position

I started with gestures: open hand, fist, finger count. They flickered at the edges and needed smoothing, which adds delay, and nobody knew which gesture did what. So position became the instrument: each zone is one octave (110-220, 220-440, 440-880 Hz) and the sound follows your hand continuously, like a theremin. The zones are drawn on screen, so it explains itself.

Small things that cost time

• Left was right. Webcam coordinates aren’t mirrored; one line fixed it (x = 1 - x).
• Silence until you click. Browsers only allow audio after a click or tap.

Decisions

• Tracking happens in the browser. Video never leaves your device; speed depends on your hardware.
• A synth instead of samples. Any pitch is available and there is nothing to download, but it does sound more like a test tone.

What I’d do differently

Keep one oscillator and glide its pitch instead of recreating it every frame, remove the per-frame logging, and measure latency instead of judging it by feel.