Concept & Artistic Intention
For my final project, I am building an interactive digital environment that revolves around a flock of autonomous agents.
The artistic intention is to explore the tension between curiosity and fear in nature. The user plays the role of a foreign, glowing entity intruding on this dark abyss. I want the environment to feel organic, slightly eerie, and highly responsive to physical presence, stepping away from standard mouse-and-keyboard inputs.
Interaction Methodology
To achieve an unconventional interface, I will use ml5.js (Handpose) to track the user’s hand via webcam. The user’s hand will act as a massive physical force field within the simulation.
The interaction is mapped to specific hand gestures:
-
Neutral / No Hand: The boids exhibit standard, calm flocking behavior (wandering, aligning).
-
Closed Fist: The sketch interprets this as a small, dense, magnetic energy source. It triggers a Curious Attraction force. The boids tighten their formation and slowly swarm toward the hand.
-
Open Hand (Fingers Spread): The sketch interprets this as a sudden, bright flash of energy or a predator. It triggers a violent repulsion force. The flock’s cohesion drops to zero, their speed spikes, and they scatter away from the hand in a panic.
Canvas Design & User Experience
The visual aesthetic will rely heavily on blendMode(ADD) to create glowing, stacking neon colors against a near-black “abyssal” background.
The webcam feed will be horizontally flipped (so it acts like a mirror) but heavily tinted and darkened so it barely registers in the background.
To give the user immediate visual feedback of where their hand is in the digital space, a glowing orb will track their palm. The orb will change color and size based on the detected gesture (e.g., a tight cyan core for a fist, a large pulsing magenta explosion for an open hand).
Initial Explorations & Technical Plan
While I am not including the code in this proposal, I have already begun prototyping the physics. I might reuse code from the boids assignment initially to get an idea.
The biggest technical challenge I anticipate is performance. Running an $N^2$ flocking simulation (where every boid checks every other boid) at the same time as a neural network (ml5.js) is heavy on the browser.
My technical roadmap involves:
-
Optimizing the boid math by limiting interaction radii.
-
Lowering the background webcam capture resolution to speed up the ML model.
-
Refining the heuristic math that determines what constitutes a “fist” versus an “open hand” by calculating the distance between the fingertip landmarks and the palm base.