The Milestone
Over the last week, I made massive headway on my project. The biggest hurdle was getting the complex ml5.js Handpose computer vision model to successfully talk to my custom physics and flocking simulation.
I’ve managed to get the core interaction loop fully functional! I wrote a custom pose-classification function that measures the distance between the palm and the fingertips to figure out what gesture the user is making.
The Working Prototype
(Make sure you are in a well-lit room and give the model a few seconds to load. Try making a tight fist, and then suddenly opening your hand!)
Technical Hurdles & Fixes
Getting the interaction to work was only half the battle; getting it to run smoothly is the real challenge. Combining an $N^2$ flocking simulation (where every agent checks every other agent) with a live neural network absolutely tanked my framerate at first.
To get this ready for user testing, I had to optimize heavily:
-
Performance Tuning: I tried lowering the hidden webcam capture resolution so the machine learning model had less data to crunch, but that caused some issues with hand detection in low lighting. I can also optimize the boids’ visual rendering, stripping out some of the heavier additive blending layers that were killing the GPU, and slightly reduce the total population, but I don’t know what I’ll actually do.
-
Jitter Smoothing: Raw webcam data is incredibly noisy. If I mapped the flock’s target directly to the raw hand coordinates, everything vibrated uncontrollably. I implemented vector smoothing (
lerp) so the digital orb that tracks your hand glides smoothly across the screen.
Next Steps
The sketch is finally in a place where I can put it in front of people. For user testing, my main goal is to see if the gestures (fist vs. open hand) feel intuitive, and if the visual feedback of the glowing orb clearly communicates what the user is doing to the swarm.
Honestly, I still think the direction of this project could do a 180 any time.