Concept:
“We are like artists or inventors uncovering meaning in the patterns of movement and connection.”– Zach Lieberman
“Shadowing Presence” is an experiment that delves into the idea of becoming physically part of the digital experience. Using particle systems that interact with the human body through a webcam, the project transforms participants into both performers and elements within the system. The particles respond in real-time, attracted by the participants’ motion, creating an immersive dialogue between the digital and the physical.
The aim is to create a playful digital space where users can experience how their physical presence influences a digital environment. The project also serves as a metaphor for how humans shape and are shaped by the digital systems they interact with. Through this project, I go on a journey to uncover what it means to be particles in the digital world interacting with other particles through a screen. This project is inspired by Zach Lieberman’s work, which focuses on making digital interactive environments that invite participants to become performers, seamlessly blending technology and human connection.
Highlight and Progress:
The program analyzes the pixels taken through a webcam and turns them into a black-and-white particle system. Then, a second particle system detects motion via the webcam and responds by moving toward the detected motion.
- Pages Design and Navigation:
- Main Page: The main page has an image I designed to match the program’s functionality. I moved the customization buttons closer to the center so they are easier for users to see and use. These buttons are for changing the color and the size of the interactive particle ‘object’. Further, I added some instructions to make it easier for the user to understand the function of the project. Buttons help in the navigation between the two main pages of the program. The ‘Start’ Button moves the users to the interaction page after they pick the colors and size of the particles they want to interact with.
-
User Interface Development -
Initial User Interface
-
- Experience Page (Page 1): This page is a real-time interactive environment where users can see themselves through a webcam. Here particles in the shape of a heart respond to the user’s motion while also reflecting their shadow in a particle-like form. There are two control buttons: the ‘reset’ and the ‘save’. The ‘reset’ goes back to the main page and users can change the color and size of particles, while the ‘save’ saves an image of the experience.
- WebCam Particle Interactive Particle System:
- The webcam is the primary input for capturing user motion and creating the particle interpretation of reality by analyzing the brightness of pixels. The brightness values of each pixel are calculated, and differences between the current frame and the previous frame are used to paint a sort of digital image of the user. For the motion, I process the video feed to detect movement and drive the particle system’s behavior. Users personalize their experience by customizing the color and size of particles, which then playfully respond to motion. Initially, the particles are randomly positioned on the canvas and when they detect motion they move toward it. Here, users see themselves as a mirrored reflection of particles and then interact with objects that respond to their motion, creating an immersive and interactive experience.
- Progress:
- I began designing the interface of the project and the basic interaction elements, which are the buttons. I had some bugs regarding when a button should appear and when it should disappear, so I had to reorganize the code to make it work better. As a result, I decided to make the size buttons () and the color buttons () an array, which made it easier for me to apply them to the particle system as the project progressed. I added more functions to handle the buttons for each page: startbutton() and resetbutton(). For the main page buttons, I added a function to create them and another to remove them, and then some buttons needed their functions, such as the save button. After that, I added the particle system, which is inspired by the ASCII text Image by The Coding Train. The particle systems are initially randomly placed and then move toward the positions. The particle’s color is based on brightness, and the size is between 2 and 7, mapped based on brightness so that the darker ones are smaller and the brighter ones are bigger. Now, in terms of how the particles are drawn. I initially load the video image pixels, then map the brightness of each pixel ( which takes four places in the index) from the RGB values, and then render it.
- Additionally, I added another particle system that is interactive. The Interactive particles visually represent the interaction between the user and the program. The Particle’s movement is a direct response to user activity, like a feedback loop where the user influences the behavior of the particles in real-time. It does this by detecting the motion. When the video is captured, the pixel data is loaded in an array RGBA then by comparing the current frame pixel data with the previous one the motion is detected. The threshold is used to see if the brightness difference is big enough to consider its motion or not. I visually made this particle system look like hearts because I wanted it to be different from the particles that reflect the user in the digital world.
-
let currentPage = "main"; //make the color and size buttons and array so its easier to use and manege let sizeButton = []; let colorButton = []; // for interactive particles let selectedColor = "white"; let selectedSize = 5; // interactive particle let interactiveParticles = []; // for motion let previousFrame; //navigation buttons switch btwn states let startButton; let resetButton; //Save Button let saveButton; //image load let Img1; //particle class array let particles = []; //video let video; function preload() { Img1 = loadImage("/image1.png"); } function setup() { createCanvas(1200, 800); //functions that handle the buttons in each page setUpMainButtons(); //for vid video = createCapture(VIDEO); video.size(80, 60); //for particles, generate a grid like to correspond to pixcels from webcam for better visualization for (let y = 0; y < video.height; y++) { for (let x = 0; x < video.width; x++) { // scale for display particles.push(new Particle(x * 15, y * 15)); } } // for interactive particles. rray of particles random on canvas for (let i = 0; i < 400; i++) { interactiveParticles.push( new InteractiveParticle(random(width), random(height)) ); } } function draw() { background(0); video.loadPixels(); // update canvas depending on current page if (currentPage === "main") { drawMainPage(); } else if (currentPage === "page1") { drawPage1(); drawParticleSystem(); } } //function for main page buttons function setUpMainButtons() { //Start experiance button startButton = createButton("Start"); startButton.size(150, 50); startButton.style("font-size", "38px"); startButton.style("background-color", "rgb(250,100,190)"); startButton.position(width / 2 - 100, height / 2); startButton.mousePressed(() => { currentPage = "page1"; // call the remove function for main page to remove them removeMainButtons(); //add the page 1 buttons function setUpPage1Buttons(); }); // color buttons colorButtons = [ createButton("Pinkish") .style("background-color", "rgb(255,105,180)") .mousePressed(() => (selectedColor = color(255, 105, 180))), createButton("Blueish") .style("background-color", "rgb(0,191,255)") .mousePressed(() => (selectedColor = color(0, 191, 255))), createButton("Greenish") .style("background-color", "rgb(0,255,127)") .mousePressed(() => (selectedColor = color(0, 255, 127))), ]; colorButtons[0].position(340, 300); colorButtons[1].position(400, 300); colorButtons[2].position(460, 300); // size buttons sizeButtons = [ createButton("Random") .style("background-color", "rgb(205,165,200)") .mousePressed(() => (selectedSize = random(2, 15))), createButton("Large") .style("background-color", "rgb(150,200,255)") .mousePressed(() => (selectedSize = 15)), createButton("Small") .style("background-color", "rgb(100,150,227)") .mousePressed(() => (selectedSize = 2)), ]; sizeButtons[0].position(610, 300); sizeButtons[1].position(680, 300); sizeButtons[2].position(730, 300); } // remove main page buttons function removeMainButtons() { startButton.remove(); for (let btn of colorButtons) btn.remove(); for (let btn of sizeButtons) btn.remove(); } //function page 1 buttons function setUpPage1Buttons() { // save button saveButton = createButton("Save Canvas"); saveButton.style("background-color", "rgb(100,150,227)").position(460, 10); saveButton.mousePressed(saveCanvasImage); // reset button resetButton = createButton("Reset"); resetButton.style("background-color", "rgb(150,200,255)").position(590, 10); resetButton.mousePressed(() => { currentPage = "main"; // remove page 1 buttons removePage1Buttons(); // add main page buttons instead setUpMainButtons(); }); } // remove page 1 buttons function removePage1Buttons() { saveButton.remove(); resetButton.remove(); } // main page content function drawMainPage() { image(Img1, 0, 0, 1200, 800); textFont("Courier New"); textSize(42); fill(200, random(100, 250), 200); text("Shadowing Presence", width / 2 - 260, height / 2 - 40); // instruction text textFont("Courier New"); textSize(16); fill(255); text("Welcome!", width / 2 - 50, height / 2 + 100); text( "Personalize your digital object by picking a color and ", width / 2 - 260, height / 2 + 120 ); text( "a size. Enjoy the way these object interacto with your motion.", width / 2 - 260, height / 2 + 140 ); text("Press 'F' to toggle fullscreen", width / 2 - 260, height / 2 + 165); } // page 1 content function drawPage1() { // flip horizontally so that it feels like a mirror translate(width, 0); scale(-1, 1); // process video for motion detection video.loadPixels(); if (video.pixels.length > 0) { if (!previousFrame) { previousFrame = new Uint8Array(video.pixels); } let motionPixels = detectMotion( video.pixels, previousFrame, video.width, video.height ); previousFrame = new Uint8Array(video.pixels); // draw particles for (let particle of interactiveParticles) { particle.setColor(selectedColor); particle.setSize(selectedSize); particle.update(motionPixels, video.width, video.height); particle.show(); } } } // save image function saveCanvasImage() { saveCanvas("Image", "png"); } // particle system // to control the behavior and appearance of a particle system. function drawParticleSystem() { video.loadPixels(); // updating and drawing each particle based on the video feed. for (let i = 0; i < particles.length; i++) { // cal the x and y of the current particle by taking the modulus (x) and div (y) of the particle index and the video width and height. This way i can map the 1D particle array index to its 2D pixel position const x = i % video.width; const y = floor(i / video.width); const pixelIndex = (x + y * video.width) * 4; const r = video.pixels[pixelIndex + 0]; const g = video.pixels[pixelIndex + 1]; const b = video.pixels[pixelIndex + 2]; const brightness = (r + g + b) / 3; particles[i].update(brightness); particles[i].show(); } } function keyPressed() { //press F for full screen if (key === "F" || key === "f") { // Check if 'F' is pressed let fs = fullscreen(); fullscreen(!fs); // Toggle fullscreen mode } }update(motionPixels, videoWidth, videoHeight) { // detect motion and move towards it let closestMotion = null; let closestDist = Infinity; //processes pixel data from a video feed to track motion and calculates the distance from position to nearest point of motion for (let y = 0; y < videoHeight; y++) { for (let x = 0; x < videoWidth; x++) { let index = x + y * videoWidth; if (motionPixels[index] > 0) { let motionPos = createVector( x * (width / videoWidth), y * (height / videoHeight) ); let distToMotion = p5.Vector.dist(this.pos, motionPos); if (distToMotion < closestDist) { closestDist = distToMotion; closestMotion = motionPos; } } } } // move towards closest motion if (closestMotion) { let dir = p5.Vector.sub(closestMotion, this.pos).normalize(); this.vel.add(dir).limit(12); } this.pos.add(this.vel); // to slow dawn this.vel.mult(0.8); }
User Testing:
https://drive.google.com/file/d/19wG0LIY-lJA8o-1Lv1v5EkUcFHZDqz2X/view?usp=sharing
I did the user testing with 3 people. Through the feedback process I decided to change two main things: 1) the shape of the interactive particle to heart to make it more distinguishable, 2) the placement of the customization buttons so that they can see them and click them.
Embedded Sketch:
https://editor.p5js.org/shn202/full/dSFLTnb4C
Future Work:
There is a lot of improvement that can go into the code mainly in the interactive particles. I think if I had time I would have added customization to how the object would look like a hear, a star, a smile face, a triangle, a number, etc. Additionally, I would make the interactive particle system have its own agency on it self by adding gravity to it, an attraction force between the particles, and maybe even make them attract to or steer away from the user depending on the nature of the motion detected.
Resources:
—. “Coding Challenge 166: ASCII Text Images.” YouTube, 12 Feb. 2022, www.youtube.com/watch?v=55iwMYv8tGI.
Zach Lieberman. zach.li.
Instagram. www.instagram.com/p/BLOchLcBVNz.
—. “11.4: Brightness Mirror – p5.js Tutorial.” YouTube, 1 Apr. 2016, www.youtube.com/watch?v=rNqaw8LT2ZU.
Particles Webcam Interaction by Kubi -p5.js Web Editor. editor.p5js.org/Kubi/sketches/Erd9Lt_Tz.
Webcam Particles Test by EthanHermsey -p5.js Web Editor.
editor.p5js.org/EthanHermsey/sketches/OzjX8uw4P.
CP2: Webcam Input by jeffThompson -p5.js Web Editor. editor.p5js.org/jeffThompson/sketches/ael8Y4YMB.
Input and Button. p5js.org/examples/input-elements-input-button.
Heart Shape by Mithru -p5.js Web Editor. editor.p5js.org/Mithru/sketches/Hk1N1mMQg.
Galanakis, George. “Motion Detection in Javascript – HackerNoon.com – Medium.” Medium, 21 Oct. 2024, medium.com/hackernoon/motion-detection-in-javascript-2614adea9325.
“Conditional (Ternary) Operator – JavaScript | MDN.” MDN Web Docs, 14 Nov. 2024, developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Conditional_operator.