Project Overview

This project grew out of Assignment 7. For that assignment, our class visited teamLab, and I picked up a pencil drawing of a butterfly, colored it yellow, and slid it under a scanner. Seconds later it showed up on the floor, glowing and moving between hundreds of other visitors’ creatures. Every other room at teamLab was something you walked through. That room was something you added to. That feeling stuck with me.

For Assignment 7, I tried to recreate that world in code. I built a glowing floor with wandering creatures drawn entirely in p5.js, flowers, fish, lizards, and swirling orbs of light, with a single yellow butterfly drifting through. It looked great. But nothing responded to you. The world never changed no matter what you did.

This project changes that. The butterfly becomes a flock. The floor grows coral based on where the flock goes. The world shifts between warm light and deep darkness depending on how the flock behaves. And it listens to the room through the microphone. A clap or any loud sound immediately disturbs the whole ecosystem.

Concept and Design

The visuals from Assignment 7 are kept exactly as they were. The glowing background, the ambient light blobs, the perspective floor grid, and all four creature designs carry over without any changes. What is new is the behavior layer on top of all of that.

The idea behind the project is simple: the user shapes the flock, and the flock shapes the world. Every system is connected to at least one other. The flock affects the coral. The coral affects the light on the floor. The mood of the world depends on how the flock is moving. The microphone affects the flock, the mood, and the coral all at once. Nothing works alone. Everything talks to everything else.

Implementation

Milestone 1 — Environment

The first step was getting the environment right before adding any creatures. The background is a gradient that goes from warm amber at the top to deep indigo at the bottom. Unlike Assignment 7 where the background cycled on its own, this one responds to the flock’s behavior every frame. Light shafts move slowly across the scene, each one a different width and brightness, giving the feeling of light coming through water. The ambient light blobs from Assignment 7 also shift between warm and cool colors depending on the mood.

Milestone 2 — Floor Creatures

The floor creatures from Assignment 7 come in next. All four designs are exactly the same. The new thing is that they now react to the flock. When the flock gets close, a floor creature turns toward it and starts glowing. The glow pulses slowly using a sine wave and fades when the flock moves away. In Assignment 7, the floor creatures had no idea anything else existed. Now they are aware.

Milestone 3 — The Flock

The single butterfly becomes a flock of 65 creatures, each one randomly given one of the four creature designs. So the flock is a mix of flowers, fish, lizards, and swirls all moving together as one group.

The flock uses three simple rules. Each creature avoids getting too close to its neighbors. Each one tries to match the direction its neighbors are heading. And each one tries to stay close to the group. Those three rules together produce the natural schooling behavior you see. On top of that, the entire flock chases the mouse cursor at all times. Move slowly and they follow calmly. Move fast and they chase frantically.

Milestone 4 — Coral Growth

The floor now has memory. A grid sits underneath the flock and tracks where they spend time. The longer the flock stays over an area, the more the coral grows. It goes through three stages: first a small dim dot, then a layered coral shape with orange and amber colors, then a full bloom with a soft glowing halo. When the flock leaves, the coral slowly fades back. So every session leaves a different floor depending on how you moved. Hovering the mouse near the floor makes the coral pulse gently. Holding the mouse still makes nearby coral grow faster.

Milestone 5 — Mood, Keyboard, and Microphone

The mood system ties everything together. Every frame, the sketch measures how spread out the flock is. A tight calm flock keeps the world warm and amber. A scattered flock pulls the scene into darkness where the coral glow is the only light left.

Four keyboard keys give the user dramatic control. S fires an explosive burst that sends every creature flying outward. C pulls the whole flock back together and shifts the world back toward warm light. D switches the world into deep night mode instantly. N drops a full coral bloom at the mouse position.

The microphone listens to the room the whole time. Any loud sound scatters the flock and darkens the world. When the room goes quiet, the flock slowly comes back together and the light returns.

How to Interact

Moving the mouse attracts the flock continuously. Left-clicking drops a food source the flock converges on, triggering rapid coral growth beneath it. Right-clicking sends a disturbance ring that scatters everything outward. The S key fires an explosive scatter from the flock’s center. C pulls every agent back together and shifts the world toward dusk. D locks the world into deep night where the coral bioluminescence dominates. N drops an instant coral bloom at the mouse position.

The three sliders in the top left panel give direct control over the simulation in real time. The flock size slider goes from 10 to 120 — dragging it right adds more creatures to the flock instantly, dragging it left removes them. The coral speed slider controls how fast the coral grows and recedes. The current strength slider controls how strongly the Perlin noise flow field pushes the flock around the canvas.

Code Highlights

The mood function produces one number that every visual system reads from:

function calcMood() {
  if (flock.length < 2) return 0;
  let total = 0; let count = 0;
  for (let i = 0; i < flock.length; i += 3) {
    for (let j = i + 1; j < min(i + 8, flock.length); j++) {
      total += dist(flock[i].pos.x, flock[i].pos.y,
                    flock[j].pos.x, flock[j].pos.y);
      count++;
    }
  }
  return constrain(map(total / count, 38, 200, 0, 1) + micLevel * 0.65, 0, 1);
}

The coral growth responds to three things at once: flock position, mouse proximity, and room volume:

if (above > 0) {
  this.cells[i] = min(1, this.cells[i] +
    growRate * above * soundSuppression * (1 + mouseFactor));
} else {
  this.cells[i] = max(0, this.cells[i] - decayRate);
}

Challenges

The slider panel was the hardest thing to fix. When you create HTML sliders inside p5.js, the canvas sits on top of them and blocks all clicks. The fix was building the panel directly in index.html, placing it above the canvas using CSS, and setting the canvas to ignore pointer events. Mouse clicks then had to be tracked manually through window event listeners.

Performance was the other challenge. Checking every flock agent against every coral cell every frame gets slow quickly. The fix was only checking agents within a short radius of each cell, which kept everything running smoothly.

Reflection

The mood system surprised me the most. Before it was connected, the sketch looked interesting but felt the same no matter how you used it. Once it was running, the same world produced completely different feelings depending on how you engaged with it. A quiet session where the flock clusters slowly and coral builds up feels calm. A chaotic session of scattering feels urgent and dramatic.

The microphone was the most satisfying addition. Clapping near the laptop produces an immediate visible effect that no mouse or keyboard interaction can match. That responsiveness to real-world sound was what made the project feel genuinely alive.

If I were to keep working on this, I would add sound output based on the flock’s behavior, and I would try WEBGL mode to add real depth to the floor, which I flagged as missing in my Assignment 7 reflection.

Embedded Sketch

Video Documentation

https://www.loom.com/share/c1f8f08127154339aac409b17878e322

References

Shiffman, D. (2024). The Nature of Code, v.2. https://natureofcode.com

Reynolds, C. (1999). Steering behaviors for autonomous characters. Game Developers Conference.

Wolfram, S. (2002). A New Kind of Science. Wolfram Media.

Gardner, M. (1970). Mathematical games: The fantastic combinations of John Conway’s new solitaire game “Life.” Scientific American, 223(4), 120–123.

Assignment 7 – https://decodingnature.nyuadim.com/2026/03/24/buernortey-assignment-7/

p5.sound library documentation. https://p5js.org/reference/p5.sound

AI Disclosure: I used Claude (Anthropic) to help organize and write parts of this documentation. I also used it to assist with debugging specific parts of the code and thinking through implementation details. The core concept, design decisions, and overall direction of the project were my own.

Project Overview

For my final project, I created an interactive desert-inspired particle system called Traces of Wind. The project is inspired by the desert as a grounding space. For me, the desert is a place where I can slow down, breathe, and let go of negative energy. I wanted to turn that feeling into an interactive environment where the user shapes the wind and watches the sand respond.

The screen shows a bright desert scene with a blue sky, a hot sun, and a field of moving sand particles. The user controls the wind using the mouse. When the mouse moves quickly, the wind becomes stronger and the sand scatters. When the mouse slows down or stops, the sand begins to settle again.

The project does not allow the user to draw directly. Instead, the user influences a natural force, and the environment responds through movement. This connects to the idea of emergence because the final visual patterns come from many small particles following simple rules.

Traces of Wind

Interaction

The interaction is simple and direct: the mouse controls the wind.

As the user moves the mouse across the canvas, the sand particles nearby react to the direction and speed of the movement. Fast movement creates a stronger force, while slower movement creates softer motion. The user can immediately see how their actions affect the environment.

I also added darker grains into the sand. These darker grains represent heavier or negative energy. When the user moves the mouse near them, they scatter and slowly fade, as if the desert wind is carrying them away.

The user can also press R to reset the darker grains and begin again.

Implementation Details

The visual scene is built in layers. First, I created the blue sky using a gradient. Then I added a bright sun with multiple transparent circles around it to create a hot glowing effect. The sand area is drawn at the bottom of the canvas with soft dune lines and many small particles.

The main interaction comes from comparing the mouse’s current position with its previous position. This gives me the direction and strength of the mouse movement.

let nowMouse = createVector(mouseX, mouseY);
let wind = p5.Vector.sub(nowMouse, lastMouse);
let windPower = wind.mag();

wind.mult(0.035);

This wind vector is then applied to the particles. Particles closer to the mouse receive a stronger force, while particles farther away are less affected.

wind(w) {
  let d = dist(this.pos.x, this.pos.y, mouseX, mouseY);

  if (d < 200) {
    let force = w.copy();
    let strength = map(d, 0, 200, 1.2, 0);
    force.mult(strength);
    this.acc.add(force);
  }
}

let n = noise(this.pos.x * 0.004, this.pos.y * 0.004, frameCount * 0.006);
let angle = map(n, 0, 1, -0.5, 0.5);

this.acc.add(createVector(cos(angle), sin(angle)).mult(0.015));

this.alpha -= w.mag() * 7;

The Abyss

Concept

My final project is an interactive, digital, abyssal ecosystem built using p5.js and ml5.js (Handpose). The core metaphor places the user in a deep-sea abyss, acting as a foreign light source (your hand) interacting with a school of highly reactive, bioluminescent organisms.

Instead of treating the user merely as a cursor, the project aims to create a living, breathing environment that responds not just to where the user is, but how they are acting. Through computer vision and audio reactivity, the ecosystem shifts between states of calm curiosity and chaotic panic, simulating a delicate, emotional underwater environment.

Sketch Link

Ml5.js won’t work on the blog so click on the link to view the project.

Video Documentation

Process, Milestones and Challenges

Building this ecosystem was a multi-step process, evolving from a simple particle system into a complex, living simulation.

Phase 1: The Core Prototype (The Abyssal Mirror)

The initial prototype focused on getting the foundation working: rendering the flocking algorithm (boids) and hooking it up to the webcam using the ml5.js Handpose model.

• The Challenge: Raw webcam data is incredibly jittery, which made the boids twitch erratically.

• The Solution: I implemented a handExistenceBuffer to prevent the hand from “disappearing” on dropped frames, and used lerp() (linear interpolation) to smooth out the tracking coordinates.

• Basic Heuristics: I started with simple interactions by calculating the average distance of the fingertips from the palm. A low distance triggered a FIST (attract), and a high distance triggered an OPEN hand (repel).

Visually, I knew I wanted it to be a bioluminescent vibe, so I used blendmode(ADD) and neon colors for the beings, taking inspiration from an old assignment. However, I wanted to make it visually differentiable from the assignment, so more on that later.

One of the challenges was getting the revolving animation working. Here is what the boids would do initially when I held up my fist, when I wanted them to orbit it out of curiosity for this new life source:

As you can see, they would just group together, intially in local groups, and then all of those would mesh into one, creating a confusing visual where all the fish were stacked on each other and moving as one. I took inspiration from Afra’s Assignment 9 where her sketch included a working revolving mechanism with the boids evenly spaced out instead of conjoining and used that sketch to make the boids orbit my fist in my sketch.

You can see how different it looks here from before, and how much closer it is to my vision of orbiting around this foreign light source. Basically, The FIST interaction was upgraded from a simple magnet to an orbital force. By calculating tangent vectors, the creatures now swirl gracefully around a closed fist like moths around a lantern.

Phase 2: Breathing Life into the Ecosystem

Once the logic worked, the geometric shapes felt too robotic and similar to what I’ve previously done. I wanted the creatures to feel organic and squishy.

• Fake Depth: To make the 2D canvas feel like an ocean, I introduced marine snow as you can see in the following visual. This added detail to the background to make it look more like an abyssal setting.

• Organic Undulation: I replaced the basic shapes with a custom beginShape() drawing that utilizes squash and stretch animation principles driven by a sin() wave. I also added ribbon trails that fade out using the boids’ movement history.

i.e. Adding tails

i.e Updating visuals from simple circles to more detailed shapes and fins

• Fake 3D: I also introduced a z multiplier. Boids and newly added marine snow particles are assigned a random depth value. This z value scales their size, speed, and opacity. By sorting the arrays (flock.sort((a, b) => a.z - b.z)), the background elements draw first, creating a beautiful parallax effect. You can see the different sizes reflected in the updated visual:

Phase 3: Deepening the Interaction (Pointing & Audio Input)

To make the environment feel truly responsive, I expanded the inputs beyond simple hand shapes.

• Pointing Logic: I added a specific pose check: if the index finger is extended but the others are curled, the system calculates a 2D vector from the knuckle to the fingertip. The boids now blast away in the exact direction you point. To differentiate it, they change colors depending on their behavior.

• Microphone Input: Deep sea creatures are sensitive to vibrations. I integrated p5.AudioIn() so that sudden loud noises (like clapping) trigger a sonar “ping”, lighting up the boids and playing a muffled heartbeat sound.

Phase 4: The Emotional Engine

The final layer was adding “Anxiety.” I introduced an anxietyLevel variable (from 0.0 to 1.0) that acts as the emotional memory of the ecosystem.

• Building Tension: Clapping spikes the anxiety. As anxiety rises, the environment shifts drastically. The water color transitions from a calm deep blue to a chaotic, flickering violet/red. The boids lose their cohesion and dart around frantically.

• Active Soothing: If the user moves their hand very slowly (distMoved < 1.5) and holds a gentle pose, a stillnessTimer activates, slowly bringing the anxiety back down, actively soothing the ecosystem. You can see them slowing down as you calm them down.

• Immersive Audio: I layered multiple audio tracks to cement the mood. An ambient underwater rumble plays constantly. Transitioning to a FIST triggers a mysterious pulsing loop, while snapping to an OPEN hand fires off a visual shockwave and an underwater explosion sound. I really think the audio experience adds a lot to the overall feel and immersion of the interactions.

Phase 5: UI

Finally, on hearing some feedback during the presentation, I added some UI to guide the user on the experience through a landing screen with instructions. I used Gemini to help with the UI as I did not want to bother with centering the text and styling boxes and whatnot (sue me).

Reflection & Future Improvements

The transition from a sterile particle system to a breathing, emotional ecosystem was incredibly rewarding. The inclusion of the anxietyLevel variable fundamentally changed the user experience. Instead of just “using” the sketch, users must actively consider their physical presence—moving too fast or making loud noises disrupts the environment, requiring physical stillness to repair it.

Future Improvements:

• Performance Optimization: Currently, the boids use nested loops to check distances for flocking ($O(N^2)$ complexity). Implementing a Spatial Hash Grid or Quadtree would allow for hundreds more boids without dropping the framerate.

• Complex Gestures: Integrating a more robust gesture recognition system to replace my simple distance-based heuristics (perhaps a trained neural net just for specific hand symbols).

References

• Libraries: p5.js, p5.sound, and ml5.js (Handpose).

• Algorithms: Craig Reynolds’ Boids (Flocking Simulation) and Afra’s Assignment 9 sketch for the orbiting logic.

• Audio Assets: All sound effects were sourced from pixbay.com.

• AI Disclosure: Large Language Models were used during the development process to assist in debugging scope issues (lagging, which I realized later was an ml5.js issue as debugging boid performance didn’t help), optimizing the audio filter logic, brainstorming the vector math for the pointing and other hand gestures, and UI. Also used for help with documentation.

Concept

Sketch Link

For the final project, I wanted to try something with WebGL for the first time. I’ve always been fascinated by how film producers make 3D space simulations for certain movie scenes, and I think a WebGL p5 sketch is a great way to do it.

How the controls look like. The red dots are Cosmic Voids and the blue dots are Great Attractors

I made an interactive, 3D generative environment that simulates the Cosmic Web—the largest known structure in the universe. In this digital cosmos, you are not merely a spectator but the architect of galactic scaffolding. The project utilizes thousands of Dark Matter points that interact through complex gravitational laws, where the true sculpture emerges from the filaments woven between them.

The core intent of this project is to provide a tactile way to explore Large-Scale Structure and Deterministic Chaos. The environment is designed to be highly responsive, allowing users to lay out Space with universal constants in real-time.

How to Architect Your Universe:

• Navigation: Click and Drag the background to rotate your perspective. The engine distinguishes between navigation and placement, so your universe won’t warp while you are just trying to find a better view.

• Creation: Click anywhere in the 3D void to place a Great Attractor (indicated by a blue sphere). This creates a localized gravity well that pulls filaments into dense superclusters.

• Manipulation: Shift + Click to place a Cosmic Void (indicated by a red sphere). These act as repulsors, pushing filaments away to create vast, empty regions of space.

• Universal Constants: Use the Control Panel in the bottom left to modify the physics:

  • Gravity: Adjusts the intensity of the pull from Attractors.

  • Reach: Sets how far a filament can stretch to find a neighboring particle.

  • Drag: Controls the “friction” of space. Low drag creates a chaotic vacuum. High drag makes the universe feel fluid.

  • Separation: Sets a “personal space” for particles to prevent them from lagging into a single point.

• System Controls: Press ‘F’ to toggle Fullscreen, ‘S’ to save a high-resolution PNG snapshot, and Space to enter Zen Mode—hiding all UI and helpers for a pure visual experience.

Implementation Details/Milestones

To maintain a high quality experience, the project moves away from standard drawing methods and employs professional-grade optimization:

• Spatial Hash Grid: The universe is divided into 3D “buckets.” Instead of every particle checking every other particle O(N² ), each node only checks its immediate 3 * 3 * 3 neighborhood, reducing complexity to O(N).

// Spatial Hash Construction
let grid = new Map();
for (let p of matter) {
    // Convert 3D coordinates into a string key (the "bucket")
    let key = `${floor(p.pos.x / GRID_SIZE)}|${floor(p.pos.y / GRID_SIZE)}|${floor(p.pos.z / GRID_SIZE)}`;
    if (!grid.has(key)) grid.set(key, []);
    grid.get(key).push(p);
}

• Vertex Batching: Rather than thousands of individual line() calls, Kinesis uses beginShape(LINES) to send all filament data to the GPU in a single “chunk,” drastically reducing the draw-call overhead.

beginShape(LINES); // Open one big batch for the GPU
for (let p of matter) {
    // ... search neighbors ...
    if (dSq < MAX_DIST_SQ) {
        stroke(190, 40, 100, alpha);
        vertex(p.pos.x, p.pos.y, p.pos.z); // Start of line
        vertex(other.pos.x, other.pos.y, other.pos.z); // End of line
    }
}
endShape(); // Close batch

• Squared Distance Logic: To avoid expensive square-root calculations, distances are compared using the formula

  d²  = (x_1 – x_2)²  + (y_1 – y_2)² + (z_1 – z_2)²

  .

The primary technical challenge was overcoming the “Density Bottleneck”—the lag created when hundreds of particles converged into a single point.

• Before Optimization O(N²): The sketch choked at 400 particles. Any attempt to create a “Great Attractor” resulted in a frame rate drop to 12 FPS as the CPU struggled with redundant distance checks.

• After Optimization (O(N) + Culling): By implementing the Spatial Hash Grid and Proximity Culling (skipping lines between particles that are too close to be seen), the engine handles 1,200+ particles at a consistent 60 FPS, even during intense gravitational collapses.

Reflection & References 

The inclusion of Zen Mode and the Architect’s UI transforms the project from a simple simulation into a dynamic tool for creative expression. The most rewarding aspect is the emergent behavior: the way filaments naturally reorganize themselves into “walls” and “voids” mirrors the actual structural evolution of our universe.

• Inspiration: The Millennium Simulation Project; The Nature of Code by Daniel Shiffman.

• Technical Foundations: p5.js WebGL and DOM documentation.

• AI Disclosure: The optimized logic and some debugging efforts were developed using Gemini 3 Flash.

Project Overview

This project explores a hand-tracked 3D interactive environment where a topology grid and boids react to user gestures.
The core concept combines three ideas I enjoyed most during the course: simple rule-based systems, satisfying emergent movement, and 3D simulation.
The final direction uses open-palm and clenched-fist gestures to attract/repel boids while the hand also influences a dynamic topology surface.

The design goal was to keep the visual language minimal and angular: mesh-line hands, a dot-based topology map, and color-coded boid states.
Instead of using gestures to rotate the world, gestures directly affect behavior inside the world.

Inspiration

I was initially unsure where to start, so I researched interactive installations and reflected on previous class work.
The project direction became clearer after revisiting hand recognition from our ml5.js session and combining it with topology and boids.

Pulse Topology by Atelier Lozano-Hemmer inspired the topology-grid idea.

Pulse Island I by the same artist inspired a calmer movement pacing.

Gesture examples from the Hand-Tracked Particle Simulator helped frame interaction possibilities, but also clarified what not to do
(gesture as pure camera/object rotation felt less natural for this project).

Seeing virtual hands in 3D (similar to headset experiences) became an important target for immersion.

Video Documentation

Final interaction and visual output demo:

Github Repository:
https://github.com/ssl9619/Decoding-Nature-Hand-Gesture-Final-Project/

Implementation Details

Milestone 1: ml5.js hand tracking in 3D (flat result)

First attempt used ml5.js with depth estimation to push landmarks in z-space.
While it partially worked for forward/back movement, rotation and depth consistency were limited.

Milestone 2: Switching to MediaPipe Hand Landmarker

I moved to MediaPipe Hand Landmarker for more reliable xyz output. This enabled more convincing 3D hand behavior while retaining similar landmark structure.

Milestone 3: Topology grid and shader optimization

The topology points are mapped to simplex noise in shader space. Height is mapped from noise values (-1 to 1), and brightness changes with height.
A shader implementation was required to resolve performance issues from CPU-heavy updates.

Milestone 4: Combining hand interaction and topology response

After integration, hand positions created indentation-like effects in the topology map (similar to pin-screen behavior).

Milestone 5: Adding boids interaction logic

Boids were integrated as a behavior system driven by gesture state and local interactions.
Final tuning focused on attract/repel responsiveness and state transitions.

Milestone 6: Hand Gesture recognition

For this milestone I focused on mapping hand poses into reliable interaction modes.
An open palm triggers attraction/follow behavior, while a clenched fist triggers repel/escape behavior.
The main challenge was keeping gesture classification stable frame-to-frame so boids did not rapidly flicker between states.

Milestone 7: Creating a design for the hand

I tried several visual designs for the hand and ultimately returned to a style close to the original skeleton, but built from 3D forms.

First, I attempted an outline-style hand, but I was not satisfied with the overall look.

Second, I tried ellipses with a billboard approach, but visibility and depth readability were weak.

Finally, I switched to spheres and cylinders with mesh lines. It is not perfect, but it gave the best balance of clarity and style among the three approaches, and it remains an area for future improvement.

Boids Behavior and State Machine

• Wander (yellow): default movement state.
• Follow (blue): triggered by open palm when boids are within interaction distance.
• Escape (red): triggered by clenched fist; boids return to wander after enough distance.
• Contagion behavior: wandering can spread to nearby boids after prolonged following.
• Close-range behavior: boids can orbit around the hand during attraction.

Hand Recognition Pipeline

The project started with ml5.js but switched to MediaPipe for stronger 3D landmark behavior.
AI assistance was used for migration and integration details.

Topology Grid

• Simplex noise drives surface displacement over time.
• Point brightness changes according to displacement height.
• Shader implementation improved runtime performance significantly.
• Grid can be circular by default, with a boolean option for rectangular mode.

3D Rendering and Navigation

• Rendering uses WEBGL canvas.
• orbitControl() allows scene inspection and confirms true 3D behavior.

Code Highlights

Highlight 1: Orientation Gizmo Drawing

function drawOrientationGizmo(center, orientation) {
  const right = p5.Vector.mult(orientation.right, GIZMO_SCALE);
  const up = p5.Vector.mult(orientation.up, GIZMO_SCALE);
  const normal = p5.Vector.mult(orientation.normal, GIZMO_SCALE);

  // For some reason (unknown to man) when commenting this out the topology grid stops working
  // so instead I just made strokeWeight(0) and wont ask any more questions
  strokeWeight(0);
  stroke(255, 90, 90);
  line(center.x, center.y, center.z, center.x + right.x, center.y + right.y, center.z + right.z);
  stroke(90, 255, 120);
  line(center.x, center.y, center.z, center.x + up.x, center.y + up.y, center.z + up.z);
  stroke(90, 150, 255);
  line(center.x, center.y, center.z, center.x + normal.x, center.y + normal.y, center.z + normal.z);
}

This is a highlight for all the wrong reasons. So really, its more of a lowlight. When changing from ml5.js to MediaPipe Hand Landmarker I relied heavily on AI to switch for me given it was my first time. In doing so it created this function. All it does is shows 3 lines perpendicular to each other with colors for each axis x,y,z and its at the center of the hand. Why? Not sure, but I tried removing the function from the update loop and suddenly the topology grid would stop working. Weird I thought, I’ll try just commenting out the lines relating to drawing the lines but I had the same issue. I realized at this point that perhaps its due to the topology grid being rendered using a shader and I didn’t really understand how that worked. I thought to myself, I don’t know exaclty why it doesn’t work, and I don’t want to find out why, because I don’t have much time or patience to do so. Lazily, I set the strokeWeight to zero and moved on.

Highlight 2: updateState() in boids3d.js

updateState(connectedWanderCount, mode, target) {
  const config = this.config;
  const stateConfig = config.state;
  const distToTarget = p5.Vector.dist(this.pos, target);

  if (this.wanderCooldown > 0) {
    this.wanderCooldown -= 1;
  }

  if (mode === "repel") {
    if (this.state !== "escape" && distToTarget < stateConfig.escapeTriggerDistance) {
      this.enterEscapeState();
    }
    if (this.state === "escape") {
      if (distToTarget >= stateConfig.escapeToWanderDistance) {
        this.enterWanderState();
      }
      return;
    }
  } else if (this.state === "escape") {
    this.enterFollowState();
  }

  if (this.state === "wander") {
    this.stateTimer += 1;
    this.wanderDurationLeft -= 1;
    if (this.wanderDurationLeft <= 0) {
      this.enterFollowState();
    }
    return;
  }

  this.stateTimer += 1;
  if (this.stateTimer < stateConfig.followBeforeWanderFrames || this.wanderCooldown > 0) {
    return;
  }

  const chance =
    stateConfig.baseWanderChancePerFrame +
    connectedWanderCount * stateConfig.neighborWanderChanceBonus;
  const cappedChance = min(chance, stateConfig.maxWanderChancePerFrame);
  if (random() < cappedChance) {
    this.enterWanderState();
  }
}

The boids and the hands are perhaps my favourite part of the simulation and in large part due to their interaction with each other. Central to their interaction is this state machine which determines whether a boid should wander, escape, or follow.

Reflection

I am very happy with the final result and how the interaction reads visually.
The topology-grid performance issue was resolved through shader-based optimization, while boids still have unresolved performance costs in denser scenarios.

One known bug remains: if the simulation starts while the palm is already open, boid state updates can behave unexpectedly until performing a clench-then-open reset.

References

• ml5.js documentation
• The Nature of Code (Daniel Shiffman)
• MediaPipe Hand Landmarker documentation
• Ultraleap documentation
• Hand landmark position/index reference image
• Hand-Tracked Particle Simulator (GitHub)
• SpaceX gesture-based design video (2013)
• Atelier Lozano-Hemmer – Pulse Topology
• Atelier Lozano-Hemmer – Pulse Island

AI Disclosure: AI assistance was used for shader-based topology optimization and migration from ml5.js hand tracking to MediaPipe Hand Landmarker integration.

Where It Started

During a class field trip to teamLab, I picked up a pencil drawing of a butterfly, colored it yellow, and slid it under a scanner. Seconds later it appeared on the floor, glowing and drifting between hundreds of other visitors’ creatures. Every other room at teamLab was something you walked through. This one was something you contributed to.

For Assignment 7, I tried to recreate that world in code. It looked alive. But it was passive. Nothing responded to the user. This project fixes that.

What It Became

The single butterfly becomes a flock of mixed creatures that chase the mouse, flock together, and trail light in night mode. The floor grows coral wherever the flock lingers and slowly recedes when they leave. The world shifts between warm dusk and deep night based on how spread out the flock is. And it listens through the microphone: a clap or any loud sound scatters the flock and darkens the world instantly.

How to Interact

Move the mouse to attract the flock. Left click to drop food. Right click to release a disturbance. Press S to scatter, C to calm, D to toggle night mode, and N to bloom coral at the mouse position. Three sliders in the top left control flock size, coral speed, and current strength.

Video Presentation

https://www.loom.com/share/c1f8f08127154339aac409b17878e322

Embedded Sketch

Project Overview

I wanted to make something that felt like a real game but was wondering which topics we took in this class can make a nice game. Feeding Frenzy seemed like the perfect fit because the whole thing runs on flocking. The small fish school together for safety using separation, alignment, and cohesion. The predators use a seek force to hunt you. And the player is just another agent in the same system, subject to the same rules about size and proximity.

The core idea is a size hierarchy: you eat what’s smaller than you, you avoid what’s bigger than you. You start as a tiny glowing fish and work your way up through six schools of progressively larger fish. When you’re small the red predators hunt you. When you grow big enough they stop hunting and you can eat them too. The final state is you as the largest thing in the ocean, every group scattering at your approach.

I really liked how the emergent behavior from flocking makes this feel alive in a way that a lot of games don’t. The fishes are not deterministic, which gives the game a taste. They’re actually making decisions every frame based on separation, alignment, cohesion, and whether they sense you as a threat. When you approach a group of fishes and it splits apart around you it looks exactly like real fish behavior, and that’s because the math is the same math.

I also chose to put a proper main screen with three difficulty levels because I wanted the project to feel like a finished game. The three levels (HARD, MEDIUM, EASY) map to different player speeds, and on HARD the player speed is actually so slow that eating anything is almost impossible. Have fun trying to eat any fish before the red ones eat you up xD

How It Works

The whole project builds on three layers that I added one milestone at a time.

The foundation is the flocking system. Every boid in every school runs the same three rules: separate from neighbors that get too close, align with neighbors moving nearby, cohere toward the average position of the group. These three forces balance against each other every frame. When a threat is nearby a fourth force kicks in, a flee force that points directly away from the threat, and its weight overrides the cohesion and alignment so the group scatters instead of holding together. I really like how this means the schooling behavior and the panic behavior are the same system, just with different weights.

The second layer is the player and eating. The player uses a steering force toward whatever direction the keys are pressing, limited by a max speed and a max force. Eating detection is a simple distance check: if the player’s size is at least equal to the boid’s size minus 1, and the distance between them is within the combined radii, the boid gets eaten and the player grows.

The third layer is the tier progression and predators. I defined six fixed schools with sizes stepping from 3 up to 42. The player starts at size 14 and grows with each eat. A progress bar tracks how close the player is to the next tier unlock size. Predators use a seek force toward the player when they’re close enough and the player is still small enough to eat. Once the player reaches 75% of the predator’s size the predator stops hunting and the player can eat it instead.

Code I’m Particularly Proud Of

This is the section of the flee behavior I spent the most time getting right:

applyBehaviors(boids) {
  let playerBigger = player.sz > this.sz - 2;
  let fleeRad      = map(player.sz, 14, 80, player.sz * 7, player.sz * 3.5);
  let fleeForce    = playerBigger ? this.flee(player.pos, fleeRad) : createVector(0, 0);
  this.fleeing     = fleeForce.mag() > 0.01;

  let sep = this.separate(boids);
  let ali = this.align(boids);
  let coh = this.cohere(boids);

  sep.mult(1.8);
  ali.mult(this.fleeing ? 0.4 : 1.2);
  coh.mult(this.fleeing ? 0.25 : 1.0);
  fleeForce.mult(4.2);
  ...
}

The part I like is the fleeRad calculation. When the player is small, the flee radius is player.sz * 7 so fish start scattering from a good distance away. As the player grows the radius shrinks proportionally down to player.sz * 3.5. This means large fish don’t scatter from across the canvas, they only react when you’re actually close to them. Without this, once I was big the entire canvas would clear every time I moved anywhere, which made the late game unplayable. Shrinking the radius as you grow is what gives the final stages their completely different feel compared to the early ones.

Building It Up: Milestones & Challenges

Milestone 1: Flocking School with Mouse Threat

I started by just getting one school of fish working with the three flocking rules, using the mouse as the threat instead of a player fish. No eating, no game loop, just the behavior. I wanted to understand what the flee force actually looks like before adding anything on top of it.

This turned out to be more useful than I expected as a first step. I could just move the mouse around and watch how the group responded without any other variables in play. What I noticed immediately is that the school feels much more alive when it’s fleeing than when it’s just flocking normally. The tighter cohesion during calm states versus the complete scatter during panic is exactly the visual I wanted for the final game. I also tuned the flee force weight here. My first value was 2.0 and it barely looked like fleeing. Going to 4.2 made the scatter feel panicked and fast, which is what I wanted.

Milestone 2: Player Fish + Eating

Once the school behavior felt right I replaced the mouse threat with an actual player fish that I could steer with arrow keys. I added the eating detection and the size growth system. Still just one group at this point and no predators.

The most annoying thing to get right here was the eating radius. My first version used player.sz as the hit radius which sounds logical but in practice felt terrible because the visual glow extends well beyond the raw sz value and you’d visually overlap a fish but not eat it. I changed the check to player.sz * 1.1 + boid.sz * 0.9 which accounts for both bodies, and that immediately felt right. You eat a fish roughly when the visual bodies overlap.

I also added the trail system at this milestone. The trails were genuinely the most visually satisfying addition in the whole project. Without trails the fish feel like sprites moving on a flat screen. With trails the canvas feels like water that things are moving through. The teal-to-gold color shift on the player trail as they grow was something I added just to see what it looked like and I ended up loving it.

Milestone 3: Tiers, Predators, Full Ocean

I expanded from one school to six schools at fixed sizes, added the tier unlock progression, and added two predator fish that hunt the player when they’re small. I also did the full ocean visuals here: the persistent dark background with plankton particles drifting on a slow current, caustic light shafts from the surface, and bioluminescent color trails for every fish.

The predator logic was simpler to write than I expected because it’s just the same seek force the vehicles used in the steering assignment. The predator seeks the player when the player is small enough. The only new thing was adding the condition to stop seeking when the player grows large enough, and then adding the reverse check so the player can eat the predator when they’re big enough. The whole predator system is about 8 lines.

What took more time was the tier progression. My first version just checked if the player was bigger than a boid before allowing an eat, which worked but meant there was no sense of stages or levels. Defining the TIER_UNLOCK array of target sizes and showing the progress bar toward the next target immediately made the game feel structured. You always know what you’re working toward.

Challenge: Fish Running Away Too Fast at Large Sizes

The hardest gameplay problem I hit was in the late game. When the player grows large, the flee radius scales up with player.sz * 7, which means at size 50 the flee radius is 350px. Every school on the visible canvas scatters the moment I appear anywhere near them. I couldn’t catch anything.

The fix was the scaling flee radius I described earlier: map(player.sz, 14, 80, player.sz * 7, player.sz * 3.5). At large sizes the multiplier drops from 7 to 3.5, halving the effective detection range. Schools don’t react until I’m actually close to them. This changed the late game from frustrating to actually interesting, because at large sizes you have to carefully approach schools rather than just running into them.

I also had to tune the eating condition a few times. The original version was player.sz > boid.sz + 2 which meant the player had to be 2 units bigger. Combined with the fact that boids flee when the player is near their size, this created a window where the player was close enough to trigger the flee behavior but not close enough to actually eat them. I changed the condition to player.sz > boid.sz - 1 so the player can eat fish that are nearly the same size, which removed that frustrating gap.

Final Result & Video Documentation

Reflection

What I find most satisfying about this project is that everything interesting in it comes from the flocking system I started with in Milestone 1. The panic scatter when I approach a group of fishes, the way they reforms after I move away, the way the predator tracks me by adjusting direction every frame, the way eating feels physically right because the distances involved match the visual sizes of the fish. All of that is just separation, alignment, cohesion, seek, and flee, balanced against each other with different weights.

I also think the difficulty system is more interesting than it might look. On HARD the player speed is 2.2 which is barely faster than the boid maxSpeed of 2.8, so catching anything requires either cutting them off or cornering them against an edge. On EASY the player speed is 9.5 which makes you feel invincible. The game is the same, the behavior is the same, but a speed difference of less than 10 units changes the entire feel of the experience.

What I’d add next:

• Sound: the eat burst should have a small pop or crunch and the predator should have an ambient low tone that gets louder when it’s close
• Power-ups: a speed boost pickup that floats in the current, giving the player a brief burst of extra speed to catch a school that’s scattering away
• More predator behavior: right now predators just seek. Adding a subtle cohesion force between predators would make them loosely coordinate, which would look spectacular and make the early game more tense

The most satisfying moment in the whole project was in milestone 3 when I didn’t have a win state implemented yet. Each time you play it you can go as big as you want until the sketch starts glitching.

References

• Daniel Shiffman, The Nature of Code — flocking rules, seek/flee steering, applyForce / update pattern
• Feeding Frenzy (2004 game by SpryFox/BigFishGames) — core size hierarchy concept
• Course material on flocking, steering behaviors, and forces
• AI Disclosure: Claude (Anthropic) assisted in idea forming, speed adjusting, coloring, and debugging.

The Concept

Fractals are the one topic this semester I had basically zero context for going in. I knew the word in a vague pop-science way, but I had no idea you could generate plant-shaped organic structures from nothing but string substitution rules. That was genuinely surprising to me when I reading about it in our book Nature of Code out of curiosity to come up with an idea for this assignment.

The thing that really clicked for me when I read was the idea that you can describe a plant as a sentence. Not metaphorically, literally as a string of characters with grammar rules that expand it. You start with the single character X. You apply a rule once and get a longer string. Apply it five times and that string is tens of thousands of characters long, and when you walk it with a turtle interpreter and treat each character as a drawing instruction you get something that looks genuinely botanical. I did not draw the plant. I did not specify any branch curves or angles by hand. The structure just falls out of the grammar and I think that is really interesting.

So the sketch ended up with two modes. A recursive fractal tree where you can adjust depth and angle in real time with the keyboard and mouse, and an L-system plant that you grow generation by generation with the G key. Both show the same underlying idea, self-similarity and self-reference, but from totally different directions. The recursive tree is top-down: the rules are explicit in the code. The L-system is bottom-up: the visual structure emerges from a grammar I never directly draw.

The Code Behind It

The recursive tree is the more intuitive one to explain. The branch() function draws one line segment, then calls itself twice, once rotated left and once rotated right, with a shorter length each time. That is literally the whole thing. It stops when depth hits zero.

function branch(d, len, pal) {
  line(0, 0, 0, -len);
  translate(0, -len);

  if (d > 0) {
    push();
    rotate(treeAngle);
    branch(d - 1, len * treeLenFactor, pal);
    pop();

    push();
    rotate(-treeAngle);
    branch(d - 1, len * treeLenFactor, pal);
    pop();
  }
}

The push() and pop() around each recursive call are the part I want to highlight because they are load-bearing here, not just visual isolation. Every time you go down a branch you save the current position and rotation with push(), draw the sub-branch, then restore with pop(). Without those, after drawing the left sub-branch the turtle would be stranded at some leaf tip with no way back to the fork. The call stack combined with the matrix stack is what makes the recursion actually work spatially. I had to mess that up once before I fully understood why it works.

What I really like about this is how much control two numbers give you. treeAngle and treeLenFactor are the only two parameters that shape the entire tree, but the space they cover is huge. A narrow angle gives you a tall thin conifer, a wide angle gives you a spreading oak, and if you push treeLenFactor above 0.8 it starts producing these weird dense spirals that do not look like trees at all. I mapped treeAngle to mouse drag so you can sweep it in real time and watch the tree morph continuously. That part I really enjoyed tuning.

The L-system works differently. The grammar is just a JavaScript object:

rules = {
  'X': 'F+[[X]-X]-F[-FX]+X',
  'F': 'FF',
};

Every generation I walk the entire sentence and replace each symbol with its expansion. The sentence starts as 'X' and by generation 5 it is around 50,000 characters. I then walk that sentence again as drawing instructions: F moves forward, + and - rotate, [ and ] push and pop the matrix stack. The plant shape is never stored anywhere explicitly, it gets assembled fresh by replaying the sentence, which is a strange way to think about drawing something but it works.

Milestones and Challenges

Milestone 1: Just Getting Recursion Working

I started with the most stripped down thing possible, just a recursive tree with white strokes on black, no color, no depth parameter, stopping when the branch length drops below 4px. The only interactive thing was adjusting the angle with arrow keys. The whole sketch was maybe 25 lines.

Here is Milestone 1:

This stage was about getting the coordinate system right before the recursion got deep enough to make bugs hard to trace. The tree was drawing upside down at first because I set up the translation at the bottom of the canvas but forgot to rotate the starting direction upward. Once I added rotate(-90) in the L-system render and just started from translate(width/2, height) pointing up in the recursive tree it clicked into place.

The other thing I had to get right was the push() and pop() pattern. I read about it in the chapter but I had to break it first to really understand it. I forgot the pop() on the left branch so the right branch started from wherever the left branch had ended, which gave me this diagonal zigzag that looked nothing like a tree. Once I wrapped both calls correctly the shape snapped into place immediately.

Milestone 2: Depth-Based Color and the L-System Plant

Once the tree was structurally solid I added two things at once: depth-based color interpolation on the recursive tree, and the L-system as a second mode.

For the color I wanted the trunk to read as warm brown and the tips to read as the accent color of the palette, with a smooth transition in between. I did this by manually lerping the three RGB channels since my palette stores them as arrays:

let r = lerp(pal.branch[0], pal.trunk[0], t);
let g = lerp(pal.branch[1], pal.trunk[1], t);
let b = lerp(pal.branch[2], pal.trunk[2], t);

Where t maps from 0 at the tips to 1 at the trunk. I think this was the change that made the sketch feel finished rather than just functional. Looking at the milestone 1 screenshot and then the colored version with depth is a pretty clear difference.

Challenge: The L-System Sentence Blowing Up at Generation 5

The hardest problem was performance on the L-system. The generate() function doubles the sentence length every generation because 'F' → 'FF' doubles every F in the string. By generation 4 the sentence is tens of thousands of characters and drawing it is fine. But I had a bug early on where I was calling generate() inside draw() instead of on a keypress, which meant it was trying to re-expand the sentence 60 times per second. By the time I noticed the tab had basically frozen and the sentence was somewhere around 50 million characters.

The fix was obvious after I saw it: move generate() entirely behind the keypress handler so it only runs once when you press G. But the lesson I took from it is that exponential growth from grammar rules is not abstract, it is immediate and it will kill your program without warning. Generation 4 is totally fine. Generation 7 would be several gigabytes of string data.

The Final Result

Two fractal modes in one sketch. Recursive tree with live mouse-drag angle control and keyboard depth adjustment. L-system plant that grows generation by generation up to gen 5. Four color palettes.

Controls (recursive tree):

• arrow up/down – increase or decrease recursion depth
• arrow left/right – adjust branch angle
• drag mouse – sweep branch angle continuously

Controls (L-system):

• G – grow one generation
• R – reset to generation 0

Shared:

• Space – toggle between modes
• P – cycle color palette
• H – hide UI
• S – save frame

Reflection and Ideas for Future Work

The thing I keep thinking about is how simple the recursive tree code actually is. The branch() function is maybe 15 lines. But at depth 9 it has drawn 512 tip segments plus all the intermediate ones, and the result looks like a real tree. That ratio between code complexity and visual complexity feels different from the other techniques this semester, and I think it is because recursion is the right language for describing things that are naturally self-similar. Trees branch. Recursion branches. They match.

The L-system surprised me more though. With the recursive tree you can look at the code and trace the structure, you can see the two recursive calls and know they will produce a Y-shape at every node. With the L-system the sentence at generation 4 is 30,000 symbols and there is no way to read it and know what shape it will draw. The shape is legible in the output but completely opaque in the representation. I find that fascinating and also kind of unsettling in a good way.

What I learned:

• push() and pop() are not just for visual isolation. They are structurally necessary when you are doing recursive drawing because the matrix stack is what makes spatial recursion possible.
• Exponential growth from grammar rules is very real and dangerous if you are not careful about where you call the expand function. (feel free to try to change the sentence in the initLSystem() function and see the output yourself)
• Two numbers fully parameterize a fractal tree and the range of shapes they cover is enormous, from a conifer to an oak to something that looks nothing like a tree at all.
• L-systems produce organic structure from combinatorial substitution rules, which makes them feel so close to how biology actually works.

What I would add next:

• 3D turtle graphics: move to WEBGL and rotate the turtle in 3D space to grow proper 3D branching structures instead of flat 2D ones.

The Concept

When I started reading through what matter.js actually does well I kept coming back to the same thing: it’s a rigid body physics engine, and the most satisfying thing you can do with one is watch things fall and bounce. That sounds simple but I think there’s something genuinely compelling about getting physical simulation right visually. The randomness of a ball path through a peg field is the kind of thing you can watch for a long time.

The reference that stuck in my head was the Galton board, those wooden statistical demonstration devices where you drop balls through rows of pegs and they collect into a bell curve at the bottom. What I really like about it is that the pattern emerging at the bottom is a direct consequence of physics, not something you program explicitly. The bell curve isn’t in the code, it falls out of the geometry. That kind of emergent result is exactly what I was interested in building toward.

The sketch is a pachinko-style peg board: balls spawn at the top, fall through seven alternating rows of pegs, and collect into nine buckets at the bottom. Wind force can be pushed left or right with the arrow keys so the distribution shifts over time. The pegs change color the more times they get hit, starting at magenta and shifting toward lime green, which ends up acting as a live heatmap of ball traffic. Clicking anywhere drops a burst of twelve balls at once.

How the Matter.js Side Works

Forces are applied every frame in applyForces(). Every live ball gets a horizontal nudge via Body.applyForce() using the current wind value. Gravity is live-editable with the up and down arrow keys and writes directly to engine.gravity.y each draw cycle, so you can feel the difference between 0.1 and 3.0 .

Collision events are set up with Events.on(engine, 'collisionStart', ...). Two things happen on every collision: pegs get a glowTimer value set to 14 which drives the flash animation, and their hitCount increments so the color can shift over time. Balls also get a tiny random horizontal force on peg impact, which adds unpredictability to each bounce so paths never feel mechanical.

Events.on(engine, 'collisionStart', function(event) {
  for (let pair of event.pairs) {
    let a = pair.bodyA;
    let b = pair.bodyB;

    if (a.label === 'peg') { a.glowTimer = 14; a.hitCount++; }
    if (b.label === 'peg') { b.glowTimer = 14; b.hitCount++; }

    if (a.label === 'ball' && b.label === 'peg') {
      Body.applyForce(a, a.position, { x: random(-0.0008, 0.0008), y: 0 });
    }
    if (b.label === 'ball' && a.label === 'peg') {
      Body.applyForce(b, b.position, { x: random(-0.0008, 0.0008), y: 0 });
    }
  }
});

I like this block because it handles three completely different concerns from the same event listener: visual feedback, statistical tracking, and physics perturbation. The hitCount accumulates over the whole run so the center pegs that get hit most go lime green first while the outer ones stay magenta, which tells you the actual distribution without needing any chart.

Building It Up: Milestones & Challenges

Milestone 1: Pegs, Balls, Physics

The first step was getting the basic setup running: engine, world, static peg bodies, dynamic ball bodies, and the p5 draw loop calling Engine.update(). No collision events, no forces, no visual polish. Just confirming that matter.js and p5 play nicely together and that ball paths through the peg rows look physically believable.

Getting the staggered peg layout right was the first real problem to solve. The rows need to alternate so that every gap in one row has a peg directly below it in the next, which is what forces balls to deflect at every level rather than falling cleanly through. I spent time getting the startX calculation right so the grid stays centered regardless of column count, and tuning the spacing so the balls are large enough to look satisfying but not so large they jam between pegs.

Restitution also matters a lot more than I expected. The default is 0, so balls just thud through the pegs with no bounce and pile up directly below the spawn point. Setting it to 0.7 gave enough bounce to spread the paths out properly.

Milestone 2: Collision Events and Wind

With the base working I added the collision event listener and the wind force. The collision detection in matter.js was straightforward, but making the peg flash readable took some iteration. A single-frame color change was too subtle to notice. Storing a glowTimer counter that counts down from 14 and drives both the color and a slight radius increase made it much more visible.

Wind was the trickiest thing to tune in the whole sketch. Initial values around 0.03 sound small but in matter.js force units they were enormous and balls would fly sideways off the canvas immediately. Getting down to the 0.0005 per-keypress step size took a few rounds of testing before it read as a gentle nudge rather than a gale.

Challenge: Getting Balls to Actually Collect

The bucket and floor setup took more work to get right than I expected. Matter.js bodies collide based on their center position plus radius, so a ground body that looks visually correct can still let fast-moving balls tunnel through it if the physics body isn’t thick enough. I made the ground body significantly taller than it visually appears and positioned its center well below the canvas edge so fast balls always hit it. The bucket dividers needed their Y position calculated precisely to sit flush against the floor without a gap balls could slip through.

There was also a collisionFilter I had on the ball bodies that was incorrectly masking out collisions with all static bodies, meaning balls were passing through both pegs and the ground without interacting. Removing it fixed everything at once.

The Final Result

Balls spawn continuously at the top, fall through seven alternating rows of pegs, and collect in nine buckets at the bottom. The peg color shifts from magenta to lime green based on cumulative hit count. Arrow keys control wind and gravity. Click anywhere to burst twelve balls at the cursor.

Controls:

• Arrow keys — left/right adjusts wind, up/down adjusts gravity
• Click — burst of 12 balls at cursor

Reflection & Future Work

The thing I kept watching was the bucket distribution. When wind is near zero it builds into a rough bell curve exactly the way Galton described, but as wind increases the whole distribution slides sideways and the center buckets start emptying. When the wind reverses there’s a brief moment where the distribution goes almost flat across all nine buckets before the new bell curve forms on the other side. None of that is programmed, it just comes out of the physics.

The hitCount heatmap on the pegs is my favorite visual element in the whole sketch. It always ends up looking the same: the center columns go lime first, then it falls off toward the edges. The physics is confirming the probability distribution in real time and you can see it happening across the peg grid.

What I’d add next:

• Different ball sizes: mixing radii would create more varied paths since larger balls interact with the peg geometry differently
• Timed wind gusts: sharp short bursts instead of a manual input would create more dramatic distribution swings automatically
• Oscillating pegs: if pegs moved slowly on the horizontal axis, the board would never settle into a predictable pattern

The Plan

I sketched the architecture in three layers before improving upon the previous code i had from draft 2. Terrain was the base: a 2D grid of 1s and 0s, updated by the CA rule every eight frames. The parallel grids sat on top of terrain: fertility, crisis state, and crisis age. These are separate arrays that reference the same coordinate space. Settlements were a list of objects that sample both layers on every update tick.

The player-facing tools needed to be implemented as targeted disruptions to this system. Earthquake needed to cure plague along a beam path. Tsunami needed to clear drought along an expanding ring. Volcano needed to destroy terrain, clear famine, and introduce high-fertility land as new volcanic soil hardened. Each tool was designed as a counter to exactly one crisis type, which meant the game had an answer to every problem if you positioned it correctly.

The UI plan was minimal. A mode bar at the top center, a HUD at the top corners for steward mode, a crisis panel at the bottom left for active crises, and a custom cursor that previewed each tool’s area of effect as a ghost overlay before firing.

I also planned the feedback particles early. Founding a hamlet gets a green sparkle. A tier upgrade gets a gold burst. A plague cure gets a green label. A settlement abandonment gets grey smoke. These are the only way the player understands what is happening inside the sim at any given moment.

Step-by-Step:

Milestone 1: Grid and CA

The first thing I got working was the CA step itself. Build a grid, apply the B5678/S45678 rule, render it. This took a day. The trickiest part was the boundary condition. I decided out-of-bounds cells count as land, which biases the CA inward and prevents the edges from eroding to water, which would’ve cut off coastal settlement placement near the canvas border.

Milestone 2: Terrain Generation

Two approaches: image-sampling for Sandbox mode, procedural island for Steward mode. The image sampler reads pixel brightness and opacity to classify land vs water. The procedural island blends a radial gradient with Perlin noise at a 70/50 split and thresholds the combined value at 0.55. Four CA smoothing passes run after generation to round the jagged initial shape into something that feels geographically plausible. Each game generates a different island because noiseSeed() randomizes on startup.

Milestone 3: Settlements

Coastal detection required a four-connectivity check: is this cell land, and does it have at least one adjacent water neighbor? The spawn function samples random grid positions, checks coastality, checks minimum distance from existing settlements, checks that no crisis is active on the tile, and places a hamlet if all conditions pass. The spawn gating on food ratio was a late addition that I’m glad I made. Without it, settlements spawn into starvation immediately if the map is small.

Milestone 4: Food and Crises

The food system was straightforward once the fertility grid was in place. Crisis spreading was harder. Drought uses eight-connectivity spread with a 25% probability per tick. Famine uses four-connectivity with 20% probability (slower and more directional, mimicking how supply disruptions travel along routes). Plague is entirely settlement-to-settlement with a distance threshold and an immunity window to prevent instant re-infection after cure. Getting these three rates balanced so each crisis was dangerous but counterable took a lot of manual tuning.

Here’s a highlight of code i am proud of:

for (let j = 0; j < rows; j++) {
  for (let i = 0; i < cols; i++) {
    if (crisisGrid[j][i] === 1) {
      crisisAge[j][i]++;
      if (crisisAge[j][i] >= 2) {
        for (let dj = -1; dj <= 1; dj++) {
          for (let di = -1; di <= 1; di++) {
            if (di === 0 && dj === 0) continue;
            let nx = i + di, ny = j + dj;
            if (inBounds(nx, ny) && grid[ny][nx] === 1 && crisisGrid[ny][nx] === 0) {
              if (random() < 0.25) toAdd.push([nx, ny]);
            }
          }
        }
      }
    }
  }
}

The crisisAge check is doing important work here. A freshly seeded drought tile cannot spread on its first tick. It has to mature for at least two spread intervals before it can infect neighbors. Without that gate, the initial seed cluster would explode outward in one frame and the player would have no reaction window at all. The 25% probability per neighbor per tick means spread is visible but not instant. You can watch the tan discoloration creep across the land in real time, which is the whole point.

Famine uses the same structure but with 4-connectivity only (no diagonals) and a 20% probability, making it slower and more directional. Plague skips the grid entirely. It checks settlement-to-settlement distance and seeds infection directly on nearby objects.

Here’s an old picture of how crises were, this is before I implemented them spreading and stuff.

Milestone 5: Disaster Tools

Earthquake beams move at 0.6 cells per frame along four cardinal directions. At every integer cell the beam tip crosses, it checks for settlements within one grid cell and cures their plague while granting 600 frames of immunity. The tsunami ring expands outward and samples the ring boundary at angular increments, clearing drought on any land cell it touches. The volcano runs a 180-frame timer, erodes a central crater, then stochastically solidifies outer cells as high-fertility land at 6% per frame per cell.

Milestone 6: Steward Mode Full Pass

I wired the energy system, the crisis spawning scheduler, the win condition, and the toast/floating label feedback. The steward mode tutorial runs once per session and front-loads the counter-tool pairings: tsunami clears drought, earthquake breaks plague, volcano ends famine. The crisis panel at the bottom left updates live and tells the player exactly which tool to use. I did not want the player to guess.

Challenges and Struggles

The hardest single problem was making the CA not eat settlements. The CA rule does not know that a given land cell has a house on it. If the terrain naturally converges toward a rule that kills that cell, the settlement disappears from the grid and the next frame the update function finds a settlement sitting on water and removes it. The fix is lockSettlements: after each CA step, the function iterates all settlement grid positions and forces them back to 1. This keeps terrain alive underneath existing settlements while still allowing CA to shape the rest of the map.

The second big struggle was food balance. In early versions, the food ratio crashed to zero almost immediately because a handful of settlements consumed more than the entire island produced. I had to tune FOOD_PER_LAND and TIER_FOOD_NEED in tandem over many test runs. The values in the final version feel natural, but they represent maybe three hours of iterative tuning that is totally invisible to anyone playing the game.

Crisis scaling gave me real trouble too. Early playtests had crises that felt either trivial or instantly fatal. The damage accumulation system (30 points to trigger a tier loss or abandonment) came from thinking about health bars in a different way. Instead of instantaneous damage, crises chip away slowly. Drought does 0.04 per frame. A full drought tile on a hamlet takes about 12 seconds to force abandonment, which is long enough to respond. Getting that number wrong in either direction completely breaks the game feel.

The cursor preview system was finicky. The ghost overlay for each tool uses low-opacity strokes to show the area of effect before the player fires. The earthquake preview draws four lines at the actual beam length. The tsunami preview draws concentric ellipses at the actual ring radii. This sounds simple but it required the preview to use the same parameters as the actual tool, which meant I had to keep those values in sync. When I changed a ring’s max radius I had to remember to update the preview too. I missed this twice.

Reflection and What Comes Next

The thing I’m most satisfied with is the emergent world behavior in Steward mode. I did not script the way crises compound. A drought reduces food production, which slows growth, which makes settlements more fragile when famine hits next. I built three independent systems and their interaction produced a cascade logic that feels scripted but isn’t.

What I want to improve: the crisis visuals feel functional but not beautiful. The drought overlay is a warm tan pulse. Famine is a grey-brown desaturation. These communicate but they don’t carry atmosphere. I want to give drought a particle system, dry cracking particles drifting off land tiles. I want famine to dim the overall palette in the affected region. Plague should feel more visceral; right now the purple ring around a settlement is subtle.

The progression in Steward mode also flattens toward the end. Once you hit three or four cities you have so much food surplus and energy regeneration that crises stop being threatening. A late-game difficulty ramp, faster crisis intervals, compound crises, or a catastrophic endgame event, would fix this.

The biggest gap in the project is sound. The CA is a visual medium by default but each of these events, the founding sparkle, the tier burst, the earthquake beam, have clear sonic signatures that are missing. A procedurally pitched tone that scales with energy level during tsunami expansion would make the game feel dramatically more alive. That’s the first thing I’d add.

References

Inspirations

• Conway’s Game of Life and its B/S rule notation — the conceptual ancestor of the B5678/S45678 terrain rule used here
• DRIFT  & GALACTIC (my previous p5.js projects, S2026) 
• Kanchan Chandra’s “The Age of New Nationalisms” JTerm course at NYUAD — specifically the framework around stratified resource access as the material basis for collective identity formation

Technical Resources

• The Coding Train — Coding Challenge #85: Game of Life — Daniel Shiffman’s CA walkthrough, useful for understanding the synchronous grid update pattern
• Using Perlin Noise to Generate 2D Terrain and Water— Perlin noise used for island generation, terrain color variation, and lava flicker
• LifeWiki — B5678/S45678 — rule family documentation and behavior classification

AI Disclosure

Claude (Anthropic) assisted with mathematical tuning of several system parameters, particularly the food balance constants (FOOD_PER_LAND, TIER_FOOD_NEED), the crisis damage thresholds, and the probability values for drought and famine spread. I used it iteratively alongside manual playtesting rather than as a one-shot solution. All system architecture, rule design, and structural decisions were my own. Also, UI. I am terrible with UI. AI carried me mostly here and I believe this particular usage is OK because UI design was not a part of our class.