Midterm – Sonic Threads

Concept:

For this project, I wanted to use what we learned in data visualization and particle systems to visualize Palestinian folklore. I want to visualize sound waves through the lens of Palestinian embroidery. 

The song I picked has an Arabic version and an English Remix.  Thus, Palestinian folklore is known to be one of the vested tools to record events and phenomena. For the Palestinians, historically, it is women who sing these songs; they are a mirror of history and society and a contribution to crystallizing the higher goals of Palestinian society.  For this project, I wanted to use Tarweeda (a type of Palestinian folklore), which is slow singing governed by a specific rhythm that depends primarily on recitation, a melodious recitation that carries its own music. This type of music became vibrant in Palestine during the Ottoman occupation, where women would sing these songs after encoding the words and delivering secret messages to each other and to the resistance fighters. The encryption process happens by adding ‘lam” in the words at the end or final syllables. Through this lens, I want to embed the traditional visual language of Palestinian embroidery with  Palestinian Tarweed in an attempt to visualize the rhythms.

 

For the visualization aspect of the project, I looked into the Palestinian embroidery, which is usually a geometric pattern of shapes, flowers, and lines. The most famous embroidery colors are red threads on black fabric. 

Code and Process:

 

I have been inspired to work with sound waves for a while. I have been watching videos of how it works and how sound wave data can be visualized. Initially, it seemed very complex to me; however, when I watched more and more videos, I got the gist of it. I had to think of a couple of things first before I took this step because I was not certain I could do it. As a result, I decided to see if P5js can visualize data. I found p5.FFT, which is Fast Fourier Transform, is an analysis algorithm that can isolate audio frequencies within a waveform, which returns an array of the analyzed data between -1 and 1. Its waveform() represents the amplitude value at a specific time. The analyze() computes the amplitude along the frequency domain from low to high between 0 to 255, and the energy () measures the amplitude at specific frequencies or ranges. 

I began by preloading a small part of the song into P5js; I tested the P5js given example before doing my own iteration because the P5js sketch was confusing. I then created a variable that takes the p5.FFT data, and a wave data variable to take the waveform. Then, in an array of index numbers, I mapped the wave data for visual purposes and visualized it within a shape similar to what we did in class. When this part was working, I decided to add a Particle system class that is responsive to the sound. To do this, I analyzed the FFT data and read the amplitude of the data to see the numbers I needed to put into the energy to enable it to respond properly. This part took some time because I had to read a little more about the sound wave data and how to make it responsive. 

After everything was working fine, I began experimenting with the shape and how I wanted it to look. My goal was to make it look similar to the embroidery patterns, which are usually floral. I realized that I needed to take the Sine value of i for both the x and the y; I was initially doing Cos(i) for x and Sin(i) for y. Then, I added interaction by making frequency values for x and y, mapping them, and then adding them into the x and y formula for the shape.

Highlight:

let song;
let currentPlaySong = 1;

let song1;
let hasPlayed = false;
let hasPlayed1 = false;
let fft; //this is to generate the wave form Fast Fourier Transform

let r1;

//For Printing https://gokcetaskan.com/artofcode/high-quality-export
//  acording to this the size of my full screen sketch is good for high quality but the particles are not printing trying to fix this i will add them in a buffer actually realized its becuse the save condition was before the particle one

particles = [];

//button to start sketch
let button;
let started = false;

function preload() {
  song = loadSound("/sound.mp3");
  song1 = loadSound("/sound1.mp3");
}

function setup() {
  createCanvas(windowWidth, windowHeight);
  // createCanvas(windowWidth, windowHeight, SVG);
  angleMode(DEGREES);
  fft = new p5.FFT();

  //   button
  button = createButton("Start");
  button.position(width / 2, 530);
  button.mousePressed(startSketch);
}

function draw() {
  background(0);

  //added this before translate to position correctly
  if (!started) {
    text("1-move the curser to change the shape.", width / 2 - 130, 450);
    text(
      "2-Press A or a to listen to the Arabic version",
      width / 2 - 130,
      470
    );
    text("3-Press E or e to listen to the English Remix", width / 2 - 130, 490);
    text("4- Press R or r to Reset & S or s to Save", width / 2 - 130, 510);
    fill(255);
    textSize(15);
  }

  translate(width / 2, height / 2);
  //particles respond to frequencies
  fft.analyze();
  amp = fft.getEnergy(20, [200]);
  // shape

  let waveData = fft.waveform();

  if (started) {
    for (let t = -1; t <= 1; t += 0.09) {
      let freqX = map(mouseX, 0, width, 1, 10);
      let freqY = map(mouseY, 0, height, 1, 10);

      let offSet = map(mouseX, 0, width, 0, 150);
      beginShape();

      for (let i = 0; i < width; i++) {
        stroke(150, 0, 0);
        fill(0);
        strokeWeight(1);
        let index = floor(map(i, 0, 180, 0, waveData.length - 1));

        r1 = map(waveData[index], -1, 1, 100, 300);

        let x1 = r1 * sin(15 * i) * t * cos(i * freqX);
        let y1 = r1 * sin(15 * i) * sin(i * freqY);

        vertex(x1, y1);
      }
      endShape(CLOSE);

      //    if (key == 's'){
      //     save("mySVG.svg");
      //     noLoop();	}
      // }
    }

    let p = new particle();
    particles.push(p);

    for (let i = particles.length - 10; i >= 0; i--) {
      if (!particles[i].edges()) {
        particles[i].update(amp > 190);
        particles[i].show();
      } else {
        particles.splice(i, 0);
      }
    }
  }
}

//function to switch from arabic to english version of the song and reset
function keyPressed() {
  if (key === "a" || key === "A") {
    switchToSong1(); // play song 1 when 'a' is pressed
  } else if (key === "e" || key === "E") {
    switchToSong2(); // play song 2 when 'e' is pressed
  } else if (key === "r" || key === "r") {
    resetSongs(); // reset the songs when 'R' is pressed
  } else if (key === "s" || key === "S") {
    // Save the canvas as PNG without stopping the sound
    save("myPNG.png");
  }
}
//I need to add this because if i dont for some reason when I press a it does not play
function switchToSong1() {
  if (!song.isPlaying()) {
    song1.pause(); // pause song 2 if it's playing
    song.play(); // play song 1
    currentSong = 1;
  }
}

function switchToSong2() {
  if (!song1.isPlaying()) {
    song.pause(); // pause song 1 if it's playing
    song1.play(); // play song 2
    currentSong = 2;
  }
}

function resetSongs() {
  // stop both songs
  song.stop();
  song1.stop();

  // Reset to the first song
  currentSong = 1;
}

function startSketch() {
  started = true;
  button.hide(); // hide the button after starting the sketch
}

class particle {
  constructor(){

  this.pos = p5.Vector.random2D().mult(200);
  this.vel = createVector(0,0);
  this.acc = this.pos.copy().mult(random(0.001,0.0001));
    // this.r = 10;
    // this.lifetime = 255;
this.PW = random(1,5);
}
update(cond){
  this.vel.add(this.acc);
  this.pos.add(this.vel);
  
  if(cond){
    this.pos.add(this.vel);
    this.pos.add(this.vel);
    this.pos.add(this.vel);
    
  }
  
}
  //remove particles that were drawn 
  edges(){
    if(this.pos.x<-width/2 || this.pos.x>width/2 || this.pos.y <-height/2 || this.pos.y>height/2
     ){
       return true;
  } else {
    return false;
  }}
show(){
  noStroke();
   
    fill(204, 0, 0);
  ellipse(this.pos.x, this.pos.y,this.PW);
  
}}

 

In this project, I had a couple of challenges. Initially, the shape I created was reflected in particles because this is how waveforms are printed out. For some reason, even when I drew lines or other shapes to map, it did not look right. I kept experimenting with the numbers within the intended shape until I figured it out. Another challenge was saving an SVG file and a PNG file. Whenever I try to save a file, the whole system lags. According to some sources linked below, it might have been because a) it was saved before the whole sketch was drawn, which made the particles disappear, and b) having the no-loop was causing issues. Even when I had a function for saving, the sound stopped playing; as a result, I added the saving part to the KeyPressed function, which magically solved the problem. 

I am mostly proud that I was able to add other versions of the song into the system and allow the users to switch with songs, reset the whole system, and start again. I did this using conditions and pressing keys instead of the cursor interaction. Further, I create a start condition (state)  button designed in CSS to start sketch.

Pen plotting:

 

As mentioned above, I had challenges saving SVG files. The p5js version I was using was the most compatible with sound, and despite my attempts to integrate the SVG into the file, it kept lagging. Apparently, sound is computationally heavy on the system, and adding SVG was not working, but I managed to get some images and then commented on the code. Another issue was that when pen plotting, the pen would pass through a specific point so many times that the center got a little damaged. I think it adds to the aesthetics of the design. I tried pen plotting twice. The first one was a disaster, but the second was good, considering that it was a new system for me. 

PenPlotP5js

https://drive.google.com/file/d/1yTITCwDRO5CCQ72peazwVLUxmkqrV-G8/view?usp=drive_link

Sketch:

Link: https://editor.p5js.org/shn202/full/9AGLnP3I8

Future work and improvements 

I am satisfied with how my project turned out. However, I think it is always good to keep experimenting and trying new things. For future improvements, I would add more user interactions where users can play with the colors, more complex shapes, and even the particle system. I would also like to create an array of songs that can be played individually or together to create a chaotic experience. 

A3 print:

Resources: