Week 5 – Painterize by Dachi

Sketch:  (can’t work without my server for now so just pasting for code and demonstration purposes)

Image based on prompt “Designing interactive media project about art generation” with Van Gogh effect.

Concept Inspiration

As a technology enthusiast with a keen interest in machine learning, I’ve been fascinated by the recent advancements in generative AI, particularly in the realm of image generation. While I don’t have the expertise to create a generative AI model from scratch, I saw an exciting opportunity to explore the possibilities of generative art by incorporating existing AI image generation tools.

My goal was to create a smooth, integrated experience that combines the power of AI-generated images with classic artistic styles. The idea of applying different painter themes to AI-generated images came to mind as a way to blend cutting-edge technology with traditional art forms. For my initial experiment, I chose to focus on the distinctive style of Vincent van Gogh, known for his bold colors and expressive brushstrokes.

Development Process

The development process consisted of two main components:

1. Backend Development: A Node.js server using Express was created to handle communication with the StarryAI API. This server receives requests from the frontend, interacts with the API to generate images, and serves these images back to the client.
2. Frontend Development: The user interface and image processing were implemented using p5.js. This includes the input form for text prompts, display of generated images, and application of the Van Gogh effect.

Initially, I attempted to implement everything in p5.js, but API security constraints necessitated the creation of a separate backend.

Implementation Details

The application works as follows:

1. The user enters a text prompt in the web interface.
2. The frontend sends a request to the Node.js server.
3. The server communicates with the StarryAI API to generate an image.
4. The generated image is saved on the server and its path is sent back to the frontend.
5. The frontend displays the generated image.
6. The user can apply the Van Gogh effect, which uses a custom algorithm to create a painterly style.

A key component of the project is the Van Gogh effect algorithm:

 

// Function to apply the Van Gogh effect to the generated image
function applyVanGoghEffect() {
  if (!generatedImage) {
    statusText.html('Please generate an image first');
    return;
  }

  vanGoghEffect = true;
  statusText.html('Applying Van Gogh effect...');

  // Prepare the image for processing
  generatedImage.loadPixels();

  // Create Poisson disc sampler and line objects
  poisson = new PoissonDiscSampler();
  lines = new LineMom(poisson.ordered);

  // Set up canvas for drawing the effect
  background(model.backgroundbrightness);
  strokeWeight(model.linewidth);
  noFill();

  redraw();  // Force a redraw to apply the effect

  statusText.html('Van Gogh effect applied');
}

This function applies a custom effect that mimics Van Gogh’s style using Poisson disc sampling and a swirling line algorithm.

Challenges

The main challenges encountered during this project were:

1. Implementing secure API communication: API security constraints led to the development of a separate backend, which added complexity to the project architecture.
2. Managing asynchronous operations in the image generation process: The AI image generation process is not instantaneous, which required implementing a waiting mechanism in the backend. Here’s how it works:
   • When the server receives a request to generate an image, it initiates the process with the StarryAI API.
   • The API responds with a creation ID, but the image isn’t ready immediately.
   • The server then enters a polling loop, repeatedly checking the status of the image generation process:

1. • This loop continues until the image is ready or an error occurs.
   • Once the image is ready, it’s downloaded and saved on the server.
   • Finally, the image path is sent back to the frontend.
   • This process ensures that the frontend doesn’t hang while waiting for the image, but it also means managing potential timeout issues and providing appropriate feedback to the user.

3. Integrating the AI image generation with the Van Gogh effect seamlessly: Ensuring that the generated image could be smoothly processed by the Van Gogh effect algorithm required careful handling of image data.

4. Ensuring smooth user experience: Managing the state of the application across image generation and styling and providing appropriate feedback to the user during potentially long wait times, was crucial for a good user experience.

Reflection

This project provided valuable experience in several areas:

1. Working with external APIs and handling asynchronous operations
2. Developing full-stack applications with Node.js and p5.js
3. Integrating different technologies (AI image generation and artistic styling) into a cohesive application

While the result is a functional prototype rather than a polished product, it successfully demonstrates the concept of combining AI-generated images with artistic post-processing.

Future Improvements

Potential areas for future development include:

1. Implementing additional artistic styles
2. Refining the user interface for a better user experience
3. Adding functionality to save generated artworks
4. Optimizing the integration between image generation and styling for better performance
5. Allowing user customization of effect parameters

These improvements could enhance the application’s functionality and user engagement.

References:

Coding Train

p5.js Web Editor | curl flowfield lineobjects image (p5js.org)