Lee Martin
Netmaker
Kesha

Building a Pixi.js Powered Web AR Camera Face Melt Effect for Kesha

2023-04-28

GAG ORDER is a suitable title for Kesha's upcoming new album because as of late the conversation surrounding her is one of litigation rather than art. When it seems that every party is preventing your “comment,” how then do you speak your mind? For Kesha, she has chosen to converse in the form of a new Rick Rubin produced album which has her digging into her uglier emotions. If you, like her, find yourself “dipping in and out of depression, gratitude, rage, and hope,” you may just connect to this new work.

Monica Seetharam and her team at RCA helped distill these themes down into an activation I was privileged to help build. We invite Kesha’s friends (and enemies) to a web based AR booth and offer them a listen of her new track “EAT THE ACID,” if they’re willing to face themselves. Naturally, the “acid” may lead to a visual that is more stipped, raw, honest, brutal, and depressing. The brief but revealing experience is recorded directly in the browser as a video and then packaged up with the GAG ORDER aesthetic for social sharing.

Visit the GAG ORDER booth today and read on to learn how this project was developed.

Camera and Face Recognition

While originally I was going to develop this project in 3D using three.js (it is a face effect after all,) I decided I could probably pull off the effect in 2D using Pixi.js instead. I figured I could save myself a bit of time this way and take a break from the string of 3D projects I've been launching lately. Also, I'm growing to love Pixi.js in my projects and knew it would be the perfect solution for a simple 2D AR effect. As with all of my camera apps, this user experience begins by gaining access to the user's device camera.

Camera Access

If you've read my dev blogs before, you've seen this pattern 1000 times but here's the Promise I use to gain access to a user's camera.

// Start camera
function startCamera() {
  // Promise
  return new Promise(async (resolve, revoke) => {
    try {
      // Get camera
      stream = await navigator.mediaDevices.getUserMedia({
        audio: false,
        video: {
          facingMode: 'user'
        }
      })

      // Replace video tag source
      video.value.srcObject = stream

      // Video ready
      video.value.onloadedmetadata = () => {
        // Resolve
        resolve()

      }

    } catch (e) {
      // Revoke
      revoke(e)

    }

  })
}

I like waiting for the onloadedmetadata event to fire because I then know the video is ready for use. Now let's figure out where the user's face is in this video stream.

Face Detection and Landmarks

I use a combination of TensorFlow and MediaPipe to determine where a user's face is located and where facial landmarks are positioned. First, we can include the necessary CDN files.

<script src="https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh/face_mesh.js" crossorigin="anonymous"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-core"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs-backend-webgl"></script>
<script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/face-landmarks-detection"></script>

Now we have the ability to load the MediaPipe face mesh model and a TensorFlow powered detector.

// Load TensorFlow model
let model = await faceLandmarksDetection.createDetector(faceLandmarksDetection.SupportedModels.MediaPipeFaceMesh, {
  runtime: 'mediapipe',
  solutionPath: 'https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh',
  maxNumFaces: 1,
  refineLandmarks: true
})

We can now use this model to estimate faces on the video. I'll call this method from a rendering loop so it is called as often as new predictions are available, storing the found face for the effect.

// Make estimation
let predictions = await model.estimateFaces(document.getElementById('video'), {
  flipHorizontal: true
})

// If predictions
if (predictions.length) {
  // Update face
  face = predictions[0]

} else {
  // No face
  face = null
  
}

Now that we have a camera video and know where a user's face is located, we can move onto creating the melting face effect in Pixi.js.

Setting Up Face Effect

I don't want to go too deep into Pixi.js in this dev blog because they have excellent docs and examples. However, here's the setup... Our final composition exists of three layers.

  1. The bottom layer is the video itself which serves as a background for the entire effect.
  2. On top of this is just the face area, which is achieved via mask using the MediaPipe data.
  3. Finally, on top of all of this is a displacement map that is used to displace just the masked face area.

Video Background

Here's how you create a video background using PIXI.Sprite and add it to the app stage.

// Initialize video background
let videoBackground = PIXI.Sprite.from(document.getElementById('video'))

// Add to app
app.stage.addChild(videoBackground)

Face

Let's also create a similar PIXI.Sprite of the video which will be used just for the face.

// Initialize face sprite
let faceSprite = PIXI.Sprite.from(document.getElementById('video'))

// Add to app
app.stage.addChild(faceSprite)

Face Mask

We'll also need a dynamic PIXI.Graphics powered polygon that will serve as the mask which will mask the user's face for selective displacement. First, let's initialize it and add it to the Pixi.js app.

// Initialize face mask
let faceMask = new PIXI.Graphic()

// Add to app
app.stage.addChild(faceMask)

Then, again in our rendering loop, we'll need to redraw the face mask as our new face data comes in. We'll do this by calculating an array of points which serve as the polygon path. Even though MediaPipe provides over 400 facial landmarks, we're only interested in the points which make up the outer oval of the face.

// Calculate path
let path = FACEMESH_FACE_OVAL.map(p => {
  return [
    face.keypoints[p[0]].x,
    face.keypoints[p[0]].y,
    face.keypoints[p[1]].x,
    face.keypoints[p[1]].y,
  ]
})

// Clear
faceMask.clear()

// Begin fill
faceMask.beginFill(0xFFFFFF)

// Draw rect
faceMask.drawPolygon(path)

// End fill
faceMask.endFill()

We now have a dynamic face mask we can use to mask the user's face.

// Mask face sprite
faceSprite.mask = faceMask

Next, let's initialize the displacement filter that will melt the face.

Displacement Filter

A PIXI.DisplacementFilter uses a displacement map to displace the pixels of an image. I wasn't too familar with displacement maps so I found just doing a bit of experimentation helped. Pure gray pixels on the displacement map means that area will not move. Whereas pure white pixels would displace the original image to the full scale of displacement declared by the filter. Since our effect is subtle, I found a series of very light white radial gradients placed around the main features of the face created a nice effect. Here's the map I ended on. Let's first create a sprite of that map.

// Displacement sprite
let displacementSprite = PIXI.Sprite.from(`/images/map.png`)

// Add to app
app.stage.addChild(displacementSprite)

Then we can initialize our displacement filter, initially setting the scale to 0 so there is no effect. We'll then add the filter to our face sprite.

// Displacement filter
let displacementFilter = new PIXI.DisplacementFilter(displacementSprite, 0)

// Add to face sprite
faceSprite.filters = [displacementFilter]

Now, there's still one thing we need to do: adjust the sizing and positioning of the displacementSprite so it sits on top of the user's face. I ended up doing this very simply by getting the bounds of the dynamic faceMask and using those values to adjust the displacement sprite. This must also occur in the app's render loop after the face mask is drawn.

// Get face mask bounds
let { x, y, height, width } = faceMask.getBounds()

// Adjust displacement sprite
displacementSprite.height = height * 2
displacementSprite.width = width * 2
displacementSprite.x = x - (width / 2)
displacementSprite.y = y - (height / 2)

I'm using double values here because it just looked better in the end. However, I did build a version of this effect that mapped the displacement map right onto the user's face using WebGL and UV maps but this simple solution worked fine.

Melting The Face

With all this setup, we can finally melt the user's face. I decided to connect the intensity of the displacement effect to the current progress of a playing audio clip. In our case, a clip from Kesha's new song "EAT THE ACID." Let's first initialize the sound object using Howler.s.

Load Audio

Here's the Promise I use to load an audio clip using Howler.js.

// Load howler
function loadHowler() {
  // Promise
  return new Promise((resolve, revoke) => {
    // Initialize sound
    sound = new Howl({
      src: ['/sounds/clip.mp3'],
      onload: () => {
        // Resolve
        resolve()

      },
      onloaderror: () => {
        // Revoke
        revoke()

      }
    })
  })
}

When we're ready, we can play the track using sound.play().

Calculate Playback Progress

Once played, in that same render loop I've been mentioning, we'll calculate a percentage of how much of the clip has played.

// Calculate percent
let percent = sound.seek() / sound.duration()

Now, we could easily use this percentage to adjust the scale of the displacement filter in a linear fashion. However, the client wanted the effect to start a little earlier so for this we'll need an ease.

Ease Face Melt

Greensock is mostly known for autoplaying animations but did you know you could pause a gsap animation initially and then interpolate over it using a provided progress? Let's do that by first initializing the animaiton we can to interpolate. In our face, we just want to adjust the y scale of the displacement filter using a "power1.out" ease. That -175 is the total amount I want the scale to adjust. Again, this value was discovered using trial and error.

// Load interpolate
interp = gsap.to(displacementFilter.scale, {
  paused: true,
  ease: "power1.out",
  y: -175
})

Now, after we calculate the sound playback percent, we can pass it to the interpolated animation.

// Adjust interp progress
interp.progress(percent)

When the clip begins to play, the displacent filter scale will adjust, causing the face to melt using our specificed ease.

If you're curious about what might also be possible when using Pixi.js for face effects, check out some of the experiments I shared while developing this project.

Record and Share

While all this is occuring, we record the Pixi.js canvas as a video using MediaRecorder and then provide that video back to the user with a one-click share button powered by the Web Share API. I love Web APIs.

Acknowledgements

Thanks again to Monica and her team at RCA for bringing me in on this. It is a massive honor to play a tiny part in such a huge moment of Kesha’s career. As it turns out, I actually built something for Kesha over 13 years ago. So long, I can’t remember what it was. Needless to say, the time of superficial and colorful Kesha is over. Don’t be surprised if shit gets ugly. GAG ORDER is out May 19. “EAT THE ACID” and “FINE LINE” out now.

Helpful?
Buy Me A Coffee