Create AR Face Filters With the Chrome Face Detection API

Welcome. My name is Jorrik Leijnsma. As Caroline said, I'm a creative front-end developer and I'm going to share something about the Chrome face detection API. It's a mouthful, but managed to say it okay.

So, I'm a front-end developer and I work for over six years as a front-end developer now. Last two years at a company called Ordina. And, what we have, we are developers. I think most of us are here. And developers come with some kind of image. Sometimes we are a bit, like, nerdy, and... But we are also problem solvers.

But there's also this shift changing where code can be fun, and also what you create with code is more and more often seen as fun. But the code part is still boring. And that's where sometimes there is this interesting thing where you also want to create no boring code. But also not the great end result. So maybe people can show hands if you have seen boring code somewhere in your life. Yeah. I think most of us have. Or have you seen code that's not boring and was more interesting? Maybe also raise hands. So maybe a few less, but still there. And the last question, maybe people have seen code that was not functional at all, but was purely for fun purposes. So also some hands over there.

With the recent years, these all kinds of developments, AI is introduced into the scene. There are more demanding users. They are expecting more and more features. And you have the problem of maybe not being noticed if you don't have these amazing features in your app or service or whatever. So you need to be divergent and show what you can do and what's possible. This Chrome Face Detection API, it's something that's behind the feature flag, meaning it's not usable right out of the box. There is a process for this shape detection API. The Chrome Face Detection API is part of that bigger shape detection API.

It includes text detection and also barcode detection. And the barcode detection is already done, it's ready to use for everyone. But that face detection and text detection are still being developed. For this, it's an important note that the face detection does not mean face recognition. So it detects your face, but is not able to recognize a face from this is this person or that's that user logging in. So it's only showing there is this face on this type of media.

So how do you enable the Chrome feature flag? So there's this link, when you open this, let me try if it works. No, that needs to open. Let me copy paste it in then. It's not not not a valid normal link, choose application and then show options. You're not seeing this. I have to select Chrome for it to open. Google Chrome browser. So over here, let me slide it over. So now it should be visible. This is the part if you open that URL. You're seeing this experimental feature thing. And you just need to enable it. It comes with some other features as well. So keep that in mind. I recently got some trouble with selecting, for example, text on GitHub, which is not showing properly, so that also seems to be part of this something buggy going on in this web platform stuff. So that's how you enable this feature. Like this. That was way easier. So now you have enabled in your browser and now you can use it.

To use it, you start to call a new face detector. It's as simple as that. It can handle two properties. It can handle the max detected faces, which can set to I think any number, but at a certain point, your machine is not going to be fast enough to detect all those faces. And there's this fast mode which can be true or false.

And it basically is about the speed of recognizing. So and you get more features if you do not use the fast mode. So you get more data from the face. And otherwise, it's more of a narrow version you'll get back.

So now we have this face detector called and now we can detect some faces. This is everything that you need. So now with this detect function you get back all the faces that it finds with that API. This slide and the previous slide are the only things that you need from the API. All the other slides will be functions added to do something with these faces because we need to put in an image as you see. We need to do something with the faces because now we got a function which has all the faces but what do we do with it?

So let's say I have this picture and you see a face and the face has eyes, has a nose, has a mouth, has a boundary box of the face. And for it to properly get feedback you see it detects those landmarks and they are displayed in this type of object you get back. That is the faces object that is returned from detect faces. So you get the location of the points for a particular type of landmark so here you see some of the points from the right eye. You're seeing here the right eye, one of those points is detected and it's in here. So that's the whole line that surrounds the eye. So you get those points back on a 2D plane, so it's starting at the image axis, so it's going from for you top left and then moving up and down, and you got those values back and you can do stuff with them.

So first of all we need to prepare the image. So I created simple image tag and imported an image. Now I need to create this new image to get a bitmap from it and with this I need to add some functions to make sure it works. I'm checking if that face detection is available otherwise where you don't have the feature flag for example enabled it will give some errors, so you want to know it's there. And after that we can check if the images are there. So we know every data is available. And after that we can just detect the faces and with what comes back we can do that stuff. So these steps are needed, these steps to get the image ready to put into the detector. That will result in these kind of landmarks. But as I showed this was only getting the image into the detection part, we still need to do something with those landmarks.

So what happens next we need the canvas because on the canvas we are going to create the lines. The canvas needs some styling because we need to display properly so we have some styling. So the canvas matches the image size and the image itself. And with that, we're going to initialize some things.

So here you see some parts where we initialize the image, but also get the element and do something with glasses. We'll come to that later. With this, we can also need to do some math. We need to define the scale, for example, of the image because that bitmap may have a different size so we need to set all the properties right.

After we've done that, we are going to render everything into the ‑‑ oh, the code goes here. This is the canvas where you get the 2D context back. You need that for rendering those points back into the canvas. When you have that, now we have this faces part where I can have to ‑‑ two functions where I can apply something with those landmarks. I have created those functions and we are going to look at that.

First of all, from this part of the code, with those functions called, I got this glasses on the picture. It is still a static picture, it's still fun to do. But how do I get the calculations for those glasses? We'll get to that now. I did some math magic. So I needed to get the efforts coordinates of those landmarks. Because like you saw, it was this big line with this amount of points. And I need to get the efforts of all those points so I know what is the middle point of my eye. So I got part.

Then maybe it's a bit small to see but I need to filter out the landmarks. Because I needed the eyes. Sometimes they mess up the eyes or they just name them eyes instead of eye left and eye right. So I need to filter that out so I know which part of the coordinates match those landmarks. After that, I'm going to find some math on the internet on how to find some right distances from two points, how to scale everything properly and put all those values back into the SVG. And with that, I apply some things. So I draw the image, I draw the rotation, and now I have those glasses rendered as an image inside that canvas. With that, it shows that image on my face. So let's see how that looks. I'm going to open the local host. I'm going to move the Chrome browser back over. So here I have an example. This is Albert Einstein.

I sometimes need to refresh the page. Here it applied a face filter with the glasses on top of the face of Einstein.

I'm going to show you the code related to it. So move back over. Get my code. Over here. Let's zoom in a bit. Not too much.

So I have this still image for now. This is the code I've been showing. So over here at the bottom, you see some styling happening. I have the image and the compass over here. And with that, I do some stuff.

So like I said, I got all the elements. Now I open up the image of Einstein which is person 2 in this case. I create some conflicts so I know what things I would like. So I create all these scaling and all the things happening over there. I detect the faces and I have this field function where I apply the landmarks to it. And I was able to update this bottom compass which is on top of the image to show this specific elements.

So with that, I have this person over here. Let's go to public persons. I just have this list of person, just a plain image of Einstein. Let's go back to this image thing. I created these configurations where I was able to easily choose which thing I would like to put on top of the image.

So now I have this glass over here. And the glasses are, again, just a cutout picture. I think it's over here. No, it's media and then glasses. So it's over here. I had those glasses.

It's just a transparent picture of those glasses. So that gets rendered on top. With that happening, let me show you this part where I draw the landmarks. So now I have this function where I just draw the outline of those landmarks. It's basically just creating a line, a stroke from every point to every point of those landmarks.

And now I was able to see what it detects. So now when I switch back to my Chrome... Oh, not to this. If I refresh it, sometimes it needs a hard refresh. It did not update very well. So I think I need to save it before I refresh it. So over here, you now see it detected those landmarks. So you see the eyes, the nose, and the mouth, and this is where I use those green lines to get that average to apply the glasses on it. This could be extended into other elements as well.

So I also have for example, this video snippet, which I used before. It's just a recording of myself, and I put that recording into a video element. So when I open this, it's basically the same thing. It's only showing this video element with a prerecorded video. After that, create a request frame animation. I'm able to detect faces on every frame of the video, then render the frames into the canvas. So when I open this, it's there. So when I now switch over. Oh, this was actually the camera. This was the last one I wanted to show. But let's see this one first.

So with a prerecorded video, you can also do it with a live recorded video. So it's now using the webcam of my laptop. That's a different part where you use a certain element where you create the – you get the user media. It's another API, a web API, where you get the media from your device, so from your phone, from your laptop, and I'm putting that image right into the video element. So with that, I'm able to use the video of my webcam to live detect the landmarks of my face.

It had some trouble with the glasses. So for now, let's turn off the landmark part because for an AR filter, you don't want to know where all the eyes are, the mouth is. So now the lighting is a bit different, but over here, now I have my glasses on. It also follows my eyes. It's a bit janky because now it takes the average of every frame. You are able to take the average of multiple frames, and with that, you are easy to just smooth it out. It is able to turn somewhat, so it also does that. And it stops detecting at some point.

So this is very fun, where you can also do this live. For when you take a picture using your device. And the last thing I wanted to show, which is basically the more easy version of this, where you have this pre-recorded video. So this pre-recorded video, and over here, it's also detecting the faces in real-time. So, I do the same thing, where I have these glasses again, and I'm just putting all the data into that video element. I'm also recording it here, so I can export it with the glasses on. Let me get to the part where I apply the images. So, here I've got that drawLandmark function I created again, which I can turn off. So, now it's only showing the glasses. And I can go back and stop adding the glasses, but only show the landmarks for example. So, this is the data where you can go wild with. So, you have the eyes, the mouth, the nose, and you are free to do anything with that. It's nice that you have this very small part of the API, which you can use to get the landmarks, and from that you are able to do more stuff with it. This is just one showcase, I think, which is nice, that you are able to do something with it, which is some lines of code, some calculations, and if you show this on a birthday for example, like this is what I created with some text and numbers. I think people can be amazed that they know this stuff from social media, know how those filters look like, but if you are able to recreate them just yourself, that's really great.

So, with that, there is this fun part of coding, which is this is maybe less functional. There are possibilities, use cases for it, but definitely with a feature flag right now, it's not that useful in production. But this is where the fun part interests you. This passion about getting code to do fun things helps you understand more and more what the subject is about and how the coding works. And that passion for problem solving, for creating something with code, that can go together with fun. And some things don't make sense other than just being fun, just being nice, and that is the fundamental thing of passion, I think. And when you let that mental part go, then that's where the fun starts.

If you stop thinking about it, just have fun with what you do. Then, yeah, that helps being excited about things. So thanks for your attention. This is some of my social media, and I'm going to invite Caroline back on to the stage.

All right. We've got the questions. So first question up, can this API handle a stream and not just a single image? I would guess yes, because of the videos, but maybe if it's changing faces, there might be limitations? Yeah, you especially have the problem with multiple faces in the stream that you have in a photo as well, but then it just picks one of them. And in a video stream, if you have multiple faces on it, it has a hard time for every frame deciding which face to pick. So it goes more over the place and randomly picking a face for every frame. So I could imagine it also takes a little bit of time sometimes to detect the landmarks, so that might be a limitation within that.

All right, so then what could I use the face detection API for beyond AR? So this is an interesting question. It can also, for example, be really helpful on the picture. For example, the important parts of the picture are most of the time when, for example, people upload a profile picture, that a face is the most important part of it. So you can now, for example, crop that picture more clearly with the face centered, for example. Yeah, we've all had that where it cuts you off on your eyes. Yeah, or especially when you have this profile picture turned into a circle when you upload it. It's nice when you have it already centered. It's not your neck. Yeah, yeah. That's great.

All right, so you kind of touched on this when you were doing the video demonstration, but how well does it handle rotated faces? Have you tried that with images? Yeah, so it has some threshold. So at a certain point, it's going to stop detect it or the rotation doesn't match because it's turning the landmark somewhat. But you are able to add your own code to it to apply for detection. So, for example, what happens if you turn at a certain amount, it stops detecting and then starts detecting again. But you are able to compensate by, for example, turning the image a bit. So if you see the face is turning, you could apply a counter-turn on the source image or video. So especially for a video, when you see the face gets turned, then you would be able to compensate for that earlier on, so it keeps detecting the faces. Yeah. Yeah. That makes sense.

Have you noticed, or have you tried, like, is it different for different landmarks? Like, I could imagine your eyes when they turn versus a nose, maybe...

Yeah. So, this helps where most of the general parts of the face are nicely separated. So when you have the both of the eyes and the mouth, you're really able to get a clear understanding of what orientation the face is, for example.

Yeah. All right. So, it is behind a feature flag, as you mentioned. Do you have any information on when people would be able to use it in production, or maybe even other browsers?

Yeah. So right now, this is in the progress state. They are also requesting for use cases, so when you go to the blog that Chrome wrote about it, they are also asking give feedback on this, share things, what you want it to be able to do in production, especially because the barcode scanner is already in production. I think this will get there somewhere as well. No really time estimation. I think the first thing I shared about this was over a year ago, almost a year ago, and it was still in progress at that time, so we all know how the browsers deal with these changes.

Oh, absolutely, yes, for sure. You mentioned this a little bit in your first answer about the rotation, but how well does it handle multiple faces and images of videos, and do you notice a difference between multiple faces and images versus multiple faces in videos? Yeah, so this is also where the hardware comes in, because it's also depending on how fast your device is able to process these tasks. I haven't tried this with a really big group, but two or three people on screen, in camera view works okay, especially if you turn for example to fast mode. Yeah, you get the two points of the eyes, yeah, you don't need the whole outline of those eyes, so that can already be very helpful, yeah.