Build a 3D Solar System with Hand Recognition and Three.js

Let's make more sense of TypeScript by using TypeScript. Let's go to the demo. Hi, everyone, welcome back from lunch. Hi to everyone watching the demo. So our talk is going to be a little bit nicer, a little bit funner. This is the slot after lunch. It's going to be lighter. I want to actually try to start with something. Let's see. Let me try and do that. Wait. Uh-oh. You know it worked when ... No. Wait a second. It always works ten times before the show. Okay. Well, and I can just rotate it with my hands. So hello, JS Nation. I hope the other demos will work better.

So let me start a little bit about myself. I'm the chief front-end architect and web enthusiast. And I'm also an analog astronaut for the Austrian space forum. I do analog missions around the globe. It's a month of isolations in the desert in order to test space equipment and space missions.

So for that we're going to switch the talk over to the main topic of the conversation, space-themed. We'll do that. And the first question that you need to ask when you see the topic of the conversation, the topic of the talk is why. Why talk about 3JS and hand recognition? Why combine everything together? Why not talk do another talk about ES modules or another talk about performance in JavaScript? And it's because there's a great saying that says if I had eight hours to chop down a tree, I'd spend ten hours to chop down a tree. What's the point of talking about technology? You need to understand the technologies you know in order to understand them better, understand them perfectly. I like this quote better. The universe is much more complicated than you think. So don't just stay with the technologies you know. Try to expand, try to browse other technologies and other aspects.

So, for that, let's do a quick 20 minutes in and out of the talk. So, what happens if you want to build a side project? Like myself, okay? I work with JavaScript all the time, work with React all the time, I wanted to experiment a little bit with different technologies, so I go to the web and I look for libraries to use, I look for tools to use and there's a plethora of things to learn. You can either learn how to write in Solidity or Unity or TensorFlow or whatever you want and you just get confused. You don't really know what to choose. And then what we do, we just turn on Netflix, right? Okay, I'm not going to learn anything new. But we can use Netflix as a method to try and to map the technologies that we want to learn, right? So, I give you the Netflix map of the technologies that I wanted to dive into. So, you have in the top line the things that I recommended for me as a JavaScript developer, like performance and hooks and tricks and mobics and read game. And in the second line it's something that's a little bit more deeper, a little bit more complex, like architecture, server worlds, metaverse. And there's a line of whole shows that we need to avoid, like NPM list and logger and missing reference kit. Yeah, so, I chose to learn about the metaverse because I heard about it, and I said okay, I want to get somewhat deeper into it, into the metaverse. So, when I talk about the metaverse, I don't talk about these scam things of NFTs, of the things that nobody understands. But I'm thinking about this. How to create 3D experiences, right? This is like a demo of a virtual H&M store, and those are the things that I wanted to create. And in the metaverse or those kind of virtual reality or AR, you know that hand recognition interface is key because this is what you use in order to navigate. And you can do really cool things with it, right? You can, like, play a bit saber like that. Don't forget to clean after yourself.

This is a demo by push metrics. And you can do super cool things once you have a hand recognition interface. So, I wanted to create beautiful 3D worlds using only hand gestures on the web. And if I isolate just the important parts, 3D worlds hand gestures on the web. If I look at the technologies that allow me to do that, 3D worlds that can use unity, 3JS, hand gestures, TensorFlow and PyTorch in the backend. And on the web we have TFJS, we're going to talk soon.

So, if I want to build some 3D scene, you see this thing, it's rendered by Unreal 5. This is a 3D rendered scene. It's not an actual movie. It's a demo for the matrix. That's super impressive. And us as developers, we think, okay, we'll just drop some lines, right? And we'll do, okay, create a 3D landscape, we have libraries for that and we'll create this beautiful 3D scene. The problem is that there are a lot of tutorials out there that explain how to build a rotating cube. You write a lot of code, and then there's a rotating cube and then the last paragraph of the tutorial is okay. Do it, continue, after that.

So, okay, it's not like in this talk I'm going to teach you more than how to build a cube, but at least that's something that we want to go deeper. And if you want to code something which is a little bit more complex, then you need to learn 3D theory and work with vertices and pixels and you have to write some WebGL code which looks like that. And the cool thing about WebGL is that you write all of this code and in the end you get, no, a rotating cube. So, in order to actually build a rotating cube with WebGL code, that's not easy and we need something to abstract that. We need some way for us as developers to write something that we don't have to have a Ph.D. and 3D math in order to write.

So, I didn't give up. It's not something that I wanted to do, to give up. So, there's Three.js, Three.js to the rescue. For those of you who know Three.js, don't know Three.js, that's a library that knows to abstract all of the code that you've just seen and allow us to build it with easy interface. The Three.js demo page, you can see a lot of examples. Most of them or all of them are really cool. I won't go into all of them. You can do really cool stuff like this website where you take 3D models and then you render them. This is web.

Render them into the web and you can try to change the angle of things. We'll see it in a second. Another example is Bruno Simon's portfolio page which is all rendered in 3D. That's super amazing. I saw it like a few years ago and it blew my mind. It's playable. It's interactive. It's really cool. And Bruno Simon also has a 3.js course, super recommended, where he goes from the beginning of Three.js, it's called Three.js journey, from the beginning, how to write how to play with cameras, with lights, with objects, with whatever you want and then you go deeper into the technology.

When I started to write Three.js, I had GitHub co-pilot activated. I wrote four words and co-pilot completed the entire thing. It was super easy and super frustrating trying to learn new technology, where you have co-pilot just completing everything, and, yeah. So co-pilot is going to take over our jobs, that's pretty sure.

So let's talk a little bit about Three.js. Okay? Let's try to break it into how it works. So in order to render something in 3D, you need to have three main components. You have to have the scene where things are happening, you have to have some sort of a camera, so, someone who is watching the scene, and you have to have a renderer because eventually you're taking a 3D scene and trying to project it into a 2D screen, so someone needs to know how to do that. Right?

So if we'll take those things and we'll try to say, okay, what do we need for every one of them? So 3js gives us a really nice interface for that. To create a scene, you create a new scene, that's it. Camera is a little bit more complicated but not as complicated. You create a perspective camera, that's for our purposes, and you need to define the degrees, right, of the camera. You need to define the near field, so everything that is nearer than the near field won't be rendered, and the far field, which is everything that is farther than the far field won't be rendered. And eventually, you need to have the renderer that renders, takes the scene, takes the camera, and says this is the camera, this is what I will display on the screen. So, again, super easy. You just define a place in your site or in your page that you want this 3D to be in, sorry, the canvas. Then you put the scene. Then this is how the camera looks and if we will try to see how it projects things, then you can see that you have the 3D scene on the right and you can see on the left how it is projected. This is the 2D projection. I am touching this point because we are going to get back into the other direction soon. This is how it looks and eventually you render the scene.

This is the code and you will be amazed but it creates nothing. Nothing, right? Because we have a scene, we have a renderer, we have a camera, but we don't have the important things in our 3D world which are objects and lights.

In order to add lights or objects, that is super easy in 3GS, you define them. You say I want ambient light which is the light that illuminates all of the scene or point light or I want it to point somewhere. Objects, again, you define the geometry and the material of the objects and then you just place them somewhere in the scene.

If you do it correctly, you can have some really cool results from the beginning, right? You can have, for example, this kind of spherical object with the right map and you have that. You can have it illuminated and reflect the light and that's pretty cool. Yeah. Okay. You can load models from any software like Blender or anything to 3GS and I think there's actually a talk about this in the talk tomorrow. So this is really cool and I'll show you how it looks.

So, again, this is our rotating cube but you can see that it's fully 3D rendered and I can play with the lights here. I can play with the lights and you see how the spotlights, they affect the lighting of the cube. So I can show you here, I can show you the spotlight. So you see these are the spotlights and when I play with them, you see how the shadows change and the scene looks different. So this is really very simple, very easy thing to do. And you can even do more complex things. This is from the demo page of 3GS. You can have a, you play with a material and have a reflective material that reflects the scene around it and then you can get to those results, which are pretty nice. Okay. So this was 3GS, which is nice, interesting. We already knew how to do that.

But what interests me more was how we can use AI or some sort of machine learning in the web in order to interact with these 3GS scene. And what triggered it was a tweet from Charlie Gerard, Charlie, where she demonstrated how with hand gestures she can create a Figma plugin in order to control Figma with hand gestures, which I thought was pretty cool. So I wanted to dive a little bit into this technology.

So when we talk about AI or machine learning, we're talking about libraries like TensorFlow. TensorFlow is a framework for AI and machine learning in the backend, in Python. And there's a port of TensorFlow to the web called TensorFlow.js by the good people at Google, which can run models, like AI models in the web, in the browser, run inference on AI models. How does it work? With WebGL, the villain from the first act now is the TensorFlow.js. It runs on your graphic card in the web and that's pretty amazing that it can do that.

There's a lot of things you can do with TensorFlow.js. I gave a talk on how you can use TensorFlow.js to play Web games for you. So it learns how to play and win the games for you, which is nice.

There are a lot of things you can do with TensorFlow.js. And there's also a project called the teachable machine where you can definitely check it out from the people at Google where you can train the model in the web to identify all sorts of things. So remember the hot dog, not hot dog? Now you can do it with dogs, which is pretty cool.

Okay. It's like the evolution of TensorFlow.js. Because running on the graphic card is not something you can do in mobile devices or in Wicked devices. So it knows how to run in Wasm with WebAssembly. And it gives us a very dedicated models for face recognition or full body recognition, which is pretty nice. And those we can run in the web. We just need to pass a camera feed into media pipe and then you have this output.

But we're interested in hand tracking, right? So they have a model called hand pose where you can just pass a video feed and it analyzes it frame by frame and then it returns you something. So it's super easy to do that. You just fire up the detector and you just you create the detector and then you say, okay, I need to estimate the hands from the video of the camera. Okay. Nice. What does it return? It returns an array of all the points it's recognized in the hand. Okay. So for every finger you're like four or five key points. And there's an array and it looks like that. So the image from the camera goes to these points, these key points, and then you can use these key points to draw something on the canvas or interact with your app or whatever you want, which is pretty nice. And you can super impose those points on the video or not. So let's see. Let's see that in action. You see, I'm super imposing those points on the video. I don't have to do that. I can just have the video and have those points somewhere else. Okay? So this is what we can do.

Magic. Okay. So we have this amazing power of controlling the web with our hands, so let's build a cube with this power. Okay, we can use the coordinates from the hand in order to create a new cube. Remember we have the coordinates of the fingers.

But what can be the best gesture to create something? If there are Marvel fans around here, they're going to be annoyed, but this is actually the best gesture to create something, right? Just a snap. And if we talk about a snap, let's dive a little bit what does it mean to have a snap. Because we only get the coordinates, remember, we only get the coordinates from media pipe, but we need to identify the snap pattern. If we take this snapping movement, right, we can divide it into two parts. One where the fingers are closed, one where the fingers are far away. If you snap right now, you'll see that the first, the thumb and middle finger are closed and then the thumbtip is above the middle fingertip. Okay? That's important. If we write some sort of pseudocode for that, then we can detect at first the distance between the thumbtip and middle fingertip and then we can say that the thumb is above the middle tip. Again, in code it's super easy. You detect this, which is the thumb and the middle, the thumb and the middle finger are closed and then you check that they're far away, but you need to check that the thumb is above the middle finger. Okay?

And then you have to do something a little bit trickier because you have those 2D coordinates, but we want to somehow create a 3D scene. So we need to convert the vector from 2D to 3D. We'll see it in a second. It's not super easy, but it doesn't matter. So let's talk about how to move from 2D to 3D. We saw that our camera can take 3D into 2D, so in order to do that, we take the projection of the camera, we reverse it, we just normalize it, and it doesn't really matter. It doesn't really matter how to move from 2D to 3D, because you can do what I did, and just take it from step to step. Okay? 2D to 3D is not fun, but you have good people around the Web that already did this math for you, and the only thing you need to do is copy and paste. So how do we create a box? Super easy, we have the coordinates, right? So we just create the box like we saw earlier, geometry, basic material, but we just need to position them in the coordinates that we have. So position it in X, Y, and Z. Let's see it in action. And this time, it better work. So let's see if it will identify my snap now. Wait. APPLAUSE Now, I have to say, the reason it's not working smoothly is because of this guy, which confuses the camera.

So, yeah. Anyway, it creates the box, which is pretty nice. So, you did this. Which is nice. But you probably noticed it's a little bit flat, the creation of the box.

So let's do something a little more interesting. Let's make it pop with Gsup or GreenSock. It is a library for creating animations and you can do really amazing things with it like this. But the most interesting thing that we want to do with GreenSock is use its easing function. So GreenSock can give you an easing function where you can say, I want some sort of value to go elastically to another value or bounce, or whatever you want. So, in order to do it bouncy box, when we create it, we just need to scale it up to 110% and then have Gsup or GreenSock, elastically reduce it to 90%. So it looks like it was created from thin air. This is the bouncing box, remember that.

And now we want some way to move it, to move the boxes around. So, again, we want to use a flick. What's the anatomy of a flick? Just do it and you'll see. First, the thumb and index fingertips are closed, then the thumb and index fingertips are far. That's a flick.

Again, pseudocode, super easy. Distance is small. Distance is far. Magic numbers all around but it's just a trial and error.

Another thing we need to do with a flick that we didn't need to do with a snap is identify if something is going to be moveable because we need to have this object move somewhere. So, in order to do that, again, first we check that the thumb and the index are close and then we iterate all the elements in the scene and we check for every one of them what is their distance to the flick candidate that we're going to do. Again, get vector from xy, that's the 2D, 3D, but it's the same thing all around. It's not that difficult.

And once we identify this is the flick candidate, we need to save the thumb location. Why? We'll see in a second. But we need to say, okay, this is the first thing that we're going to do with the flick. And then we need to identify that the flick is being released, right? So, the distance is large. And now we get the vector, the direction of the flick. So, we have the flick candidate. We have the thumb location. And we have the vector, which is again, Euclidean math, super easy. And we'll use our friend Gsup here to animate the move of earth or whatever we use it. Like that.

And let's see that in action. Ooh, a lot of boxes. Okay. I'm going to try. Let's see if it pops now. Is it? No. It works so much better on my machine.

So, you see, actually, it flickers because of this. So I'll just try. And maybe. Wait. Oh, you would have been so impressed right now. Okay. So you've seen the pop. And now when I flick it. The problem that you see here, the flickering of the hands, it's because of the background. And because it recognizes the hands that are here. So let me try and do something else. Maybe do like that. Okay. So now we try and flick it. Come on. Be impressed. Okay. That was a good example. Okay? Thank you. And live demos are hard.

So you probably think to yourself, okay, do I need to do all this math myself? So MediaPipe does have modules or does have some sort of logic in order to detect gestures. So you can teach MediaPipe to detect this gesture or this gesture. But because we needed something which is continuous, we had to do it ourselves.

Okay, now let's go Solar. We wanted to create a Solar System, hopefully. So with Solar, we can't use cubes anymore. We have to use spheres. And how do we create a sphere with 3JS? First of all, we need to do to create a sphere material or geometry, sorry, and we can define how many triangles will this sphere consist of. And then we have to have the material of the sphere. We have to have the texture. So how does it look? The texture is actually the image of the sphere. And you can also put additional textures on it.

So we have the texture, which is how Earth looks from afar. And we can also have a black and white image that says what is the topography of the Earth. So how the shadows will behave. So we have a white sphere. Put a texture and then we can have this texture, which is a little bit flat. Put a topography. And hopefully you notice that now the mountains have shadows. You can put a shininess because the oceans reflect light different than the land, right? So I can put shininess and then I have this like glare going on. And I can add another layer of clouds, which is nice.

Now we want to create the stars. In order to create the stars we need to create a sphere and be inside the sphere. Imagine stars are around you. You need the camera to be inside the sphere. So we'll use an actual star data from our galaxy. And for the magnitude of the star we'll define the size of the point that we're going to put. So we're just going to create an array of points. And then we're going to just put them on the sphere. Again spherical geometry.

And then we have the star field. Now it's interesting because we used snap to create the boxes before. We want to use snap to create the stars. So we can just replace create box with create stars right? But that's not good. We want to use some sort of an event system. And now this can be much more scalable. We can have a snap detector that only detects snap. And it just published the event of a snap. And we can have somewhere else, the stars just subscribe to the snap and creates the stars.

Okay, this is going to be really handy later. In order to create the sun, it's a bit more complicated. Now we need some sort of a state machine, first snap for the stars, second snap for the sun. So what is the most complex state machine you can think of? No, we just use a switch case because again, side project weekend. So, create the stars, then move to sun, then end of time.

In order to create the sun, we create a point light that is within the sun and will illuminate everything around it. And to enter the sun, again, we'll make it pop, right. Scale it 110%, then scale it back to 90%. Okay, we need to flick it to the center, which is super easy with the sun because we have events, remember? So, we need to subscribe to flick and we need to move the sun to 0000, the center of the scene. And the last gesture that I wanted to introduce is to make it spin, make the sun, like, rotate itself. So, let's look at the anatomy of a spin. You need to detect straight finger, then you need to find overlapping element, right, which is, again, easy. And then, you need to detect the straight and diagonal finger. So, it need to be straight, but the angle shouldn't be 180 degrees. And then you need to trigger the event of the spin for that element. And how do we spin? We need to identify if it's the right hand or the left hand in order to know the direction of the spin. Again, events are awesome. And then we have the animate loop that just says, okay, the sun needs to spin with some sort of a speed. So, every animation frame, I'll just add the speed to the sun rotation. It looks like that. It's really, really simple.

You just put it in a rotation set, and then the animation loop makes it rotate. And again, apologies in advance.

Okay, let's try to create the stars. No. Wait. Something's not working. Oh, maybe the lanyard is making it worse.

Now let's try to create our sun. Oh... No. No. Thank you.

Okay, creating Earth is super easy. You don't need seven days, you need seven seconds because we have this switch case, we just add Earth and then we need to somehow put it into orbit. To put it into orbit, we'll flick it, but we need to flick it only to Y equals zero, right? So we don't need to flick it to the center, just to the same plane as the sun. And then we have to have some sort of a loop in order to make it rotate the sun, so, again, every animation frame will just add to the angle of Earth.

Okay, no, we won't see because we need to add Mars as well, right? We forgot about Mars. In order to add Mars, we just add it to the switch case. Again, it's super easy to do that. And we'll add another gesture which is rotating the entire scene with our hands. So in order to rotate everything, we just need to detect straight fingers. So I know that I want to do like this, like in Minority Report. So everything is straight and vertical. And then I need to detect the movement of the fingers in order to change the camera. Either rotate it or go backward and forward.

Okay, let's see it all. And now... Good energies. Okay. Come on.

This can take a while. Go get the sandwiches.

Wait. Sorry, I don't know why it's taking so long. Yeah, it shouldn't.

Wait, wait, wait. Yeah, you can snap with me. Okay. One. Snap with me. It will help. Nice one. Come on. Could have been a really cool demo. Well, okay.

Sun first. Then flick. Spin. Spin. Earth. Spin. Flick. Back, back, back, back, back. Wait. Wait, wait, wait. Come on Mars. Come on Mars. Elon is counting on you. Come on Mars. Spin.

And... Wait. Come on. I mean. It's less weird than it looks. Okay. One last chance, Mars. Come on. And I lost my hand. Okay, Mars won't spin today. But maybe we can make it pop. Where's my hand? Anyway.

Now we can use our hand to rotate. And we can use the second hand. When it will be recognized. To dolly the camera back and forth. Right? So, this is our scene. Yeah. It was pretty fun to create it.

And where do we go? Thank you. Applause. Where do we go from here? Amazing. Yes. Okay. This is cool. A lot of things we can do with it. There's a guy called Oz Ramos which uses hand gestures in order to navigate the web and control websites, which is pretty cool. And we can do really cool things with body recognition, like creating an avatar that moves according to your web cam. If this was too much, because a lot of things to soak in, we can go back to Netflix and just relax with a familiar and nice avatar. And that's it.

The future is amazing. You're awesome. And my code is over there. It works much better when not live-demoing it. Thank you very much. Thank you. Thank you. Thank you very much. You did such a nice live demo. Let's give one more round of applause. Yes.

Now, I kindly ask our AV team to put questions on the stage. I think we have time for maybe one of them. Do we have? Okay. Technical issues but I remember the question. I remember the main question from audience which is what are the limits for performance. Is there a chance to render something really expensive in the browser? What's your experience with that? Yeah. So there are a lot of performance considerations when you try to render something with javascript. But 3JS uses GPUs. So if you make if you are able to make the GPU render the scene it's very smooth. Actually when I built the demo I didn't use GPUs and it was a little bit choppy but with GPUs it's super smooth. Okay. Cool, cool, cool. And take a chance to catch Liat in the hybrid chat area and folks who watch us remotely you have a chance to ask Liat also. Yeah, now we quickly switch to the next speaker.

Come on, come on, come on. Whoa, whoa, whoa. Ah, ah, ah, ah. Ah, ah, ah, ah. Ah, ah, ah, ah.

Ah, ah, ah. Ah, ah, ah. Ah, ah, ah. Ah, ah, ah.

Come on, come on. Come on, come on. Ah, ah, ah. Ah, ah, ah. Ah, ah, ah. Ah, ah, ah. Ah, ah, ah. Ah, ah, ah.

Come on, come on. Come on, come on. Eh, eh, eh. Eh, eh. Eh, eh. Eh, eh. Eh, eh. Eh, eh. Eh, eh.