Bun, Deno, Node.js? Recreating a JavaScript runtime from Scratch - Understand magic behind Node.js

Today I'm going to talk to you about some experiments, some mad sciences I've been doing using javascript and a lot of stuff. And I hope you really like this content because it was a real effort to create everything here. So to get started, everything that I'm going to show you today is already online. So after all the talk I'm going to show you some links so you can go there. But please, if you can, take a picture of this talk, mention the event, mention myself, because this helps us a lot with the work we've been doing. All right. I'm so excited. I'm going to talk about node.js and the node.js creator is here, the bun and so on. So it's pretty amazing. Well, first of all, I've been doing a lot of other experiments. I was trying to reimplement node.js, reimplementing WebSockets, reimplementing a code coverage too. So I've been making a lot of specific questions. So I'm very curious. And all these tutorials are there so you can find it as well. Well, all these experiments started when I started asking myself, well, do I really know what is node.js? So I started researching and I figured some posts were telling that V8 does one thing, Libv does another, javascript is another role, and sometimes one post was controversial to another. So I was like, hmm, maybe I should learn more. Maybe I should understand better. So I don't really know how it's going behind the scenes, how it's really working. So I started researching a bit and I figured out there's no content about this. No one has ever done like recreating the whole stuff, compiling the whole libraries. But I started researching on the node.js websites and those links helped a lot to learn how the event loop, how the concurrency module in node.js was working, but still I wanted more. So that's why I created this tutorial. This is a full step-by-step tutorial, which is this talk is based. So this talk is going to be a lot of highlights because I cannot show all the hands-on right here, okay? So you can try out later. Just a disclaimer, before we go on, I'm going to tell you everything here is part of my research, okay? I'm not a C++ developer. You might see a lot of like bad practices right there, but it's something that I was enjoying and doing. Also this is part of my own research. As I tell you, there is no content on the internet. So I started asking some friends, looking at the source code and getting some assumptions. And just a heads out, JS run timers, our authors are amazing. I start to give more value to them as I saw how complex is it behind the scenes so we are able to use javascript in there. All right, let's go to the fun part, right? So let's understand the magic behind the node.js. So everything here, I did like a Git pod. I did all the environment for you right there. It's binaries and a lot of stuff that you can start using right away. To get started, I was like, what if I go to the node.js repo and try finding how Ryan Dahl was doing this stuff. So I found out a lot of files and I found out like, oh, maybe I should try reproducing this. But if you see, it's 14 years ago. Like a lot of tools, it's not even working anymore. But still, has anyone seen this website before? No one. This is so nice. This was the first version released of the node.js. It's the V1, V001. And you can see there was no console.log back then. It was puts. Very nice. All right. I know this is a very complex subject, so I won't make you sleep right here, okay? So I tried to split the main components so you can understand each individual role. If you are trying to search for jobs, this is kind of cool to tell on the interview. So first of all, we're going to talk about V8. V8 is grammar. It's the javascript data types. It's how javascript is interpreted, which means a class, a variable, a data type. It's all on V8. We also have the libuv. Libuv is the async thing that we've been listening a lot. But just think of it as a while true asking for new events. And if so, if there are any pending events, it's going to dispatch them all, and you can start receiving more data and so on. And here for me is the mind-blowing part, the C++ bridge. So when you try finding, you're going to realize node.js is almost everything on C++. I'm going to try a magic with you, trying to implement a new feature on the V8 side. So let's take a look at our JS code. When you're starting using the V8 from scratch, our context, our global disk is empty. So there's nothing there that we can use. But I'm going to try implementing the print function. Print doesn't exist in javascript. So if I want to be able to execute this function from the javascript side, this must be on the V8. So using a C++ bridge, I'm going to create a print function on C++, and then I'm going to bind it to the context. See, I would say every time I see this string, I'm going to call this C++ function. Behind the scenes, V8 is like the evolve. It's evaluating everything that you want. Okay, let's go try doing something harder. A setTimeout function. A setTimeout function is something that is asynchronous. It depends on the environment. So we can use the uv start, which is leave uv functions. We do exactly the same thing. We map this string to this C++ function, and then it's already available on the V8. I'll tell you, this was the part for me like, oh my god. This is why this is so good because it's extensible, right? Most of the js runtimes follow the same idea. They are extending the javascript runtime environment and doing a lot of cool stuff. So right here, I'm going to try doing some experiment using our javascript code. So here I can see an index.js file, and here's the whole project in C++. So if you go there, we're going to take a look of the workflow, how it's working. First of all, I have to read the file. As I was telling you, the index.js file is just a string in the end. I'm going to grab this file, and then I'm going to use compile. So the compile will grab all the string and transform to C++ instances. And after that, we're going to go and see the run, the actual execution behind it. And then we have our wait for events. So we just evaluate all the code, and then we schedule and wait for events. That's why I was telling you. And just a matter of curiosity, if you go to the source code, you're going to see the read file from C++. So I'm not using libuv here because the entry point, I just need the string. So I don't want to wait or don't need to wait for any other process. All right. To make this project, I spent a month trying to create some structure there so we can reuse it. So I had to compile the V8 library, which is in C++, and the libuv, which is in C. So you can see all the headers there. So if you're not C++ developer, neither do I, right? So you can just take a look at the structure. This is a very nice workaround. You could see I was using libuv. I was using basically NodeMod to create all this stuff there, right? So let's just jump a bit so you can see the NodeMod. So I don't know anything about the tooling on the javascript or on the C++ side. So I was like, what if I just use NodeMod to wait for any changes and then refresh our application? So I use make to run all the steps from my configuration file. It generates a binary, and then I can execute a javascript file. First of all, what happens if we try calling the console.log? It was the first thing I was trying to do there, and I honestly was struggling a lot. So if you run console.log, we could see in the logs nothing happens, right? I was like, why is not happening? However, if I get the print function I implemented myself and I print the console, I could see an object there. And if I try seeing the all keys, they're all there. However, I would say they're just an interface, right? But they are not doing anything. Behind the scenes, setTimeout, setInterval, and console are not javascript. So anything that relies on the environment is part of something else. It's actually expanding javascript. So if I try using setTimeout here, I could see setTimeout is not defined. Or what if setInterval? Not defined either. Everything that is a part of it is actually from another working group. So we can see console, we can see other specifications, but it's not actually like ECMAScript, right? We don't have something that runtimes must follow in this case. Well, let's try doing something nicer here. Let's take a look at the print. For me, this was like, oh my God, I'm using printf, which I used on the first day of the university. It's nice to see something so complex and still using something which for me was really primitive. So I have those functions, right? I have the print one. The same idea I used on the whole drawing I did. So we have a string and then I map to a C++ function. Every function that we have on node.js is actually instantiated by using this same approach. Now we can check what is javascript. So a new date is javascript, even though time relies on the environment, right? It's the only exception. Template strings we can use there. We could also use maps. We could use spread operators. Everything is built in on V8. And even promises. Promises are not async operations. They are just wrappers for callbacks to help us developers to write better code. Okay? Well, let's try doing something more interesting. What about setTimeout and setInterval? Remember the complexity. I want to schedule a function which we will be executing in the future, and those variables, those references must be there when I have to call them back. And then I have to call back the javascript side, okay? This took me a while, my friend. This was really crazy to build. But let's take a look at the final result. First I'm going to have a class. Don't pay attention to the C++ side. Let's just see the arguments. So I'm going to grab the slip, the interval, and make sure the third argument is a callback function. And then I'm going to create something to store those values so I can execute in the future. And here I'm creating a timer on the libuv. So I start, I just send, and I tell which function is going to be executed in the end. And then I can go to this function and see when the timer has ended, it goes back to C++, I can grab all the context, and then I can generate the result that I'm going to send back to javascript and call our callback function. It's crazy to see it, right? It's not, for me it was like some magic with javascript, but in the end they are just abstracting the whole complexity for us so we can start using what we already know. And then if you try to look at the UV code, which is the while true I told you, we have just the run and it makes sure all the events is going to be dispatched later. And we follow the same idea. We compile the code and we wait for new events. And then I'm going to set the timeout exactly the same way I did for the print and other ones. Now what I told you that promises are javascript and not LibioV or async operations. I can grab the same function I created that uses a callback, I can make a wrapper and use async await because async await is V8, right? It's part of the language. And one interesting thing, if you go to the first version on the V011, this is how Ryan Dahl had made the first version of it, okay? He was exposing the timers and executing from a javascript file and actually adding more complements. So we should know that V8, what javascript does by default, is known by the ECMAScript spec. That's why we have ECMAScript modules to replace like the common JS or the required JS. It makes our lives easier and also the lives of the authors of the other runtimes. Okay, we have now a lot of other new runtimes coming, right? The first question I had was like, is that easy to create a runtime? Why do we need the runtime now? Node has been there like for almost 10 years. So I start researching, going on their code and I had some assumptions. First deno. If you go to the deno source code, you're going to see a very similar code that we had in the C++ site written in rust. He's grabbing a string, compiling it and executing it, okay? The same idea, exactly the same idea on C++. And then you can see that he's injecting deno core which is extending V8 and injecting more code, exactly the same interesting idea. Okay, what about Bun? Well, Bun is, I would say, more mad science idea because he used javascript core instead of V8 which has no docs. I don't know how he has written it and it's written in ZIG. But looking at the whole code, you're going to see the same approach. He's extending the natural bindings of the javascript runtime and executing a lot of stuff. So you, from the back, you may be wondering right now, okay, we have a lot, right? cloudflare workers, deno and so stuff. What makes one better than others? This was something that I was wondering how are they faster or what is the whole point of creating a new javascript runtime? Well, the point is, remember that C++ is the bridge. So there, in node.js, we have all the modules, this is the print for FS and what makes one runtime better than others, it's how they handle this whole complexity. So I'm receiving results from the operating system and I'm trying to ping other services, calling processes, going back and forth with the strings and we can see how they are making this approach. So, Bun claims to be faster because he used better algorithms to write and to execute functions. Nice, isn't it? In my opinion, what makes one better than others is developer experience. deno has came with like testing native, they have typescript and a lot of stuff to help developers to write code faster and in my opinion, this matters a lot. But is it a competition? Like is deno replace Node? Is Bun gonna replace Node or deno? My opinion is they are going to collaborate with each other. So Jared Summer, who's speaking also in this conference, joined the node.js organization. Also Colin Rigg, he used to work on deno and has been working node.js since the beginning. So they are all collaborating together so we can be benefit. And also, there's a working group so they can share ideas. You saw, in the world of js runtimes, we don't have ECMAScript, right? Everyone could write C++ functions as they wish. So this is how they would do it. Nice, isn't it? I'm gonna see like, oh. If you want to follow more and go deep and try to implement your own FS, I strongly recommend the video. There I spent like almost two hours to build this tutorial for you so you can even complement it, okay? All the content I've been teaching you, all the ideas, all the talks, is already on my website. So if you wanna go further, study more, you can go there. And actually I released an e-book about this whole talk as well so you can go step by step. Before I finish this talk, we have a minute left. So we have a tradition to take a selfie and make the others on the other stage envy, okay? So I'm gonna count on three and I'm gonna make some noise, all right? All right. Oh, nice. So, let's get started. One, two, three. Thank you so much for having me. I'm gonna also be on the stage which Ryan Doll is right now to discuss more if you wanna gather and throw some questions and suggestions, would be nice. Well, thank you. Please join me. We only have one question yet so please bring your question on the slider. The question is why the demo project is called Capybara? Well, we Brazilians, we tend to show the world that we are not only samba, right? So capybara is a very known animal there. It looks like a really big pig and I really love to call capybara because of it to get some references, right? Nice. What's a favorite thing that you can share about your work? Okay, I would say like Anna, she's there. This girl helped me a lot and this is, I would say, the power of the community and actually the power of being curious, right? I was making a lot of questions and she was like, come on, what do you want to do? Why do you want to recreate something? I was like, I want to learn, all right? So I start asking a lot of people and they're like, I have no idea. I had some talks with people at Google, at V8, they were like, I only work with something working there, the environment working, but I've never thought about recreating the whole stuff. So right now, my next step, it's even harder, but I got some ideas. I'm going to try recreating the react Native. So I want to use like an alert and see the toaster on iOS and a toaster on Android. But the problem is we have to compile the whole thing, right? So environment, it's really, really hard to handle, but it's fun, right? The next question is, how do you find the courage to even do this? Which code base do you have more fun with, same author Dino or Node? Well, I like history, right? I would say this is why I love Berlin, people, everything is so historical. And I was going through, like I produce content on a daily day and I was looking at the brands, the very ones, and if you go there, you're going to enjoy a lot. Like Ryan Dahl, this is working on the committee and then this is not working anymore. Like sometimes we find those heroes sometimes and we're like, no, maybe these people are genius and so on. But sometimes they have the same problems that we face, right? So I would say in general, I put myself in this challenge, I would say because there was no content on the internet. So it was like, if I put myself on this, maybe other people could benefit as well and we could help our ecosystem. Cool. Have you looked into writing your own event loop for a toy runtime like this to use instead of LibioV? Well, never. Actually, LibioV was written for node.js, right? Before it was, I think, Libio, which only worked on Linux and Mac operating systems and LibioV came to help on the Mac system. But I was looking at the operating system APIs, my God, this is not human readable, right? I don't know if there is any C++ developer here. I really value a lot your job because it's pretty hard. Yeah, looks like. What is the recreation that you are more proud about? It's hard to say. I would say I try always to teach something that I'm not 100% sure that I know. So I can challenge myself a lot. The last thing was, I would say, I was trying to use machine learning on the browser using the tensorflow APIs and I was like, what if I can use the webcam to click on the elements, like the VR was doing, right? So the api was amazing doing this. So I had to map everything, but the browser, they don't let you to simulate a hover. So it was a hacky thing to find the css of the element to get to click, but it was so fun. And just a matter of curiosity, after two weeks, Zuckerberg published, like he was playing the guys, oh, Zuckerberg is stealing from you. Wow. Come on, give him a rise, man. Sorry. Any funny findings from node.js repo archaeology? I have no idea, but maybe the Atlantic stuff, right? I don't know if it is. No idea. Yeah. I don't know if we can call archaeology, but it's there, right? The branches and all this stuff, old stuff. Yeah. Will be there any videos about creating your own Node Docker image like Alpine, et cetera, please? Nice. I mean, I would say, I don't know, there's so many good ones right now installing all the packages. I'm not sure if I would add more value creating this, but I'm not sure. Wow. How much money do you make from YouTube? Not listening, right? That was fast, man. Yeah. No, let me tell a story. YouTube is horrible. So you have people to edit your video, to create a good thumbnail, people to edit or to review it. In the end, YouTube is, I'm paying to do this stuff there. However, it's not just like that, right? It's because I'm producing a lot of content. I can make like a quality product or a premium stuff so people can get value, change jobs a lot. And this is how I survive nowadays, right? Cool, cool. This is not a question, but I had no idea SataMount or console were not part of ES. I was like evangelist. I was talking to my mom. Mom, do you know console.log is not javascript? It was crazy because I've seen this since the beginning. This is the first thing we do in javascript and actually it's not javascript. This is crazy, isn't it? It says Senhora So Demais. I guess that, I don't know what's that. It's Portuguese or I have no idea. I have no idea. Oh, Senhora Zão. Oh, yeah, I guess it's Portuguese. This is something about English, right? Because in Portuguese, we have a lot of slangs and when we go to English, we cannot use them all. So, Senhora Zão is like big senior, high level. So we use this a lot. Oh, this is really nice. Okay, and they said, amazing talk. In your opinion, which is one of the best runtime for microservices right now? You want to put me in trouble, right? Oh my God. It's hard to say, right? I would say, no, I don't. It's hard because I've seen how BAN is evolving a lot and how node.js is so mature. How deno is being nice. I would say the better is what makes your experience better. Because in performance way, we had some benchmarks that shows, oh, this is faster. But 90% of the applications are crude applications, right? So it's hard to feel this whole improvement. And my opinion in the end is the best developer experience and what your team like most. I would say it's more a personal opinion than for the runtimes. But this is my opinion, right? I have more experience with node.js than others runtimes as well. People can think differently, right? Definitely. We have one more question. What does the WTF prefix mean in the BAN GSA code we saw? So include WTF slash model dot H. Well, I would say WTF is a very specific text, right? I'm going to think it's a template. So in C++, we use include to link the libraries, to say, oh, I'm going to call this function. For example, the V8 compile. When I use V8 compile, I had to include the headers. So when is an execution time, it's going to refer from the binary, right? If you look at the whole gate pod I built, V8 has two gigabytes. So this is something that I was talking to Anna. How two gigabytes comes to 200 gigabytes in the end in node.js. This is really interesting. I was like, oh my God, tell me everything. Now is the last question. Why did you choose node.js? What's your inspiration? I chose node.js. I used to be a.NET developer. And back then, I wouldn't say the language was a problem, but I felt like I was in a cage. I remember working and I had some update on my Visual Studio there, and I couldn't work. I was like, come on, I'm tied to some tool. I'm not a real developer, I was thinking. With node.js, when it came, I was using Sublime or Notepad in a terminal. So for me, it was like, this is the thing that I'm going to learn the real thing in node.js or learn how the things is working behind the scenes. Because if the framework is not working anymore or the library or something itself, I would be able to fix myself, right? Or at least understand it. Cool. Thank you very much for your talk and answering all the questions. It was a great talk. Thank you. Thank you. Thank you.