Bun, Deno, Node.js? Recreating a JavaScript runtime from Scratch - Understand magic behind Node.js

Today, I'm gonna talk to you about some experiments, some mad sciences I've been doing using JavaScript and a lot of stuff, and I hope you really like this content because it was a real effort to create everything here. So to get started, everything that I'm gonna show you today is already online, so after all the talk I'm gonna show you some links so you can go there, but please, if you can, take a picture of this talk, mention the event, mention myself because this helps us a lot with the work we've been doing.

All right. I'm so excited, I'm gonna talk about Node.js and the NodeJS creator is here, the bun, and so on. So it's pretty amazing. Well, first of all, I've been doing a lot of other experiments. I was trying to re-implement Node.js, re-implementing WebSockets, re-implementing a code coverage, too, so I've been making a lot of specific questions, so I'm very curious, and all these tutorials are there so you can find it as well.

Well, all this experiment started when I started asking myself, well, do I really know what is Node.js? So I started researching, and I figured some posts were telling that V8 does one thing, Libv does another, JavaScript is another role, and sometimes one post was controversial to another, so I was like, hmm, maybe I should learn more, maybe I should understand better. So I don't really know how it's going behind the scenes, how it's really working. So I started researching a bit, and I figured out there's no content about this. No one has ever done recreating the whole stuff, compiling the whole libraries, but I started researching on the Node.js website, and those links helped a lot to learn how the event loop, how the concurrency module in Node.js was working, but still, I wanted more. So that's why I created this tutorial. This is a full step-by-step tutorial, which is this talk is based, so this talk is going to be a lot of highlights, because I cannot show all the hands-on right here. So you can try it out later.

Just a disclaimer, before we go on, I'm going to tell you, everything here is part of my research, okay? I'm not a C++ developer. You might see a lot of bad practices right there, but it's something that I was enjoying in doing. Also, this is part of my own research. As I tell you, there's no content on the internet. So I started asking some friends, looking at the source code and getting some assumptions. And just a heads out, JS Runtimers, our authors, are amazing. I started to give more value to them as I saw how complex is it behind are able to use JavaScript in there.

All right, let's go to the fun part, right? So let's understand the magic behind Node.js. So everything here, I did like a Gitpod. I did all the environment for you right there. It's binaries and a lot of stuff that you can start using right away. To get started, I was like, what if I go to the Node.js repo and try finding how Ryan Doll was doing this stuff? So I found out a lot of files. And I found out like, oh, maybe I should try reproducing this, but if you see, it's 14 years ago. Like a lot of tools, it's not even working anymore. But still, has anyone seen this website before? No one. This is so nice.

This was the first version released of the Node.js. It's the V001. And you can see there was no console.log back then. It was puts. Very nice.

All right. I know this is a very complex subject, so I won't make you sleep right here, okay? So I tried to split the main components so you can understand each individual role. If you are trying to search for jobs, this is kind of cool to tell on the interview.

So first of all, we're going to talk about V8. V8 is grammar, it's the JavaScript data types, it's how JavaScript is interpreted, which means a class, a variable, a data type, it's all on V8. We also have the libuv. Libuv is the async thing that we've been listing a lot. But just think of it as a while true asking for new events, and, if so, if there are any pinned events, it's going to dispatch them all and you can start receiving more data and so on.

And here, for me, is the mind-blowing part. The C++ bridge. So when you try finding, you're gonna realize, Node.js is almost everything on C++. I'm gonna try a magic with you, trying to implement a new feature on the V8 side. So let's take a look at our JS code. When you're starting using the V8 from scratch, our context, our global disk is empty. So there's nothing there that we can use, but I'm gonna try implementing the print function. Print doesn't exist in JavaScript, all right? So if I want to be able to execute this function from the JavaScript side, this must be on the V8. So using the C++ bridge, I'm gonna create a print function on C++, and then I'm gonna bind it to the context. See, I would say, every time I see this string, I'm gonna call this C++ function. Behind the scenes, V8 is like the evolver, right? It's evaluating everything that you want.

Okay, let's go try doing something harder. A setTimeout function is something that is asynchronous, it depends on the environment, so we can use the uvstart, which is leave uv functions. We do exactly the same thing, we map this string to this C++ function and then it's already available on the V8. I'll tell you, this was the part for me like, oh my god, this is why this is so good, because this is extensible, right? Most of the JS run times follow the same idea, they are extending the JavaScript runtime environment and do a lot of cool stuff. So right here, I'm gonna try doing some experiment using our JavaScript code. So here, I can see Index.js file and here's the whole project in C++.

So if you go there, we're gonna take a look of the workflow, how it's working. First of all, I have to read a file, ok? As I was telling you, the Index.js file is just a string in the end. I'm gonna grab this file and then I'm gonna use compile. So the compile will grab all the string and transform to C++ instances. And after that, we're gonna go and see the run, the actual execution behind it and then we have our wait for events. So we just evaluate all the code and then we schedule and wait for events. That's why you're through, I was telling you.

And just a matter of curiosity. If you go to the source code, you're gonna see the read file from C++. So I'm not using libuv here because the entry point, I just need the string. So I don't want to wait, I don't need to wait for any other process. All right.

To make this project, I spend a month trying to create some structure there so we can reuse it. And I've had to compile the V8 library, which is in C++ and the libuv, which is in C. So you can see all the headers there. So if you're not C++ developer, neither do I, right? So you can just take a look at the structure.

This is a very nice workaround. You could see I was using libuv, I was using basically NodeMod to create all this stuff there, right? So let's just jump a bit so you can see the NodeMod. So I don't know anything about the tooling on the JavaScript or on the C++ side. So I was like, what if I just use NodeMod to wait for any changes and then refresh our application? So I use make to run all the steps from my configuration file, it generates a binary, and then I can execute a JavaScript file. Okay?

First of all, what happens if we try calling the console.log? It was the first thing I was trying to do there, and I honestly was struggling a lot. So if you run console.log, we could see in the logs, nothing happens. Right? I was like, why is not happening? However, if I get the print function I implemented myself, and I print the console, I could see an object there. And if I try seeing the all keys, they're all there. However, I would say they're just an interface, right? But they are not doing anything. Behind the scenes, setTimeout, setInterval, and console are not JavaScript. So anything that relies on the environment is part of something else. It's actually explaining JavaScript. So if I try using setTimeout here, I could see setTimeout is not defined. Or what if setInterval, not defined either. Everything that is a part of it is actually from another working groups. So we can see console, we can see all the specifications, but it's not actually like ECMAScript, right? We don't have something that runtime must follow in this case.

Well, let's try doing something nicer here. Let's take a look at the print. For me, this was like, oh my God, I'm using printf, which I use on the first day of the university, right? It's nice to see something so complex and still using something which, for me, was really primitive. So I have those I have the print one, the same idea used on the whole drawing I did. So we have a string and then I map to a C++ function.

Every function that we have on Node.js is actually instantiated by using the same approach. Now, we can check what is JavaScript. So a new date is JavaScript, even though time relies on the environment, right? It's the only exception. Template strings we can use there. We could also use maps. We could use spread operators. Everything is built in on V8. And even promises. Promises are not async operations. They are just wrappers for callbacks to handle us, to help us developers to write better code, okay?

Well, let's try doing something more interesting. What about setTimeout and setInterval? Remember the complexity. I want to schedule a function, which is we will be executing in the future, and those variables that those references must be there when I have to call them back, and then I have to call back the JavaScript side, okay? This took me a while, my friend. This was really crazy to build it. But let's take a look at the final result. First, I'm gonna have a class. Don't pay attention on the C++ side. Let's just see the arguments. So I'm gonna grab the sleep, the interval, and make sure the third argument is a callback function. And then I'm gonna create something to store those values, so I can execute in the future. And here I'm creating a timer on the libuv. So I start, I just send, and I tell which function is gonna be executed in the end. And then I can go to this function and see when the timer has ended, it goes back to C++, I can grab all the context, and then I can generate the result that I'm gonna send back to JavaScript and call our callback function. It's crazy to see, right? It's not, for me it was like some magic with JavaScript, but in the end they are just subtracting the whole complexity for us so we can start using what we already know. And then, if you try to look at the UV code, which is the while true I told you, we have just the run and it makes sure all the events is gonna be dispatched later. And we follow the same idea, we compile the code and we wait for new events. And then I'm gonna set the timeout exactly the same way I did for the print and other ones. Now, what I told you that promises are JavaScript and not LibuV or async operations. I can grab the same function I created that uses a callback, I can make a wrapper and use async await because async await is V8, right? It's part of the language. And one interesting thing, if you go to the first version on the V0.1.1, this is how Ryan Doll had made the first version of it, okay? He was exposing the timers and executing from a JavaScript file and actually adding more complements. So, we should know that V8, what JavaScript does by default is known by the ECMAScript spec.

That's why we have ECMAScript modules to replace like the common JS or the required JS and makes our lives easier and also the lives of the authors of the other runtimes.

Okay, we have now a lot of other new runtimes coming, right. The first question I had was like, is that easy to create a runtime? Why do we need a runtime now? Node has been there like for almost 10 years. So, I start researching, going on their code and I had some assumptions. First, DNO. If you go to the DNO source code, you're going to see a very similar code that we have in the C++ site written in Rust. It's grabbing a string, compiling it, and executing it. Okay? The same idea. Exactly the same idea on C++. And then you can see that he's injecting DNO core which is extending V8 and injecting more code. Exactly the same interesting idea. Okay? What about Bunn? Well, Bunn is I would say more math science idea because he used JavaScript core instead of V8 which has no docs. I don't know how he has written it, and it's written in Zig. But looking at the whole code, you're going to see the same approach. He's extending the natural bindings of the JavaScript runtime and executing a lot of stuff.

So you, from the back, you may be wondering right now. Okay. We have a lot, right? CloudFlare workers, Dino and so stuff. What makes one better than others? This was something that I was wondering how are they faster or what is the whole point of creating a new JavaScript runtime? Well, the point is, remember the C++ is the bridge. So there, in Node.js, we have all the modules. This is the print for Fs and what makes one runtime better than others, it's how they handle this whole complexity. So I'm receiving results from the operating system and I'm trying to ping all the services calling processes, going back and forth with the strings and we can see how they are making these approaches. So, Bunn claims to be faster because he used better algorithms to write and to execute functions. Nice, isn't it? In my opinion, what makes one better than others is developer experience. Dino has came with, like, a testing native. They have TypeScript and a lot of stuff to help developers to write code faster and in my opinion, this matters a lot. But is it a competition? Like, is Dino replace Node? Is Bunn going to replace Node or Dino? My opinion is they are going to collaborate with each other. So, Jarrett Sommer who is speaking also in this conference joined the Node.js organization. Also, Colin Rigg who used to work on Dino has been working on Node.js since the beginning. So, they are all collaborating together so we can be benefit. And also there's a working group so they can share ideas. You saw in the word of JS runtimes, we don't have ECMAScript, right? Anyone could write C++ functions as they wish. So, this is how they would do it. Nice, isn't it? I'm going to see like, oh. If you want to follow more and go deep and try to implement your own FS, I strongly recommend the video. There, I spend like almost two hours to build this tutorial for you so you can even complement it, okay? All the content I've been teaching you, all the ideas, all the talks is already on my website. So if you want to go further, study more, you can go there. And actually, I released an e-book about this whole talk as well, so you can go step by step.

Before I finish this talk, we have a minute left. So we have a tradition to take a selfie and make the others on the other stage angry, okay? So I'm going to count on three and I'm going to make some noise, all right? All right. Oh, nice. So let's get started.

One, two, three. Ahh! Thank you so much for having me. I'm going to also be on the stage which Ryan Dow is right now to discuss more if you want to gather and questions and suggestions would be nice. Well, thank you. Please join me. We only have one question yes, so please bring your question on the slide.

And the question is, why the demo project is called Capybara? Well, we Brazilians, we tend to show the world that we are not only samba, right? So, capybara is a very known animal there, it looks like a really big pig. And I really love to call capybara because of it, to get some references, right? Nice. What's a favorite thing that you can share about your work? Okay, I would say like Anna, she's there. This girl helped me a lot. And this is I would say the power of the community and actually a part of being curious, right? I was making a lot of questions and she was like, come on, why, what do you want to do? Why you want to recreate something? I was like, I want to learn. All right. So I start asking a lot of people and they're like, I have no idea. I had some talks with people at Google, at V8. They were like, I only work it with something working there, the environment working, but I've never thought about recreating the whole stuff. So right now, my next step, it's even harder, but I got some ideas. I'm going to try recreating the React Native. So I want to use like Alert and see the Toaster on iOS and a Toaster on Android. But the problem is, we have to compile the whole thing, right? So environment, it's really, really hard to handle, but it's fun. The next question is, how do you find the courage to even do this? Which code base do you have more fun with? Same author Dino or Node? Well, I like history, right? I would say this is why I love Berlin. People, everything is so historical. And I was going through, like, I produce content on a daily day and I was looking at the brands, the very ones, and if you go there, you're going to enjoy a lot. Like Ryan Dahl, this is working from the committee and then this is not working anymore. Like, sometimes we find those heroes sometimes and we're like, no, maybe these people are genius and so on. But sometimes they have the same problems that we face, right? Yeah. So, I would say, in general, I put myself in this challenge.

I would say because there was no content on the Internet. So, if I put myself on this, maybe other people could benefit as well and we could help our ecosystem. Cool.

Have you looked into writing your own event loop for a toy runtime like this to use instead of libuv?

Well, never. Actually, libuv was written for Node.js, right? Before it was, I think, libio which only worked on Linux and Mac operating systems and the libuv came to help on the mac system. But I was looking at the operating system APIs, my God, this is not human-readable, right? I don't know if there is any C++ developer here. I really value a lot your job, because it's pretty hard. Yeah, it looks like.

What is the recreation that you are more proud about? It's hard to say. I would say, I try always to teach something that I'm not 100% sure that I know so I can challenge myself a lot. The last thing was, I would say, I was trying to use machine learning on the browser using the TensorFlow APIs. And I was like, what if I can use the webcam to click on the elements like the VR was doing, right? So the API was amazing doing this so I had to map everything. But the browser, they don't let you to simulate a hover. So it was a heck of a thing to find the CSS of the element to get to click, but it was so fun. And just a matter of curiosity, after two weeks, Zuckenberg published, like, the guys, oh, Zuckenberg is stealing from you. Come on, give him a rise, man. What? Sorry.

Any funding findings from Node.js, brepro, archaeology? Oh, I have no idea, but maybe the Atlantic stuff, right? I don't know if it is. No idea. Yeah. I don't know if you can call archaeology, but it's there, right, the branches and all this stuff, old stuff. And will be there any videos about creating your own Node-Docker image like Alpine, et cetera, please. Nice. I mean, I would say I don't know, there's so many good ones right now installing all the packages. I'm not sure if I would add more value creating this, but I'm not sure. Wow. How much money do you make from YouTube? Not listening, right? No. Let me tell a story. YouTube is horrible. So, you have people to edit your video, to create a good thumbnail, people to edit or to review it.

In the end, I'm paying to to do this stuff there. However, it's not just like that, right? It's because I'm producing a lot of content. I can make like a quality product or a premium stuff so people can get value, change jobs a lot and this is how I survive nowadays, right? This is not a question but I had no idea that Saitama out or console were not part of ES. So I was like evangelists. I was talking to my mom. Mom, do you know console.log is not Javascript? It was crazy because I've seen this since the beginning. This is the first thing we do in Javascript and actually, it's not Javascript. This is crazy, isn't it? It says Señor da so the mind. I guess that, I don't know what's that. Is it Portuguese? I have no idea. Señor da so. This is something about English, right? Because in Portuguese we have a lot of slangs and when we go to English, we cannot use them all. So Señor da so is like senior, high level. So we use this a lot. This is really nice.

Okay, this is an amazing talk. In your opinion, which is one of the best runtimes from microservices right now? Oh, you want to put me in trouble, right? Oh my god, it's hard to say, right? I would say, no, I don't – it's hard because I've seen how BAN is evolving a lot and how Node.js is so mature, how Dino, it's been nice. I would say the better is what makes your experience better because in performance way, we had some benchmarks that shows, oh, this is faster, but 90% of the applications are crude applications, right? So it's hard to feel this whole improvement and my opinion in the end is the best developer experience and what your team likes most. I would say it's more a personal opinion than for the runtimes. But that's my opinion, right? I have more experience with node than others, runtimes as well. People can think differently, right? Definitely.

We have one more question. What does the WTF prefix mean in the GSA code we saw? Example include WTF slash model dot h. Well, I would say WTF is a very specific text, right? I'm going to think it's a template. So in C++, we use include to link the libraries to say, oh, I'm going to call this function, for example, the V8 compile. When I use V8 compile, I had to include the headers. So when it's in execution time, it's going to refer from the binary, right? If you look at the whole gate pod I built, V8 has two gigabytes. So this is something that I was talking to Anna, how two gigabytes comes to 200 gigabytes in the end of Node.js. This is really interesting. I was like, oh my god, tell me everything.

And now it's the last question. Why did you choose Node.js? What's your inspiration? I chose Node.js. I used to be a .NET developer. And back then, I wouldn't say the language was a problem, but I felt like I was in a cage. I remember like working, and I had some update on my Visual Studio there, and I couldn't work. I was like, come on. I'm tied to some tool. I'm not a real developer, I was thinking. With Node.js when it came, I was using Sublime or Notepad in a terminal. So for me it was like, this is the thing that I'm gonna learn the real thing in Node.js or learn how the thing is working behind the scenes. Because if the framework is not working anymore or the library or something itself, I would be able to fix myself, right? Or at least understand it. Cool, thank you very much for your talk and answering all the questions. It was a great talk. Thank you.