Node.js Compatibility in Deno

Hi, folks! My name is Bartek and I'm a member of the Deno core team. Today, we're going to discuss Node.js compatibility in Deno. In this talk, I would like to give you a quick rundown of what is Deno, and try to answer a few questions regarding compatibility with Node.

So, can Deno run apps and libraries offered for Node.js? What are the trade-offs? How does it all work and what is next on the plate for us? So, let's get right to it!

Deno one-on-one. Deno is a new command-line runtime for JavaScript and for TypeScript. It's similar to Node.js, that you can execute your JavaScript code, but the difference is that we also support TypeScript as a first-class citizen. Word of caution here, Deno doesn't actually execute your TypeScript. It just transpiles it to JavaScript underneath, behind the scenes, so you don't need to set up a build chain or anything like that. Deno also aims to be fun and productive. We like to think of it as a bag of LEGO blocks.

Deno is a single, executable file. This file is all you need to run any Deno program, and we will always distribute Deno as such. Deno works on Linux, it works on Mac and on Windows, so wherever you go, new server, new laptop, whatever, there's only a single command you need to execute to get Deno and you're ready to go.

Deno has something we like to call secure execution by default. What that means is none of your programs can access your file system, your environmental variables, sponsor processes, or even open an internet connection unless you explicitly allow it to. In this short example we can also see that you can directly link to third-party code and you can execute it even though you don't trust this code. So, in here we are executing an example from DenoSTD, which is Deno's standard library, it's a set of modules that are guaranteed to work with Deno, but it could have been any URL in the world you can think of, and in this case where there are no flags provided besides run sub-commands, this program cannot do anything to your computer. But if you want to access a file system or spawn a sub-process or open an internet connection, you can give Deno some flags and tell it to allow your program to actually do these things. And as you can see here some of these flags allow you to scope this even further. So for example, we can allow our program to read from the file system, however only from the slash tmp directory. Or we can tell your program to allow open an internet connection, however, only scoped to google.com domain.

Deno allows you to link to any third-party code using URLs. It works just like it does in browser. In this example we can see our es module import. We are importing a magenta function, again from Deno STD. This time we are doing this from fmt slash colors module. So this module resolution is simple and compatible with browsers and Deno has no notion of node modules or indexJS. Or actually that's what this talk is about after all. Deno doesn't depend on a centralized server. That is, you are not explicitly tied only to npm if you want to link to your packages or install new packages or a private registry if you have one. You can pull code from anywhere on the internet, either public servers or private servers if you provide proper authorization headers. Deno as a runtime fetches, caches and compiles this code, or transpiles it, transparently for you on the first run. After the first run, the code is already in cache, so there is no need to download it again. In fact, Deno won't download the code again unless you explicitly ask it to.

So, why are we even thinking or working on Node.js compatibility? Well, npm is arguably the biggest package ecosystem in the world, and users just want to use this package ecosystem. When Deno first came out with v1.0, we stated explicitly that we are not compatible with npm. And that was fine. Here, some users still find lack of support for npm a huge hurdle for adoption. So some users want to try Deno instead of Node.js. However, the cost to switch from one runtime to the other often proves too high to even try and do it.

So, by providing Node.js compatibility, we want to make the migration path smoother and easier for users. So let's try to answer this question. Can Deno actually run apps, libraries or packages that were written for Node.js? And yes, but... And before we get to this but, images speak more than a thousand words.

So I'm going to give you a quick example, or actually two examples of code offered for Node.js that is already running in Deno. In the first example, we can see the very popular tool called ESlint that is executing directly in Deno. So we pulled the sources for ESlint. For the sake of this example, say, we already used npm. And now we are executing this code using Deno. Important thing to notice here is that we are providing this --"compat flag", which tells Deno to actually provide this Node.js compatibility. And then we are passing --"unstable", which is currently required due to some limitations in the API. So some of the APIs are not yet set in stone and may change in the future. And this flag conveys the message that, you know, this is unstable, so something might change, but don't worry, it will be okay.

The second example is actually npm itself. We can already execute npm in Deno and use it to install some other packages. One thing you might notice here is that there are some prompts here saying, not implemented. For example, process.onCodeException or process.onUnhandledRejection. We'll get to these in just a moment. So yes, Deno can execute apps, libraries, packages offered for Node.js. However, not yet all of them are supported. We are focusing our efforts on enabling users to run the most popular packages and tools at the moment. So we already saw ESLint and npm, but we are also making great progress on getting Yarn to work, Vite, Next.js, which is probably the most popular framework at the moment, and Mocha, which is also one of the most popular testing frameworks at the moment. So yes, Deno can run code offered for Node.js, but not all of it.

The reason is we are still working on porting or providing polyfills for some of the built-in Node.js modules, such as TLS or ChildProcess. These are being worked on, however, their API service is huge, so you can spawn subprocess just like you would do in Node with Deno, however not all of the APIs will work at the moment. Another important thing is CJS and ESM integration. Deno from day one was ESM first. And they didn't understand CommonJS modules. However, due to the fact that a lot of code offered for Node.js is still using CommonJS, it's crucial to provide this integration or interoperability between the two formats. And by the way, this is actually supported out of the box by Node.js, so we're just trying to catch up.

Even though we are executing code offered for Node.js, this code is still subject to the same permission rules or allow flags that we saw at the beginning of the presentation. So that means that you can execute code offered for Node.js in Bino and get an additional benefit of being able to sandbox your programs to certain directories or certain internet domains or don't allow it to spawn sub-processes.

So what are the trade-offs of this compatibility? Well, some APIs that exist in Node.js and in Deno and in browsers, they are just not compatible. A good example is setTimeout. This is a global API. It is available in Node.js, it is available in browsers, it is available in Deno. In Node.js you can also import it from the Timers module. However, the difference is that in Node.js, when you call setTimeout, you will get back a TimeoutClass instance. However, in browsers and in Deno, you will get back a number, which is an identifier for this Timeout. In Node.js, due to this return type, a lot of packages rely on it to, for example, unrev the timer. That is, tell Node.js that if this Timeout is the last thing to execute or you don't wait on anything else, just don't wait for this Timeout and let's finish running right now. Similarly, in browsers and in Deno, there's a lot of code that relies on the fact that you might want to do a typeof check based on this return value. There's a lot of code that does typeof ID equals to number. This is a really hard problem. We've been scratching our head for three months already on it. If any folks from Node's technical steering community are listening to this talk, please reach out and let's figure out how we can make progress on getting this more compatible between the environments.

Another trade-off is that the developer experience is not so great. But at the moment, we're working on this, we're making progress and are thinking about how this could be made nicer for users to switch between Node.js and Deno transparently. And why this DX is not so good? So you need to pass a lot of flags to Deno to actually tell Deno to behave like Node.js. You need to pass ____ compat. So this flag actually sets up globals like process. Process is a global variable that is available in Node.js environment. However, it's not available in Deno by default. But if you pass this ____ compat, it will tell Deno to actually prepare this global variable for you. You need to pass ____ hands table. Once again, some of the APIs are not yet done, and they will undergo some changes or vetting process. So this is just to signal to users that, you know, there might be some dragons here and there, but that's okay. We're moving fast. Most often, you also need to pass ____ allow read flag, and this is required for CJS modules, for CommonJS, because CommonJS resolution relies heavily on probing the file system and Node modules directories. And since it's implemented 100% in user land at the moment, it's subject to Deno permission checks.

Another trade-off is that some APIs might not be supported at all in the future. So an Inspector module, or a V8 module, they might not make it to Deno at all. We are still discussing this. This is both a technical problem, and in fact, if we want to support these APIs, it's because we think some of them are questionable.

One rather big caveat is that native add-ons will not be supported. Native add-ons in Node.js rely on Node.JPG, and are basically C++, while Deno is actually authored in Rust. So, it's not so easy to reconcile this problem due to ABI stability. However, I might have a good answer to this problem on the next slides.

So, before we get to that, let's briefly talk how does it all work. You might be surprised, but most of these APIs coming from Node.js are polyfilled in userland. Once again, I'm going to refer back to this project called DinoSTD, which is a standard library that we ship with Dino. It's a collection of modules offered in TypeScript that are guaranteed to work with Dino. And in there we have this Node module, and this Node module contains a lot of files that basically polyfill all of the APIs available in various built in Node.js modules. I encourage you to go to this link and check it for yourself. You might find this very educational.

One problem with this is the code that is the runtime code that provides built in modules and Node.js is offered as CommonJS modules, whereas all of these codes in DinoSTD is actually ES modules. So not only did we have to rewrite most of the code to TypeScript and make it type safe, we also had to do a rather significant reorganization to make the code work because of different semantics of CommonJS and ES modules.

So this is all fine and great that we could do with this in Userland. However, there are some very intricate APIs that are not really possible to be done in Userland. And two of the examples would be a process dot unhandled rejection. So, an event handler that is fired off when there is a promise that is being rejected but has no catch handler. In Deno up to this point, this was always a hard failure, your program would exit. Similarly, process dot next tick allows you to schedule a callback to being called after next tick of the event loop. And we tried polyfilling these APIs in Userland. However, due to very strict semantics of where they should be called or when or what should happen, that is altering how Deno behaves, we couldn't just do this in Userland. So we had to add some direct support for these APIs in Deno's core.

So what's next for Node.js Compatibility in Deno? Well, we want to ship the first version, the first publicly available version of this Compatibility layer and second quarter of this year. And some of the things we would like to do is firstly, explore so-called superset approach where Deno basically supports Node.js APIs out-of-the-box without any special flags or different modes running that Deno would run. And so this dash-dash compact flag might not be necessary in the future. Another idea that we are exploring is installation of packages from npm to be handled transparently.

Similarly to how Deno fetches, caches and transpiles code it discovers from URLs, we would like to provide similar functionality for fetching code from npm directly. So you wouldn't need to do npm install before running your code. You could just Deno run it and Deno will figure out if it's missing some packages and fetch them from npm.

And circling back to the native add-ons, we are actually making great strides on the NAPI integration and we are very confident that we could provide this integration to at least somehow bridge the gap between the two runtimes in terms of binary runtime extensions.

Your feedback on this whole compatibility layer is extremely us and would be very, very helpful for our efforts and what should we prioritize first. So, I encourage you very much to give this dino run dash dash compat a go and let us know what you think. If you had any problems running some of the packages you use or everything just worked out of the box, please try and let us know.

And lastly, if you ever wanted to contribute to Open Source, dino is always appreciating all contributions. We have over 600 contributors. We have tens of different projects, spanning from REST projects, which are very low level parts of their runtime, through higher level functionalities like AST parsing, formatters, linters, bundlers, to projects written in TypeScript, JavaScript, and even projects that are completely front end and don't touch backend at all. So yeah, please go to this URL and see if there's something that interests you and, again, all contributions are extremely welcome in dino and we appreciate all of them.

Right, so I went pretty fast, we had a lot of ground to cover but hopefully you found this talk interesting. Should you have any questions, please feel free to send me an email at this email address, and this is our website at dino.lab. Thank you for having me here. And see you soon. Thank you.

Hi. Thanks for having me here. Yeah, thank you for this great talk. So let's see your poll, which knowledge is packages? Would you like to run on denim? So we have next please, next JS web pack remix mocha parcel in that order next day is winning within 34%. Why do you think about this? To be honest, I'm not really surprised. And I have good good news for everyone who voted for next JS, we are actively working on getting next GS support and hopefully it should arrive within a few weeks. I'm actually kind of surprised that next GS is second and web pack is third. But oh, well, it's great to get feedback from from our attendees. And this will help us tremendously in our efforts for better compatibility with node. Yeah, I would have thought web pack will be more popular. But yeah, those are the results.

Okay, so the question is, Christina is asking, why use Dino instead of node.js? Is they not ready for production? Right. So one of the biggest advantages of Dino is that we not only give you a runtime that can execute JavaScript, but JavaScript and TypeScript, but also we give you a full tool chain. So if you're starting a new project, you need to worry about setting up a formatter, linter, test runner, bundler, documentation, generator, all of this. This is already set up for you. And we ship it to you in a single executable. So you can get your project with proper continuous integration and tests in a matter of minutes. So no more bike shedding on which test runner we're going to use this time. And what are the prettier settings that we want to apply? This is all done for you and it's ready to go as soon as you download Dino. And is Dino production ready? Yes, definitely. We very we dogfood Dino heavily in our Dino land company and we use it heavily for a multitude of purposes, but there are also many companies using Dino in production already, you can see this list going to the wiki on our Dino land Dino repository. Thank you. And you are is asking, why do you try making node.js compatibility in Dino? Should this be an effort of the developer support the code for Dino? Right. So why we're doing this? Well, the answer is simple. NPM is the biggest package library ecosystem in the world.

It's bigger than Ruby's gems or Python's PyPy. And a lot of users provided us with feedback that they would love to use Dino, but since they cannot use libraries from NPM, it's not that compelling of a reason. So that's why we're working on this.

But to answer the second part of the question, should you be working code or writing especially for Dino? Well, it's really up to you. But I have, I can suggest you to check dnt, which is another project from from the Dino company. And it's a tool that allows you to offer the code for Dino first. But then it will transparently be set up to be published to NPM. So you can actually target both Dino and node using a single Dino code base. And again, since we're shipping, shipping a full tool chain, you might find setting up and maintaining your project easier than having to rely on multiple dependencies from NPM to do your formatting, linting, etc.

OK, thank you. Yeah, that makes total sense. Also, what about making a converter instead of making compatibility? Yeah, we explored this avenue. However, it sounds a lot easier than it could be done. So a lot of code that is there and authored for Node, it still relies on the callback-based API. And some of that flow of code, in this case, is not so obvious, and converting it automatically would be extremely error-prone compared to providing first-class compatibility. So instead of trying to do some conversions and then trying to fix errors, which you would probably have to do on your own, it seems easier and more bulletproof to provide these APIs that are available in Node already. And we will test and make sure that they will work the same in Deno. Thank you.

What level do you think a dev should be to collaborate in Deno? Sorry, can you repeat the question, please? What level do you think a dev should be to collaborate in Deno? I guess the person is trying to ask, it should be maybe senior or junior. I don't know what, maybe it's that? What level to collaborate in Deno? Sorry, to code in Deno or to contribute to Deno? To collaborate. Collaborate, right. Well, everyone is very welcome. We've got tens of different repositories and they span multiple projects because Deno is not a single thing. It's actually built out of different blocks which are either Rust grades or JavaScript or TypeScript projects. So everyone can find something that they could contribute to and doesn't matter how skilled you are. So if you feel comfortable in JavaScript, we got some projects that use only JavaScript. If you prefer TypeScript, we've got this huge project called Deno STD which is a standard library that you can contribute to and either help us with the node compatibility or provide a new module. Or if you prefer Rust, you can help us with the Deno itself or some of the tooling because all of the tools like formatter and linter, et cetera they're written in Rust. So drop to our Discord server and I'm sure we can point you to the right direction based on what you're interested in and how confident you feel in either of these languages. No, thank you.

Are there any tools to write robust and report friendly tests in Deno? That's a question from Ivan. To test what, sorry? Are there any tools to write robust and report friendly tests in Deno? Right, so I would say that what we provide in Deno itself, the built-in utility is robust enough. So there's a simple API, a single Deno.test function that you can use to register a test. And it doesn't matter if it's a unit test or an integration test, we've got a bunch of helpers like assertions for utilities to work with sub-processes.

So you can test anything you want. And the great thing about Deno test runner is that it's not magical. It does exactly what it's supposed to do. It executes this test file and reports any errors. It doesn't mess with your global scope like sometimes Jess does. It doesn't add any new APIs. It's exactly the same thing as running just your program with this only one change that you've got this dno.test API available to you. And we actually provide a few helpers that make sure that your code doesn't leak memory or that you clean up after you're done with some resources like files from file system or network sockets. So definitely check out Deno's built in test runner. I think you will find it robust enough and simple enough to pick it up in a matter of minutes. Thank you.

Will we be able to run all existing NPM packages in Deno? No, unfortunately no. We are aiming for say 80 to 90% compatibility. There's a problem with some of packages that they, for example, rely on the native add-ons which we won't be really able to support because add-ons offered for Node.js are mostly written in C++ and we don't really have a way to provide compatible API. However, if you use an API, then we think we might have an answer to that and we are actually on the prototype stage of providing an API integration which adding Wasm would provide two different ways to run native code, both in Deno and in Node.js. Thank you.