Node.js Compatibility in Deno

Hi folks, my name is Bartek and I'm a member of the deno core team. Today we're going to discuss node.js compatibility in deno. In this talk I would like to give you a quick rundown of what is deno and try to answer a few questions regarding compatibility with Node. So can deno run apps and libraries offered for node.js? What are the trade-offs? How does it all work? And what is next on the plate for us? So let's get right to it. deno 1.1. deno is a new command line runtime for javascript and for typescript. It's similar to Node that you can execute your javascript code, but the difference is that we also support typescript as a first class citizen. Word of caution here, deno doesn't actually execute your typescript. It just transpiles it to javascript underneath, behind the scenes, so you don't need to set up a build chain or anything like that. deno also aims to be fun and productive. We like to think of it as a bag of Lego blocks. deno is a single executable file. This file is all you need to run any deno program and we will always distribute deno as such. deno works on Linux, it works on Mac and on Windows, so wherever you go, new server, new laptop, whatever, there's only a single command you need to execute to get deno and you're ready to go. deno has something we like to call secure execution by default. What that means is none of your programs can access your file system, your environmental variables, spun-stock processes, or even open an internet connection unless you explicitly allow it to. In this short example, we can also see that you can directly link to third-party code and you can execute it even though you don't trust this code. In here we are executing an example from deno STD, which is deno's standard library. It's a set of modules that are guaranteed to work with deno, but it could have been any URL in the world you can think of. In this case, where there are no flags provided besides run subcommand, this program cannot do anything to your computer. If you want to access a file system or spawn a subprocess or open an internet connection, you can give deno some flags and tell it to allow your program to actually do these things. As you can see here, some of these flags allow you to scope this even further. For example, we can allow our program to read from the file system, however, only from the slash tmp directory, or we can tell your program to allow open an internet connection, however, only scoped to google.com domain. deno allows you to link to any third-party code using URLs. It works just like it does in browser. In this example, we can see ES module import. We are importing magenta function again from deno STD. This time we are doing this from fmt slash colors module. So this module resolution is simple and compatible with browsers, and deno has no notion of node modules or index.js. Or actually, that's what this talk is about, after all. deno doesn't depend on a centralized server, that is, you are not explicitly tied only to npm if you want to link to your packages or install new packages, or a private registry if you have one. You can pull code from anywhere on the internet, either public servers or private servers, if you provide proper authorization headers. deno as a runtime fetches, caches, and compiles this code or transpiles it transparently for you on the first run. After the first run, the code is already in cache, so there's no need to download it again. And in fact, deno won't download the code again unless you explicitly ask it to. So why are we even thinking or working on node.js compatibility? Well, npm is arguably the biggest package ecosystem in the world, and users just want to use this package ecosystem. When deno first came out with version 1.0, we stated explicitly that we are not compatible with npm. And that was fine, however, some users still find lack of support for npm a huge hurdle for adoption. So some of our users want to try deno instead of node.js. However, the cost to switch from one runtime to the other often proves too high to even try and do it. So by providing node.js compatibility, we want to make the migration path smoother and easier for users. So let's try to answer this question. Can deno actually run apps, libraries, or packages that were written for node.js? And yes, but. And before we get to this but, images speak more than a thousand words, so I'm going to give you a quick example, or actually two examples of code offered for node.js that is already running in deno. In the first example, we can see the very popular tool called eslint that is executing directly in deno. So we pooled the sources for eslint. For the sake of this example, say we already used npm. And now we are executing this code using deno. Important thing to notice here is that we are providing this dash-dash-compat flag, which tells deno to actually provide this node.js compatibility. And then we are passing dash-dash-unstable, which is currently required due to some limitations in the api. So some of APIs are not yet set in stone and may change in the future. And this flag conveys the message that, you know, this is unstable, so something might change, but don't worry, you'll be okay. The second example is actually npm itself. We can already execute npm in deno and use it to install some other packages. One thing you might notice here is that there are some prompts here saying not implemented, for example, process.onCodeException or process.onUnhandledRejection. We'll get to these in just a moment. So yes, deno can execute apps, libraries, packages, output for node.js. However, not yet all of them are supported. We are focusing our efforts on enabling users to run the most popular packages and tools at the moment. So we already saw ESLint and npm, but we are also making great progress on getting yarn to work, vite, next.js, which is probably the most popular framework at the moment, and Mocha, which is also one of the most popular testing frameworks at the moment. So yes, deno can run code offered for node.js, but not all of it. And the reason is we are still working on porting or providing polyfills for some of the built-in node.js modules, modules such as TLS or child process. These are being worked on. However, their api service is huge. So you can spawn subprocess just like you would do in Node with deno. However, not all of the APIs will work at the moment. Another important thing is CJS and ESN integration. deno from day one was ESM first. It didn't understand CommonJS modules. However, due to the fact that a lot of code offered for node.js is still using CommonJS, it's crucial to provide this integration or interoperability between the two formats. And by the way, this is actually supported out of the box by node.js. So we're just trying to catch up. Even though we are executing code offered for node.js, this code is still subject to the same permission rules or allow flags that we saw at the beginning of the presentation. So that means that you can execute code offered for node.js in deno and get an additional benefit of being able to sandbox your programs to certain directories or certain internet domains or don't allow it to spawn subprocesses. So what are the trade-offs of this compatibility? Well, some APIs that exist in node.js and in deno and in browsers, they're just not compatible. A good example is setTimeout. This is a global api. It is available in node. It is available in browsers. It is available in deno. In node.js, you can also import it from the timers module. However, the difference is that in node, when you call setTimeout, you will get back a timeout class instance. However, in browsers and in deno, you will get back a number, which is an identifier for this timeout. In node, due to this return type, a lot of packages rely on it to, for example, unrev the timer, that is, tell node that if this timeout is the last thing to execute or you don't wait on anything else, just don't wait for this timeout and let's finish running right now. And similarly, in browsers and in deno, there's a lot of code that relies on the fact that you might want to do a type of check based on this return value. So there's a lot of code that does type of ID equals to number. And this is a really hard problem. We've been scratching our heads for three months already on it. So if any folks from node's technical steering committee are listening to this talk, please reach out and let's figure out how we can make progress on getting this more compatible between the environments. Another trade off is that the developer experience is not so great, but at the moment we are working on this, we are making progress and are thinking about how this could be made nicer for users to switch between node.js and deno transparently. And why this DX is not so good? So you need to pass a lot of flags to deno to actually tell deno to behave like node.js. You need to pass dash dash compat, so this flag actually sets up globals like process. Process is a global variable that is available in node.js environment. However, it's not available in deno by default. But if you pass this dash dash compat, it will tell deno to actually prepare this global variable for you. You need to pass dash dash unstable. Once again, some of the APIs are not yet done and they will undergo some changes or vetting process. So this is just to signal to users that, you know, there might be some dragons here and there, but that's okay. We're moving fast. And most often you also need to pass dash dash allow read flag. And this is required for Cjs modules, for CommonJS, because CommonJS resolution relies heavily on probing the file system and Node modules directories. And since it's implemented 100% in user land at the moment, it's subject to deno permission checks. Another trade off is that some APIs might not be supported at all in the future. So an inspector module or a V8 module, they might not make it to deno at all. We are still discussing this. This is both a technical problem and a fact. If we want to support these APIs, because we think some of them are questionable. And one rather big caveat is that native add-ons will not be supported. Native add-ons in node.js rely on Node.JPG and are basically C++, authored in C++, while deno is actually authored in rust. So it's not so easy to reconcile this problem due to api stability. However, I might have a good answer to this problem on the next slides. So before we get to that, let's briefly talk how does it all work. You might be surprised, but most of these APIs coming from node.js are polyfilled in user land. Once again, I'm going to refer back to this project called DenoSTD, which is a standard library that we ship with deno. It's a collection of modules authored in typescript that are guaranteed to work with deno. And in there, we have this Node module. And this Node module contains a lot of files that basically polyfill all of the APIs available in various built-in node.js modules. I encourage you to go to this link and check it for yourself. You might find this very educational. One problem with this is the code that is the runtime code that provides built-in modules in node.js is authored as common JS modules, whereas all of these code in DenoSTD is actually ES modules. So not only did we have to rewrite most of the code to typescript and make it type safe, we also had to do a rather significant reorganization to make the code work because of different semantics of common JS and ES modules. So this is all fine and great that we could do with this in user land. However, there are some very intricate APIs that are not really possible to be done in user land. And two of the examples would be process.unhandledRejection, so an event handler that is fired off when there is a promise that is being rejected but has no catch handler. In deno, up to this point, this was always a hard failure. Your program would exit. Similarly, process.nextStick allows you to schedule a callback to being called after nextStick of the event loop. And we tried polyfilling these APIs in user land. However, due to very strict semantics of where they should be called or when or what should happen that is altering how deno behaves, we couldn't just do this in user land. So we had to add some direct support for these APIs in deno's core. So what's next for node.js compatibility in deno? Well, we want to ship the first version, the first publicly available version of this compatibility layer in the second quarter of this year. And some of the things we would like to do is firstly explore so-called superset approach, where deno basically supports node.js APIs out of the box without any special flags or different modes running that deno would run in. So this dash-dash compat flag might not be necessary in the future. Another idea that we are exploring is installation of packages from npm to be handled transparently, similarly to how deno fetches, caches, and transpiles code it discovers from URLs. We would like to provide similar functionality for fetching code from npm directly. So you wouldn't need to do npm install before running your code. You could just deno run it and deno will figure out if it's missing some packages and fetch them from npm. And circling back to the native add-ons, we are actually making great strides on the NAPI integration, and we are very confident that we could provide this integration to at least somehow bridge the gap between the two runtimes in terms of binary runtime extensions. Your feedback on this whole compatibility layer is extremely important to us and would be very, very helpful for our efforts and what should we prioritize first. So I encourage you very much to give this deno run dash-dash compat a go and let us know what you think. If you had any problems running some of the packages you use or everything just worked out of the box, please try and let us know. And lastly, if you ever wanted to contribute to open source, deno is always appreciating all contributions. We have over 600 contributors. We have tens of different projects, spanning from RAS projects, which are very low-level parts of the runtime, through higher-level functionalities like AST parsing, formatters, bundlers, to projects written in typescript, javascript, and even projects that are completely frontend and don't touch backend at all. So yeah, please go to this URL and see if there's something that interests you. And again, all contributions are extremely welcome in deno, and we appreciate all of them. Right, so I went pretty fast. We had a lot of ground to cover, but hopefully you found this talk interesting. Should you have any questions, please feel free to send me an email at this email address, and this is our website at deno.lan. Thank you for having me here, and see you soon. Thank you. Hi. Thanks for having me here. Yeah, thank you for this great talk. So let's see your poll, which node.js packages would you like to run on deno? So we have next.js, Nest.js, webpack, remix, Mocha, Parcel, in that order. next.js winning within 34%. What do you think about this result? To be honest, I'm not really surprised, and I have good news for everyone who voted for next.js. We are actively working on getting next.js support, and hopefully it should arrive within a few weeks. I'm actually kind of surprised that Nest.js is second and webpack is third, but oh well. It's great to get feedback from our attendees, and this will help us tremendously in our efforts for better compatibility with Node. Yeah, I would have thought webpack would be more popular, but yeah, those are the results. Okay, so the question is, Christina is asking, why use deno instead of next.js? Is deno ready for production? Right, so one of the biggest advantages of deno is that we not only give you a runtime that can execute javascript and typescript, but also we give you a full tool chain. So if you're starting a new project, you don't need to worry about setting up a formatter, linter, test runner, bundler, documentation generator, all of this. This is already set up for you, and we ship it to you in a single executable, so you can get your project with proper continuous integration and tests in a matter of minutes. So no more bike shedding on which test runner we're going to use this time and what are the prettier settings that we want to apply. This is all done for you, and it's ready to go as soon as you download deno. And is deno production ready? Yes, it definitely is. We dogfood deno heavily in our DenoLand company, and we use it heavily for a multitude of purposes, but there are also many companies using deno in production already. You can see this list going to the wiki on our DenoLand deno repository. Thank you. And you are asking, why do you try making node.js compatibility in deno? Should this be an effort of the library developers to port the code for deno? Right. So why we're doing this? Well, the answer is simple. npm is the biggest package library ecosystem in the world. It's bigger than Ruby's gems or Python's PyPy, and a lot of users provided us with a feedback that they would love to use deno, but since they cannot use libraries from npm, it's not that compelling of a reason. So that's why we're working on this. But to answer the second part of the question, should you be porting code or writing especially for deno? Well, it's really up to you, but I can suggest you to check DNT, which is another project from the deno company. And it's a tool that allows you to offer the code for deno first, but then it will transparently be set up to be published to npm. So you can actually target both deno and Node using a single deno code base. And again, since we're shipping a full tool chain, you might find setting up and maintaining your project easier than having to rely on multiple dependencies from npm to do your formatting, linting, et cetera. Okay, thank you. Yeah, that makes total sense. Also, what about making a converter instead of making compatibility? Yeah, we explored this avenue. However, it sounds a lot easier than it could be done. So a lot of code that is there and authored for Node, it still relies on the callback based api. And some of the flow of code in this case is not so obvious and converting it automatically would be extremely error prone compared to providing first class compatibility. So instead of trying to do some conversions and then trying to fix errors, which you would probably have to do on your own, it seems easier and more bulletproof to provide these APIs that are available in Node already. And we will test and make sure that they will work the same in deno. Thank you. What level do you think a dev should be to collaborate in deno? Sorry, can you repeat the question, please? What level do you think a dev should be to collaborate in deno? I guess the person is trying to ask, like, it should be like maybe senior or junior. I don't know what, maybe it's that, like what level to collaborate in deno? Sorry, to code in deno or to contribute to deno? To collaborate. Collaborate, right. So, well, everyone is very welcome. We've got tens of different repositories and they span multiple projects because deno is not a single thing. It's actually built out of different blocks, which are either RAS crates or javascript or typescript projects. So everyone can find something that they could contribute to and doesn't matter how skilled you are. So if you feel comfortable in javascript, we got some projects that use only javascript. If you prefer typescript, we've got this huge project called deno STD, which is a standard library that you can contribute to and either help us with the Node compatibility or provide a new module. Or if you prefer RAS, you can help us with the deno itself or some of the tooling because all of the tools like Formatter, Linter, et cetera, they're written in RAS. So drop to our Discord server and I'm sure we can point you to the right direction based on what you're interested in and how confident you feel in either of these languages. Thank you. Are there any tools to write robust and report-friendly tests in deno? That's a question from Ivan. To test what? Sorry. Are there any tools to write robust and report-friendly tests in deno? Right. So I would say that what we provide in deno itself, the built-in utility is robust enough. So there's a simple api, a single deno.test function that you can use to register a test and it doesn't matter if it's a unit test or an integration test. We've got a bunch of helpers like assertions or utilities to work with sub-processes. So you can test anything you want. And the great thing about deno Test Runner is that it's not magical. It does exactly what it's supposed to do. It executes this test file and reports any errors. It doesn't mess with your global scope like sometimes Jest does. It doesn't add any new APIs. It's exactly the same thing as running just your program with this only one change that you've got this deno.test api available to you. And we actually provide a few helpers that make sure that your code doesn't leak memory or that you clean up after you're done with some resources like files from file system or network sockets. So definitely check out deno's built-in test runner. I think you will find it robust enough and simple enough to pick it up in a matter of minutes. Thank you. And will we be able to run all existing npm packages in deno? No. Unfortunately no. We are aiming for say 80 to 90% compatibility. There's a problem with some of packages that they, for example, rely on the native add-ons, which we won't be really able to support because add-ons offered for node.js are mostly written in C++ and we don't really have a way to provide compatible api. However, if you use an api, then we think we might have an answer to that. And we are actually on the prototype stage of providing an api integration, which adding Wasm would provide two different ways to run native code, both in deno and in node.js. Thank you. Can deno run native add-ons? Yes. So one of the ways is Wasm, if you consider it as native enough. Another one is this NAPI integration that we are working on. And the third way is we actually have a proper FFI or foreign function interface layer. So you can call into any code that uses C calling convention. So if you have an existing C++ library or something out for it in NIM or ZIG or basically anything that uses C calling convention, you can use it from deno. And this is actually quite a new thing. We decided to replace rust only native extensions with something that is available to use from many different languages. And we are actually seeing some exciting developments in this space, like running Python interpreter directly from deno using this FFI api. So check out this api. You might find it very powerful and interesting. And of course, we're looking for feedback if some of the things are not suited for your use case. Yeah, thank you. And the last question we have for today is, will deno support NAPI? Yeah, so I already answered this several times. It's not set in stone that we will definitely do it, but we have a proof of concept working. We are able to run multiple modules from Parcel that use NAPI and some other modules like Dprint or XUtter. And this is using NAPI. So we are fairly confident that we'd be able to provide this integration. We are still discussing how this fits into deno's permission model, et cetera. But yeah, it's definitely possible. And be on the lookout for news about this in the coming weeks and months. Thank you. Thank you very much. This was very, very interesting. And I want to invite everyone to a special chat to Bart's special room. So thank you very much. And I'll see you there. Bye. Thank you.