Can Deno run apps and libraries authored for Node.js? What are the tradeoffs? How does it work? What’s next?
Node.js Compatibility in Deno
AI Generated Video Summary
Deno aims to provide Node.js compatibility to make migration smoother and easier. While Deno can run apps and libraries offered for Node.js, not all are supported yet. There are trade-offs to consider, such as incompatible APIs and a less ideal developer experience. Deno is working on improving compatibility and the transition process. Efforts include porting Node.js modules, exploring a superset approach, and transparent package installation from npm.
1. Introduction to Deno and Node.js Compatibility
Hi, folks! My name is Bartek and I'm a member of the Deno core team. Today, we're going to discuss Node.js compatibility in Deno. Can Deno run apps and libraries offered for Node.js? What are the trade-offs? How does it all work and what is next on the plate for us?
Hi, folks! My name is Bartek and I'm a member of the Deno core team. Today, we're going to discuss Node.js compatibility in Deno. In this talk, I would like to give you a quick rundown of what is Deno, and try to answer a few questions regarding compatibility with Node.
2. Deno Execution and Node.js Compatibility
Deno is a single, executable file that works on Linux, Mac, and Windows. It offers secure execution by default, preventing unauthorized access to file systems and internet connections. Deno allows linking to third-party code using URLs and fetches, caches, and compiles or transpiles the code transparently. While Deno is not initially compatible with npm, some users find the lack of npm support a hurdle for adoption.
Deno is a single, executable file. This file is all you need to run any Deno program, and we will always distribute Deno as such. Deno works on Linux, it works on Mac and on Windows, so wherever you go, new server, new laptop, whatever, there's only a single command you need to execute to get Deno and you're ready to go.
Deno has something we like to call secure execution by default. What that means is none of your programs can access your file system, your environmental variables, sponsor processes, or even open an internet connection unless you explicitly allow it to. In this short example we can also see that you can directly link to third-party code and you can execute it even though you don't trust this code. So, in here we are executing an example from DenoSTD, which is Deno's standard library, it's a set of modules that are guaranteed to work with Deno, but it could have been any URL in the world you can think of, and in this case where there are no flags provided besides run sub-commands, this program cannot do anything to your computer. But if you want to access a file system or spawn a sub-process or open an internet connection, you can give Deno some flags and tell it to allow your program to actually do these things. And as you can see here some of these flags allow you to scope this even further. So for example, we can allow our program to read from the file system, however only from the slash tmp directory. Or we can tell your program to allow open an internet connection, however, only scoped to google.com domain.
Deno allows you to link to any third-party code using URLs. It works just like it does in browser. In this example we can see our es module import. We are importing a magenta function, again from Deno STD. This time we are doing this from fmt slash colors module. So this module resolution is simple and compatible with browsers and Deno has no notion of node modules or indexJS. Or actually that's what this talk is about after all. Deno doesn't depend on a centralized server. That is, you are not explicitly tied only to npm if you want to link to your packages or install new packages or a private registry if you have one. You can pull code from anywhere on the internet, either public servers or private servers if you provide proper authorization headers. Deno as a runtime fetches, caches and compiles this code, or transpiles it, transparently for you on the first run. After the first run, the code is already in cache, so there is no need to download it again. In fact, Deno won't download the code again unless you explicitly ask it to.
So, why are we even thinking or working on Node.js compatibility? Well, npm is arguably the biggest package ecosystem in the world, and users just want to use this package ecosystem. When Deno first came out with v1.0, we stated explicitly that we are not compatible with npm. And that was fine. Here, some users still find lack of support for npm a huge hurdle for adoption. So some users want to try Deno instead of Node.js. However, the cost to switch from one runtime to the other often proves too high to even try and do it.
3. Deno and Node.js Compatibility
By providing Node.js compatibility, Deno aims to make the migration path smoother and easier for users. Deno can run apps, libraries, and packages offered for Node.js, but not all of them are supported yet. Efforts are focused on enabling users to run the most popular packages and tools. Deno is working on porting or providing polyfills for some built-in Node.js modules and ensuring CJS and ESM integration.
So, by providing Node.js compatibility, we want to make the migration path smoother and easier for users. So let's try to answer this question. Can Deno actually run apps, libraries or packages that were written for Node.js? And yes, but... And before we get to this but, images speak more than a thousand words.
So I'm going to give you a quick example, or actually two examples of code offered for Node.js that is already running in Deno. In the first example, we can see the very popular tool called ESlint that is executing directly in Deno. So we pulled the sources for ESlint. For the sake of this example, say, we already used npm. And now we are executing this code using Deno. Important thing to notice here is that we are providing this --"compat flag", which tells Deno to actually provide this Node.js compatibility. And then we are passing --"unstable", which is currently required due to some limitations in the API. So some of the APIs are not yet set in stone and may change in the future. And this flag conveys the message that, you know, this is unstable, so something might change, but don't worry, it will be okay.
The second example is actually npm itself. We can already execute npm in Deno and use it to install some other packages. One thing you might notice here is that there are some prompts here saying, not implemented. For example, process.onCodeException or process.onUnhandledRejection. We'll get to these in just a moment. So yes, Deno can execute apps, libraries, packages offered for Node.js. However, not yet all of them are supported. We are focusing our efforts on enabling users to run the most popular packages and tools at the moment. So we already saw ESLint and npm, but we are also making great progress on getting Yarn to work, Vite, Next.js, which is probably the most popular framework at the moment, and Mocha, which is also one of the most popular testing frameworks at the moment. So yes, Deno can run code offered for Node.js, but not all of it.
The reason is we are still working on porting or providing polyfills for some of the built-in Node.js modules, such as TLS or ChildProcess. These are being worked on, however, their API service is huge, so you can spawn subprocess just like you would do in Node with Deno, however not all of the APIs will work at the moment. Another important thing is CJS and ESM integration. Deno from day one was ESM first. And they didn't understand CommonJS modules. However, due to the fact that a lot of code offered for Node.js is still using CommonJS, it's crucial to provide this integration or interoperability between the two formats. And by the way, this is actually supported out of the box by Node.js, so we're just trying to catch up.
4. Trade-offs and Developer Experience
Even though we can execute Node.js code in Deno, there are trade-offs to consider. Some APIs are not compatible, like setTimeout, which returns different types in Node.js, browsers, and Deno. The developer experience is not ideal, as it requires passing multiple flags to Deno. However, efforts are being made to improve this and make the transition between Node.js and Deno smoother. Passing flags like ____ compat sets up globals like process, and ____ allow read is necessary for CommonJS modules.
Even though we are executing code offered for Node.js, this code is still subject to the same permission rules or allow flags that we saw at the beginning of the presentation. So that means that you can execute code offered for Node.js in Bino and get an additional benefit of being able to sandbox your programs to certain directories or certain internet domains or don't allow it to spawn sub-processes.
So what are the trade-offs of this compatibility? Well, some APIs that exist in Node.js and in Deno and in browsers, they are just not compatible. A good example is setTimeout. This is a global API. It is available in Node.js, it is available in browsers, it is available in Deno. In Node.js you can also import it from the Timers module. However, the difference is that in Node.js, when you call setTimeout, you will get back a TimeoutClass instance. However, in browsers and in Deno, you will get back a number, which is an identifier for this Timeout. In Node.js, due to this return type, a lot of packages rely on it to, for example, unrev the timer. That is, tell Node.js that if this Timeout is the last thing to execute or you don't wait on anything else, just don't wait for this Timeout and let's finish running right now. Similarly, in browsers and in Deno, there's a lot of code that relies on the fact that you might want to do a typeof check based on this return value. There's a lot of code that does typeof ID equals to number. This is a really hard problem. We've been scratching our head for three months already on it. If any folks from Node's technical steering community are listening to this talk, please reach out and let's figure out how we can make progress on getting this more compatible between the environments.
Another trade-off is that the developer experience is not so great. But at the moment, we're working on this, we're making progress and are thinking about how this could be made nicer for users to switch between Node.js and Deno transparently. And why this DX is not so good? So you need to pass a lot of flags to Deno to actually tell Deno to behave like Node.js. You need to pass ____ compat. So this flag actually sets up globals like process. Process is a global variable that is available in Node.js environment. However, it's not available in Deno by default. But if you pass this ____ compat, it will tell Deno to actually prepare this global variable for you. You need to pass ____ hands table. Once again, some of the APIs are not yet done, and they will undergo some changes or vetting process. So this is just to signal to users that, you know, there might be some dragons here and there, but that's okay. We're moving fast. Most often, you also need to pass ____ allow read flag, and this is required for CJS modules, for CommonJS, because CommonJS resolution relies heavily on probing the file system and Node modules directories. And since it's implemented 100% in user land at the moment, it's subject to Deno permission checks.
5. Node.js Compatibility and Future Plans
Some APIs might not be supported in the future, and native add-ons in Node.js will not be supported in Deno due to ABI stability. Most Node.js APIs are polyfilled in userland using the DinoSTD project. However, some intricate APIs cannot be polyfilled and require direct support in Deno's core. The next steps for Node.js compatibility in Deno include shipping the first version of the compatibility layer and exploring a superset approach and transparent package installation from npm.
Another trade-off is that some APIs might not be supported at all in the future. So an Inspector module, or a V8 module, they might not make it to Deno at all. We are still discussing this. This is both a technical problem, and in fact, if we want to support these APIs, it's because we think some of them are questionable.
One rather big caveat is that native add-ons will not be supported. Native add-ons in Node.js rely on Node.JPG, and are basically C++, while Deno is actually authored in Rust. So, it's not so easy to reconcile this problem due to ABI stability. However, I might have a good answer to this problem on the next slides.
So, before we get to that, let's briefly talk how does it all work. You might be surprised, but most of these APIs coming from Node.js are polyfilled in userland. Once again, I'm going to refer back to this project called DinoSTD, which is a standard library that we ship with Dino. It's a collection of modules offered in TypeScript that are guaranteed to work with Dino. And in there we have this Node module, and this Node module contains a lot of files that basically polyfill all of the APIs available in various built in Node.js modules. I encourage you to go to this link and check it for yourself. You might find this very educational.
One problem with this is the code that is the runtime code that provides built in modules and Node.js is offered as CommonJS modules, whereas all of these codes in DinoSTD is actually ES modules. So not only did we have to rewrite most of the code to TypeScript and make it type safe, we also had to do a rather significant reorganization to make the code work because of different semantics of CommonJS and ES modules.
So this is all fine and great that we could do with this in Userland. However, there are some very intricate APIs that are not really possible to be done in Userland. And two of the examples would be a process dot unhandled rejection. So, an event handler that is fired off when there is a promise that is being rejected but has no catch handler. In Deno up to this point, this was always a hard failure, your program would exit. Similarly, process dot next tick allows you to schedule a callback to being called after next tick of the event loop. And we tried polyfilling these APIs in Userland. However, due to very strict semantics of where they should be called or when or what should happen, that is altering how Deno behaves, we couldn't just do this in Userland. So we had to add some direct support for these APIs in Deno's core.
So what's next for Node.js Compatibility in Deno? Well, we want to ship the first version, the first publicly available version of this Compatibility layer and second quarter of this year. And some of the things we would like to do is firstly, explore so-called superset approach where Deno basically supports Node.js APIs out-of-the-box without any special flags or different modes running that Deno would run. And so this dash-dash compact flag might not be necessary in the future. Another idea that we are exploring is installation of packages from npm to be handled transparently.
6. Fetching Code from npm and NAPI Integration
We are working on fetching code from npm directly, eliminating the need for npm install. NAPI integration is being developed to bridge the gap between Deno and Node.js in terms of binary runtime extensions. Your feedback is crucial. Try dino run --compat and let us know if you encounter any issues. Contributions to various projects in Deno are highly appreciated.
Similarly to how Deno fetches, caches and transpiles code it discovers from URLs, we would like to provide similar functionality for fetching code from npm directly. So you wouldn't need to do npm install before running your code. You could just Deno run it and Deno will figure out if it's missing some packages and fetch them from npm.
And circling back to the native add-ons, we are actually making great strides on the NAPI integration and we are very confident that we could provide this integration to at least somehow bridge the gap between the two runtimes in terms of binary runtime extensions.
Your feedback on this whole compatibility layer is extremely us and would be very, very helpful for our efforts and what should we prioritize first. So, I encourage you very much to give this dino run dash dash compat a go and let us know what you think. If you had any problems running some of the packages you use or everything just worked out of the box, please try and let us know.
Conclusion and Node.js Compatibility
We covered a lot of ground in this talk. If you have any questions, feel free to contact us via email or visit our website at dino.lab. Thank you for having me. Let's discuss the poll results. Next.js is the most popular choice, and we're actively working on adding Next.js support. It's interesting to see the feedback from attendees. Now, let's address a question about why to use Dino instead of Node.js. Dino provides a full toolchain along with the runtime, making it easy to start new projects. It's production-ready and already used by many companies. The focus on Node.js compatibility is driven by the importance of the NPM ecosystem.
Right, so I went pretty fast, we had a lot of ground to cover but hopefully you found this talk interesting. Should you have any questions, please feel free to send me an email at this email address, and this is our website at dino.lab. Thank you for having me here. And see you soon. Thank you.
Hi. Thanks for having me here. Yeah, thank you for this great talk. So let's see your poll, which knowledge is packages? Would you like to run on denim? So we have next please, next JS web pack remix mocha parcel in that order next day is winning within 34%. Why do you think about this? To be honest, I'm not really surprised. And I have good good news for everyone who voted for next JS, we are actively working on getting next GS support and hopefully it should arrive within a few weeks. I'm actually kind of surprised that next GS is second and web pack is third. But oh, well, it's great to get feedback from from our attendees. And this will help us tremendously in our efforts for better compatibility with node. Yeah, I would have thought web pack will be more popular. But yeah, those are the results.
Collaborating in Deno and Compatibility
It's bigger than Ruby's gems or Python's PyPy. And a lot of users provided us with feedback that they would love to use Dino, but since they cannot use libraries from NPM, it's not that compelling of a reason. So that's why we're working on this.
But to answer the second part of the question, should you be working code or writing especially for Dino? Well, it's really up to you. But I have, I can suggest you to check dnt, which is another project from from the Dino company. And it's a tool that allows you to offer the code for Dino first. But then it will transparently be set up to be published to NPM. So you can actually target both Dino and node using a single Dino code base. And again, since we're shipping, shipping a full tool chain, you might find setting up and maintaining your project easier than having to rely on multiple dependencies from NPM to do your formatting, linting, etc.
OK, thank you. Yeah, that makes total sense. Also, what about making a converter instead of making compatibility? Yeah, we explored this avenue. However, it sounds a lot easier than it could be done. So a lot of code that is there and authored for Node, it still relies on the callback-based API. And some of that flow of code, in this case, is not so obvious, and converting it automatically would be extremely error-prone compared to providing first-class compatibility. So instead of trying to do some conversions and then trying to fix errors, which you would probably have to do on your own, it seems easier and more bulletproof to provide these APIs that are available in Node already. And we will test and make sure that they will work the same in Deno. Thank you.
Tools for Writing Tests in Deno
Deno provides a built-in utility for writing robust and report-friendly tests. The Deno.test function allows you to register tests, whether they are unit tests or integration tests. Deno's test runner executes test files and reports any errors without affecting the global scope or adding new APIs. It provides helpers to prevent memory leaks and clean up resources. Deno's built-in test runner is robust, simple to use, and worth exploring.
Are there any tools to write robust and report friendly tests in Deno? That's a question from Ivan. To test what, sorry? Are there any tools to write robust and report friendly tests in Deno? Right, so I would say that what we provide in Deno itself, the built-in utility is robust enough. So there's a simple API, a single Deno.test function that you can use to register a test. And it doesn't matter if it's a unit test or an integration test, we've got a bunch of helpers like assertions for utilities to work with sub-processes.
So you can test anything you want. And the great thing about Deno test runner is that it's not magical. It does exactly what it's supposed to do. It executes this test file and reports any errors. It doesn't mess with your global scope like sometimes Jess does. It doesn't add any new APIs. It's exactly the same thing as running just your program with this only one change that you've got this dno.test API available to you. And we actually provide a few helpers that make sure that your code doesn't leak memory or that you clean up after you're done with some resources like files from file system or network sockets. So definitely check out Deno's built in test runner. I think you will find it robust enough and simple enough to pick it up in a matter of minutes. Thank you.
Compatibility with NPM Packages and Native Add-ons
We aim for 80-90% compatibility with existing NPM packages in Deno. However, packages relying on native add-ons pose a challenge as they are mostly written in C++. We are exploring API integration and considering the addition of Wasm to support running native code in both Deno and Node.js.
Will we be able to run all existing NPM packages in Deno? No, unfortunately no. We are aiming for say 80 to 90% compatibility. There's a problem with some of packages that they, for example, rely on the native add-ons which we won't be really able to support because add-ons offered for Node.js are mostly written in C++ and we don't really have a way to provide compatible API. However, if you use an API, then we think we might have an answer to that and we are actually on the prototype stage of providing an API integration which adding Wasm would provide two different ways to run native code, both in Deno and in Node.js. Thank you.