JS Nation

JSNation 2023

Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring. While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.

ESM Loaders: Enhancing Module Loading in Node.js

EFM Loaders, enhancing module loading in Node.js. Hi, I'm Gil Tayar. I'm pretty old. We go all the way back to the 80s. I always was a developer. I did lots of stuff, but basically that is what I love to do. Another thing I love to do is use NPM and ESM and all that to enhance my code using extreme modularity and testing code. I'm currently software engineer at Microsoft and working on a cool stuff like the Azure Data Explorer, which is a real-time analytics database engine. Nothing to do with ESM. Let's get a few things out of the way before we talk about enhancing module loading. Let's talk about what ESM is. ESM is the standard JavaScript module system and it is now native to Node.js. If previously we had CommonJS, module.exports to export something and require to import it, in ESM, you use standard syntax, export and import to do module export and import, native syntax works in Node.js. Most people use today, still use CommonJS, but ESModules is getting more and more frequently used. What is module loading in Node.js? Let's talk a little bit about that module loading. First of all, what is a module? A module is basically a glorified file, it's just called a module. When I run a module or import a module, I'm basically executing the code inside the module. It's glorified execution module loading. Even when I'm importing module.js on the top, it's executing module.js at the bottom and then continuing executing main.js. It's all about glorified execution. First of all, a note to TypeScript users. If you're using TypeScript in Node.js, you're using the import export syntax, but you're probably transpiling that to CommonJS because ESM is pretty new to the game. TypeScript usually by default transpiles to CommonJS. If you're trying all the things I'm doing here with ASM loaders on your TypeScript code, it won't work. You have to not translate the ESM using module node next and you have an example in the GitHub repo. You don't worry, at the end, you'll have a link to the GitHub repo with a nice QR code and a link to this presentation. What are the phases of module execution? How does Node.js execute a module? It's not simple. First of all, we resolve the URL. If we're doing import.module.js, we have to resolve it to the absolute path on the disk. ESM doesn't talk about paths, it talks about URLs. Usually, the URLs are file URLs, but they're URLs nonetheless. Once they resolve the URL, once Node.js resolves the URL, it reads the file from the disk. Then, this is different from CommonJS, but let's not talk about what. It recursively loads all the modules in the file without executing them. Notice that we haven't executed the ESM module above. Once it recursively loaded all the files, it executes the code of all the modules bottom up in the right order. These are the phases of module execution in Node.js, and basically in the browsers too. Anywhere ESM is implemented, these are the phases. This is different from CommonJS, we don't get into where. Let's talk about module loaders. Module loaders enhance the first two, enhance and change and transform how Node.js does a resolution and how it reads the file, the module. Let's talk about enhancing the resolution phase. We'll give an example, and then we'll give an example for the loading phase. Let's talk about the resolution phase. Let's give two examples. One is PNPM. It's a package managers like Yarn and NPM, but PNPM changes the way modules are found. It searches for them in a central cache repository on your disk. Currently, because they don't have loaders, they do various hacks using hard links to make it work. But if they had loaders, if it was native ESM, then they could have modified the resolution phase to search wherever they want. Another example is import maps, overriding bare specifiers, and this is what we'll be trying to do. Let's write a loader that does this and you'll understand what it is in a second. Now, notice the sample code is demonstration code. Do not use it in production. It doesn't have error handling, it doesn't have tangled edge cases. It's just toy examples. Let's talk about overriding modules. This is main.js and we're doing import a module to overwrite. A module to overwrite doesn't exist anywhere in Node modules or everywhere. If I run it, I get error module not found, obviously. But let's say I have an overrides.json file which says a module to overwrite actually exists in./.moduleoverride.js. This is moduleoverride.js. We console.log module overridden and we want this to work. We want to run the main.js with a loader. How do we do that? We add //loader and point to the loader. If we run it, boom, it works. This is the syntax, //loader equals or space, it doesn't matter, and./.loader.js. Note that the./. is important. If you say loader.js, it will look for the package loader.js, not the file loader.js. The./. is essential. Let's look at loader.js. Don't worry, we'll go one by one and understand it. This is the loader. It's very, very small. As you can see, writing a loader is not that difficult. First of all, we have to read the overrides.json. We just read it using top level 08 and parse it to get the overrides.json. Perfect, easy, no problem. Now, we export a function. That function has to be named resolve because this is what, when Node.js is loading a loader, it looks for that function, that exported function, and it will receive three parameters. We'll see in a second, and it's async. You can do whatever you want in there. You're not limited to synchronicity. Let's talk about the three parameters. First of all, the module specifier is what is inside the quotes. In our example, a module to override, just as it appears in the code. The context we'll see later, it has all sorts of things, but mainly we'll talk about. Rex resolve, if you can't resolve something or don't want to resolve, and you want to pass it on to Node.js, you can use next resolve to resolve. Let's look at the code. If the specifier is an override, remember overrides is the JSON file, then if it's not in the overrides, we call next resolve with a specifier. If we don't know what to do with it, we pass it on to Node.js. Easy. But if we do want to know what we do with it, we take the overrides specifier. We take whatever is in the overrides and pass it on to Node.js. Now, Node.js will not get the bare specifier like the module override, but it will get dot slash module override stock JS. But we still pass it on to Node.js. Boom, there we go. It works. See how easy it is to write an ASM loader. It really is that simple. Obviously, edge cases, error handling, blah, blah, blah. But in essence, this is it. We've seen resolution phase. Let's talk about the loading phase and how to override that. Remember, resolving is taking the specifier and resolving it to a full URL, and then reading the file is where we're talking about loading. Why enhance the loading phase? Well, there are lots of examples. We're going to do two things. Loading directly from HTTP instead of from a file and loading TypeScript code without building it. We will be able to give it a TS file and it will just work. Let's talk about loading from HTTP. This is the code. We're loading from HTTPS, ESM.SH, whatever. ESM.SH and there are others out there is a service that gives you ESM modules of all the native, all the NPM packages out there. It's really, really, really cool. This is what we want to run. If we run it like this with no loader, we will get error unsupported URL scheme because Node.js is telling us I don't know what to do with HTTPS. Perfect. But if we run it with a loader, boom, it works. We're very, very happy. Let's see the loader. First of all, just like there's an export of load, we have an export of resolve for the resolve URL. Wait, something happened to the, I'm stopping. Yeah. Now I see what it is. It's my, there we go. Yeah. There we go. This is the loader. As you can see, it's as simple. We'll go over it one by one just like previously. Just like in the resolver, the loader has a load function which is exported async, just like example, like the previous one. Let's talk about the module URL. The URL is the module URL. Notice this is the module URL after resolution. The resolvers, all the resolvers are finished with it. We get the full absolute URL. Context we'll see later. Next load, if you can't load something, you can use next load to load it. It's usually the Node.js. If the URL isn't an HTTP or HTTPS URL, well, we just pass it on to next load. Very, very easy. If it is, we use fetch. Fetch is now native to Node.js, just like in the browser, you can use it without importing anything from Node V18, I think, or something like that. We use fetch to fetch the source code. We follow redirects because ESM.sh has redirects, and we have the source code. What do we do with it? We return the results to Node.js. The source code of the module is in source. Format is what is the format of the module. Is it module, which means ESM, common.js, JSON, WASM. In this case, we're always saying, if it's coming from HTTPS, it has to be module. Short circuit. Short circuit is telling Node.js, look, we didn't call next load. We know what the source is, but just so you know, we didn't call next load and it's fine. If we don't add short circuit true, Node.js will fail and say, are you sure you didn't want to call next load? If you didn't, please send short circuit true, and then you add short circuit true, and you're good. Will this work? Let's see. No. Because the Node.js resolver throws an error. It's not the loader that says, I don't know what to do with HTTPS. It's the resolver that says, I don't know what to do with HTTPS. This is infuriating actually, because why should it care? Why should the resolver say, I don't know what to do with HTTPS? Maybe somebody else wants to know what to do with HTTPS. I actually found this out when I was working on this talk, and opened a bug, and implemented a pull request, so in Node 20, if this pull request passes, we will not need the next phase, the next thing that fixes this, because the Node.js resolver says, I don't know this URL, but it's fine, I'll throw in the loader. But we still have this problem, so we need a resolver in the HTTP loader that solves this problem. We have to override resolution too, and this is infuriating, but this is what it is. This is where the code become interesting. So please, please, please, please pay attention. Specifiers. Remember, these are specifiers. Our resolver will need three kinds of specifiers, relative URL specifiers, absolute URL specifiers, and what are called bare specifiers. We will need to deal with all of them just like Node.js does. So relative are these kinds of specifiers, bares are the ones on the bottom, and absolute, yes, theoretically, somebody can give us an absolute URL. So bare specifiers, what do we do? We let Node handle it. We don't know how to handle bare specifiers. So if is bare specifier, then we continue on to next resolve. There's no problem there of HTTP URLs, so we can just let Node handle it. This is bare specifier, it's ugly. Maybe somebody else has a better way. So I'd say if this specifier starts with dot, then it's not a bare specifier, otherwise, parse it. If it's not a URL, then it's a bare specifier, otherwise, it's a URL, so not a bare specifier. So if it's bare specifier, pass it on to Node.js. Now, we need to deal with relative URLs. How do we absolutize a relative URL? We take the module's URL and add it to the specifier and absolutize it. So in this case, module.js, we have the absolute URL of the module, and we absolutize it to get the module.js. We can do it with new URL in Node.js very easy, and this is what we do. We take the specifier, we take the context.parentURL, which is where the parent URL resides, and we get a URL in the back. This also takes care of absolute URLs. Now, if the URL starts with HTTP, we do not want to send it to the Node.js resolver because then it will throw, so we just return the URL as is and short-circuit it. And otherwise, it's not an HTTP URL, we can pass it on to the Node.js resolver. This takes care of the problem with HTTP and resolve. And there you go. The loader works. Let's talk about another one, transpiling TypeScript. The previous loader added support for loading other sources. This loader is more of a transformer. Let's see the TypeScript code. This is a main.ts. It imports from another TypeScript code, but it also imports from JavaScript code, and as you can see, TypeScript code. And this is the files, nothing very interesting here. Running it without a loader, will we get an unknown file extension because it doesn't know what to do with.ts? Perfect. Running it with a loader will work. Let's see how a TypeScript loader works. If the URL ends with.ts, first of all, it calls next load. It says, I don't know how to load the file. I'll pass it on to Node.js. Node.js will load the file and return the source to us. Once we have the source, we can transform it using esbuild. You can use whatever transpiler you want. I use esbuild because it's easy. Get back the new code and return it as this transpiled code with return source and format. Very, very easy. And boom, you get transpilation with like 10 lines of code. Very easy. Look at ts-node.js, which does the same thing, but robustly, you can see it has hundreds of lines of code. But in essence, it's doing what I saw, what I showed previously. Okay. We've seen three loaders. One that overrides modules. It does the resolving phase. And two, the loading phase, HTTP and TypeScript transpiling. Can we chain loaders? Can we add them one after the other? And the answer is absolutely. So let's do HTTP and TypeScript loaders together. So I import HTTP, but one of them is a TypeScript code, and I want the two loaders to make it work. As you can see, we're using HTTP and we're using TypeScript. And these are the modules, nothing interesting here, but let's add the override. Okay. So instead of giving the full URL, we'll just give, you know, bare specifiers, A and B, and specify in the overrides that the URLs are HTTP URLs. We add the loaders with multiple, notice the commas between them. We can also use dash dash loader, HTTP loader, dash dash loader, TS loader, dash dash loader, override loader, same thing. We add them together and boom, it works. Let's see how, because this is interesting. Notice the order, HTTP loader, TS loader, override loader. First of all, it does the resolution. The resolution gets a specifier and returns a URL, if you remember. So notice that HTTP loader is first, but it will call the override loader. It will call the last loader. Okay. Why? Because the override resolver, sorry, override resolver, when it says next resolve, it will call the HTTP resolve. And when HTTP resolve calls next resolve, it will call the Node.js resolve. So the override resolve reads the overrides.json and resolves via overrides.json and passes it on to the next loader. And this HTTP resolver, if you remember, passes non-HTTP specifiers to Node.js and deals with HTTP specifiers on its own. Node.js resolver was absolutize everything and pass it on and boom, we get the URL. Loaders, same thing. It will call the last loader, this one. This is the TS loader will pass it on to the HTTP loader, which passes on to Node.js, et cetera, et cetera. HTTP loader will fetch the HTTP URLs and pass on non-HTTP URLs and boom, it works. Now, my time is short, so I'll skip this, but you can see everything in my presentation, but you can actually use loaders with API. So specify a loader that also has an API. It's a more advanced scenario, not used a lot, but you can do that. Okay, let's summarize. Let's summarize. Loaders enhance these two phases, not the execution phase, but the resolution of the URL, which is finding where the module is and reading the URL or what we call loading the file. Loaders can be used to load from different sources, transform source code, transform resolved URLs and resolve URLs differently. Lots of uses for that. A loader always has at least, well, not at least, at most two exported functions, resolve and load. It can have only one of them. One deals with the resolution, which takes a specifier and resolves it to a URL. It can use the next resolve in the chain if it doesn't know what to do with something. And loading takes the URL, the resolved URL, and returns the source code, and if it wants, it can use the next load to load it. To use loader, use dash dash loader. You can chain them, which is great. And to commute, well, that's something we passed. You can communicate with the loader using various methods. And the future is bright. We finally have a formal way of doing module loading enhancements. It has withstood the test of time. It's still experimental, so small changes can still happen. Be careful. It is very simple to use and can be used in a large variety of use cases. Thank you very much. ♪
