1. Introduction to Modules and Terminology
Hello, everyone. My name is Andrew and I've been on the TypeScript team for about the last four years. Today, we'll be discussing module system complexities and the terminology associated with it, such as ESM and common JS. It's important to be familiar with module resolution options and the differences between Node.16 and Node.Next.
All right. Hello, everyone. And thanks for joining me to talk about modules. My name is Andrew. I've been on the TypeScript team for about the last four years, and we have a lot to talk about today in a short amount of time, so that's enough about me. And I think we all know that module system complexities are super interesting and super fun, so no time to sell you on it, but if you watched Mark Erickson's talk and couldn't get enough, this is definitely the talk for you.
I'm also going to have to say a few file extensions during this talk. And since those can overlap with other acronyms, I'll try really hard to remember to pronounce them with a dot at the beginning. It will be somewhat helpful if you're at least vaguely familiar with some module resolution options and typescripts. Node.16 and Node.Next are currently identical, so I might call them by either name. They're the only good choice for modern Node, and they don't just mean emitting ES modules. That's a common misconception. Historically, the one called Node was what everyone used for pretty much everything, but it really hasn't kept up with modern features.
2. Understanding the Typescript Error with Helmet
We're going to take a look at a confusing typescript error with the NPM dependency Helmet. When compiling down to common JS, everything runs fine. However, when switching to ES modules, an error occurs where helmet isn't callable anymore. Let's investigate further by running the code.
We're going to take a look at a bit of a confusing typescript error here. I'm looking at the NPM dependency called Helmet. I've just copied and pasted the README example here. This builds with TSC and runs in Node just fine. You can see that I'm currently compiling down to common JS. If I want to move to ES modules here, I can do that by adding the type module field to my own package JSON for the project. But when I do that and come back here, I'm now getting this confusing typescript error that says helmet isn't callable anymore. If I mess around with this enough, I'll see typescript I'm running. This doesn't look right to me, but let's run it and see what happens.
3. Understanding the TypeScript and Node Disagreement
Yeah, so this crashes in node. On the other hand, if we go back to the way it was where typescript is complaining, but we run this, this works just fine. So there seems to be a disagreement here between typescript and node, and we can't satisfy both at the same time. Seeing this, you might reasonably wonder, is this a typescript bug, or are the types wrong?
In order to determine that for ourselves, we need to learn a few pieces of background first. The first one is how typescript and node even know what file we're talking about when we say import helmet or require helmet. It's going to be easier to explain this one, this exports field and package JSON when we're back in the code, so let's leave it behind for a moment. The second is module format detection. I think I mentioned that node supports both ESM and common JS, so it needs a way to determine which file is which format, and it does that purely by file extension. A .mjs file is always going to be ESM, a .cjs file is always going to be common JS, and a .js file, we have to look up at the nearest package JSON and if it has this special type module field, it's going to be ESM, otherwise common JS. All right.
The next thing we need to learn is, what happens if we were to try to compile this export default down to CommonJS? Default exports in ESM are just a special form of named import with special syntax attached. So all we're going to do is just attach a named property assignment here on module.exports with the value that we're exporting. And then we'll also define this flag here that just says, hey, I've been transpiled down from ESM to CommonJS. Our declaration file, on the other hand, kind of retains this esm syntax. Okay, almost done here. The final thing that we need to look at is what happens when we try to import that same transpiled export default that we just looked at. So here we have our export default transpiled down to CommonJS again. And if we're importing this in another file that is also being compiled down to CommonJS, then it makes sense that what we should get with a default import there is going to be the value that we default exported.
4. Understanding Default Imports and Exports in Node
The default import and default export need to match up. When importing a CommonJS module in Node, the whole module.exports object is returned. To diagnose the error between TypeScript and Node, we need to examine the files they are looking at, starting with Helmet's package.json file.
The default import and default export need to match up. So I should just get hello world. And the transformations that are applied as we compile this from a default import to a require statement ensure that that holds. On the other hand, when we do this in a real ES module in node, node doesn't understand that our exporting module here is pretending to be ESM. And in node, when you default import a CommonJS module, what you'll get every time is just the whole module.exports object.
So over here, you can see I've logged the module.exports and I get an object with a property called default. And that's the same thing I would get in node if I were to do a real default import of this fake default export. I'll get the module.exports property with this default. OK, so we should know enough now to be able to diagnose this error for ourselves. If we want to understand the difference between TypeScript and node, we need to know which files each of them are looking at.
5. Resolving Format Disagreement and Fixing Types
By the way, there is a command line flag for TypeScript for TSC called trace resolution. So that will spell out all of this that's going on. So you don't have to remember this if you just want to see what TypeScript is resolving to.
The node file index dot MJS we know has to be ESM and the type declaration file we know is representing a common JS file. And if we take a look at the types themselves, we're concerned with this export helmet as default part. And this kind of finishes explaining what we were seeing. If TypeScripts knew that this file was attempting to type a real ES module, then it would see this export as helmet as default and know that when we default import that it's going to work as we expect. You know, we have a default export. We default import it and we can use it no problem. But what TypeScript thinks is that this file is representing a common JS file that is doing exports.default equals helmet. And if that's the case, when we default import it in Node, we still are going to need to do that extra .default in order to access that exports.default property since we're just getting the view of module.exports as a whole.
6. Fixing the TypeScript Error with Helmet
So let's just delete it. And I think we should have agreement now. So coming back to our input file, recall that we didn't touch the types at all. So we are still expecting to see this TypeScript error, but what we should see is that if TypeScript is giving us an error, Node also is going to crash. So let's build and run that. And yeah, this time, helmet is not a function. And if we instead do what TypeScript is telling us we should be doing, helmet dot default, and then build and run this, this now works.
7. Analyzing Module Resolution and Type Disagreements
So the cool thing about doing all of this analysis on npm packages is that it gives us a natural way to look at a lot of data over time. I ran rthetypes wrong on the top 6,000 or so npm packages as they existed on the first of each month since January of last year, broken down by a module resolution algorithm. So we can see that things are kind of improving for users in all groups here, with the slight exception of the recent uptick in node 10, which just comes from packages dropping support for this. On the other hand, things are not looking great for node ESM users, with over a quarter of types dependencies showing some sort of error. So this kind of puts us in a tough spot, because if the community is going to migrate to ESM, and I think that it should, but regardless, it's already happening. It would be really nice if we could encourage users to make that transition first before libraries, but that's a pretty tough sell as a TypeScript user if making the switch is going to break a quarter of your dependencies.
8. Challenges with TypeScript and Node Support
A lot of the problems we see can be traced back to TypeScript's lagging support for Node. I've been focusing on this problem space for about a year, working on getting information out there and improving the Are the Types Wrong tool. Now, I'm looking to make a more direct impact on the existing problems in popular packages.
It's only an observation about what works or doesn't work now. And in fact, a lot of the problems that we see can trace their roots to TypeScript's lagging support for Node. So this problem space has been kind of my main thing for the better part of a year. And up until now, most of what I've been doing has been in service of just getting information out there from Are the types wrong to official TypeScript docs. I think that phase of my work is finally kind of wrapping up and I have some improvements to Are the Types Wrong plan that I'm excited about. But I anticipate kind of shifting into looking for ways to make a more direct impact on some of the existing problems that we see out there in popular packages.