Understanding Package Resolution in Node.js

Rate this content
Bookmark

Everytime we import or require a package, we follow a set of rules to resolve the package. This talk will cover the different ways Node.js resolves packages and how to debug when things go wrong. We will also cover the new features in Node.js 20 that make package resolution faster and more reliable.

 Yagiz Nizipli
Yagiz Nizipli
11 min
04 Apr, 2024

Comments

Sign in or register to post your comment.

Video Summary and Transcription

In this Talk, the speaker discusses package resolution in Node.js, covering topics such as CommonJS, ES modules, package.json structure, and package.json loader. The Talk also touches on conditional loading and file extension resolution, module import and export, module type determination based on file extensions and package.json, module resolution strategies in Node.js, and tips for improving loading time in ESM applications.

Available in Español

1. Understanding Package Resolution in Node.js

Short description:

In this part, we will discuss CommonJS, ES modules, ESM, package.json structure, and package.json loader in Node.js. CommonJS is the most widely known module resolution strategy in Node.js.

Hi, I'm going to be talking about understanding package resolution in Node.js today. I'm Yalcin Zipli. I am a senior software engineer at Sentry. I am a Node.js Technical Stream Committee member and an OpenJS Foundation Cross-Project Council member. You can reach me from my GitHub account, github.com, and from Twitter, axe.com.

So in summary, today we're going to talk about CommonJS, ES modules, ESM, package.json structure, and package.json loader in Node.js.

Let's start with CommonJS. CommonJS is the first and the most widely known module resolution strategy in Node.js. It includes files with require, export implementations with module.export or exports. It can have an extension of .cgs or .js. Require calls does not have to include the extension of the file, and loading a file is synchronous.

2. Conditional Loading and File Extension Resolution

Short description:

If you want to do conditional loading where a file is loaded and used in a function, you need to use require inside the function. However, this implementation can block the execution and IOU depending on the file size. The extension is not required, and the loader checks for different extensions in a specific order. Loading a file without an extension involves synchronous calls to the file system, which can impact performance.

So basically, if you want to do a conditional loading, let's say you have a function called read file, and just when this function is executed, you want to load a file and use that implementation in your function, then you need to use require inside this function.

The caveat of this implementation is that in the first line of read file, the lazy loaded module is equal require.reader, that's going to block the execution of this file because it's a synchronous call. And it's going to block the IOU depending on the size of the file itself.

As you can see, the extension is not required, .reader. This means that first the loader is going to check for .gs extension, then .json and so on and so forth. And then it will return a value and then we can run this function.

So in order to load this file without an extension, this implementation makes a synchronous call to the file system in order to understand if reader.js file exists. If not, it automatically checks for reader.json, if it exists and so on and so forth, and then it will return an error if it's not found.

This is particularly slow because in order to understand if that file exists, you need to make additional file system calls and it will impact your performance, small or big, it will impact it.

3. Module Import and Export

Short description:

In CommonJS, additional modules are included using requires and exports. ES modules use import and export statements. The import statement returns a promise and should be evaded if inside a function. Modules can be exported with export or export default. File extensions for ES modules can be .mgs or .gs.

So despite CommonJS implementation, as I said, CommonJS, in order to include additional modules with CommonJS, you need to call it with requires. Inside the function that you want to require, you need to export the implementation with module.express and so on and so forth. So it can have an extension of .cgs, which is CommonJS, and .jsimplementation. But in ES modules, it's a lot different, the idea behind it. It's introduced in Node.js 8.5.0 with an experimental flag. In order to include files, you need to call it with import. If this import statement is on top of the implementation, like if it's not inside the function, then you can use import blah blah. But if it's inside a function, you need to evade the import statement because import returns a promise. Export implementations with export and export default, and it can have an extension of .mgs or .gs and async loading a module file.

4. Module Type and Package.json

Short description:

In order to load modules conditionally, use lazyLoadedModule?equal and evade the import statement. This allows for async loading without blocking I/O. Node.js determines module type based on file extensions and the presence of a package.json file. The type attribute in the package.json specifies if it's a module or a CommonJS.

So in the base, it's async loading structure. So in order to load the module conditionally, just like the previous implementation, you need to call it with lazyLoadedModule?equal, evade import, and this makes this particular function an async call, whether or not the lazyLoadedModule.do something is async or not. This is the main reason for this, that when you evade an important module, it doesn't block the I.O.

On top of that, we talked about CommonJS file extensions and MGS file extensions, so how do we actually know what's going on? So we have this package.json file in all of our projects. It contains metadata about a project, but Node.js only cares about 5 of those fields. Name, main, imports, exports, and type. For the sake of this presentation, I'm just going to focus on the type attribute. Type can be a module or a CommonJS.

So how does Node.js know if you're using ESM or CommonJS? So first, it checks for file extensions, Node.js checks for file extensions. If it's MGS, then it's ESM. If it's CGS or JS, then it is CommonJS. So finding the package.json, so we first check for the extensions. If the extension is basically a .js file, then we don't know what it is, then we need to look for the package.json that we are executing this file in. So we need to know the context.

5. Module Resolution and Tips

Short description:

If the extension is a .js file, Node.js checks for the package.json in the file's directory. If found, it checks the type field in the package.json. If type is module, it uses ESM loaders; otherwise, it's CommonJS. If type is not present, Node.js checks for the experimental detect module flag. For files required from ESM implemented in CommonJS, Node.js follows module depth resolution. To improve loading time, use the experimental default type CLI flag for ESM applications.

If the extension is basically a .js file, then we don't know what it is, then we need to look for the package.json that we are executing this file in. So we need to know the context. So Node.js tries to find the closest package.json in the directory until root. So it checks for app, my, project, package.json, and so on, so forth, until the root. If it's not found, then it's going to assume something else.

Node.js checks it, and whenever the package.json value is found, Node.js checks the type field in the package.json. And if it's module, it uses ESM loaders, the loader implementation in Node.js. And if it's not, then it's CommonJS. So if type is not present, we couldn't find the package.json, what happens? If type is not present, we check for an experimental flag, experimental detect module. This is pretty new in Node 20 or 21. It automatically checks if the file that you're running is a CommonJS file or an ESM file. This is particularly new because this is an experimental flag, and there are known issues with it, but we are working on it.

So if we have experimental detect module, then we detect if the file is a required, if it's an ESM or a CommonJS. If not, then we fall back to CommonJS. So then we know how our application starts, because we know the initial script is written in ESM or CommonJS. But the issue is, what happens if you want to require a file from ESM that's implemented in CommonJS? Or you want to call a function that's CommonJS or an ESM from a CommonJS module? So what about Node.js module depth resolution? Like if you have Node modules implemented, a package inside your dependency list, and it's implemented in either CommonJS or ESM. So Node.js checks the type field in the package.json of the dependency. If it's module, it uses ESM, otherwise CommonJS. If type is not present, it uses the type of the parent package, which is the root package that our project's package.json contains.

So let's continue with some tips to improve the loading time. Because we talked about all of these package.json loaders, system calls, we talked about the detection, the extensions, and so on and so forth. So if you want to avoid all of those things, and if you want to start Node.js as soon as possible, what you could do is that, if you have an ESM application, you can use an experimental default type, CLI flag, which will automatically eliminate all of those checks, and it will always return ESM. It will not check for the extension, it will not check for anything else, it will just load the ESM loader.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Top Content
Do you know what’s really going on in your node_modules folder? Software supply chain attacks have exploded over the past 12 months and they’re only accelerating in 2022 and beyond. We’ll dive into examples of recent supply chain attacks and what concrete steps you can take to protect your team from this emerging threat.
You can check the slides for Feross' talk here.
Towards a Standard Library for JavaScript Runtimes
Node Congress 2022Node Congress 2022
34 min
Towards a Standard Library for JavaScript Runtimes
Top Content
You can check the slides for James' talk here.
Out of the Box Node.js Diagnostics
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
In the early years of Node.js, diagnostics and debugging were considerable pain points. Modern versions of Node have improved considerably in these areas. Features like async stack traces, heap snapshots, and CPU profiling no longer require third party modules or modifications to application source code. This talk explores the various diagnostic features that have recently been built into Node.
You can check the slides for Colin's talk here. 
ESM Loaders: Enhancing Module Loading in Node.js
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring.
While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
pnpm – a Fast, Disk Space Efficient Package Manager for JavaScript
You will learn about one of the most popular package managers for JavaScript and its advantages over npm and Yarn.A brief history of JavaScript package managersThe isolated node_modules structure created pnpmWhat makes pnpm so fastWhat makes pnpm disk space efficientMonorepo supportManaging Node.js versions with pnpm
Node.js Compatibility in Deno
Node Congress 2022Node Congress 2022
34 min
Node.js Compatibility in Deno
Can Deno run apps and libraries authored for Node.js? What are the tradeoffs? How does it work? What’s next?

Workshops on related topic

Node.js Masterclass
Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Top Content
Workshop
Matteo Collina
Matteo Collina
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
Build and Deploy a Backend With Fastify & Platformatic
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Matteo Collina
Matteo Collina
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
0 to Auth in an Hour Using NodeJS SDK
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Asaf Shen
Asaf Shen
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
Building a Hyper Fast Web Server with Deno
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Matt Landers
Will Johnston
2 authors
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
GraphQL - From Zero to Hero in 3 hours
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
Pawel Sawicki
Pawel Sawicki
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.
Mastering Node.js Test Runner
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Marco Ippolito
Marco Ippolito
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.