Ever wondered what happens after you hit npm install and go to grab a coffee? Let's deep dive into Npm and Yarn installation process, and how you can put this knowledge into practice.
The Secret Life of Package Managers
AI Generated Video Summary
npm install can be a mysterious process, but understanding how package managers work is essential. NPM solved problems like large node_modules, circular dependencies, and multiple instances of the same package. Managing package versions and conflicts is crucial for consistency across projects. Alternative approaches to package management, like PNPM and Yarn2, provide insights into the hidden complexities of package managers.
1. The Secret Life of Package Managers
npm install can be a mysterious process, but understanding how package managers work is essential. When you install a package, it creates a node_modules folder and adds the necessary code. However, this can lead to issues like large node_modules, circular dependencies, and multiple instances of the same package. NPM solved these problems by deduplicating packages and using a hierarchical structure. This ensures efficient package retrieval and eliminates the need for redundant copies.
So, my name is Tali Barak. I work for Youbeek and let me tell you about the secret life of package managers. This is what happens in your project, this is the basics of npm.
So, you have your project and you need a package called foo. So, you are running npm install foo and that's add npm install to your package json and create a node modules in your project and put this code of foo inside it. But what if foo requires buzz? Okay, no problem. It will create another node modules folder under foo and it will put buzz. And what if buzz requires bugs? Well, same thing. It will put it and add it there. And then what happens if your foo requires a buzz but also your bar requires buzz. In this case, in the naive implementation of npm, you will have two packages of buzz in the same project.
And this actually creates the whole structure of your file system that is replicating the package structure that you have in your project. And this was nice, but it created quite a few problems. For example, it made your node models huge. Also, it created a problem with circular dependencies. That means that if your foo needed a buzz, that needed a buzz, that needed a buzz, that needed bar again, that needed buzz, and needed buzz, you would go into an infinite loop. And this is, by the way, quite common. It's not as rare as you might think it is. Another issue which is common is with singletons. If you need a single instance of a certain package, like the debug package, for example, in this structure, you would have multiple instances, and that can cause bugs when you execute the packages. And the last one is no longer with us. Thank God for that. It was a Windows 8 problem that the path, the file path was limited to 256 characters. This is less common.
So what did NPM do in order to solve that? So they decided to do a dedup, de-duplicate the packages that were multiple times. So instead of having buzz twice, they would put it in the highest possible level and use it there. And the reason this worked is because the way Node is requiring packages. So if buzz needed buzz, it would go under the Node modules and search for it. If it doesn't find it, it will go one level up and search for a third.
2. Managing Package Versions and Conflicts
When packages have different versions and conflicts arise, the package structure becomes a graph. NPM and Yarn tackle this issue by taking a snapshot of the node modules, ensuring consistency across projects.
And then if it doesn't find it, it will go another level up. And there it is. It actually found buzz there and it will use it.
Next, they decided, well, let's take it one step further. If we can move packages up the tree, why only the duplicate one? We can actually do that for all the packages. And they made a very flat tree with all the packages. And this was good. This solved the problem. You now had smaller packages, shorter paths because it didn't go that deep. It was only unidirectional, no circulars. It made every package unique.
Was good, but then we had a problem. Each package might have a different version that it requires. So your tree, the tree that you need doesn't actually look something like this one. You have different versions of the same packages required in different places in the tree. And even worse, in some cases, the versions could conflict. That means that your foo might require buzz in version one, but bar requires buzz in version two. And how do you flatten that? What do you put at the top level? In fact, we have an issue here that your file structure, your package structure, is no longer a tree. It is actually a graph.
And the way it was solved is by different versions of NPM had different solutions. Sometimes it would take a popular one. Sometimes it would take a first one and put it at the top of the tree. While the other version that was required was left under the package that required it. Like in this case here, where you could promote two or one, depending on the order. And this is a problem because now we get a very shaky and unpredictable tree. And the way that NPM and also Yarn in version 1 solved it is by actually taking a snapshot of your whole node modules. And this is the famous log file. NPM has it as a shrink wrap file. And then Yarn added the Yarn log file. And this is the way for NPM to make sure that the package structure that the node module files in one project is the same as the one on the CI or is the same as your colleagues run.
3. Alternative Approaches to Package Management
PNPM and Yarn2 offer alternative approaches to package management. PNPM keeps the original package structures in a cache, using hard links to point to the required versions. Yarn2, codenamed berry, maintains a map of all packages and their dependencies, intercepting file requests from Node to provide the required packages. These solutions provide insights into the hidden complexities of package managers.
But here is another solution. PNPM is actually doing something smart and different. PNPM is keeping the original structures, the tree structures that we talked about, keeping all the packages and all of their versions. But they don't really install the packages. They store all the packages and all the different versions in a cache. And then the node modules is actually pointing at the operation system level, using hard links to the cache and storing the exact version and pointing to the right version where it is required.
So this is one solution that was used and the other solution is what yarn2 which is codenamed berry introduced. They basically say that we are the package manager and we know exactly what package is required by each package. So they have a map of all the node modules and all the packages and what they require. And they patch, they change the way node is requesting the files. So if node is now requiring a package it will no longer go directly to the file system and get the package but instead it will access yarn and request the specific package that it needs and yarn will look into this map of modules and will return the exact file that are showing that.
So that was a very, very short and very quick peek into the secret life of package manager.