The Secret Life of Package Managers

So you are running npm install and you go and grab a cup of coffee and then you come back and you have no idea or maybe you don't even care what npm did during this time. So my name is Tali Barak, I work for Ubique and let me tell you about the secret life of package managers. This is what happens in your project. This is the basic of npm. So you have your project and you need a package called foo. So you're running npm install foo. And that's add npm install to your package JSON and create a node modules in your project and put this code of foo inside it. But what if foo requires buzz? Okay, no problem. It will create another node modules folder under foo and it will put buzz. And what if buzz requires bugs? Well same thing, it will put it and add it there. And then what happens if your foo requires a buzz, but also your bar require buzz? In this case, in the naive implementation of npm, you will have two packages of buzz in the same project. And this actually creates the whole structure of your file system that is replicating the package structure that you have in your project. And this was nice, but it created quite a few problems. For example, it made your node modules huge. Also it created a problem with circular dependencies. That means that if your foo needed bar that needed buzz that needed bugs that needed bar again that needed buzz and needed bugs, you would go into an infinite loop. And this is by the way, quite common. It's not as rare as you might think it is. Another issue which is common is with singletons. If you need a single instance of a certain package, like the debug package, for example, in this structure, you would have multiple instances and that can cause bugs when you execute the packages. And the last one is no longer with us. Thank God for that. It was a Windows 8 problem that the path, the file path was limited to 256 characters. This is less common. So what did npm do in order to solve that? So they decided to do a de-dup, de-duplicate the packages that were multiple times. So instead of having buzz twice, they would put it in the highest possible level and use it there. And the reason this worked is because the way node is requiring packages. So if bugs needed buzz, it would go under the node models and search for it. If it doesn't find it, it will go one level up and search for it there. And then if it doesn't find it, it will go another level up. And there it is. It actually found buzz there and it will use it. Next, they will decide that, well, let's take it one step further. If we can move packages up the tree, why only the duplicate one? We can actually do that for all the packages. And they made a flat tree with all the packages. And this was good. This would solve the problem. You now had smaller packages, shorter paths, because you didn't go that deep. It was only unidirectional, no circulars, and it made every package unique. It was good, but then we had a problem. Each package might have a different version that it requires. So the tree that you need doesn't actually look something like this one. You have different versions of the same packages required in different places in the tree. And even worse, in some cases, the versions could conflict. That means that your foo might require buzz in version one, but bar requires buzz in version two. And how do you flatten that? What do you put at the top level? So in fact, we have an issue here that your file structure, your package structure is no longer a tree. It is actually a graph. And the way it was solved is by different versions of npm had different solutions. Sometimes it would take a popular one. Sometimes it would take a first one and put it at the top of the tree, while the other version that was required was left under the package that required it. Like in this case here, where you could promote two or one, depending on the order. And this is a problem, because now we get a very shaky and unpredictable tree. And the way that npm and also yarn in version one solved it is by actually taking a snapshot of your whole node modules. And this is their famous log file. npm had it as a shrinkwrap file. And then yarn added the yarn log file. And this is the way for npm to make sure that the package structure that the node module files in one project is the same as you run on the CI or is the same as your colleagues run. But here is another solution. PMPM is actually doing something smart and different. PMPM is keeping the original structure, the tree structure that we talked about, keeping all the packages and all of their versions. But they don't really install the packages. They store all the packages and all the different versions in a cache. And then in the node modules is actually pointing at the operation system level using hard links to the cache and storing the exact version and pointing to the right version where it is required. So this is one solution that was used. And the other solution is what yarn 2, which is codenamed Berry, introduced. They basically say that we are the package manager and we know exactly what package is required by each package. So they have a map of all the node modules and all the packages and what they require. And they patched, they changed the way node is requesting the files. So if node is now requiring a package, it will no longer go directly to the file system and get the package. But instead it will access yarn and request the specific packages it needs. And yarn will look into this map of modules and will return the exact file that they're showing that. So that was a very, very short and very quick peek into the secret life of package manager. Thank you so much.