Levelling up Monorepos with npm Workspaces

Rate this content
Bookmark

Learn more about how to leverage the default features of npm workspaces to help you manage your monorepo project while also checking out some of the new npm cli features.

25 min
20 Jun, 2022

AI Generated Video Summary

This Talk provides an introduction to NPM Workspaces and covers various aspects of using them, including starting a workspace with npm init, adding dependencies, forcing correct library versions, running scripts and orchestrating in a monorepo, and using npm pkg and npm exec. The examples provided demonstrate real-life use cases and highlight the flexibility and control that NPM Workspaces offer. The speaker also mentions improvements and future developments in the NPM CLI, emphasizing the importance of correct declarations in the package.json file and the ability to manage data across all workspaces.

1. Introduction to NPM Workspaces

Short description:

Hello, and welcome to Leveling Up Monorepos with NPM Workspaces. Let me start by introducing myself. My name is Rui Adorno. I'm a software developer in the NPM CLI team of GitHub. I have been working on the NPM CLI for almost three years now. I work on the NPM v7 rewrite and a lot of those features that we're going to be talking about here today, and especially workspaces.

Hello, and welcome to Leveling Up Monorepos with NPM Workspaces. Let me start by introducing myself. My name is Rui Adorno. I'm a software developer in the NPM CLI team of GitHub. I have been working on the NPM CLI for almost three years now. I work on the NPM v7 rewrite and a lot of those features that we're going to be talking about here today, and especially workspaces. So, I'm super excited to be sharing some of that work with you here today.

And let's start with an overview what we're going to be covering here today. Let's start with an intro to what is NPM Workspaces and then cover some very practical examples with trying to get as close to real-life usage as possible and also note some of the stuff that you should be looking forward to. So, let's get started.

NPM Workspaces. What is it? So, let's start with Workspaces, which is basically this concept introduced by Yarn a while ago, and basically it is a way to help you manage many packages within a single repo. And for NPM, NPM Workspaces is kind of the broad name we gave the set of features that help you achieve that. Basically help you manage multiple nested packages within a single top level package. So, it's going to help you centralize the installation of all the dependencies into a single Node Modules folder, so you're going to end up also having only a single package log file. And there are some advantages to that. It is definitely the ideal way for managing Monorepos, and thanks to that, you can have a single place to manage all your issues and basically manage your project and your community. So, but also into the technical side, you can also centralize, have a single place to run all your tests, peer, linting, anything that you can kind of imagine. You kind of need everything together to get your project running with all the packages. You can have it all in a single place. So it might help a lot depending on the style of the project you're trying to achieve. So another thing I would like to note here is timeline just to give a little bit of this notion of how we've been iteratively improving on NPM workspaces. And you can see it all started in August, 2020 with the release of the release 7.0 of the NPM CLI that first introduced support to installing NPM the workspaces. And then in version 7.7, another big one which kinda first introduced the configuration properties of a workspace or targeting all the workspaces and the first command that support them with NPM run and NPM exec. So there weren't many more, but I just kinda wanted to illustrate how we've been improving and you can definitely keep waiting, you can definitely wait to see more improvements coming on NPM workspaces.

So let's get started with some examples here. I have this project that I have on the left-hand side here that is a sample app that I've put together trying to get as close as possible for real-life project. This is kind of like a web app that consists in two different services. So you can see, I have my user sync service, my web app service, and I even configure a third workspace here, which is the slides that you're seeing right now. So they're all part of this monorepo app that I'm managing here.

2. Using npm init to Start a Workspace

Short description:

And with that, I hope to put some examples that are real close to how you would use this in your real life. So you can see they all have a bunch of dependencies here using Next.js, the services using Fastify. Moving forward, I'd like to highlight npm init. It's probably the best way of just starting your workspace because it's going to make sure every step that is needed is going to end up being there.

And with that, I hope to put some examples that are real close to how you would use this in your real life. So you can see they all have a bunch of dependencies here using Next.js, the services using Fastify. So I can just quickly start by running NPM ls here and we can see how, and ls is a command that is aware of workspaces and it's going to highlight each of the workspaces in my project and even list the dependencies of each workspace when I run an NPM ls. So it's the first command to know that has first class support to workspaces.

So moving forward here, I'd like to highlight npm init. It's probably the best way of just starting your workspace because it's going to make sure every step that is needed is going to end up being there. So basically everything you need to track a nested package as a workspace is make sure you have the folder with a package JSON inside it, inside your project, and then just add the folder name to your workspaces field of your package JSON in the top level. So using npm init is going to make sure that those requirements are there. So I'm gonna run a quick example here in my sample app, I'm gonna create a new workspace. Let's say I'm gonna call it the website. So I can run npm init, I'm using dash Y here to just accept all the defaults and I'm targeting a website. So that's going to create a website folder with a package.json file inside it, and it's going to print the contents of the package.json file over here. And as I mentioned before, the other really important point that npm init takes care of is placing the reference to the website inside my package.json file in my top level folder. So with that, it is all set up. If I npm install here, now the install tree is going to be tracking the website. And if I run npm ls again, I'm going to see website listed as one of the workspaces here highlighted in green. So that is definitely the recommended way to get started with a new workspace inside your project.

3. Adding Dependencies to Workspaces

Short description:

And we can move on to another very important aspect. How do you add dependencies to one of your workspaces? So a real quick example here, I'm going to target the website I just created and install the simple Slider package into my website. That's one of the most important concepts with npm workspaces - everything tries to be hoisted as much as it can. However, if there are duplications, you might end up with a node modules folder inside your workspace that deduplicates those packages. For NPM workspaces, we are currently tracking a new feature to give users more control over where dependencies are placed. In my sample app, I hit a scenario where the Prisma Client library didn't work when hoisted to the top of my app. Duplicating it in both service folders solved the issue.

And we can move on to another very important aspect. How do you add dependencies to one of your workspaces, right? So a real quick example here, again, I'm going to shift around the workspace configuration here. So you can see that it can practically go anywhere. So I can just declare npm workspace. I'm going to target the website I just created. And let's say I want to install a package. I'm going to install this simple Slider package into my website. So I can run that command.

What it's going to do, it's going to install the package name, simple Slider from the registry. It's going to put in as a dependency off my website workspace. So I can look up its package JSON. And I can see that the simple Slider is declared here as a dependency. But if I look up on the node modules folder inside the top level of my monorepo app, I can see that simple Slider was put there. So that's basically one of the most important concepts with the npm workspaces is that everything tries to be hoisted as much as it can. It only avoids that hoisting or that bringing one level up each dependency if it finds duplications. So if there are duplications, then you might end up with a node modules folder inside your workspace that deduplicates those packages so that it can work as expected.

So all that to mention, one thing that I've seen as a common repeating with workspaces is that some people used to have this no hoist configuration option that was an option that Yarn introduced in order to basically have workspaces say, okay, I need this dependency, but I need to make sure it is inside a node modules folder inside my workspace. So because of the nature of some modules, some packets, they really need to rely on the fact that they are at that position in the file system. So that's something that you already come up with in order to basically just give a scape hatch to users to kind of have a little bit more control over where it's placed in their dependencies. So for NPM workspaces, we are currently tracking that as a new feature. We're discussing it in your RFC processes. And like, we are looking forward to provide users with a way of doing it. We're just not sure yet, if it is going to be using that same no-hoist config value, or is it going to be something different? But meanwhile, I wanted to basically provide a way to unblock yourself, if you ever hit that scenario. In my sample app here, I actually hit it. I'm using this Prisma Client, which is this library to manage basically, it's an ORM for that basis. And I'm using it in both my services here of the sample app. And when it got hoisted to the top of my app, it basically didn't work. So a quick way of unblocking me and making it work, is to duplicate it and have it being in both of my services folder, inside of their node modules. So one quick way to achieve that duplication, was basically that I put some mocked package here, that basically would conflict with the namespace of the package there they're using. So I've used Prisma and Prisma Client, and declared them using NPM aliases and pointing to just something else.

4. Forcing Correct Library Versions

Short description:

In order to ensure that the correct versions of libraries are installed, it is important to have the correct declarations in the package.json file. This will ensure that the necessary folders are placed in the correct location and prevent any issues with hoisting. While there are currently workarounds available, it is worth noting that improvements to the default NPM-CLI support may be on the horizon.

In my case here, I'm using this abraham module, but it can be anything really. And so this is in order to make sure that when I declare in like my user saying service here, I have it inside my package.json, I just have the correct declaration of the expected versions I have for these libraries. So that when NPM installs, it is going to be actually putting the Prisma client folders inside my user sync node modules. So it's basically a way of forcing that same behavior of no hoist and it might help them block other users there, so it's a very practical way of get you going today, but definitely keep an eye because there's definitely things going on. It might improve very soon in the default NPM-CLI support.

5. Running Scripts and Orchestrating in a Monorepo

Short description:

So moving forward, I wanted to cover running scripts and provide examples of how to run them. You can run scripts by using the workspace name or the workspace path name. I also show how to run tests for a specific workspace and how to orchestrate scripts in a more complex monorepo. I use the npm run all package to run multiple scripts and provide flexibility. I demonstrate running test and dev scripts for different workspaces, and how to run dev scripts in parallel using npm run all. The result is a running development server with a functional web app.

So moving forward, I wanted to also cover running scripts and here I also place a few examples of how you can run it. You can run it by, or by using the workspace name, also an example if your workspace has a scope, then you also have to define scope, but you can also call it using the workspace path name.

And basically here, in my project here, I can show real quick if I target that user sync service workspace that I have and I run its tests, you can see it's going to run tests for that workspace and it's basically the same thing as navigating to that workspace folder and running NPM test inside it. Yeah, and I want to basically move forward here to a little bit more complex examples that I put together, basically showing how in a more complex monorepo you can orchestrate your scripts in a way that they can become much more useful.

So starting real quick here, following up from the test example I just gave. This is an example. These are the scripts from my top level package JSON. And I'm declaring this test column web app and test column user saying, named after each of my workspaces. So they're basically calling the tests and pointing to each of the workspaces there. So this way I have these multiple scripts in my top level command. And I'm gonna be using this run-ass binary file that it actually comes from this userland package named npm run all. So I noted it up here. It's a very handy package. Helps you kind of run multiple scripts and gives you much more flexibility on the way you do it. So basically here, I'm telling run-ass to run all of my test column targets defined here on my file. So let's jump back to my root folder here and run npm test and see what happens. So you can see it runs the test web app target first, which is running the test inside that workspace. And then after that, you run the test user sync, which ran the test for that usersync workspace, and you can see it succeeded. So, a little bit more evolved, but kind of following up from this first example, let's run also my dev web servers here. So I have kind of similar thing, dev column web app and usersync, and I'm pointing npm run to my workspaces and I'm running the dev script of each one of them. But the particularity of the servers is that I need all of them running at the same time, right? So, this is also something provided by npm run all, which I'm going to be using to run, using run-p, which is going to be running both of these targets in parallel. It's going to just start them all at the same time, keep them running so that I can have everything running and actually use my development server, navigate to my UI. So, let's try it real quick here. I'm going to run npm run dev. You can see it's going to run all the scripts and their dependencies. So, I'm going to advance here. This next slide is actually pointing to the server running on my local host. And so now, once everything puts up, I can see my user list is back up again. I can see some users, I can click around, navigate, and I can see that my web app is working as expected. So, here I kind of want to highlight this new command somewhat new.

6. Using npm pkg and npm exec

Short description:

We introduced a command to help set and retrieve keys and values from package.json files. It's super useful in the context of workspaces. You can use it to print out the content of package.json and filter properties. It can retrieve the name and version of all workspaces, set data in package.json files, and manage data across all workspaces. npm version and npm publish also support workspaces, allowing you to cut new patch versions. Keep in mind that npm version works differently now, with less committing and tagging. Improvements to npm version are expected. To wrap up, here's a quick example using npm exec.

We kind of introduced it end of last year in version 7.20, and it's a command to help you set and retrieve keys, values from your package.json files. So it is not only useful in the context of workspaces, but it's super useful when it's used in the context of workspaces.

So it can be used to, let me highlight here real quick. If you run like npm pkg get, it can be just a simple package. It's going to basically print out the content of package.json. And you can also filter out the properties, let's say, name, and then I get the name of my package there.

So when you bring on the support to the workspaces properties, it can actually get really powerful. Like in this example here, I'm going to retrieve the name of the version of all of my workspaces. So this config option here, dash dash WS, it's basically a way of saying, okay, target all of my configured workspaces. So if I run that, you can see it return the name of the version of each of my workspaces. And they're even keyed by the workspace name. So it can be really useful.

And also to highlight a little bit more of how they can be useful, let me set also some data into these package.json files. So let's say I'm managing this project and it's an open-source project. I just want to surface info about how users can fund my work. So I can go ahead and use npm pkg set. And let's say, I'm gonna set a founding key to all of these package.json. Then I'm gonna point it to my GitHub sponsors URL and I'm going to target all of the configured workspaces. So I can run that. And if you look up, let's say my usersync package.json, you can see that the founding info is there now. So same thing goes to my web app package.json. So it can be an incredible powerful way of just help you manage all of that data in your package.json across all of the configured workspaces.

So also highlight here that npm version, npm publish also have support to workspaces. So if you want to cut out, let's say, a new patch version of a workspace, that's possible today. And something to keep in mind is, as you can see here, I bumped my version to v1.0.1. I can look up my package.json there and see that the version is there, but one thing to keep in mind that has a bit different now from the way npm version works by default is that there's not a lot of committing and tagging when it's running npm version. So it's something to keep in mind, but you can definitely be looking forward to improvements to npm version specifically.

So to kind of wrap up everything, I wanted to run a quick example here. I'm using npm exec. It's basically npx.

7. Using npm exec and npm run in Workspaces

Short description:

It kind of got promoted into a subcommand in the npm CLI since version seven. Let's call it the print current working directory. I'm going to be using exec to round this workspace as a command across all of the other workspaces. I want to highlight the nature of how npm exec or npm run works in the context of each folder. I set a bin value of Index.js to the print to CWD workspace, ensuring proper tracking of the binary file. By running npm exec, I can call my module to run inside each workspace, producing the expected result of printing the path for each workspace. For more information, refer to the official docs or find me on Twitter, GitHub, or my website.

It kind of got promoted into a subcommand in the npm CLI since version seven. And I wanted to kind of like create real quick another workspace. Let's call it the print current working directory. And I'm going to be using exec to basically round this workspace as a command across all of the other workspaces I have configured, just kind of putting everything together using all of the commands that we just saw.

So let me jump real quick into that folder and start index.js file. Index.js file. And this is going to just basically print the current working directory. This is gonna be useful because I wanna highlight the nature of how npm exec or npm run. They're running in the context of each folder. So I wanna print the current working directory just to make sure of that. So there next time I'm gonna npm pkg set a bin value of Index.js to that just created print to CWD workspace so that it properly track that binary file, Index.js. And now I'm gonna npm install to put everything together, make sure my bin is properly linked to my Node modules folder. And now I can npm exec, and I'm gonna be calling my module, and I'm gonna tell it to run inside each of the workspaces. And you can see I got the expected result here. It basically printed the path for each one of my workspaces when it executed inside them. So just a quick example here, to kind of close, wrap up and put everything together, how you can use all of these commands.

So if you want to learn more about it, definitely look up the official docs. We have a section exclusive on the workspaces, but there is also documentation across each of these commands explaining a little bit more how to use them. And if you want to find me, Rui Adorno on Twitter or GitHub. And here's also my website. I really hope this was useful and very good real-life examples. And I hope you enjoy. Thank you.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
React Summit 2022React Summit 2022
21 min
Scale Your React App without Micro-frontends
As your team grows and becomes multiple teams, the size of your codebase follows. You get to 100k lines of code and your build time dangerously approaches the 10min mark 😱 But that’s not all, your static CI checks (linting, type coverage, dead code) and tests are also taking longer and longer...How do you keep your teams moving fast and shipping features to users regularly if your PRs take forever to be tested and deployed?After exploring a few options we decided to go down the Nx route. Let’s look at how to migrate a large codebase to Nx and take advantage of its incremental builds!
JSNation 2022JSNation 2022
25 min
The Age of Monorepos
The history of the web can be divided into evolutionary development leaps. The age of inline scripts, the age of jQuery, the age of SPAs, the age of JAMStack...We are now entering the next stage that has been carefully prepared in the past few years. Let me invite you to the world of modern monorepo solutions and share with you the benefits you will reap by using them in every project size and setup. It's time you automate those boilerplate tasks and reduce the bottlenecks so you can focus on what truly matters.Get ready for the next leap! Welcome to the age of monorepos!
Remix Conf Europe 2022Remix Conf Europe 2022
22 min
Remixing Your Stack in a Monorepo Workspace
Remix entered the stage with a unique and refreshing take on how to develop on the web. But how do you integrate it into your existing ecosystem of applications? Do you want to test-drive Remix on a small project, or do you want to go full-in, but it is tricky to do a big-bang migration from your existing React app? In this talk, we're going to explore how a monorepo-based code organization can help integrate Remix with your existing React and TypeScript infrastructure, facilitating high code reuse and a migration path to Remix.

Workshops on related topic

React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Featured WorkshopFree
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
WorkshopFree
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
JSNation 2022JSNation 2022
99 min
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
WorkshopFree
npm and security, how much do you know about your dependencies?Hack-along, live hacking of a vulnerable Node app https://github.com/snyk-labs/nodejs-goof, Vulnerabilities from both Open source and written code. Encouraged to download the application and hack along with us.Fixing the issues and an introduction to Snyk with a demo.Open questions.
React Summit 2022React Summit 2022
51 min
Build Web3 apps with React
WorkshopFree
The workshop is designed to help Web2 developers start building for Web3 using the Hyperverse. The Hyperverse is an open marketplace of community-built, audited, easy to discover smart modules. Our goal - to make it easy for React developers to build Web3 apps without writing a single line of smart contract code. Think “npm for smart contracts.”
Learn more about the Hyperverse here.
We will go over all the blockchain/crypto basics you need to know to start building on the Hyperverse, so you do not need to have any previous knowledge about the Web3 space. You just need to have React experience.