Package-based Monorepos - Speed Up in Under 7 Minutes

Rate this content
Bookmark
SlidesGithub

They gave me just 7 minutes! So this is a no-slides, just code talk! I'm gonna show you a Yarn/NPM/PNPM workspaces-based monorepo implementation and how we can speed it up. In particular, I'm using Nx on top to get better parallelization, the ability to define task pipelines, caching, and more. Curious? Join me!

Juri Strumpflohner
Juri Strumpflohner
9 min
01 Jun, 2023

Comments

Sign in or register to post your comment.

Video Summary and Transcription

The Talk discusses speeding up MonrayBus in a pmpm workspace by organizing packages and considering dependencies. It covers installing and configuring the nx package, including choosing cacheable scripts. The nx-graph command is introduced for analyzing dependencies and optimizing the build process.

1. Introduction to MonrayBus in a pmpm workspace

Short description:

They want to talk about how to speed up MonrayBus in a pmpm workspace. Packages are organized in a folder with their own node modules, package JSON definitions, entry files, and scripts. Dependencies between packages are possible. The package can be located in either the examples or packages folder. Running scripts directly in the packages folder is not convenient, so filters can be used. Dependencies between packages need to be considered when building. The UI, product list, and application need to be built in order.

So, they told me I have seven minutes, so no code, just slides, just code, no slides, and what I want to talk a bit through in this short lightning talk is how we can speed up MonrayBus, all right?

So, what I have here is a pmpm workspace, and so it uses pmpm as the package manager system. It's the same thing almost like for npm or YARN workspaces, so there's no big difference there. So, what you can see here in this file, we specify where our packages live, which is here in this packages folder, where each of the packages basically has its own node modules, they have their own kind of package JSON definitions, entry files, also their own scripts, and most importantly down here, they might even have dependencies between them, right?

So, in a MonrayBus scenario, and this is a special case where it is very often used in open source libraries where you want to publish multiple packages and they might have some Like, if you look at React, GitHub repo, VEET, Vue, Angular, a lot of them use these types of approaches. So, how does this work? Now, in PMPM specifically, there's this prefix, and so that means you should resolve the package locally within the Monray but not go out for some MMPM registry, and in your MMPM workspace you don't need that, but that star there will be the same, and so that simply means I always want to depend on the latest version because it is locally in my workspace, right? Don't do this for production apps. This is just within the Monray repo.

So where does this package live? That's where this comes into place again because it can figure out, can it even live in this examples folder or in a packages folder? In this case, this UI library here. And again, this is a package with some build scripts, test scripts, and potentially authors as well, and then we have at the top here some example applications. Now, if you develop this as your application Monray repo, this might be the actual production apps. In open source package-based Monray repos, this is most often example pages where you test out your product or your packages, or it might even be deployed alongside your documentation for some live demos. So how do you run things in such a Monray repo? Well, you could obviously cd into the packages and just trigger these scripts, right? That would totally work. It's not really handy, though.

And so, for instance, PMPM has a concept such as filters. And so, for instance, I can filter me all the packages in this packages folder, and run me that script here, which is that build script. So it recursively traverses these packages. Now, in this case, it is a simple setup. So we just have two of them and it would build those. Now, I can also go and just build the single one. For instance, I have here that remix up there, and I target the dev script. So that would then launch my remix application, and it can serve it then at local 3,000. And I can kind of browse my remix application here, right? So I have, it just renders the components that are in here. Yeah. It's super right. All right. So, you might be wondering, like, what is wrong with this setup? And there's nothing really wrong here, right? Because it works, and especially in a smaller setup, it totally works fine and scales nicely. However, there are already some kind of things that we could optimize. For instance, these packages depend potentially on each other. So, for instance, the next, our remix app imported product list, which internally we have seen depends on the UI library. So what happens, for instance, if I delete this folder, because they consumed this folder, because in the package json we referenced these files, which is compiled output of our typescript files. Then if I serve, for instance, again the remix app, it would kind of break, because it cannot resolve those entry points anymore, and so it breaks, right? And so I would need to kind of run that filter again, rebuild it, and so then it would work. The things I need to keep in mind to order how I build them, I need to first build the UI, then the product list, and then the application.

2. Installing and Configuring nx Package

Short description:

To install nx, you can run nx add latest init or add the package manually. Running the script helps configure the workspace structure. Choose the built script as it needs to respect the order. Determine which scripts in the package monorepo are cacheable. Skip the outputs as they are already captured.

Plus, if I rerun this filter again, it would always keep running, right, even though we didn't touch anything, we didn't change anything. And this is where, for instance, a package like nx comes in, which can help you speed up some of these things.

So, how can you install nx? First of all, you can just run nx add latest init or just add the package on your own. The advantage of running this script here is that it looks at your workspace and the structure of it, and it kind of walks you through a couple of steps of kind of trying to configure it for you.

For instance, one of the first questions here is what scripts need to run in order? And we just learned that the built one might need to run in order because we need to respect that ordering of those things. And so I would choose the built one here. Next up, it asks me what type of scripts of those here in your package monorepo are cacheable. And so here, for instance, probably the built would be cacheable. Linting, type check and test, potentially. Usually, the dev server and the start scripts are not cacheable because that's the development environment you want to kick off. And then it asks me for outputs. We can just skip them because it already captures most of the common outputs, such as dist and build, and it would already monitor those for the output there.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Levelling up Monorepos with npm Workspaces
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Levelling up Monorepos with npm Workspaces
Top Content
Learn more about how to leverage the default features of npm workspaces to help you manage your monorepo project while also checking out some of the new npm cli features.
The Zen of Yarn
DevOps.js Conf 2022DevOps.js Conf 2022
31 min
The Zen of Yarn
In the past years Yarn took a spot as one of the most common tools used to develop JavaScript projects, in no small part thanks to an opinionated set of guiding principles. But what are they? How do they apply to Yarn in practice? And just as important: how do they benefit you and your projects?
In this talk we won't dive into benchmarks or feature sets: instead, you'll learn how we approach Yarn’s development, how we explore new paths, how we keep our codebase healthy, and generally why we think Yarn will remain firmly set in our ecosystem for the years to come.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
Federated Microfrontends at Scale
React Summit 2023React Summit 2023
31 min
Federated Microfrontends at Scale
Top Content
The talk will be a story of how Personio went from rendering through a Monolithical PHP architecture, to a microfrontend oriented Next JS app, powered by Module Federation and the NX monorepo toolchain.
Scale Your React App without Micro-frontends
React Summit 2022React Summit 2022
21 min
Scale Your React App without Micro-frontends
As your team grows and becomes multiple teams, the size of your codebase follows. You get to 100k lines of code and your build time dangerously approaches the 10min mark 😱 But that’s not all, your static CI checks (linting, type coverage, dead code) and tests are also taking longer and longer...How do you keep your teams moving fast and shipping features to users regularly if your PRs take forever to be tested and deployed?After exploring a few options we decided to go down the Nx route. Let’s look at how to migrate a large codebase to Nx and take advantage of its incremental builds!
The Age of Monorepos
JSNation 2022JSNation 2022
25 min
The Age of Monorepos
The history of the web can be divided into evolutionary development leaps. The age of inline scripts, the age of jQuery, the age of SPAs, the age of JAMStack...We are now entering the next stage that has been carefully prepared in the past few years. Let me invite you to the world of modern monorepo solutions and share with you the benefits you will reap by using them in every project size and setup. It's time you automate those boilerplate tasks and reduce the bottlenecks so you can focus on what truly matters.Get ready for the next leap! Welcome to the age of monorepos!

Workshops on related topic

React at Scale with Nx
React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Top Content
Featured WorkshopFree
Isaac Mann
Isaac Mann
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Monorepos with Nx
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
Top Content
WorkshopFree
Isaac Mann
Isaac Mann
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
JSNation 2022JSNation 2022
99 min
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
WorkshopFree
Matthew Salmon
Matthew Salmon
npm and security, how much do you know about your dependencies?Hack-along, live hacking of a vulnerable Node app https://github.com/snyk-labs/nodejs-goof, Vulnerabilities from both Open source and written code. Encouraged to download the application and hack along with us.Fixing the issues and an introduction to Snyk with a demo.Open questions.
Build Web3 apps with React
React Summit 2022React Summit 2022
51 min
Build Web3 apps with React
WorkshopFree
Shain Dholakiya
Shain Dholakiya
The workshop is designed to help Web2 developers start building for Web3 using the Hyperverse. The Hyperverse is an open marketplace of community-built, audited, easy to discover smart modules. Our goal - to make it easy for React developers to build Web3 apps without writing a single line of smart contract code. Think “npm for smart contracts.”
Learn more about the Hyperverse here.
We will go over all the blockchain/crypto basics you need to know to start building on the Hyperverse, so you do not need to have any previous knowledge about the Web3 space. You just need to have React experience.