Fast React Monorepos with High Quality DX

Rate this content
Bookmark

Monorepos have been around for some time but only recently gained popularity in the JavaScript community. The promise of easily sharing code, better enforcing organizational standards, greater developer mobility due to common tooling, and more is very appealing. Still, if approached naively, a monorepo will quickly turn into a huge mess: skyrocketing slow CI times, spaghetti dependencies among projects, hard to navigate, and ultimately leading to frustration. In this talk, we will look at the available tooling, how to kickstart a new React monorepo in particular, and we will learn the key ingredients required to build a successful, long-running monorepo that scales.

22 min
21 Jun, 2022

AI Generated Video Summary

Welcome to a talk about fast React monorepos with high quality DX. Monorepos allow for collaboration and code sharing between packages, providing a more organized development environment. Leveraging caching and distribution in CI can improve speed and efficiency. NX provides a feature-rich monorepo setup for React, improving developer experience. Monorepo tools like NX console extension and project graph visualization enhance capabilities and enforce code quality.

1. Introduction to React Monorepos

Short description:

Welcome to my talk about fast React monorepos with high quality DX. I'm Joris Schumfler, a Director of Developer Experience at Narwhal. We provide consulting services for Fortune 500 companies, specializing in monorepos. NX is a smart, fast, and extensible build system that helps you succeed in the long run. It's hugely popular, crossing $1 million in December and about to reach $2 million per week in June.

Hey, and welcome to my talk about fast React monorepos with high quality DX. But before we go ahead, let me first introduce myself. My name is Joris Schumfler, I'm a Director of Developer Experience here at Narwhal. I'm also a Google Developer Expert in Web Tech. I'm an ECHI instructor and Cybers Ambassador.

Narwhal is the company behind NX, and we provide consulting services for Fortune 500 companies, but not only actually. And our range goes from helping with Angular development, React development, but in particular, helping with monorepos, so we help migrate to monorepos scenarios, set up new monorepos, but especially help succeed those monorepos in the long run. We're also the creators of the open source toolkit and NX. And if in case you missed it, most recently, we also took over Stewardship of Leerna, which is now joining forces with NX.

So what is NX? NX is a smart, fast and extensible build system and can be used for monorepos. And today I'm not going specifically into NX directly, but I'm using it as an example of a feature set like a fully feature complete monorepo tool that can help you not just getting started fast, but also succeed in the long run. Now NX is hugely popular. So we have seen crossed $1 million in December and are about to cross $2 million per week in June probably. So it is super exciting, and it shows how much traction the monorepos space recently got.

2. Understanding Monorepos and Polyrepos

Short description:

Now, what are monorepos? Rich Harris recently expressed dissatisfaction with the term monorepo, as it implies a single repository for the entire organization. However, in reality, large companies often have multiple monorepos, divided by department or domain, alongside polyrepos. These repositories can share components through internal registries, allowing polyrepos to benefit from the libraries built within monorepos. On the other hand, a polyrepo is a more traditional scenario with a single repository containing one project.

Now, what are monorepos? Rich Harris, a couple of weeks ago, posted a tweet, which I'm 100% agreeing with, because I have the same question from people, and I have to answer and explain that over and over again. So he was basically not happy with the term monorepo. And the problem is that monorepo implies, or what many people think, is that you need to have one single repository for the entire organization. Now, it's perfectly clear why people think that, but in reality, what we see when we work with now, for instance, with large companies, is more something like this. So you have large monorepos, couple of them inside your company, maybe split by department or organization or domain, and then you have also the existing polyrepos, or new polyrepos even, that come up within that organization. So if you have a couple of such a mix of landscape, and you share stuff over registries, over internal registries, even monorepos share some of the parts for the outside. Because if you have some polyrepos that might want to benefit from, let's say, the component library you build within a given monorepo, you might want to publish that to a registry as well, apart from just using it within that monorepo. So that's perfectly fine. Now, probably having that a term like multiproject repo versus single project repo would be more useful or more meaningful, but I'm not going to coin a new term here. So whenever I'm speaking about monorepos, what I intend is basically a single Git repository with two or more distinct projects, and Polyrepo on the other side is more classic scenario, which is a lot half-basic, which is like single repository with one project in it.

3. Exploring Monorepo Architecture

Short description:

Monorepos come in different forms. In some cases, they are used for code collocation and reuse of CI setups or common utilities. However, the real value of a monorepo lies in the collaboration and code sharing between packages. By converging separate polyrepos into a monorepo, you can split the codebase into smaller subprojects, each with specific libraries. These libraries can be shared across domains, providing a more organized development environment and allowing for fine-grained APIs. It is also possible to create dedicated sharing libraries and general-purpose libraries that can be published outside the monorepo. To ensure fast development, it is important to have tooling that allows for building only what has changed, such as using a project graph to determine dependencies and affected applications.

So how do monorepos look like? They come in different forms. So a lot, especially in the open source world, what you see is just like one repo with code collocation. So you have a couple of projects in there, they don't necessarily relate to each other or have relationships between them, so most of the time it's just for reusing CI setups or common utilities for being quicker in publishing to NPM, things like that.

But the real value you get out of a monorepo is actually if those packages relate to each other to some degree. So they share code and facilitate basically collaboration within the monorepo. And in fact, a monorepo is all about architecting your system. So, many times what you have is scenarios like this. You have different domain areas within your organization, and very often they exist as separate polyrepos. But if you converge them into a monorepo, a scenario might look like the following. So you have these domains, and as you evolve your monorepo, you split it up into smaller subprojects. So you have like an application per such domain, which you can deploy independently, and each domain has some specific libraries that are not shared with other domains or within other parts of the monorepo, but they're specific to that single domain.

The main reason of having those libraries is to split up the code base, have teams more organized and be able to work on single projects within that workspace, basically within that single domain, and you can also have much more fine-grained APIs in that case, which help you create more maintainable code in the long run. Now, obviously, you get more out of it once you also start reusing things. So at the very bottom of this image here, for instance, you see those libraries, which are used a lot by the upper-level domains, and those libraries are usually more code utilities, like authentication libraries, logging libraries, maybe for calculating dates, or also maybe even some more specific parts to your domain, to your organization. For instance, it could be like a UI component library that lives down there, which is used by the various parts of the domains to have like a uniform UI appearance and look and feel.

So what you rather do is you split up rather than creating a monolith, in this case, now down here, as you can see here, we have the more utility libraries, which I've just explained, but those are not the only ones you can even have connection between domains. And also in those cases, it is very useful to create, for instance, dedicated sharing libraries, which expose like a common data object, it can be used by other domains and common API interfaces, such that domains can communicate with each other. For instance, the checkout domain might need utilities, be it like UI utilities, but also communication utilities for the product part of the domains of your organization. And finally, as I mentioned before, you might even have those very leaf nodes, which are some very, very general purpose libraries, such as UI components, which you might also want to publish outside the monorepo, such as like polyrebus can consume those. Now, as you can see my title, the fast part is very important in the monorepo because if you approach them naively, it might be easy to get started, but like a year in or so, things might get very, very slow, very quickly. So if you PR on CI takes over an hour to build, that hinders new feature development and slows down your teams. So basically create a counterpart rather than having some more collaboration. You basically start to be detrimental. So you definitely need to have tooling that allows you to build just what changed and most of the development or most of monorepo tools have something that is called a project graph and what can be done with a project graph is, for instance, if such library as here in the picture in the middle changes, those tools can walk up those projects graphs to understand what needs to be built or tested or linted. So in this case, you can see we can already cut out quite some libraries and applications that don't need to be touched. So obviously, it will be much faster than also running tests for those. You can even leverage this, for instance, to understand what needs to be redeployed, such as we have always the latest version and production. So in this case, you can see all of the applications in our domains are affected because it's a very central library in this case that got changed.

4. Leveraging Caching and Distribution in CI

Short description:

You can leverage this to understand what needs to be redeployed. Caching makes the process faster by reusing results from previous runs. NX Cloud allows for distributed caching and task execution, improving speed and efficiency in CI. By distributing caches and tasks uniformly based on historical data, NX Cloud optimizes resource utilization and reduces overall execution time.

You can even leverage this, for instance, to understand what needs to be redeployed, such as we have always the latest version and production. So in this case, you can see all of the applications in our domains are affected because it's a very central library in this case that got changed. And so we would probably want to all deploy those applications as well.

Now, affected is just one part. Caching is what makes that even faster. For instance, let's take that same scenario where. We changed those libraries and they all get run in CI and one of the lib, like in the upper left corner, failed the test. So the developer looks at them, pulls it down, fixes the test, pushes it up again. Now we don't have to run all of them again because all of the other parts are not affected by the change that was just made. They're still affected compared to the main branch like master or main. However, we only need to run tests again or computation again for the lib and the app in the upper left corner. Because all the other results can be fetched out of the cache. So obviously you get much more speed.

Now the distribution is really the key here because right now the cache, for instance, in NX specifically, by default it's just local. So local to ever developers workstation, which means the local development gets quicker and speeds up but you cannot really leverage it, especially in CI. And that's where we see a lot of folks use actually the caching because you want to make sure PRs are fast to get merged in. And so for being able to have a distributed cache, you need some central server-side cloud counterpart, the NX cases and NX Cloud, which allows you to distribute that cache among team members as well as CI. And NX Cloud goes even a step further. So it doesn't only distribute the actual caching, but it also distributes the task execution, which makes it even faster. So normally what you do on CI is you get that project graph that was affected by the change. And then you have a set of agents because what you want to do is you want to parallelize as much work as possible. And so you can actually split it up in equal batches. That's usually what happens. That's the most naive approach, like just split them up in equal batches and distribute them to the agents. What might happen, though, is that some agents are super fast because they got tasks to complete in a couple of seconds while other agents run for minutes. And so still the entire CI run has to wait until all the agents finish. And so therefore the utilization of the agents is suboptimal. And also the actual time is suboptimal in the end. So it's not that fast as you would expect. Now, NX Cloud, for instance, distributes the caches or distributes not only the caches, but also the tasks uniformly based on historical data.

5. Improving Developer Experience with NX

Short description:

NX provides a fast and feature-rich monorepo setup for React. It offers a core setup for migration scenarios and pre-configured templates for new monorepos. The core setup leverages the Fast task scheduler, caching, and distributed task execution with NX Cloud. Setting up NX is easy, and it can be integrated with existing monorepos or used for new projects. NX also improves developer experience with beautiful output that focuses on what is most important and reduces cognitive load.

And so it knows based on the graph, which processes can be built in parallel, which basically where they may just assign a single task to an agent because it's a long running one and which one needs to be serially executed. And in the end, the DX part, it comes in here is that all the logs and artifacts are being collected and sent back to the actual main node and grouped together. And so from a developer perspective, if you get an error or want to look at the actual run, you can just go to the logs and you will see them the same way as if they would have been run on a single machine.

And it's even cooler because like all those tasks now are distributed across the agents. You here, you can see a screenshot of NX cloud showing the utilization of the agents, and you can see how they are balanced out also based on the previous data that NX cloud has gotten and therefore it knows how to best parallelize those tasks. And you get even like a nice visualization. So whenever a PR runs, you can in real time understand how many agents are currently running, which tasks are running on which agent, which is particularly important for debugging purposes.

Now, this was kind of the fast already a bit interleaved with the DX part, as we have just seen, like the visualization for instance, that helps you from a developer experience perspective to debug things. But a DX is very important in React, more in the monorepo scenario in general, because there's nothing like having a fast, super like, worst feature rich monorepo setup, but it's super hard to use as a, from a developer perspective or to configure. And so in NX specifically, the important part is the developer experience part. And so, first of all, things need to be incrementally adopted, right? So NX can be set up in two different main ways. So you can get started with a core setup, which means you don't use any plugins that come with NX, you just install NX and use it on your existing monorepo.

And so for migration scenarios, this is ideal because you can still use the same infrastructure that you had before, and NX will just make it much faster to run the tasks. So the thing you leverage is the Fast task scheduler, the caching, and also the distributed task execution if you use NX cloud, which you've just seen. The other approach is that if you start new, you can actually benefit from setting up NX with some pre-configured things. So you can say, okay, I know that I'm using a react monorepo setup because react is my main focus. So you can already use some pre-configured templates that NX comes with. And so then NX would make sure that you get like Jest setup, ESLint, Prettier, Cypress configuration set up for you. So you don't have to worry about a lot of that stuff. It's usually that is the best option if you start a new monorepo right now.

So for the core setup, it is super easy to adjust. You basically just run the npx add NX to monorepo command, which would add it to any npm, yarn, pnpm workspace, and just make it a lot faster. Interestingly, also, as I mentioned initially, we took over Seo Chip for Lerna, and now we can do some very interesting things, especially for your Lerna workspaces. For instance, right now, if you're using 5.1 plus of Lerna, you can just install NX and set the use NX to true on your Lerna JSON, and it would automatically defer the task scheduling to NX, making Lerna super fast without you having to change anything else, which is super important I think, from a developer economics.

Another thing is beautiful output. Now, this might not sound as important initially, but if you think like how often you look at the terminal outputs as a developer, and so in order to reduce the cognitive load, NX really just shows you what is most important right now. So it doesn't show you, for instance, the execution of depending tasks as in this animation here, but just what gets executed, unless, of course, some error occurs. That would be highlighted, obviously, big and then in red, and even if you rerun the task and they get cached, the output is exactly the same. So you basically have a much, much less lower cognitive load when you parse those logs because you just see what you need right now. Also ID integration.

6. Exploring Monorepo Capabilities

Short description:

This part explores the capabilities of monorepo tools, such as the NX console extension for Visual Studio Code. It also discusses the visualization of project graphs and the benefits of co-generators for onboarding junior developers. Additionally, it highlights the prevention of spaghetti dependencies through a tagging system.

This is specifically interesting for newcomers, but also like if you want to explore simply the capabilities that you have, right? So, for instance, for Visual Studio Code, but there's also community extensions for WebZone, for instance, we have the NX console extension, which allows you to navigate within an NX workspace and in the future even a learner workspace, allowing you to browse commands that you can execute, things that you can generate in that workspace visually from within Visual Studio Code.

Another part, the visual part is very important also when exploring the workspace. So, every or many of those monorepo tools have a so-called project graph underneath. Now in NX we went even a step further and we visualized that as a dynamic web application that you can start just from your CLI. And so here you can see how you can, for instance, take two nodes and basically have an NX visualize the shortest path between those nodes as well as all potential paths. And you can imagine how that can help to understand why some nodes connect with each other, like why the connection actually exists, but also for instance for debugging circular dependencies.

And finally, a very important part is not just like getting started quickly as we have seen in terms of being fast, but also continued support as your monorepo grows. So, co-generators are one thing that can help you with that, which NX comes with, meaning for instance, if you have like for easier onboarding junior developers, you have already some co-generators in place such that like all the libraries and applications are generated in a uniformly way as you would expect. So, you don't have them copy and paste, which is very error prone, of course, existing libraries remove existing code, and then basically continue working on that. But you can actually even not just use the NX provided generators, but tailor them really down to your own needs. So, you can customize them, so it only gets generated whatever you need in your current workspace. Easily, that can also be triggered by Visual Studio Code. And so, here for instance, you can see in this animation example of setting up that distributed task execution environment for CI. So, you can just generate such a CI setup, which is like super useful. And it's even just a couple lines of code, if you take a look in the end. So, this is an example of how powerful those generators can be without you having to write long docs explaining how to set up those things. Similarly, also, for instance, if you want to set up a React Webpack modification setup, NX comes with already such a plugin that you can use for setting up the main host and remote applications. And then you can use that to build up your existing stuff and continue developing on top of those generators.

Finally, also preventing Spaghetti dependencies. And this is something people underestimate quite a lot if they start in monerepos. Because they get started very excited in the beginning. But then if you are in like half a year where you have multiple teams working on a monerepo, things might get messy very, very quickly. So, people importing from Libs that they shouldn't import, and maybe even just by accident, because they saw some cool utility library and they were not supposed to use that. So, your code quickly might become unmaintainable. In the next, for instance, as you can see here, for instance, like if we have those domains, like you can use those library as domains, right? Where you can clearly expose a public API, such that other people from a different domain can then leverage those functions within their own. Now, how can you, however, prevent that someone imports just a different Lib? In theory, you can just do that because you live in the same monerepo. Now, in the next we have so-called tagging system. This is simply strings. So, you can say, like, I have a scope column, some name, which for instance could be the domains and to give to all the apps and Libs in a certain domain, you give those tags. You can add that in your configuration and all those Libs would get that.

7. Enforcing Dependencies and Code Quality

Short description:

You can use specific tag APIs to mark APIs that can be consumed from outside the domain. ESLint rules can define source tags to enforce dependencies within specific domains. Running these checks in CI and editors helps maintain code quality.

And then you also give to that API, for instance, specific tag API to mark it as something that can be consumed from outside the domain. And then the next comes just with a specific ESLint rule. So, in this ESLint rule here, for instance, you can see how we define a source tag and say, OK, something that is called scope checkout, which is our specific checkout domain, can only depend on other apps and Libs that also live in that scope checkout so that also have that tag or that they have a scope API. Because, like, if there's a scope API, there is something that can be explicitly used. So, that's fine. And this can then be those checks run in CI. They run in your editor. So, you immediately see if you import something wrong and are, therefore, very, very powerful to keep your code maintainable.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
GraphQL Galaxy 2022GraphQL Galaxy 2022
31 min
Your GraphQL Groove
Building with GraphQL for the first time can be anywhere between daunting and easy-peasy. Understanding which features to look for in your client-side and server-side tooling and getting into the right habits (and ridding yourself of old habits) is the key to succeed with a team of any size in GraphQL.

This talk gives an overview of common struggles I've seen numerous teams have when building with GraphQL, how they got around common sources of frustration, and the mindset they eventually adopted, and lessons learned, so you can confidently stick with and adopt GraphQL!
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!

Workshops on related topic

React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Featured WorkshopFree
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
WorkshopFree
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching
React Advanced Conference 2021React Advanced Conference 2021
168 min
How to create editor experiences your team will love
Workshop
Content is a crucial part of what you build on the web. Modern web technologies brings a lot to the developer experience in terms of building content-driven sites, but how can we improve things for editors and content creators? In this workshop you’ll learn how use Sanity.io to approach structured content modeling, and how to build, iterate, and configure your own CMS to unify data models with efficient and delightful editor experiences. It’s intended for web developers who want to deliver better content experiences for their content teams and clients.