Managing Large-Scale Node.js Projects with Monorepos

Rate this content
Bookmark

Unlock the secrets of managing Large-Scale Node.js Projects with Monorepos. Discover best practices, scalability strategies, effective code organization, and collaboration tools for streamlined development on a larger scale. Join me for insights that turn project management challenges into opportunities.

Fortune Ikechi
Fortune Ikechi
19 min
04 Apr, 2024

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Monorepos are a development strategy that allows you to store multiple projects in one repository, facilitating code sharing and simplifying dependency management. They provide simplified dependency management, improve code reusability, and enable a consistent build, test, and deployment process across projects. Tooling support like Nix, Yarn Workspaces, and NPM Workspaces streamline monorepo development. Code organization and scalability strategies involve fine-grained models, consistent directory structure, and selective dependency installation. Streamlining build processes can reduce build time, and optimizing performance and collaboration involves profiling tools and effective code review processes.

Available in Español

1. Introduction to Monorepos

Short description:

Hi, everyone. I'm Fortune Ikechi, a software engineer and technical writer. Today, I'll be talking about managing large-scale Node.js projects with monorepos. Monorepos are a development strategy that allows you to store multiple projects in one repository, facilitating code sharing and simplifying dependency management. They also enhance collaboration across teams.

Hi, everyone. My name is Fortune Ikechi. I'm a software engineer and technical writer. When this was created, I was a developer engineer at Storyblok. I'm not anymore. You can find me on Twitter at underscore, at codoc underscore. And today I'll be talking about, I'll be giving a lightning talk on managing large-scale Node.js projects with monorepos.

And yeah, so I'm going to look at this table of contents. Number one is Introduction to Monorepos and their role in Node.js development. Right. Also look at the best practices for organizing your code. I will also talk about the scalability strategies for Node.js applications, and I'll end it all with collaboration tools and workflows that you can use for Node.js application in monorepos.

Now, let's get started. So number one is the Intro to Monorepos and their role in Node.js development. Now before we get started, I have to mention what is a monorepo, right? So a monorepo in full is actually called monolithic or monolithic repository. It's a development strategy that you have so much projects and you store so many projects in one repository, acting like a single source of truth for multiple projects. So think of it as when you have your monolithic applications, which is where you have like everything joined in one particular project without separating concerns and all of those things. Basically the frontend, backend is all in one place. But in this case, a monolithic repository or a monorepo is basically where you have three, four projects in a single project, and then that becomes the source of truth for the entire project.

Now, the thing that monorepos help do is that it helps to facilitate code sharing, right? Because with this, you can share code between the different projects, right? And then reuse the different components of models or packages around those projects. And the second thing that it does is that it helps to simplify dependency management. And this is where you try to have multiple dependencies, right? For probably one dependency of folder or file or JSON handling all of the entire projects. And the last thing that monorepos do is that they enhance collaboration across different teams. Now, because of the fact that monorepos are different projects in one particular setup, you can have all of those projects, engineers or people who work on those projects, or the teams, again, collaborating to use the probably different models or packages around all of the projects.

2. Benefits of Monorepos

Short description:

Monorepos provide simplified dependency management and improve code reusability. They enable a consistent build, test, and deployment process across projects. Node.js follows a modular approach that complements monorepos, making it easy to manage interdependencies. Additionally, monorepos facilitate cross-project refactoring and ensure consistency through shared configurations.

Now, I'm going to look at the benefit of monorepos, and here I listed out the simplified dependency management. I already explained that you can have, again, a centralized package for your dependencies. And this is really, really essential Node.js environment because Node.js works with the whole idea of shared models, right? And now when you have all of these dependencies, probably from a single file, you can share them across the different models on your Node.js project in a monorepo. And you can also import all of them from a single, like, file, basically.

The next thing I mentioned is that unified to linear scripts. Now, with monorepos, you can have a consistent build, test and deployment process across all your projects. And this is because you can have basically a single file, probably a single script there, so JSON file, packet to JSON file, that handles all of your scripts, your tooling, your tests for all of the projects. And with these, your engineers and teams can build consistently for you to follow that particular test format or that build format or deployment process.

The next benefit of monorepos is that it helps to improve code reusability. This is because it's very easy to share code across the projects. I'm going to share this later on or spend more of it later on because of the talk and then we can see it. And I added here that this reduces redundancy and fosters consistency. When you have basically like one single code that you use across multiple projects, there is a consistent version of that code that is being used, right? It's basically like the same code reused across multiple places. And then it's you and your team can be sure of the consistency across multiple code being used across multiple projects or a single code being used across multiple projects.

Now, I wanted to mention Node.js monorepos and or monorepos in Node.js development. Now, I did hear that Node.js follows a modular approach, right? And this complements how monorepos basically work. And this makes it very easy to manage interdependencies across models because, again, you have one single dependency working across the different models in your application. I also added here the modular architecture compatibility, right? And with Node.js, Node.js works on a very small reusable packages or models that you can use across projects. And this works again with the way monorepos actually work, because now in this case, you can have a folder probably called utils and that folder is like all of the packages for the authentication or all of the packages for production logins and all of that. And then you can reuse this particular module across all your projects, right? In different folders in that particular monorepo. And now I'm going to mention the cross project refactoring, right? Now, with this, you don't have to be scared of making changes and then worrying about how it affects the entire project, because all your changes are basically working in one single place. And dependencies are again one single dependency. So you can do the updates, graphical code without worrying about a side effect or probably the change in breaking changes and all of those things. Because, again, this is just singular models that affects multiple projects. And you can be sure of how they all work because it's in box and it's collaboratively with a monorepo approach. And added shared configurations.

3. Tooling Support for Monorepos

Short description:

Monorepos facilitate code updates without side effects and enable consistent testing and code quality. Tooling support like Nix, Yarn Workspaces, and NPM Workspaces streamline monorepo development by providing semantic support, dependency installation, and efficient package management.

So you can do the updates, graphical code without worrying about a side effect or probably the change in breaking changes and all of those things. Because, again, this is just singular models that affects multiple projects. And you can be sure of how they all work because it's in box and it's collaboratively with a monorepo approach. And added shared configurations.

I think we see this every day with the linting, the formatting and testing config across different projects within monorepo. Now, with this consistency, right? You can be sure of maybe one single test case working across all the projects or your team building the project sync with that particular test-driven development or that particular test case. And this can make it really easier to how you evolve new packages because you just have to be sure that it works across all the projects and then they can be used safely across all the projects in your monorepos.

I added a few tooling supports for Node.js monorepos. And the first I wanted to look at was Nix. I think they call it Nix. Some people call it Nix, but it's Nix for me. So Nix is basically like a custom-age development tool that makes it easier for you to follow a monorepo approach. And the thing that Nix does well is that it simulates your workflow and makes it very efficient for you to build Node.js applications. Now, Nix does this by providing a semantic support and a CLI that makes it easy to automate some repetitive tasks, right? And again, this enhances your Node.js approach. But what are the support that it provides? Like those JS, Cypress, ES links. And this helps your code with high-quality and consistent code. But then how does this actually help your monorepo? Now, because you're sure that you can write a single test case that could affect all of the projects, you can be sure to not have to worry about writing multiple projects or multiple test cases for multiple projects, or even having to define different linting strategies for different projects in your monorepo. And this is one of the things that Nix helps you do.

I also added the Yarn Workspaces. And now the thing Yarn Workspaces does really, really well is that it allows you basically to have like a single root package of JSON file. And this is this helps with your dependency installation, right? I just have to install one single dependency that would work across all of my projects at all times. And then you can also link the different of dependencies to one single like package of JSON file. And that is in case you have like different JSON files that you can use. And again, another thing that the Yarn Workspaces help to do is that it actually helps you to, with a single command, work with all of the package of JSONs in your different projects. Because this is because all the packages are hosted. There is like links to one particular main or root file. And that helps you with, again, your monorepo, right? And with these, it just reduces the amount of space, the space that your computer uses when you're working with Yarn Workspaces and the monorepo approach.

The last I wanted to mention, I think, yeah, the last I wanted to mention is the NPM Workspaces. And I think a lot of you already use this because I see this in a number of open source packages. And what this does is that it just offers a native support for monorepos, right? It allows you to set up automatic installation to link your different packages within the repo.

4. Code Organization and Scalability Strategies

Short description:

The NPM CLI enhances monorepo management with a single CLI, automatic installation, and interdependent packages. Best practices include fine-grained models for single functions or features, a consistent directory structure, and shared utilities and libraries. Scalability strategies involve duplication of dependencies to reduce size and selective dependency installation at the root level.

And of course, it gives you the NPM CLI, which is one of the best developer friendly CLIs that we have in the Node ecosystem. And this enhances, again, how you manage your monorepos because you now have a single CLI that can work across multiple boards in your monorepo. You also have access to the automatic installation and, again, interdependent packages within your repo. And I think I'll just add a link here for those who want to check the NPM Workspaces. But this has been a good try. I see this a lot in, again, such open source projects.

The next thing we want to eye on is what are the best practices for organizing code in the monorepo. Now, I added fine-grained models. And this is an advice that you should break down your code base into small and reasonable models that perform a single function or a feature. Example here, I have in this code image or code block, is a logger image that basically logs for information and it logs the error, right? And you can then import this across all the multiple applications in your monorepo. I also have here a validator, which again, is a regular expression that is for your email or your phone number. And again, this can be used across all the multiple files or the multiple packages or models or projects you have in your monorepo. And that's the excellency of organizing code in a Node.js monorepo. This is also what I add here, is a directory structure, which allows you to create a consistent directory structure, right? That basically reflects how your application is and the relationship between your packages or your applications or projects. Here in the code, we have a packages folder and in that particular folder, we have authentication, which has a source and a package.json. And next, in there again, the package folder, we have a user management, which handles all of the user management features in this particular project. We also have image service, which handles all of the image service of users in this particular project or monorepo. And this is an excellent way of organizing your directories, right? Because with this, you can then link the different package.jsons or even different applications across each other. For example, I can use the auth to work with user management. I can use authentication to work with the image service. I added here shared utilities and libraries. And now, this is in collaboration with our first one, right? Which is a modular, fine-grained library or approach. And now, with these, you can basically create, again, small size packages that can be imported and used across different packages or protocols or even projects.

The next on our slide is the scalability strategies for Node.js applications, right? Basically, what are the strategies you're going to use to implement a Node.js repo and make sure the application is also scalable, its performance improves the way it should and it's actually sustainable. And the first one is the duplication of dependencies. Now, we have packages and tools like npm-dedu or Yarns Automatic Duplication, which again helps you to reduce the size of your node models by resizing and flattening your dependencies. And now, what this does is that it helps you basically to speed up your installation time and to reduce your disk usage. Because what Node.js does is it takes four duplicate packages and removes them, allowing you to just use one single package when due, right? And this can seriously cut down on your node model sizes. The next thing I added here is selective dependency installation. Basically, when you use Yarn Workspace or npm workspace, you can install only one dependencies at the root level, right? Instead of duplicating them across multiple packages, like for example, you want to use probably like bcrypt or you want to use like bcrypt or any other packages, right? You can basically have one of these two on your Yarn Workspace npm package in your root level, and then use it across all the different packages, models and projects in this monorepo.

5. Streamlining Build Processes

Short description:

Parallel build processes and incremental builds can reduce the build time of a Node.js monorepo by compiling only the changed files, making it easier to run the projects.

Now the next one I have is streamlining build processes. Now, Parallel build like Lena help you, you can actually run Parallel build flags or even npmx to run build processes in a parallel approach. And this would help to reduce the build time of your Node.js monorepo. I think time is a bit normal. Sorry, I'm going to go a bit faster. I also have incremental builds and this is where you use tools like WebPaths incremental build or watch mode or typescript incremental option to compile all the files that have changed. Instead of rebuilding your entire project, for example, if you want to build your projects for probably hosting all of those needs, you only run certain files that have changed, right? Using incremental build ideology from web path or even using type build incremental option. And this helps you basically make it easier to run your projects, right? You don't have to run the entire seven projects in one monorepo. You only run maybe two that have their files changed in the last couple of minutes.

6. Optimizing Performance and Collaboration

Short description:

To optimize performance for a Node.js monorepo, you can use profiling and monitoring tools like NodeSpec, Chrome DevTools, and New Relic. Implement caching strategies for frequently accessed data, either in memory or using distributed caching systems like Redix. Collaboration in a Node.js monorepo can be improved by adopting a branching model like GitFlow or trunk-based development and making small, focused commits. Code review processes in monorepos can include clear pull request guidelines, automated code quality checks with tools like Sonocue, ESLink, and CodeCov, monitoring for pull requests, and code review assignments using tools like GitHub Codonas or GitLab Approval Rules.

Number three is how do you optimize performance for a Node.js monorepo. And here I added profiling and monitoring, right? So you can basically like regularly profile your applications, use tools like NodeSpec or Chrome DevTools or you can even use third-party solutions like New Relic. I'm a big fan of New Relic. You can again implement caching strategies for frequently accessed data. And this could be either in the memory or using distributed caching systems like Redix. And I think a lot of us have already come across Redix and New Relic. If you're not, you can check that documentation. They're actually excellent.

Now, how do you collaborate with other developers in a Node.js monorepo, right? What are the tools and workflows that can help us collaborate better? The first thing I wrote here is a branching model. And I wrote that you adopt the branching strategy, like GitFlow or trunk-based development to manage development and releases in a structured manner. Basically, I said that trunk-based development can help with minimizing main request and facilitating continuous integration. But the next thing that it does that helps you making small commits, right? So with a Node.js monorepo, you can only make small or is recommended that you make only small and commits, right, your GitHub repository for a particular change or feature. And what this helps you do is that it gives other developers within the clarity of what that particular feature does and why it's necessary.

Now, what are the code review processes that you can use in monorepos? One is pull request guideline. And I said that you can establish clear guidelines for creating pull request, including naming conventions, required option, information in the description. Basically, this allows you to fix unrelated issues or documentation for your pull requests. And this can also help you create a new guideline for those pull requests. Added automated code quality checks. Now we have tools like Sonocue and ESLink and CodeCov, which help you just CI CD pipelines to perform code quality checks and reverse coverage on pull requests. I think I see this on GitHub too. Now, what is DOS that helps you, just monitors your application for when there is pull requests, right? Also check the quality of the code you receive in that pull request. We also have code review assignments. Now with tools like GitHub Codonas or GitLab Approval Rules, you can assign reviewers to pull requests based on the area of the code base that they are most comfortable with, right? Or they are more expertizing or experts in. I added here that this is an example of a Codonas file. And what this file does that the Auth folder in the packages directory basically is assigned to the Auth team, while the API is assigned to the API team and the UI is assigned to the UI team.

7. Code Review Processes and Conclusion

Short description:

Code review processes in a monorepo can be improved by using tools like GitHub Codonas or GitLab Approval Rules to assign reviewers based on their expertise. In conclusion, monorepos streamline Node.js development by offering a unified workflow, facilitating modularization, directive structure, dependency management, and a manageable code base. Collaboration can be enhanced through effective code review processes and using Git or trunk-based flow. My name is Fortune Iketchi, a software engineer and former DevRel Engineer at Storyblok. Find me on Twitter at CodeDog underscore. Thank you.

Now, what is DOS that helps you, just monitors your application for when there is pull requests, right? Also check the quality of the code you receive in that pull request. We also have code review assignments. Now with tools like GitHub Codonas or GitLab Approval Rules, you can assign reviewers to pull requests based on the area of the code base that they are most comfortable with, right? Or they are more expertizing or experts in. I added here that this is an example of a Codonas file. And what this file does that the Auth folder in the packages directory basically is assigned to the Auth team, while the API is assigned to the API team and the UI is assigned to the UI team. And that is an excellent way to basically have code review processes in a monorepo.

And now for the conclusion, I already explained how monorepo streamline Node.js development by offering a unified workflow for managing your project. We also saw the product of modularization. I mentioned directive structure. I also mentioned dependency management, right? I also mentioned a manageable code base. For referencing, I mentioned scalability strategies and how you should have efficient build and performance optimization for your code process. I ended this entire talk with a collaboration that on how you can use Git or trunk-based flow, basically store your source contracts, right? Also ended this effective code review processes. And that is excellent because this is something that we all do, even with or without monorepos, and not just in Node.js ecosystem, but also in other applications that we build.

Once again, my name is Fortune Iketchi. I'm a software engineer and until recently as a DevRel Engineer at Storyblok, you can find me on Twitter at CodeDog underscore. Thank you very much.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Levelling up Monorepos with npm Workspaces
DevOps.js Conf 2022DevOps.js Conf 2022
33 min
Levelling up Monorepos with npm Workspaces
Top Content
Learn more about how to leverage the default features of npm workspaces to help you manage your monorepo project while also checking out some of the new npm cli features.
End the Pain: Rethinking CI for Large Monorepos
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
Federated Microfrontends at Scale
React Summit 2023React Summit 2023
31 min
Federated Microfrontends at Scale
Top Content
The talk will be a story of how Personio went from rendering through a Monolithical PHP architecture, to a microfrontend oriented Next JS app, powered by Module Federation and the NX monorepo toolchain.
Scale Your React App without Micro-frontends
React Summit 2022React Summit 2022
21 min
Scale Your React App without Micro-frontends
As your team grows and becomes multiple teams, the size of your codebase follows. You get to 100k lines of code and your build time dangerously approaches the 10min mark 😱 But that’s not all, your static CI checks (linting, type coverage, dead code) and tests are also taking longer and longer...How do you keep your teams moving fast and shipping features to users regularly if your PRs take forever to be tested and deployed?After exploring a few options we decided to go down the Nx route. Let’s look at how to migrate a large codebase to Nx and take advantage of its incremental builds!
The Age of Monorepos
JSNation 2022JSNation 2022
25 min
The Age of Monorepos
The history of the web can be divided into evolutionary development leaps. The age of inline scripts, the age of jQuery, the age of SPAs, the age of JAMStack...We are now entering the next stage that has been carefully prepared in the past few years. Let me invite you to the world of modern monorepo solutions and share with you the benefits you will reap by using them in every project size and setup. It's time you automate those boilerplate tasks and reduce the bottlenecks so you can focus on what truly matters.Get ready for the next leap! Welcome to the age of monorepos!
Remixing Your Stack in a Monorepo Workspace
Remix Conf Europe 2022Remix Conf Europe 2022
22 min
Remixing Your Stack in a Monorepo Workspace
Remix entered the stage with a unique and refreshing take on how to develop on the web. But how do you integrate it into your existing ecosystem of applications? Do you want to test-drive Remix on a small project, or do you want to go full-in, but it is tricky to do a big-bang migration from your existing React app? In this talk, we're going to explore how a monorepo-based code organization can help integrate Remix with your existing React and TypeScript infrastructure, facilitating high code reuse and a migration path to Remix.

Workshops on related topic

React at Scale with Nx
React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Top Content
Featured WorkshopFree
Isaac Mann
Isaac Mann
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
Node Monorepos with Nx
Node Congress 2023Node Congress 2023
160 min
Node Monorepos with Nx
Top Content
WorkshopFree
Isaac Mann
Isaac Mann
Multiple apis and multiple teams all in the same repository can cause a lot of headaches, but Nx has you covered. Learn to share code, maintain configuration files and coordinate changes in a monorepo that can scale as large as your organisation does. Nx allows you to bring structure to a repository with hundreds of contributors and eliminates the CI slowdowns that typically occur as the codebase grows.
Table of contents:- Lab 1 - Generate an empty workspace- Lab 2 - Generate a node api- Lab 3 - Executors- Lab 4 - Migrations- Lab 5 - Generate an auth library- Lab 6 - Generate a database library- Lab 7 - Add a node cli- Lab 8 - Module boundaries- Lab 9 - Plugins and Generators - Intro- Lab 10 - Plugins and Generators - Modifying files- Lab 11 - Setting up CI- Lab 12 - Distributed caching