Managing Large-Scale Node.js Projects with Monorepos

Hi, everyone. My name is Fortune Ikechi. I'm a software engineer and technical writer. When this was created, I was a developer engineer at Storyblok. I'm not anymore. You can find me on Twitter at underscore, at codoc underscore. And today I'll be talking about, I'll be giving a lightning talk on managing large-scale Node.js projects with monorepos.

And yeah, so I'm going to look at this table of contents. Number one is Introduction to Monorepos and their role in Node.js development. Right. Also look at the best practices for organizing your code. I will also talk about the scalability strategies for Node.js applications, and I'll end it all with collaboration tools and workflows that you can use for Node.js application in monorepos.

Now, let's get started. So number one is the Intro to Monorepos and their role in Node.js development. Now before we get started, I have to mention what is a monorepo, right? So a monorepo in full is actually called monolithic or monolithic repository. It's a development strategy that you have so much projects and you store so many projects in one repository, acting like a single source of truth for multiple projects. So think of it as when you have your monolithic applications, which is where you have like everything joined in one particular project without separating concerns and all of those things. Basically the frontend, backend is all in one place. But in this case, a monolithic repository or a monorepo is basically where you have three, four projects in a single project, and then that becomes the source of truth for the entire project.

Now, the thing that monorepos help do is that it helps to facilitate code sharing, right? Because with this, you can share code between the different projects, right? And then reuse the different components of models or packages around those projects. And the second thing that it does is that it helps to simplify dependency management. And this is where you try to have multiple dependencies, right? For probably one dependency of folder or file or JSON handling all of the entire projects. And the last thing that monorepos do is that they enhance collaboration across different teams. Now, because of the fact that monorepos are different projects in one particular setup, you can have all of those projects, engineers or people who work on those projects, or the teams, again, collaborating to use the probably different models or packages around all of the projects.

Now, I'm going to look at the benefit of monorepos, and here I listed out the simplified dependency management. I already explained that you can have, again, a centralized package for your dependencies. And this is really, really essential Node.js environment because Node.js works with the whole idea of shared models, right? And now when you have all of these dependencies, probably from a single file, you can share them across the different models on your Node.js project in a monorepo. And you can also import all of them from a single, like, file, basically.

The next thing I mentioned is that unified to linear scripts. Now, with monorepos, you can have a consistent build, test and deployment process across all your projects. And this is because you can have basically a single file, probably a single script there, so JSON file, packet to JSON file, that handles all of your scripts, your tooling, your tests for all of the projects. And with these, your engineers and teams can build consistently for you to follow that particular test format or that build format or deployment process.

The next benefit of monorepos is that it helps to improve code reusability. This is because it's very easy to share code across the projects. I'm going to share this later on or spend more of it later on because of the talk and then we can see it. And I added here that this reduces redundancy and fosters consistency. When you have basically like one single code that you use across multiple projects, there is a consistent version of that code that is being used, right? It's basically like the same code reused across multiple places. And then it's you and your team can be sure of the consistency across multiple code being used across multiple projects or a single code being used across multiple projects.

Now, I wanted to mention Node.js monorepos and or monorepos in Node.js development. Now, I did hear that Node.js follows a modular approach, right? And this complements how monorepos basically work. And this makes it very easy to manage interdependencies across models because, again, you have one single dependency working across the different models in your application. I also added here the modular architecture compatibility, right? And with Node.js, Node.js works on a very small reusable packages or models that you can use across projects. And this works again with the way monorepos actually work, because now in this case, you can have a folder probably called utils and that folder is like all of the packages for the authentication or all of the packages for production logins and all of that. And then you can reuse this particular module across all your projects, right? In different folders in that particular monorepo. And now I'm going to mention the cross project refactoring, right? Now, with this, you don't have to be scared of making changes and then worrying about how it affects the entire project, because all your changes are basically working in one single place. And dependencies are again one single dependency. So you can do the updates, graphical code without worrying about a side effect or probably the change in breaking changes and all of those things. Because, again, this is just singular models that affects multiple projects. And you can be sure of how they all work because it's in box and it's collaboratively with a monorepo approach. And added shared configurations.

So you can do the updates, graphical code without worrying about a side effect or probably the change in breaking changes and all of those things. Because, again, this is just singular models that affects multiple projects. And you can be sure of how they all work because it's in box and it's collaboratively with a monorepo approach. And added shared configurations.

I think we see this every day with the linting, the formatting and testing config across different projects within monorepo. Now, with this consistency, right? You can be sure of maybe one single test case working across all the projects or your team building the project sync with that particular test-driven development or that particular test case. And this can make it really easier to how you evolve new packages because you just have to be sure that it works across all the projects and then they can be used safely across all the projects in your monorepos.

I added a few tooling supports for Node.js monorepos. And the first I wanted to look at was Nix. I think they call it Nix. Some people call it Nix, but it's Nix for me. So Nix is basically like a custom-age development tool that makes it easier for you to follow a monorepo approach. And the thing that Nix does well is that it simulates your workflow and makes it very efficient for you to build Node.js applications. Now, Nix does this by providing a semantic support and a CLI that makes it easy to automate some repetitive tasks, right? And again, this enhances your Node.js approach. But what are the support that it provides? Like those JS, Cypress, ES links. And this helps your code with high-quality and consistent code. But then how does this actually help your monorepo? Now, because you're sure that you can write a single test case that could affect all of the projects, you can be sure to not have to worry about writing multiple projects or multiple test cases for multiple projects, or even having to define different linting strategies for different projects in your monorepo. And this is one of the things that Nix helps you do.

I also added the Yarn Workspaces. And now the thing Yarn Workspaces does really, really well is that it allows you basically to have like a single root package of JSON file. And this is this helps with your dependency installation, right? I just have to install one single dependency that would work across all of my projects at all times. And then you can also link the different of dependencies to one single like package of JSON file. And that is in case you have like different JSON files that you can use. And again, another thing that the Yarn Workspaces help to do is that it actually helps you to, with a single command, work with all of the package of JSONs in your different projects. Because this is because all the packages are hosted. There is like links to one particular main or root file. And that helps you with, again, your monorepo, right? And with these, it just reduces the amount of space, the space that your computer uses when you're working with Yarn Workspaces and the monorepo approach.

The last I wanted to mention, I think, yeah, the last I wanted to mention is the NPM Workspaces. And I think a lot of you already use this because I see this in a number of open source packages. And what this does is that it just offers a native support for monorepos, right? It allows you to set up automatic installation to link your different packages within the repo.

And of course, it gives you the NPM CLI, which is one of the best developer friendly CLIs that we have in the Node ecosystem. And this enhances, again, how you manage your monorepos because you now have a single CLI that can work across multiple boards in your monorepo. You also have access to the automatic installation and, again, interdependent packages within your repo. And I think I'll just add a link here for those who want to check the NPM Workspaces. But this has been a good try. I see this a lot in, again, such open source projects.

The next thing we want to eye on is what are the best practices for organizing code in the monorepo. Now, I added fine-grained models. And this is an advice that you should break down your code base into small and reasonable models that perform a single function or a feature. Example here, I have in this code image or code block, is a logger image that basically logs for information and it logs the error, right? And you can then import this across all the multiple applications in your monorepo. I also have here a validator, which again, is a regular expression that is for your email or your phone number. And again, this can be used across all the multiple files or the multiple packages or models or projects you have in your monorepo. And that's the excellency of organizing code in a Node.js monorepo. This is also what I add here, is a directory structure, which allows you to create a consistent directory structure, right? That basically reflects how your application is and the relationship between your packages or your applications or projects. Here in the code, we have a packages folder and in that particular folder, we have authentication, which has a source and a package.json. And next, in there again, the package folder, we have a user management, which handles all of the user management features in this particular project. We also have image service, which handles all of the image service of users in this particular project or monorepo. And this is an excellent way of organizing your directories, right? Because with this, you can then link the different package.jsons or even different applications across each other. For example, I can use the auth to work with user management. I can use authentication to work with the image service. I added here shared utilities and libraries. And now, this is in collaboration with our first one, right? Which is a modular, fine-grained library or approach. And now, with these, you can basically create, again, small size packages that can be imported and used across different packages or protocols or even projects.

The next on our slide is the scalability strategies for Node.js applications, right? Basically, what are the strategies you're going to use to implement a Node.js repo and make sure the application is also scalable, its performance improves the way it should and it's actually sustainable. And the first one is the duplication of dependencies. Now, we have packages and tools like npm-dedu or Yarns Automatic Duplication, which again helps you to reduce the size of your node models by resizing and flattening your dependencies. And now, what this does is that it helps you basically to speed up your installation time and to reduce your disk usage. Because what Node.js does is it takes four duplicate packages and removes them, allowing you to just use one single package when due, right? And this can seriously cut down on your node model sizes. The next thing I added here is selective dependency installation. Basically, when you use Yarn Workspace or npm workspace, you can install only one dependencies at the root level, right? Instead of duplicating them across multiple packages, like for example, you want to use probably like bcrypt or you want to use like bcrypt or any other packages, right? You can basically have one of these two on your Yarn Workspace npm package in your root level, and then use it across all the different packages, models and projects in this monorepo.

Now the next one I have is streamlining build processes. Now, Parallel build like Lena help you, you can actually run Parallel build flags or even npmx to run build processes in a parallel approach. And this would help to reduce the build time of your Node.js monorepo. I think time is a bit normal. Sorry, I'm going to go a bit faster. I also have incremental builds and this is where you use tools like WebPaths incremental build or watch mode or typescript incremental option to compile all the files that have changed. Instead of rebuilding your entire project, for example, if you want to build your projects for probably hosting all of those needs, you only run certain files that have changed, right? Using incremental build ideology from web path or even using type build incremental option. And this helps you basically make it easier to run your projects, right? You don't have to run the entire seven projects in one monorepo. You only run maybe two that have their files changed in the last couple of minutes.

Number three is how do you optimize performance for a Node.js monorepo. And here I added profiling and monitoring, right? So you can basically like regularly profile your applications, use tools like NodeSpec or Chrome DevTools or you can even use third-party solutions like New Relic. I'm a big fan of New Relic. You can again implement caching strategies for frequently accessed data. And this could be either in the memory or using distributed caching systems like Redix. And I think a lot of us have already come across Redix and New Relic. If you're not, you can check that documentation. They're actually excellent.

Now, how do you collaborate with other developers in a Node.js monorepo, right? What are the tools and workflows that can help us collaborate better? The first thing I wrote here is a branching model. And I wrote that you adopt the branching strategy, like GitFlow or trunk-based development to manage development and releases in a structured manner. Basically, I said that trunk-based development can help with minimizing main request and facilitating continuous integration. But the next thing that it does that helps you making small commits, right? So with a Node.js monorepo, you can only make small or is recommended that you make only small and commits, right, your GitHub repository for a particular change or feature. And what this helps you do is that it gives other developers within the clarity of what that particular feature does and why it's necessary.

Now, what are the code review processes that you can use in monorepos? One is pull request guideline. And I said that you can establish clear guidelines for creating pull request, including naming conventions, required option, information in the description. Basically, this allows you to fix unrelated issues or documentation for your pull requests. And this can also help you create a new guideline for those pull requests. Added automated code quality checks. Now we have tools like Sonocue and ESLink and CodeCov, which help you just CI CD pipelines to perform code quality checks and reverse coverage on pull requests. I think I see this on GitHub too. Now, what is DOS that helps you, just monitors your application for when there is pull requests, right? Also check the quality of the code you receive in that pull request. We also have code review assignments. Now with tools like GitHub Codonas or GitLab Approval Rules, you can assign reviewers to pull requests based on the area of the code base that they are most comfortable with, right? Or they are more expertizing or experts in. I added here that this is an example of a Codonas file. And what this file does that the Auth folder in the packages directory basically is assigned to the Auth team, while the API is assigned to the API team and the UI is assigned to the UI team.

Now, what is DOS that helps you, just monitors your application for when there is pull requests, right? Also check the quality of the code you receive in that pull request. We also have code review assignments. Now with tools like GitHub Codonas or GitLab Approval Rules, you can assign reviewers to pull requests based on the area of the code base that they are most comfortable with, right? Or they are more expertizing or experts in. I added here that this is an example of a Codonas file. And what this file does that the Auth folder in the packages directory basically is assigned to the Auth team, while the API is assigned to the API team and the UI is assigned to the UI team. And that is an excellent way to basically have code review processes in a monorepo.

And now for the conclusion, I already explained how monorepo streamline Node.js development by offering a unified workflow for managing your project. We also saw the product of modularization. I mentioned directive structure. I also mentioned dependency management, right? I also mentioned a manageable code base. For referencing, I mentioned scalability strategies and how you should have efficient build and performance optimization for your code process. I ended this entire talk with a collaboration that on how you can use Git or trunk-based flow, basically store your source contracts, right? Also ended this effective code review processes. And that is excellent because this is something that we all do, even with or without monorepos, and not just in Node.js ecosystem, but also in other applications that we build.

Once again, my name is Fortune Iketchi. I'm a software engineer and until recently as a DevRel Engineer at Storyblok, you can find me on Twitter at CodeDog underscore. Thank you very much.