Scalable React Development for Large Projects

Hey everyone. My name is Jason. I'm here to talk to you about Scalable React Development Made Easy by NX. So first I'm going to go through an introduction of myself and I'm also going to introduce the concept of a moderator to you and then I'm going to go more in detail about what NX is and how it can help you scale your React development. Then we'll do a live demo at the end given that we have time. So let's get started. A little bit about myself, you can find me on Twitter at FerzandPandas. I'm an architect at Narwhal and I'm also a part of the NX core team. So I work on NX as part of my full time job. At Narwhal we consult for Fortune 500 companies. We're experts in moderator development and we also create dev tools to accelerate moderator development. So my whole job is to accelerate companies work faster and more efficiently. And that's where NX comes from. But before I go into NX, let me tell you a bit about monorepos. So first I want to say what a monorepo is not because there's a lot of misconceptions. Then I'll tell you what it is and then I'll tell you why one might want to use monorepo. So a misconception about a monorepo is that it is not a monolithic application. This means that you don't have to build it all at once. You don't have to build one big application out of a monorepo and you also don't have to deploy the whole monorepo at once. And let me go into more about what a monorepo is so that you see where this is a misconception. A monorepo is just a single repository that has multiple projects. Meaning there's one piece of the code living in one directory and another piece of the code living in another directory but it's all one version control. I think of it like a warehouse for code. In a warehouse you have packages that are packed separately by different people. They're stored together in one warehouse but then you ship them separately to different people. And the same is true for projects within a monorepo. They're developed independently with one repository but they're deployed independently. Who uses monorepos? So Facebook, Microsoft, Google, they all use a monorepo as well as other large tech companies. And I don't think it's a coincidence that a lot of these successful large tech companies have gravitated towards a monorepo approach to development. So let's take a look at why they choose to do so. Let's take a look at how we usually develop apps without monorepos. So I have my app but I also split out the functionality into different pieces of the code so that my code is organized. I stick everything into one Git repo. I set up NPM and I set up Jenkins or Travis and everything works together. I'm able to work on different pieces of the code as well as the application itself at the same time and then I'm able to test everything all at once. And this is the workflow that many people are familiar with. However, soon we're going to need to share the code between different applications within our organization. Not only do we have a cart application but we might have an API and a shop application as well. And they may want to access parts of the code that we had written previously for the cart application. So this is really hard to do because everything is in one repo and you can't really bring in the pieces of cart without bringing in cart as a whole. So I'm going to talk about sharing code in multiple repos, how many of you are probably used to doing things. And this is going to be the more complicated way of the two. So when you realize you want to share code, you decide, okay, that means I have to split out all my code into different repos. So you go ahead and make all these GitHub repos and people work on them. So this immediately introduces the need for duplicate tooling. Not only do you have multiple repos, you also have multiple CI flows, you also have multiple versions of NPM dependencies and all that. And for every repo that you set up, you need this. So along with duplicate tooling, you also have to separate your workflow. When you work on this library over here, you can only run those tests because the other tests live in different repos. So you're stuck with running only these tests. Therefore, you don't know if car application is broken or not. So you have to do some work over there as well. Then you can run those tests. But you'll also notice that API also depends on its library. So you have to do the same thing over there. And this is kind of like three different phases of your workflow. It's very discontinued. And you lose a lot of context when switching from one place to the other. Not like before, where we could work on the car application and our library at the same time. But because people want to share code, this is what we have to deal with. Also, those clear lines that I have drawn between these projects, they are less analyzable when one is using multiple repos. Multiple repos are usually linked through methods like NPM, through package JSON, and stuff like that. And it's really hard to be able to analyze them. So those lines become less visible when we're looking at our organization as a whole. All we know is that we have six different repos, but those are related in ways that we can't easily analyze through code analysis. Then we also can grow out of sync. A lot of times when we split things into different repos, we publish different versions. So at the beginning, everything might be a version one that works together. It's great. But then in the bottom library over here, we might release a second version. That has some changes. And other libraries have to pull those changes in. So the one on the right will pull it in and upgrade to version two of this bottom library. But the library on the left will choose to do it later. So they're going to stay on version one right now. And that might seem like a fine decision at this view of the world. But when we go back to our organization, the car application that uses both of these libraries is very confused right now about which version is the correct version. And more than likely, it will actually bring in both versions. And we kind of run into the jQuery days where there's three versions of jQuery on the page. And this goes the same for external dependencies as well, such as React. When things are different repos, they can depend on different versions of React. And in our organization, we might have version 16, version 15, and version 0.14 all at the same time. And each application might use a different version. It might be confused about which version it should bring in and things like this. So let's talk about how we can do this easier and less complicated. Using a monorepo, we will store all this code under one repo. And I know what everyone's thinking, this goes out of scale. But later on, I'll show you that it doesn't. One thing we see is that our tooling is once again unified. We have one repo, one CI flow, and one NPM dependencies. Also, those dependencies that we had drawn are very analyzable now. They look like import from this file, from this file, and we're able to see what projects depend on other projects. That way, I can work on both of these at the same time. I can work on my library. I can work on my application. And when I push it up to see if my changes look good, I can also run the specific set of tests that are sufficient for verifying that my change is accurate and good. So the other libraries towards the left of the screen, those don't have to run because my changes don't affect them at all. There's also a single version of every project and all your dependencies. So it just moves ahead. Whenever you check out a new branch, you have the same version that works together with everything. This also means that you can commit to multiple projects under one commit and you don't have to worry about keeping it safe, the versions. And the same is true for React. So once we upgrade to version 16 of React, our whole repository and all of our applications are on version 16. So I went through a lot of information. Let's just take a step back and review. So using multiple repos, you don't have clear dependencies. You have multiple versions of everything and you have a lot of duplicated tooling. When you do the same thing via a monorepo, you have one set of tooling, you have one version of dependencies, and you're also able to test only what's necessary to be tested at the time. Sounds pretty cool. So at the end of the day, why use a monorepo, right? Isn't it just how you store your code? And I don't think so. I think it's actually about the people. It's not just about the code. And our people are organized under one company and one organization. And we're all one team, even though we might have different responsibilities within our organization, we work as a team and have one goal. So let's take a look at how we can use NX to implement a monorepo workflow. NX is an open source toolkit that enables you to scale your development across multiple teams. So that means multiple applications, back end, front end. It allows you to do it all. The first thing that's a major feature is it allows you to develop like Facebook. We'll go into more details about what this means. The second thing is that it has an intelligent build system. So like before, we were able to see our different projects be analyzed and tested appropriately, and NX is able to do that. It also supports the use of modern tools, such as Storybook, and I'll go into more detail about each of these. So for developing like Facebook, as you saw with the monorepo approach, you're able to share code very easily between different teams. This also gives you a holistic development experience. You have all these different projects under one hood, and you're able to see how different pieces work with one another, and what's going on in each part of the repo. It may not be important at all times, but when you're working on a specific task that involves another library, it's wonderful to see what's going on there. So the next thing is how you can stay consistent. Through code generation, we're able to create a workspace with zero manual setup. We run npx create NX workspace with a preset of React, and we get this workspace generated with our new app for Carts. Then you can add shared functionality very easily. Before you'd have to create a different repo, you'd have to set up a CI flow, publish the NPM, but now you can run NX generate lib share button, and now you have a shared button that you can use across multiple different applications without publishing the NPM and without setting up CI. You can also generate readout state. Everyone knows there's a lot of boilerplate, and even with the changes that are in the Redux toolkit, there's still a lot of code that you have to write to get started. Well now you can generate it using NX generate Redux cart, directing it to the project of cart state. So this is automatically generated for you, and you can see that we're able to create a slice with a cart, and it even has some actions predefined for you that you can change after you've generated the code. We can also add things like Next. So using Yarn add at RWA Next and NX generate at RWA Next app shop, you're able to generate a new shop app in the same repo as cart application without much setup at all, and NX is fully configured and ready to go. What if you don't like what NX generates? There's so many tools in the ecosystem that we all love and want to use, but it's understandable that you might not like what NX decides to generate for you. And we've also taken that into consideration. You're able to write workspace schematics, which are custom code generation that suits your organization using NX workspace best practices. And this really allows your organization to stay consistent by generating the code that implements the best practices that you want to have. This is really useful when you want to integrate with different projects. You want to have a consistent way of laying out your state and stuff like that. This is one of my favorite features of NX. Also, NX helps you enforce standards and best practices throughout your organization. So before we were talking about generating code in the right format, but what about the code that we write? One thing that is confusing about a monorepo sometimes is that people think that any library can depend on any other library and you get this ball of mud that you can't untangle and understand. With NX, you can tie different projects. So the one on the left is tied to state and the one on the right is tied to UI. And we all know that state should not depend on UI. It should go the other way around. So if we make a dependency from the state to the UI, we get a linting error within our term and within our editor. And let me zoom in for that so that you can see the whole thing. A project tied with state can only depend on state. It can't depend on state or a type of UI. NX is also smart enough to infer your dependencies. As we were seeing before, projects depend on one another. And it's important to know how those dependencies are structured. So taking the code like this, where I import product UI and also product state from my application, NX is able to infer that this application, the current application, depends on these two libraries without you having to explicitly say it. Then once we know these dependencies, we're also able to visualize it for you. Running NX Step Graph, you're able to have an interactive view of all the dependencies within your project. So on the right, you see product's homepage depends on product UI. These dependencies are also crucial for the intelligent build system. When people think of monorepos, a lot of the times they think of just running everything, right? And this is what Lerna does. And sometimes this grows to 60 minutes or even more. And this isn't very scalable if we wanted to make a PR and check that it works within five minutes. So let's take a look at some ways that we can improve this. NX allows you to build only the affected projects. As we saw before, for a little change in this really big monorepo, we don't need to test everything. So we can take out the bits of the project that are relevant to us at this time and test only those, cutting our times by more than half to about maybe 25 minutes in this hypothetical scenario. Also because we have different projects that are run independently from one another, we're also able to build in parallel now on the same machine. So we can run across two different threads and you can see that our times once again have lowered by more than half. Then you can also distribute builds across multiple agents. Not only can you parallelize them on the same machine, but you can actually distribute them. So if I have five different workers, I can now test these five different projects across different nodes in my CI. And as you can see here, the red bar over there is still quite long. In order to make that shorter, we can also cache the previous builds. If we've already done this build before, even though my change actually affects it, it can realize, hey, I've already done this build before, let me not do it again. And it can pull that from a cache, bringing the times down to two minutes. Now the best way to do this is via remote cache, which you can add by adding NX Cloud, which is our paid add-on to NX. But NX is open source again, so you don't necessarily need to. Six minutes is more than a favorable time going down from 60 minutes. But NX Cloud allows you to never build the same thing twice. So the last thing is being able to use modern tools. NX comes by default set up with TypeScript, Jest, Cypress, and Storybook. Sure everyone is familiar with Jest, and you might even have your own opinions of TypeScript, but you can also use JavaScript within your repo with NX. Two things that you might not be familiar with are Cypress, which is an end-to-end testing experience. So not only does it allow you to write your tests, but it actually gives you a UI where you can time travel through those tests and make sure that you're not running into issues when you're running your end-to-end tests. Storybook allows you to develop your components in isolation, the way that we like to think about it during our React developments. And this is really important to be able to see how these UI components look when they're isolated from the application that they're developed under. You can also install all these plugins a la carte. If you don't want Storybook, you don't have to install Storybook. You can install only React and Next if you want to use it. You can also install Node.js, Express, and HTML if you wanted. And you can even install Angular and Nest.js plugins. All this is a la carte, so you pick and choose what you want, and you don't have anything extraneous that you're not using. We also have community plugins. So we have the ability for people in the community to create plugins for their favorite tools, such as migrating from Karma to Jest or deploying serverlessly. On our wall, we can't support every single technology out there in every single community project, but the community has done a great job of creating plugins for other people to use. And you can actually create your own plugin. If you do mpx create-nx-plugin, it'll actually create a different workspace for you that's suitable for plugin developments. So with that all out of the way, let's jump into a live demo. So I'm going to jump into a regular workspace here. This is already laid out, so I'm not going to show you all of the code generation steps, although we saw that it was fairly easy. Most of this code here was set up through code generation, but I want to take a look into our car application. In our car application, in our main component, we see that this file right here imports from a different project called the shared header. So we know as developers that, oh, obviously the car application depends on the functionality and shared header. So that when we have our nx-example header here, the functionality doesn't live within this directory right here. It actually lives in a separate directory. So let's follow that import, and we have our header elements. Being this is our header elements, we're able to see that this is the way it works without having to go to a different repo. If I wanted to change the way this works, I can add a console block here, hello React Summit, save this, see that my changes are in Git. And so this is great. I can make a PR. I can check my car application here by running run start cart, and now I can see my application running with our latest changes. So if I open this, give it one second. So we're able to see that my console log shows up here. We have the header, and our car application still looks good. Going back to the code, now that we've changed this, we don't really know for sure until we run our tests that the car application looks good within CI. So I could go and test my shared header, and then after that I can do share. I can test my cart. So I can let those run. I'm going to trust that they both work, but it's kind of tedious to have these dependencies in my mind at all times. And X can actually show that to you. So if I run a threaded test, it'll look at my changes in Git and realize that, hey, I need to check my cart, and I also need to test my header. So it automatically knows it's for you, and you don't need to tell it. And this is great to be used in CI, where you have an arbitrary number of changes across multiple projects, and you want to test only what's necessary. And I told you earlier that you can actually see the depth graph. So by running nx-depth-graph, this will open the interactive depth graph in a browser. So you can see a lot of projects here, and they all depend on one another. But let's say I'm only working on my cart page right now. I can go in and focus on my cart page and see that these are the only dependencies that my cart page has, separated from the rest. And if I don't really care about my product date data here, I can exclude that from the view and see an even clearer picture. I can also group things by folder so that you can see where things live within the repository and stuff like that. This is really useful because in the past, an architect would go into a Wikipedia page and update with what they think is depending on one another, and it very quickly goes out of date. So that's about all the time I have to show you stuff in the code for now. So let's continue on with the presentation. You can find that repo that I was showing you examples from at narwhal.com. Please check it out. It has a cart application written in React, and the product application is actually written in Angular. So you can see how multiple frameworks can be used inside the same repo, but still share code between the two seamlessly. That's it for my talk. You can follow me on Twitter at Verzopandas, and also please make sure to check out NX at nx.dev. NX.app is for NX Cloud, and follow us at Twitter at NX DevTools. All right. Thank you very much, and I'd love to answer your questions now. That was great. Thank you so much, Jason, Gene, for that talk. It was outstanding, and it reminds me, when I was working at IBM, we had lots of microservices, and it took a long time for us to get those things deployed. And something like NX really would have been helpful. So let's welcome back JJ. Thank you for coming on. We got some great questions from the audience, so I'm just going to launch right into it. The first one is really interesting to me because I think it's almost the most important part of software development that we always don't talk about, which is how do you convince leadership and other people on the team to make a switch like this? And more specifically, how do you convince leadership that a monorepo isn't a tight coupling of code? Yeah. So I think from the leadership standpoint, is that you're going to see a lot of collaboration across multiple teams, and there's a lot of dependencies between different teams that create friction when they're not working together well, and a monorepo drastically changes that. So companies like Facebook are able to collaborate between multiple different teams, and we actually have a case study with T-Mobile about how they were able to use a monorepo and deploy their new shop really quickly, relatively speaking. And about tight coupling, you have a lot of code, but you can feel free to make multiple different copies of that code, like Kent was talking about in his talk and stuff like that. So you can still have different relationships between your code, not necessarily tightly coupled. Okay. So another great question here is, if you start using NX, how dependent does the codebase become on NX? Like if you decide later that you want to stop using it, is it something that you can migrate away from easily? Yeah. NX is a dev tool, right? So it doesn't have to do anything with your production code. And the things that are under NX, while it's really smart and we have it available for everybody, you can do it yourself, right? Or you can use another tool that also works with this. So companies like Facebook don't use NX internally, but they have their own tools. I think it's called Pants or Buck, one of them, right? And it does very similar things, but it's much harder to set up. But yeah, if one day you decide NX is not working for you and you need something else, then you could move away from it. It doesn't change the way that you write your code, it just works around it. Yeah. So let's see. Does NX support deployment to different environments, like if you wanted to deploy to Heroku? Yeah, NX supports. Every different project can be deployed differently depending on what they need, right? So there is actually a community plugin for Netlify and a lot of other services like Firebase. So it's down to the project level. If you want to deploy an API this way, you want to deploy a frontend another way, another frontend another way, it's free room for you. A question that I saw that I really liked here is how could monorepos and microfrontends work together? Yeah, so I think there's a misconception that they're mutually exclusive and people talk about breaking up their monolith into different repositories, but you can have all these different microfrontends, different projects into this monorepo. I actually think monorepo actually makes it easier to manage microfrontends, right? Because when you break everything into separate frontends, when you get the task to change the universal header across all of them, you just have this huge plate of work going in different repos and making the same change multiple different times. And with the monorepo, you make the change, you do all the fixes in one go, and every microfrontend is updated, right? So you've actually just gotten rid of that huge hurdle of working across these microfrontends by putting them in the same place. And I actually have a blog post about it, so you can check out React Microfrontends with monorepos, a perfect match. Okay, yeah. So a question that I'm seeing here, I'm going to rephrase it a little bit so that it's not calling anybody out, but if you've got a team where the pattern has been to create a new repo for each small thing, what is your recommendation for helping teams work within the NX monorepo framework when that's been the pattern before, is to break everything into small codebases? Yeah, and I'm guessing that as you break code into small codebases, you're also publishing those to a registry somewhere so that people can pull them down. I think the first thing that I would do is that I would move the publishable parts into NX, and then have the different projects within NX, and you can see the relationships between them, but then you can publish those back up to artifactory, whatever registry, and then you can start bringing it in from higher and higher nodes of your dependency graph. So a question that I have is, what's the difference between something like NX and other monorepo management tools like Lerna? Are they the same, or are they solving different problems? Yeah, Lerna is really popular, and it's really, really good for, I think, package management. So NX has a lot of different packages that we manage, and we don't use Lerna, but it would be nice to have something that just runs everything, and that's what Lerna does. It allows you to have separate projects, but it allows you to run builds across all of them, test against all of them. But at the end of my talk, I was going through, running everything is going to take a really long time once you can scale up five, six different teams. So when you're talking about organizations like Facebook, running all the tests takes a huge amount of time. So NX has the intelligence, along with that, to be able to run only a portion of your projects when necessary to verify that the change is safe for production. Got it. So that kind of dovetails into another question here, which is, how do you support big teams that are working on one repo if you have the need of something like a merge queue? I might not be as familiar with the term of a merge queue, but maybe it's like, you know, a lot of people working in the same monorepo, you might imagine a lot of merge conflicts, right? I think it's also a misconception because merge conflicts only happen when two people work on the same thing. It's not exactly like, you know, a thousand people working on the same piece of code, right? Everyone's working on their little slice of the monorepo. So many times you won't get merge conflicts, right? Because you're working on something completely different from other people. The only thing is when you're working on a change that touches everything in the monorepo, say like you change this really core thing, then yeah, there's going to be merge conflicts when people merge your code in. But hey, you're going to have to do it once rather than like 50 different times across all those different teams. So at the end of the day, that's still more efficient from a workflow standpoint, right? And yeah, the only other recommendation I have is that maybe you need this like, you know, I need this change in, let me just force push it in without, you know, getting all my approvals or like there's a separate team that handles that kind of change. So that's how I recommend going about it. Not sure if I understood the merge queue correctly. So the question has been kind of rephrased to talk about like a deployment queue. And so I'm going to talk about my own experience because I know for sure what I'm asking, just to make sure. And what I've seen is that in a lot of companies, when you get ready to do a deployment of a certain thing, so let's say that the app's front end is about to do a deployment, there might be multiple branches going, but they will lock deployment for a window of time to make sure that the code gets merged and it goes live and everything is okay. The workflow that we use now wouldn't work in a monorepo because we'd be locking the entire company, most of which that code is unaffected by the front end deploying. So how would you, how is a workflow like that where I need to say no one can merge to the UI for this window of time? How would that, how would that work? Yeah, so a development workflow that works really well with monorepos is called trunk-based development. So that means everyone develops based off of master or develop branch, but all the commits go into there. When it comes time to do releases, and as I talked about, you can release a separate app by itself, right? You don't have to release the whole monorepo at once. So what you can do is you can cut a release branch based off of master, right? And say like release for April 17th. Don't release on a Friday, but let's just say that you've cut this release branch, you deploy this to like a QA environment, a staging environment, whatever, right? You do those tests and then you deploy based off of that branch, right? Meanwhile, master keeps going, but you don't have to worry, right? Everyone's still breaking code, right? And stuff like that. And then, you know, when another team wants to do their own deployment, like they're free to just cut a branch off of master, right? And then use that for releasing. So I recommend looking into trunk-based development and what that means. It works hand in hand with the monorepos. So great question. Great question. Cool. And that is all the time that we have. JJ, thank you so much for the talk. Thank you for hanging out with us and answering questions. And remember, if you are a paid ticket holder, JJ is about to go to a special room where you can get FaceTime with him. So make sure that you check your ticket details and go join if you've got follow-up questions.