Using React to Build Performant Game UIs in Minecraft

Hello, welcome to Using react to build performant game UIs in Minecraft. My name is Paolo, I'm a tech lead at Mojang Studios here in Stockholm, and I work in a game that you're probably familiar with, which is Minecraft. I'm part of this fantastic team composed of artists, designers, C++ developers, javascript developers, and a producer, and our main goal is to change how UIs are built in Minecraft. And we're doing that not only by changing the text deck, but also by introducing a design system into the product. So this is some of the components that we have, and you might see these rolling out in the upcoming years. And yes, we are actually already even in production. We rolled out a screen last year, which is the achievement screen. So if you play Minecraft Bedrock Edition on your Xbox or PlayStation, you probably have seen the screen, and this screen is completely based on standards/talks">web standards and it's built using react. And this is all the platforms that we have support, which is also one of the main challenges for our project. So we need to run on the Xbox One, on the PlayStation, on the Switch. We have Android phones and tablets, iOS phones and tablets, Windows 10, MacOS, almost all platforms that exist, Minecraft runs on it, and then we need to support it. And it's not only about device capabilities, but also different input types. So we need support touch, gamepad, all sorts of things. VR even is supported by Minecraft. But why are we using web technologies to, or like standards/talks">web standards, should I say, to build UIs for Minecraft? There are a lot of reasons. And some of those I did cover on this preview talk that I gave in 2018. So if you're curious, you can also check that out. But if you don't want to take a look at that, here's a too long dinner read. So basically Minecraft is a completely custom engine game. So it doesn't use something like Unreal or Unity. So the UI is completely built in house. It's completely custom. And currently that's been a problem for us where when we onboard new developers, it takes a long time for them to get used to the tech and figure out how to use it. And we wanted to move to a solution that will lead us to better maintainability and would make it easy for us to get people onboarded and also improve the iteration speed. So in short, we wanted to make something that was based on open standards, W3C, and want to get the benefit of all the great tooling that the javascript ecosystem has. But how are we doing this? Are we just taking Chromium and embedding it within Minecraft? So that could technically be a possibility, and there are some games that actually do this. What we're doing is instead something else. So there's a company called Korean Labs, and they've been doing solutions based on Chromium for a while for game UIs. But for a while already, they built another solution, which they call Game Face, where they took the standards, which are open of the web, and then they built just a subset of them into something they call Game Face. So Game Face, it could be considered kind of like a tiny browser, I'd say. If you build a UI for Game Face, it will most likely run in Chrome or Firefox, but it might not be the case if you just pick a random UI that you build on Firefox and put it on Game Face. It might not work. For example, it has support for Flexbox, but it doesn't have support for floats. And it's built with the main goal for targeting game UI. So performance is the main thing. It's one of the main goals. But otherwise, taking Game Face out of the equation is a fairly standard stack. We use react, we use webpack, we do unit tests with Jest. So if you're a web developer working in a shop today, and you join Mojang, you will probably be productive from day one, and you will be able to contribute with a UI for the game, which is pretty awesome. And it's a very different experience from what we have today in the studio. But before I go and talk about performance, which is the main thing we want to discuss on this presentation, first we need to talk about how state management works, and how that's different when we are in a game environment. So when we look at a traditional web application, we can think about it like this. We have your browser, like for example here I have Firefox running the UI, and that has its own javascript runtime. And then we have, let's say, a server on the internet running node.js. And then we have HTTP requests and responses coming back. So it's a very kind of like request-response operation. And then we have a copy of the state in the browser. When you go into a game world, what we usually have is just a single process, right? We have the javascript engine still running, which will have our react code and our UI code in there. But we have the C++ game engine right there. And that's what we call our backend. And that's usually the holder of the state. If you want to understand this, it's better to look at an example. And a good example is Minecraft. So if you look at this screen, which is basically what you see if you're playing the game, this is the HUD. We see basically the information about how much health your character has, the fullness, because Minecraft is a survival game. So if you don't eat, you start to starve and you can die. We also see the items that you have access in your hotbar. So these are items that you can quick switch using the gamepad, and which item you have selected. So here you can see we have the first item selected, which is the sword. But a player can also, by interacting with the game, can switch to change the selected item to be the rotten flesh. And that not only affects the game itself, but also needs to be reflected in the UI. And the player can also take damage, right? Let's say a mob comes and a creeper blows up next to the player. And then we need to take some health damage. And here the health has dropped to eight. So we can see here that we have two categories of data, two groups of data. And we call those facets. So facets is a term that we coined internally in the studio, because we needed something that didn't exist within the code base. So we couldn't use model, for example. But this is kind of like what it is. It's just a slice of data. But other than that, facets conceptually, they're very similar to an observable, where they are likely available over time. So we subscribe to a facet, like let's say the player stats, and then we keep getting updates about new values as they change, as the player plays the game, right? So the bottom there, we can see as the time progresses and the player loses health, we have a hypothetical player health component updating with how much health the player has. Usually then what we want is that the state shouldn't live in the javascript side, but it rather should live on the C++ side, which is our backend, but it's living right there within the same process. We want kind of like a global store, and that global store is the C++ side of the game. And we think about global stores, and we look at the javascript ecosystem, there are a couple solutions available. Some of the popular ones these days are kind of like Recoil or Jotai, and they're very kind of like on this fashion where you have like a global store. And we took a very similar approach to these solutions, and we got inspired by Recoil's api to build something that feels very familiar, so that anyone that would come to the studio would be, would understood what the concepts are trying to imply. But there are some differences, of course, that we're going to go through in Next. So react Facet is actually a pack, a collection of packages. We have a lot of packages. The part that deals with the communication between the javascript bits and the C++ bits, we call react Facet Remote, and that's the first part I'm going to talk about now. And I guess the first thing that we need to look at is how do we define the facets, right, the state bits that we have that is shared between the C++ side and the javascript side. And if you look at, if you used Recoil or JS, for example, at one point, this might feel a bit familiar. The only thing is that we're using typescript here. So the first thing that we start with is defining the api contract, which is just an interface for this data type. This is something that we usually talk with the C++ developers in a line. So we say, for example, that we want to have a health that has a number, and we want to have fullness that has a number. Then we have a unique identifier, and this is how we, from the javascript side, can request a particular piece of data to the C++ side. Then we have the definition itself, which is very similar to an atom on a Recoil JS lingo. And then we also have selectors, which allow us to pick the full facet and slice just a piece of it, kind of like have a map function. And here we're picking, for example, just the health attribute from the entire place stats facet. And when we want to consume it, it's also very familiar. We would have just some extra hooks and some different naming here, but we basically need to first use the use remote facet hook to kind of like subscribe to this remote facet. This will actually kick off and start listening to the events. Then we turn it into a react state, which is the health there. It's going to be the actual string where the number containing the health, and then we can just render the result as a normal react component. The key difference between our solution and the existing solutions in the open sourcing space is subscription. Our facets, they don't really have any value until we start subscribing to it. And after that, they act pretty much like observables. So in our solution, we do have to have a provider that will wrap the application. And this provider, we have the responsibility of implementing the glue from the javascript side to the C++ side. In this case here, we have an engine, which is like a global object that we have access to, and is kind of like an event emitter, where I need from the javascript side, I need to set up a listener, which in this case is listening for the event facet updated player stats. Let's say I want to access the data for the player stats. And then I need to emit an event to notify the C++ side or the game engine that I'm interested in that data. And finally, I can return a function that can do a cleanup. So what happens in practice is that when a component subscribe to, let's say, the player health selector that we showed in the previous example, if it's the first time this function will be called, we'll set up the subscription, we'll start listening to that data. And once that component gets unmounted, the cleanup function gets called, and we stop listening to it, and we tell the C++ side to stop notifying us from updates. That's pretty cool. And it works pretty well for us. So the next step, which is the interesting part for all of you, is, of course, how we handle performance. Where do you optimize stuff for a solution so that it can work well in a gaming development world? And the first thing, I guess, to really make it clear is that react by itself is already plenty fast. If you're building web applications, it's usually sufficient just to use react as it is. And solutions like Recoil or Jotite, they help kind of like narrow down and funnel updates so that only specific parts of your react tree are updated. And when we run into performance issues in react, usually the recommendation is that we take a look at our components and we see if we have any kind of like re-rendering that we're doing unnecessarily, and see if we can avoid that at all costs. So when we look at this concept, it's kind of like, I mean, what if we instead of just avoid unnecessary renders, we could avoid all re-renders at all? Could we have a situation where we basically don't trigger the react reconciliation? So here's kind of like how it's happening as our previous example. The facet gets updated through an event from the C++ side that triggers the, that turns, that of course updates some state internally in react, which triggers the reconciliation and then react through the virtual DOM figures out, oh, I need to actually update this text node. And that's how an update happens. So what we're proposing is that we could just skip the middleman and just go straight from the facet update to the DOM update. So if we go back to the example that we had before, what we're proposing is instead of like taking the facet, which is kind of like this nice data structure that holds a value over time, instead of unwrapping it and turn it into a state, a react state, can we just pick the facet and pass it directly to the react render and let the react render figure out how to unpack the value, unwrap the value from the facet? And that's exactly what we did. So we want to pass the facet and that directly updates a text node. So let's see how we've done that. So taking the example from before, I just expanded a bit and turned it a bit more interesting. So here I have now a bit more complex progress bar. So I'm actually rendering some divs to make it like a nice style. You can see at the top there how visually it will look. And then we see the hardcoded result. We have like a hardcode variable there with the progress set to two. So that's very simple react component, just rendering a progress bar. So if you want to take this example and make it so that it skips reconciliation altogether, how could we do that? So I think the first step, of course, is to subscribe to the facet in the first step. So we change that to do the use remote facet to take the value from the selector and grab it to be used in a component. But progress here is not actually a number per se. It's actually just the facet still. Like, it's the data structure that holds the value inside of it. So I'm getting actually a type error here because I can't make an arithmetic operation with a facet. It's not a number. So we need to then figure out a way now to like, OK, I have this facet, and I want to do some transformation to it and turn it into something else. If this was a recoil in a global state, I would probably just use a selector. Because this is inside a react component, what we need is some way of making a selector that is defined within a react component. And we do support that through a custom hook that we have that's called use facet map. And it's very similar to a selector conceptually. It takes an input facet. It can actually take multiple ones, but in this case, we're only passing one, which is the progress that we subscribed to in the line before. Then we have a map function that will take the progress, which is the number coming from the facet, and transforming it to the string containing the pixel unit. And then the result of it is also another facet, which is holding now in the width variable. So the next step now is, of course, is to take this width and pass it to the div. But if we do that, react is not going to be happy. It's going to be like, hey, I don't know what to do with this data type, right? You're trying to give me something that's not a string. It's not a number. I don't know how to render this. So the next step in this is we need to change react render itself to support this data type. And that's what we did by introducing a new div, which we call fastdiv. And we actually created a whole series of components that match kind of like one-to-one. So we have a fastP, a fastSpan, that instead of accepting just regular numbers, strings for the props, they also accept facets as a prop. And in the end, what we have is basically this, right? A component that is subscribing to a remote facet, turning to like a variable progress that holds a facet of progress. Then we're mapping that into a width facet that has the width. And then I'm taking that and passing it to the fastdiv that actually then updates that prop in the DOM. So we have basically created a renderer that natively supports facets, or you could say a lightweight observable, as values for the props. But we could also go one step further and make custom components that also natively supports facets as value. So let's see how that would look like. So if we go back again and take a look at this example of the player health, what if we want to do also the fullness, right? We want to render also the fullness progress bar. Could we do that? I guess the first thing is that we want to extract the components so we can reuse in both places, right? So we want to take this part here that handles the progress bar and extract that to a progress bar component. But we want to pass the progress there as a facet, right? We don't want to pass as a number because if we unwrapped that in the player health to turn it into like a regular react state, we would go back to basically triggering conciliation and we would defeat all the benefits we're trying to achieve. So here we have them, basically the pair health and the player fullness, two components using the progress bar and both of them passing a progress, each with their own facet containing their own data. And the progress bar implementation looks like this. Instead of taking a progress that's just a number, it's taking instead a facet of a number. And we have the type specifically available for there. But otherwise the implementation here in the body is exactly the same as we had before. So this is kind of like what's happening then in the end, right? We have at the top, our player stats remote facet. Once that gets notified of an update from the C++ side, that data then gets passed down to the player health remote selector that then picks just the health from the data. That then is passed to their use remote facet, which kind of like triggers the subscription. Then it goes through the use facet map that transforms that. And then the way that finally gets down to the fast diff and then the DOM is actually updated. And the cool thing about all of this is that all of this is happening behind the scenes. We're triggering no re-require renderings at all. And this whole implementation is built as a custom renderer, as I said, and it's available in this package react-facet-dom-fiber. But this is all fantastic and it's pretty cool, but like how well, how much does it really improve the performance? So comparing, and I put here two different platforms just to give it some perspective. So I'm running these scenarios on both an Xbox One and also on Chrome. So this Xbox One is running the Minecraft game and we have also Chrome on my MacBook. This example that I'm showing here is basically taking the progress bar implementation that we did before and just changing its value every frame and see how much time I'm taking between each implementation. On the left, we have implementation that's state-based from react and it's taking like a hundred percent of the time. And then on the Xbox One, we can see that a facet-based solution takes only 32% of the time to do the same work. When we look at Chrome, the difference is a little smaller, but it's also a lot faster when we look at the facets implementation. Another scenario that's very interesting to look at is that imagine if you have like a big list and you're rendering that through a bunch of components and you're memoizing it. So each component is memoized. When we look at the solution based on facets on the Xbox, the difference becomes massive, taking just 12% of the time. And on Chrome, it's also about 30% of the time. This example I brought up because it's quite interesting to look at if we have like a big list, but all the items are being updated at once. Here we can see that the Xbox is actually a lot faster when we compare it to the standard react implementation. When I looked at the Chrome on my MacBook, the difference is not that big. And the main thing for this that is worth bringing up is that when we're running V8 in our Chrome, in our Macs, we have just-in-time compilation. But unfortunately in some platforms like the Xbox, PlayStation, and Switch, we cannot have JIT enabled. So optimizations and libraries that we're used to will behave and will have different performance characteristics when we run them. And also another important thing is like the more complex the react trees become, then the wider the gap will be and the faster, actually the faster the implementation will be in relation to the standard react implementation. We have more data, of course, available at our GitHub repository. All the examples that I mentioned here, you can go there and see their actual implementations to understand better exactly what they're doing. I didn't have much time to go into the details here, but we do have all of that available on the GitHub repository. But not only that, I mean, if you look at facets, we have global stores, we actually have a bunch of more hooks that turn into a much more react-friendly api. So for all of the react hooks that are on the right here, like useState, useReduce, useEffect, we have counterpart implementations on the facet world, but instead of like, you know, the difference being that it actually can accept facets as input and they might return a facet as an output. And this is all available in the react Facet Core package. Just to go really quickly, here's how that looks like, where I have this component here, where I'm using the useFacetState, that is very similar to useState, but instead of value there being just a string, it would be, for example, a facet of a string. Then I have also a useFacetCallback here that is very similar to useCallback, but instead it allows you to pass value there, which is a facet instead. There's more hooks and documentation available also at our GitHub page. So in summary, facets not only in just the library, but it's a pack of libraries. We have the new data type that we introduced, which we call facets, of course, which is a lightweight observable that lives inside the react Facet Core. And you can use just react Facet Core to build something without necessarily using the custom renderer or anything like that, or the remote package. We have the remote package, which is the mechanism that we developed to have like this global store state to share state between the javascript side and the C++ side. And we have our custom renderer that we built within the DOM fiber. So this is all the packages that we have. And you can take a look at the docs to figure out what they do. I don't have much time on the talk to go through them. But it seems a bit daunting, but it's actually quite easy to get started if you want to just get your feet wet with facets. We have a package called DOM Components, and what it allows you to do is it implements fast components that we did. But instead of being like host components in a custom react renderer, they're actually just regular react components. So you can see here them being used on a regular react DOM to render this component. And all of this is being open sourced. We're open sourcing under the Oryui brand. Oryui is kind of like an initiative that we're doing in Mojang Studios to kind of like try to standardize UI development in the gaming industry. So the stuff they're doing is trying to push that. So Oryui will have react facets to start, but we'll push some other packages that we're developing as well. So this is the fantastic logo. And this is our GitHub repository. So if you want to look at facets code base, if you want to learn more, go to GitHub and take a look at Mojang slash Oryui. And of course, I had to mention that we are hiring at Mojang. It's quite an exciting project. I mean, it's not everywhere you can go and use your web skills to contribute to one of the biggest games in the world. It's quite exciting to be able to work here. And if you want to do it, we have positions open for tech leads, web developers. Feel free to reach out to me or go straight to jobsmojang.com. So yeah, thank you so much. I had to run a bit through the slides. But yeah, if you have any questions, I think we're going to have a Q&A soon. So thank you so much. And feel free to reach out to me as well if you have any questions. Bye bye.