This talk will go briefly about the history of how Mojang Studios decided to use web standards and React to build the new UI stack for Minecraft (Bedrock Edition), the challenges we faced around performance and how we landed on a solution that allowed us to continue to write declarative UI, but without the virtual DOM.
Using React to Build Performant Game UIs in Minecraft
AI Generated Video Summary
This Talk introduces the use of React and web technologies for building UIs in Minecraft. It discusses the challenges of onboarding new developers to the current tech and the benefits of using open standards. The speaker explains the use of Gameface, a solution for building game UIs with React and Webpack. The Talk also covers state management in a game environment and the use of facets for performance optimization. It concludes with an overview of the Oryui brand and the availability of resources on GitHub.
1. Introduction to Using React in Minecraft
Welcome to Using React to Build Performant Game UIs in Minecraft. I'm Paolo, a tech lead at Mojang Studios. Our goal is to change how UIs are built in Minecraft by introducing a design system. We have already rolled out the achievement screen based on web standards and React. We support multiple platforms and input types, including VR.
Welcome to Using React to Build Performant Game UIs in Minecraft. My name is Paolo. I'm a tech lead at Mojang Studios here in Stockholm. And I work in a game that you're probably familiar with, which is Minecraft.
This is some of the components that we have, and you might see these rolling out in the upcoming years. And yes, we are actually already even in production. We rolled out a screen last year, which is the achievement screen. So if you play Minecraft Bedrock Edition in your Xbox or PlayStation, you probably have seen the screen. And this screen is completely based on web standards, and it's built using React. And this is all the platforms that we have support, which is also one of the main challenges for our project. So we need to run on the Xbox One, on the PlayStation, the Switch. We have Android, phones and tablets, iOS, phones and tablets, Windows 10, Mac OS, almost all platforms that exist, Minecraft runs on it. And then we need to support it. And that's not only about device capabilities, but also different input types. So we need to support touch, gamepad, all sorts of things. VR, even, is supported by Minecraft.
2. Using Web Technologies for Minecraft UIs
Why are we using web technologies to build UIs for Minecraft? Minecraft is a custom engine game, and the UI is completely built in-house. Onboarding new developers to the current tech takes a long time, so we wanted a solution based on open standards for better maintainability and improved iteration speed.
3. Using Gameface for Minecraft UIs
We're not just taking Chromium and embedding it within Minecraft. Instead, we're using a solution called Gameface by Korean Labs. Gameface is a subset of open web standards, built for game UIs with a focus on performance. It's like a tiny browser that runs UIs built with React and Webpack. Web developers joining Mojang can be productive from day one and contribute to the game's UI.
But how are we doing this? Are we just taking Chromium and embedding it within Minecraft? So that could technically be a possibility, and there are some games that actually do this. What we're doing is instead something else. So there's a company called Korean Labs, and they've been doing solutions based on Chromium for a while, for game UIs. But they, for a while already, they built another solution which they call Gameface, where they took the standards which are open of the web, and then they build just a subset of them into something they call Gameface. So Gameface, it could be considered kind of like a tiny browser I'd say. If you build a UI for Gameface, it will most likely run in Chrome or Firefox, but it might not be the case if you just pick a random UI that you build on Firefox and put it on Gameface, it might not work. For example, it has support for Flexbox, but it doesn't have support for Floats. And it's built with a main goal for targeting game UI, so performance is the main thing, it's one of the main goals. But otherwise taking Gameface out of equation is a fairly standard stack. We use React, we use Webpack, we do unit tests with Jest. So if you're a web developer working in a shop today and you join Mojang, you will probably be productive from day one and you'll be able to contribute with a UI for the game, which is pretty awesome and it's a very different experience from what we have today in the studio.
4. State Management and Facets in Game Environment
But before I go and talk about performance, which is the main thing we want to discuss on this presentation, first we need to talk about how state management works and how that's different when we are in a game environment.
So if you look at this screen, which is basically what you see if you're playing the game, this is the HUD. We see basically the information about how much health your character has, the fullness because Minecraft is a survival game, so if you don't eat, you start to starve and you can die. We also see the items that you have access to in your hotbar, so this is items that you can quick switch using the gamepad, and which item you have selected. So here you can see we have the first item selected, which is the sword, but a player can also by interacting with the game, can switch to change the selected item to be the Rotten Flesh and that not only affects the game itself, but also needs to be reflected in the UI. And the player can also take damage, right? Let's say a mob comes and a creeper blows up next to the player, and then we need to take some health damage and here the health has dropped to eight. So we can see here that we have kind of like two categories of data, two groups of data. And we call those facets.
5. Using Remote Facets and Performance Optimization
So the next step, which is the interesting part for all of you is, of course, how we handle performance. Where do you optimize stuff for a solution so that it can work well in a gaming development world? And the first thing I guess to really make it clear is that React by itself is already plenty fast.
6. Optimizing React Reconciliation with Facet Updates
When building web applications with React, we often try to avoid unnecessary re-renders and improve performance. In this case, we propose skipping React reconciliation by directly updating the DOM from facet updates. We demonstrate this approach with an example of a progress bar. To achieve this, we subscribe to facets using the use remote facet hook and transform the facet value using the use facet map hook. However, when passing the transformed value to the div, React may not recognize the data type.
If you're building web applications, it's usually sufficient just to use React as it is. And solutions like Recoil or Jotite, they help kind of like narrow down and follow updates so that only specific parts of your React tree are updated. And when we run into performance issues in React, usually the recommendation is that we take a look at our components and we see if we have any kind of re-rendering that we're doing unnecessarily and see if we can avoid that at all costs.
So when we look at this concept, it's kind of like, I mean, what if instead of just avoid unnecessary renders, we could avoid all re-renders at all? Or could we have a situation where we basically don't trigger the React reconciliation? So here's kind of how it's happening as our previous example, right? The facet gets updated through an event from the C++ side that triggers the, that turns, that of course updates some state internally in React, which triggers the reconciliation and then Reacts through the virtual DOM, figures out, oh, I need to actually update this text node and that's how an update happens. So what we're proposing is that we could just skip the middle man and just go straight from the facet update to the DOM update. So if we go back to the example that we had before, what we're proposing is instead of like taking the facet, which is kind of like this nice data structure that holds a value over time, instead of unwrapping it and turn it into a state, a React state, can we just pick the facet and pass it directly to the React render and let the React render figure out how to unpack the value, unwrap the value from the, from the facet? And that's exactly what we did. So we want to pass the facet, and that directly updates a text node. So let's see how we've done that. So taking the example from before, I just expanded it a bit and turned it a bit more interesting. So here I have now a bit more complex progress bar. So I'm actually rendering some divs to make it like a nice style. You can see at the top there how visually it will look. And then we see the hard-coded result. We have like a hardcode variable there with the progress set to two. So that's a very simple React component, just rendering a progress bar. So if you want to take this example and make it so that it skips reconciliation altogether, how could we do that? So I think the first step of course is to subscribe to the facets in the first step. So we changed that to do the use remote facet, to take the value from the selector, and grab it to be used in a component. But progress here is not actually a number per se, it's actually just the facet still, like it's a data structure that holds the value inside of it. So I'm getting actually a typo error here because I can't make an arithmetic operation with a facet. It's not a number. So we need to then figure out a way now to be like, okay, I have this facet and I want to do some transformation to it and turn it into something else. If this was a recoil in a global state, I would probably just use a selector. Because this is inside a React component, what we need is some way of making a selector that is defined within a React component. And we do support that through a custom hook that we have that's called use facet map, and it's very similar to a selector conceptually. It takes an input facet, you can actually take multiple ones, but in this case, we're only passing one, which is the progress that we subscribed to in the line before. Then we have a map function that will take the progress, which is the number, coming from the facet, and transforming it into the string containing the pixel unit. And then the result of it is also another facet, which is holding now in the width variable. So the next step now, of course, is to take this width and pass it to the div. But if we do that, React is not going to be happy. It's going to be like, hey, I don't know what to do with this data type, right? You're trying to give me something that's not a string, it's not a number.
7. Rendering with FastDiv and Facets
We introduced a new div called FastDiv and created a series of components that accept facets as props. This allows us to create a component that subscribes to a remote facet and maps it to a width facet. We have created a renderer that supports facets as values for props and can even make custom components that natively support facets.
I don't know how to render this. So the next step in this is, we need to change React Renderer itself to support this data type. And that's what we did, by introducing a new div which we call FastDiv. And we actually created a whole series of components that match, kind of like one-to-one. So we have a FastB, a FastSpan. Instead of accepting just regular numbers, strings for the props, they also accept facets as a prop. And in the end, what we have is basically this, right? A component that is subscribing to a remote facet, turning to like a variable progress that holds a facet of progress. Then we're mapping that into a Width facet that has the width. And then I'm taking that and passing to the FastDiff that actually then updates that prop in the DOM. We have basically created a renderer that natively supports facets, or you could say a lightweight observable, as values for the props. We could also go one step further and make custom components that also natively supports facets as value.
8. Using Facets for Improved Performance
Let's see how using facets improves performance. By extracting the progress bar component and passing the progress as a facet, we avoid triggering unnecessary re-renders. The implementation behind the scenes uses a custom render and is available in the react-facet-dom-fiber package. Performance tests on Xbox One and Chrome show that the facet-based solution is significantly faster, taking only 32% of the time compared to the state-based React implementation.
Let's see how that will look like. If we go back again and take a look at this example of the player health, what if we want to also do the fullness? We want to render also the fullness progress bar. Could we do that? I guess the first thing is that we want to extract the components so we can reuse in both places. We want to take this part here that handles the progress bar and extract that to a progress bar component. But we want to pass the progress there as a facet. We don't want to pass as a number. Because if we unwrapped that in the player health to turn it into like a regular React state, we would go back to basically triggering conciliation, and we would defeat all the benefits we were trying to achieve.
So here we have them, basically the player health and the player fullness, two components using the progress bar, and both of them passing a progress, each with their own facet containing their own data. And the progress bar implementation looks like this. Instead of taking a progress that's just a number, it's taking instead a facet of a number. And we have the type specifically available for there. But otherwise, the implementation here in the body is exactly the same as we had before. So this is kind of like what's happening then in the end. We have at the top our player stats remote facet. Once that gets notified of an update from the C++ side, that data then gets passed down to the player health remote selector that then picks just the health from the data. That then is passed to their use remote facet, which kind of like triggers the subscription. Then it goes through the use facet map that transforms that. And then the way that it finally gets down to the facet div, and then the DOM is actually updated. And the cool thing about all of this is that all of this is happening behind the scenes. We're triggering no rewrapped renderings at all. And this whole implementation is built as a custom render, as I said, and it's available in this package, react-facet-dom-fiber.
But this is all fantastic and it's pretty cool, but like how well, how much does it really improve the performance? So comparing, and I put here two different platforms just to give it some perspective. So I'm running these scenarios on both an Xbox One and also on Chrome. So this Xbox One is running the Minecraft game, and we have also Chrome on my MacBook. This example that I'm showing here is basically taking the progress bar implementation that we did before and just changing its value every frame and see how much time I'm taking between each implementation. On the left, we have implementation that's state-based from React and it's taken like 100% of the time. And on the Xbox One, we can see that a Facet based solution takes only 32% of the time to do the same work. When we look at Chrome, the difference is a little smaller, but it's also a lot faster when we look at the Facet's implementation. Another scenario that's very interesting to look at is that imagine if you have like a big list and you're running that through a bunch of components and you're memoizing it. So each component is memoized.
9. Performance Comparison with Facets
When we compare the solution based on Facets on the Xbox to the standard React implementation, the difference is massive, with the Xbox being significantly faster. Platforms like Xbox, PlayStation, and Switch, which do not have JIT enabled, may have different performance characteristics. The more complex the React trees become, the wider the performance gap. More information and examples are available on our GitHub repository.
When we look at the solution based on Facets on the Xbox, the difference becomes massive, taking just 12% of the time. And on Chrome, it's also about 30% of the time. This example I brought up because it's quite interesting to look at if we have like a big list, but all the items are being updated at once, here we can see that the Xbox is actually a lot faster when we compare it to the standard React implementation. When I looked at the Chrome on my MacBook, the difference is not that big. And the main thing for this that is worth bringing up is that when we're running V8 in our Chrome, in our Macs, we have just-in-time compilation. But unfortunately in some platforms like the Xbox, PlayStation, and Switch, we cannot have JIT enabled. So optimizations and libraries that we're used to will behave and will have different performance characteristics when we run them. So, and also another important thing is like the more complex the React trees become, then the wider the gap will be and the faster, actually the faster the implementation will be in relation to the standard React implementation. We have more data, of course, available at our GitHub repository. All the examples that I mentioned here, you can go there and see their actual implementations to understand better exactly what they're doing. I don't have much time to go into the details here, but we do have all of that available on the GitHub repository.
10. Introduction to Facets and Oryui
But not only that, I mean, if you look at facets, we have global stores, we actually have a bunch of more hooks that turn into like a much more React friendly API. So for all of the React hooks that are on the right here, like use state, useReduce, or useEffect, we have counterpart implementations on the facet world, but instead of like, you know, the difference being that it actually can accept facets as input, and they might return a facet as an output. And this is all available in the react-facet-core package.
Just to go really quickly, here's how like that looks like, where I have this component here, where I'm using the useFacetState, that is very similar to useState, but instead of like value there being just a string, it would be, for example, a facet of a string. Then I have also, useFacetCallback here, that is very similar to useCallback, but instead it allows you to pass value there, which is a facet instead. There's more hooks and documentation available also at our GitHub page.
But it seems a bit daunting, but it's actually quite easy to get started if you want to just get your feet wet with facets. We have a package called DOM components. What it allows you to do is it implements this fast components that we did, but instead of being like host components in a custom React renderer, they're actually just regular React components. So you can see here them being used on a regular React DOM to render this component. And all of this is being open sourced. We're open sourcing under the Oryui brand. Oryui is kind of like an initiative that we're doing at Mojang studios to try and standardize UI development in the gaming industry. So the stuff we're doing is trying to push that. So Oryui will have React facets to start, but we'll push other packages that we're developing as well. So this is the fantastic logo. And this is our GitHub repository. So if you want to look faster at the codebase, if you want to learn more, go to GitHub and take a look at Mojang slash Oryui. And of course I have to mention that we are hiring at Mojang. It's quite an exciting project. I mean, it's not everywhere you can go and use your web skills to contribute to one of the biggest games in the world. It's quite exciting to be able to work here. And if you want to do it, we have positions open for tech leads, web developers. Feel free to reach out to me or go straight to jobsmojang.com. So yeah, thank you so much. I had to run a bit through the slides, but yeah, if you have any questions, I think we're going to have a Q and A soon. So thank you so much and feel free to reach out to me as well if you have any questions.