React Summit

React Summit 2023

A lot of recent talks at other conferences have talked about how to start using React and THREE.js to create 3D experiences on the web. In my career I've built several complex applications using those technologies, including a big Figma-like 3D editor, and I've learnt a lot of patterns and tricks on how to make it actually fast and scalable. Unfortunately, though, nobody is also talking about those tricks an what it really takes to build an rendering engine similar to the one that Figma uses. My goal with this talk is to give people an overview of the challenges that may arise when you try scaling your application, and what are some lesser known tricks to overcome them: instancing, memory usage reduction techniques, SDF-based text rendering, GPGPU and tree data structures. Everything of course done with Javascript, React and THREE.js.

Scaling React-Three-Fiber Applications beyond the Hello World

Hi, my name is Giulio, I'm a software engineer and I've worked with React and JavaScript for quite a while now. Recently in my career I've been developing some stuff with WebGL and today I wanted to talk about my experience doing it. To talk about it, I think it makes sense first to discuss about what is it and its history. WebGL is a browser technology that allows using the GPU to build 3D graphics based applications with JavaScript. It has been around for quite a while now, almost 12 years, and its usage changed a lot during that time. It all started in the beginning with some power users building cool 3D graphics demos, which were showcasing the technology but not yet delivering value to users. After a while though, we started building real applications that use WebGL to enhance user experience for some specific use cases, which answered a specific business need and enhanced the user experience. Those experiences though were still delivered as special applications, an isolated section of a website with a clear scene from the rest of it, not something that you would use every day. This changed though with time, and today WebGL is used in all sorts of applications that we use every day, and professionally even. Think about Google Maps or Figma, you don't even know that WebGL is there, but it's using the GPU to accelerate the rendering, both for 3D and 2D applications, and it's what makes the application possible in the first place. So since WebGL is today so widespread, can we say that 3D graphics on the web are nowadays a commodity, something that is easy to build? Well to answer this question, I think we have to look at what is the current tooling ecosystem that is available for it. One example of this is Three.js, a library that accelerates the development of WebGL applications by providing utilities and by wrapping its API, providing an API that is more similar to the DOM. And nowadays we can even use React together with Three.js to build WebGL applications. And this really boosted its adoption recently. In fact, thanks to a library called React 3 Fiber, we are now able to use React declaratively to build applications and scenes using Three.js. The power of React is that it allows to easily reuse components and bind your data without imperative mutations. What's really powerful about React 3 Fiber is the ecosystem that is built around it. There are many open source projects that facilitate some common 3D rendering tasks, such as post-processing, layouting, physics, and accessibility. There are even some component libraries, such as Try, which provide advanced rendering techniques, such as lighting, skyboxes, shadows, and keyboard and mouse controls as reusable React components. This is really powerful since it allows everybody to build really cool applications with very few lines of code. For example, this demo really showcases what you can do with it. Physics, mouse controls, advanced materials, and reflections. The code that is needed to build something like this seems a lot, but actually is surprisingly low. This demo is in fact around 60 lines of code. This is possible because it's able to reuse a lot of code from the component library. We have seen a lot of cool demos like this one recently around on Twitter, and a lot of talks have been made in conferences on how to get started with this new cool technology. But does this mean that now I can easily build my Figma or my Google Maps just by using those libraries and writing a few lines of code? Well, unfortunately, not really. For example, if we look at something that implements a similar thing as what Google Maps is doing, such as procedural.js, we can see how it quickly turns into a giant library with over 10,000 lines of code. This is because if you want to develop something that is more data intensive and uses 3D graphics, you have a lot of challenges to overcome. For example, you have to efficiently load and unload lots of data, triangles and textures, which represent the terrain. You have to stream your data in tiles from a server, and you need to handle level of details between zoom levels so that if you zoom out, you get a lower resolution of your scene that is renderable. And you also want fast collision checking for your mouse-based interactions. Unfortunately, the answer to those questions is not something that you're really going to find on a web development book. Something that is way more common between game developers, not web developers. Game developers use different languages and different tools, such as C++. So there is not much around on the web that you can find. But all this insider knowledge is what made open-world video game possible in the first place. For example, algorithms that dynamically simplify a complex scene when you look at it from a distance. Using these kind of tricks is crucial as you're dealing with massive amount of data. If you look at complex applications on the web, such as Figma, they're actually doing the exact same thing after the hood. If you artificially slow down your computer and zoom into a complex Figma document, you'll see how it is actually rendering stuff asynchronously and showing you the lower resolution first while it's loading. Let's look at it again. You zoom in and you see how you can get the pixelated version first and then it loads when ready without blocking the main thread or without blocking user interactions. Using a lot of techniques like these is what makes Figma possible and not painfully slow. Unfortunately, developing these kind of tricks, it's not easy. It involves writing a lot of code that is tailor-made for your data, for your specific use case. So, it's difficult to reuse and to encode into a library. We can say that for this reason, building application like those, it's actually a data problem. You need efficient data structure to represent your data. You need efficient algorithms to get information you need from the data structures. You also need indexed representation of data, such as binary trees, so that you can accelerate your algorithms. You also need streaming to be able to dynamically load data from the web. Most importantly, you need to pre-process as much data as possible so that when it's loaded, it's ready to be rendered on the client machine without the need for further processing. We have seen the challenges that building complex 3D graphics applications on the browser takes. I want to show you what's my experience with it by discussing a real-world scenario of a complex 3D graphics application. I'm working on the rendering engine of a new product called Flux. It's an online browser-based collaborative CAD editor, similar to Figma, but instead focused on electronic engineering and printed circuit boards. In Flux, we want to be able to handle very complex engineering documents. It needs to render very complicated scenes with lots of components. Each component consists of 2D shapes and 3D objects. We want users to be able to design complex electronic circuits and even display them in 3D while keeping always a smooth user experience. Our application is built with React, 3JS, and React 3 Fiber. Flux, when compared to Figma and Google Maps, we can say that it comes with its own set of challenges. In fact, we want to be able to support very big documents with up to 10,000 electronic components. We want each electronic part to have a lot of different shapes inside of it, like circles, rectangles, text, and 3D models. The entire document also can be viewed at once. We want the user to be able to zoom out and see all of it without performance drops. We also want to provide, as seen before, a 3D view of your document, not only 2D, so tricks like caching tiles cannot work. And we also want real-time speed picking so that mouse interactions are fast and snappy. And all this needs to be able to run at 60 frames per second, even on mobile and low-end hardware. So, when you want to overcome challenges like those, it's important to keep a performance methodology. I wanted to share with you what I found are the most important points to follow when scaling up your React 3 Fiber application. And those are the steps that we are following internally to ensure that the application stays fast. We want to try to limit React renders as much as possible. We continuously profile the rendering and interaction times. We try to profile and reduce the memory usage as much as possible. We use efficient algorithms and data structures to represent our data and for some tasks. And last, we use a lot of GPU optimization to exploit the hardware that we have at our disposal as much as possible. The first things are the React renders. As described in the React 3 Fiber documentation, you want to use transit values, refs, and imperative mutations in your frequently running code, which can be sometimes hacky, but it's the only way to obtain smooth animation without going through React renders. We went even further than this guide by optimizing our state management solution to use partition store to reduce the cost of subscriptions. Even if you apply all the React-level optimizations, though, you still need to optimize your render loop. We use the concept of frame budget. This means that if you want to reach 60 frames per second, you have 15 milliseconds to render your entire scene. So it's crucial to go there with the Chrome profiler and see what is really taking up a lot of time in your frame loop and optimize those things first. Sometimes the result of those profiles can be even unexpected, finding out that you introduced a function call with too much complexity in your critical path. Another thing that we do is memory profiling, as keeping a low memory footprint is crucial when you're handling lots of data. By using the Chrome profiler, you can take a snapshot of the memory and see what is taking up too much space. By using this method, for example, we're able to notice how we're using hundreds of megabytes just to store string-based UIDs. This has suggested us to use numbers instead, which we're able to save up a lot of space and make the load of certain documents possible. Another thing you need are efficient algorithms and data structures. This really helps, for example, for optimizing ray casting, which is the process of finding out which objects are present in your scene at the given position. This is crucial for mouse interactions, as you need to know what's under the cursor pointer. The naive approach to this problem, as it's also implemented in ReactryFiber, is by doing a linear search, which unfortunately can be very expensive when you have thousands of objects. Since this is something that needs to run in real time, having faster data structures, such as R3 and BinaryTree, can really speed up your application a lot. Another thing you have to be mindful about are GPU optimizations. GPUs are really powerful parallel processors, but you need to be mindful of the CPU-GPU bottleneck. In fact, the communication between the CPU and the GPU is not free. Every time you ask the GPU to do something, also called a draw call, you get some overhead. For this reason, you want to keep your draw calls as low as possible, especially on low-end and mobile devices. This means that if you want to draw, for example, 10,000 instances of the same object over and over again, doing that the naive way, by using a for loop, can be very inefficient. You can see that if we want to render 10,000 instances of the same object, we will end up with pretty poor performance, around 33 FPS. Luckily, there is a way to optimize that, and it's called instance rendering. With this technique, we can draw several instances of the same object with a single draw call to the GPU, reducing the bottleneck by a lot. This can yield great performance improvements in some cases, like here, for example, where we were previously limited at 33 FPS on average. The cool thing about instance rendering is that it's very customizable. Even if you're limited to have the same geometry material for each instance, you can program how the GPU is processing each vertex and pixels, and even pass custom parameters for each one. For example, you can give each instance a different position or size, and program on the GPU how to interpret those parameters. Instance rendering was crucial in Flux to optimize our text rendering, even though it required some special setup. You see, text rendering is unfortunately not provided by WebGL, and it's something that you have to handle with your own custom solution. It's also quite complicated, since it involves shapes and geometry that can have a lot of triangle and vertices, which can become expensive to draw. For Flux, we needed a solution that was able to scale up to thousands of characters a screen, while keeping each character as a 3D extruded shape. For this reason, a simple sprite-based 2D solution wouldn't have worked. We needed real 3D geometry. To optimize our text rendering, we needed to use instance rendering. But as we said previously, instance rendering only works when you want to repeat the same geometry multiple times. And this is obviously not the case if you have different characters. We also said, though, that we can customize how each instance is displayed by providing custom parameters, which can encode stuff like position and size. So we asked ourselves, what if we represented the desired glyph as an instancing parameter? If that was possible, we could encode our string to display into an instancing parameter and have the GPU figure out how to draw each character by itself. At this point, we also needed to pass to the GPU the data about the geometry of each glyph as sequences of 3D points. We did that by using textures. Normally, textures are used for attaching images to the surfaces. But since they are, after all, a big array of numbers, we can use them also to encode 3D positions so that the GPU can use them to represent characters like this one. We can use the RGB values to encode our X, Y, Z positions so that we can encode geometry into a texture. The end result is something like this in the example here. We're able to display 20,000 3D characters at the same time with a single draw call and animate each one independently while keeping a steady frame rate. You may ask, well, but how would have this behaved without those optimizations? We tried it, and even with a fraction of the characters on the screen, it was painfully slow, leaving almost no processing power left for other, more interesting stuff. At this point, I can already hear someone saying maybe, have a lot of text to render too. And for all you guys, we just published an NPM, or 3D and 2D text rendering engine, which provide high performance and efficiency with a React-friendly API. Just wrap your scene with a provider component and use the instanced text component wherever you like. Thank you for following my talk. I hope it gave you an overview of what it means to build a complex, data-intensive WebGL application nowadays. And I hope my experience I described can be useful for your developments in the future.
