Scaling React-Three-Fiber Applications beyond the Hello World

Rate this content
Bookmark
Slides

A lot of recent talks at other conferences have talked about how to start using React and THREE.js to create 3D experiences on the web. In my career I've built several complex applications using those technologies, including a big Figma-like 3D editor, and I've learnt a lot of patterns and tricks on how to make it actually fast and scalable. Unfortunately, though, nobody is also talking about those tricks an what it really takes to build an rendering engine similar to the one that Figma uses.
My goal with this talk is to give people an overview of the challenges that may arise when you try scaling your application, and what are some lesser known tricks to overcome them: instancing, memory usage reduction techniques, SDF-based text rendering, GPGPU and tree data structures. Everything of course done with Javascript, React and THREE.js.

20 min
06 Jun, 2023

AI Generated Video Summary

WebGL has evolved from showcasing technology to being used in everyday applications like Google Maps and Figma. React and 3.js can be used together to build WebGL applications, allowing for reusable components and declarative development. Building complex 3D graphics applications requires efficient data structures, algorithms, and rendering techniques. The Flux CAD editor uses React, 3.js, and React ReFiber to handle complex engineering documents and optimize GPU utilization. Optimizing the render loop and GPU performance is crucial for improving WebGL application performance. Instance rendering can be used to optimize text rendering in WebGL applications, achieving efficient rendering of thousands of 3D characters.

1. Introduction to WebGL and its History

Short description:

Hi! My name is Giulio. I'm a software engineer, and I've worked with React and JavaScript for quite a while now. Recently in my career I've been developing some stuff with WebGL, and today I wanted to talk about my experience doing it. WebGL is a browser technology that allows using the GPU to build 3D graphics-based applications with JavaScript. It has been around for quite a while now, almost 12 years, and its usage changed a lot during that time. It all started in the beginning with some power users building cool 3D graphics demos, which were showcasing the technology but not yet delivering value to users. After a while, though, we started building real applications that used WebGL to enhance user experience for some specific use cases, which answered a specific business need and enhanced the user experience. Those experiences, though, were still delivered as special applications, an isolated section of a website with a clear scene from the rest of it, not something that you would use every day. This changed, though, with time, and today WebGL is used in all sorts of applications that we use every day and professionally, even. Think about Google Maps or Figma. You don't even know that WebGL is there, but it's using the GPU to accelerate the rendering, both for 3D and 2D applications, and it's what makes the application possible in the first place. So since WebGL is today so widespread, can we say that 3D graphics on the web are nowadays a commodity, something that is easy to build? Well, to answer this question, I think we have to look at what is the current tooling ecosystem that is available for it.

Hi! My name is Giulio. I'm a software engineer, and I've worked with React and JavaScript for quite a while now. Recently in my career I've been developing some stuff with WebGL, and today I wanted to talk about my experience doing it. To talk about it, I think it makes sense first to discuss about what is it and its history.

WebGL is a browser technology that allows using the GPU to build 3D graphics-based applications with JavaScript. It has been around for quite a while now, almost 12 years, and its usage changed a lot during that time. It all started in the beginning with some power users building cool 3D graphics demos, which were showcasing the technology but not yet delivering value to users. After a while, though, we started building real applications that used WebGL to enhance user experience for some specific use cases, which answered a specific business need and enhanced the user experience. Those experiences, though, were still delivered as special applications, an isolated section of a website with a clear scene from the rest of it, not something that you would use every day.

This changed, though, with time, and today WebGL is used in all sorts of applications that we use every day and professionally, even. Think about Google Maps or Figma. You don't even know that WebGL is there, but it's using the GPU to accelerate the rendering, both for 3D and 2D applications, and it's what makes the application possible in the first place. So since WebGL is today so widespread, can we say that 3D graphics on the web are nowadays a commodity, something that is easy to build? Well, to answer this question, I think we have to look at what is the current tooling ecosystem that is available for it.

2. Using React and 3.js for WebGL Applications

Short description:

One example of this is 3.js, a library that accelerates the development of WebGL applications by providing utilities and by wrapping its API, providing an API that is more similar to the DOM. Nowadays, we can even use React together with 3.js to build WebGL applications. Thanks to a library called React ReFiber, we are now able to use React declaratively to build applications and scenes using 3.js. The power of React is that it allows to easily reuse components and bind your data without imperative mutations. React ReFiber has an ecosystem built around it, with many open source projects and component libraries that provide advanced rendering techniques. This allows developers to build cool applications with minimal code.

One example of this is 3.js, a library that accelerates the development of WebGL applications by providing utilities and by wrapping its API, providing an API that is more similar to the DOM. Nowadays, we can even use React together with 3.js to build WebGL applications. And this really boosted its adoption recently.

In fact, thanks to a library called React ReFiber, we are now able to use React declaratively to build applications and scenes using 3.js. The power of React is that it allows to easily reuse components and bind your data without imperative mutations. What's really powerful about React ReFiber is the ecosystem that is built around it. There are many open source projects that facilitate some common 3D rendering tasks such as post-processing, layouting, physics, and accessibility. There are even some component libraries, such as Tri, which provide advanced rendered techniques such as lighting, skyboxes, shadows, and keyboard and mouse controls as reusable React components.

This is really powerful since it allows everybody to build really cool applications with very few lines of code. For example, this demo really showcases what you can do with it. Physics, mouse controls, advanced materials, and reflections. The code that is needed to build something like this seems a lot, but actually it's surprisingly low. This demo is in fact around 60 lines of code. This is possible because it's able to reuse a lot of code from the component library.

3. Building Complex 3D Graphics Applications

Short description:

Building complex 3D graphics applications on browsers is a data problem. It requires efficient data structures, algorithms, indexed representation of data, streaming, and pre-processing of data for efficient rendering.

We have seen a lot of cool demos like this one recently around on Twitter, and a lot of talks have been made in conferences on how to get started with this new color technology. But does this mean that now I can easily build my Figma or my Google Maps just by using those libraries and writing a few lines of code? Well, unfortunately, not really. For example, if we look at something that implements a similar thing as what Google Maps is doing, such as procedural.js, we can see how it quickly turns into a giant library with over 10,000 lines of code. This is because if you want to develop something that is more data intensive and uses 3D graphics, you have a lot of challenges to overcome. For example, you have to efficiently load and download lots of data, triangles and textures which represent the terrain. You have to stream your data in tiles from a server, and you need to handle level of details between zoom levels so that if you zoom out, you get a lower resolution of your scene that is renderable. And you also want fast collision checking for your mouse-based interactions. Unfortunately, the answer to those questions is not something that you're really going to find on a web development book. It's something that is way more common between game developers, not web developers. Game developers use different languages and different tools, such as C++, so there is not much around on the web that you can find. But all this insider knowledge is what made open-world video games possible in the first place. For example, algorithms that dynamically simplify a complex scene when you look at it from a distance. Using these kinds of tricks is crucial as you're dealing with massive amounts of data. If you look at complex applications on the web, such as Figma, they're actually doing the exact same thing after the hood. If you artificially slow down your computer and zoom into a complex Figma document, you'll see how it is actually rendering stuff asynchronously and showing you the lower resolution first while it's loading. Let's look at it again. You zoom in and you see how you can get a pixelated version first and then it loads when ready without blocking the main thread or without blocking user interactions. Using a lot of techniques like these is what makes Figma possible and not painfully slow. Unfortunately developing these kinds of tricks is not easy. It involves writing a lot of code that is tailor-made for your data, for your specific use case, so it's difficult to reuse and to encode into a library. We can say that for this reason building applications like those is actually a data problem. You need efficient data structures to represent your data. You need efficient algorithms to get information you need from data structures. You also need indexed representation of data, such as binary trees, so that you can accelerate your algorithms. You also need streaming to be able to dynamically load data from the web. Most importantly, you need to pre-process as much data as possible so that when it's loaded it's ready to be rendered on the client machine without the need for further processing.

4. Building Complex 3D Graphics Applications

Short description:

I'm working on the rendering engine of a new product called Flux. It's an online, browser-based, collaborative CAD editor. In Flux, we want to handle very complex engineering documents with lots of components. Our application is built with React, 3JS, and React 3 Fiber. We need to support big documents with up to 10,000 electronic components, provide a 3D view, and ensure real-time speed picking. To overcome these challenges, we follow a performance methodology, including limiting React 3 renders, profiling rendering and interaction times, reducing memory usage, using efficient algorithms and data structures, and optimizing GPU utilization.

We have seen the challenges that building complex 3D graphics applications on browsers takes. I want to show you what's my experience with it by discussing a real-world scenario of a complex 3D graphics application.

I'm working on the rendering engine of a new product called Flux. It's an online, browser-based, collaborative CAD editor. Similar to Figma, but instead focused on electronic engineering and printed circuit boards.

In Flux, we want to be able to handle very complex engineering documents. It needs to render very complicated scenes with lots of components. Each component consists of 2D shapes and 3D objects. We want users to be able to design complex electronic circuits and even display them in 3D. While keeping always a smooth user experience.

Our application is built with React, 3JS and React 3 Fiber. Flux when compared to Figma and Google Maps, we can say that it comes with its own set of challenges. In fact, we want to be able to support very big documents with up to 10,000 electronic components. We want each electronic part to have a lot of different shapes inside of it, like circles, rectangles, text and 3D models. The entire document also can't be viewed at once. We want the user to be able to zoom out and see all of it without performance drops. We also want to provide, as seen before, a 3D view of your document, not only 2D, so tricks like caching tiles cannot work. And we also want real-time speed picking so that mouse interactions are fast and snappy. All this needs to be able to run at 60 frames per second, even on mobile and low-end hardware.

So when you want to overcome challenges like those, it's important to keep a performance methodology. I wanted to share with you what I found are the most important points to follow when scaling up your React 3 Fiber application. And those are the steps that we are following internally to ensure that the application stays fast. We want to try to limit React 3 renders as much as possible. We continuously profile the rendering and interaction times. We try to profile and reduce the memory usage as much as possible. We use efficient algorithms and data structures to represent our data and for some tasks. And last, we use a lot of GPU optimization to exploit the hardware that we have at our disposal as much as possible. The first things are the Reactor Renders. As described in the React 3 Fiber documentation, you want to use transit values, refs, and imperative mutations in your frequently running code, which can be sometimes hacky, but it's the only way to obtain smooth animation without going through Reactor Renders. We went even further than this guide by optimizing our state management solution to use partition store to reduce the cost of subscriptions.

5. Optimizing Render Loop and GPU Performance

Short description:

Even if you apply all the React level optimizations, you still need to optimize your render loop. Memory profiling is crucial for handling lots of data. Efficient algorithms and data structures help optimize raycasting. GPU optimizations, like instanced rendering, can greatly improve performance.

Even if you apply all the React level optimizations though, you still need to optimize your render loop. We use the concept of frame budget. This means that if you want to reach 60 frames per second you have 15 milliseconds to render your entire scene. So it's crucial to go there with the Chrome profiler and see what is really taking up a lot of time in your frame loop and optimize those things first. Sometimes the result of those profiles can be even unexpected. Finding out that you introduced a function call with too much complexity in your critical path.

Another thing that we do is memory profiling as keeping a low memory footprint is crucial when you're handling lots of data. By using the Chrome profiler you can take a snapshot of the memory and see what is taking up too much space. By using this method for example we're able to notice how we're using hundreds of megabytes just to store string-based UIDs. This suggested us to use numbers instead which were able to save up a lot of space and make the load of certain documents possible.

Another thing you need are efficient algorithms and data structures. This really helps for example for optimizing raycasting which is the process of finding out which objects are present in your scene at a given position. This is crucial for mouse interactions as you need to know what's under the cursor pointer. The naive approach to this problem as it's also implemented in ReactryFiber is by doing a linear search which unfortunately can be very expensive when you have thousands of objects. Since this is something that needs to run in real time having faster data structures such as R3 and BinaryTree can really speed up your application a lot.

Another thing you have to be mindful about are GPU optimizations. GPUs are really powerful parallel processors but you need to be mindful of the CPU GPU bottleneck. In fact the communication between the CPU and the GPU is not free. Every time you ask the GPU to do something also called a draw call you get some hoverhead. For this reason you want to keep your draw calls as low as possible especially on low-end mobile devices. This means that if you want to draw for example 10 000 instances of the same object over and over again doing that the naive way, by using a for loop can be very inefficient. You can see that if we want to render 10 000 instances of the same object we will end up with pretty poor performance around 33 FPS. Luckily there is a way to optimize that and it's called instanced rendering. With this technique we can draw several instances of the same object with a single draw call to the GPU reducing the bottleneck by a lot. This can yield great performance improvements in some cases like here for example where we were previously limited at 33 FPS on average. The cool thing about instanced rendering is that it's very customizable. Even if you're limited to have the same geometry material for each instance you can program how the GPU is processing each vertex and pixels and even pass custom parameters for each one. For example you can give each instance a different position or size and program on the GPU how to interpret those parameters. Instanced rendering was crucial in Flux to optimize our text rendering even though it required some special setup. You see text rendering is unfortunately not provided by WebGL and it's something that you have to handle with your own custom solution.

6. Optimizing Text Rendering in WebGL Applications

Short description:

To optimize text rendering in WebGL applications, instance rendering can be used. By representing each character as an instancing parameter and encoding the desired geometry into a texture, we can achieve efficient rendering of thousands of 3D characters. This approach allows for smooth animation and frees up processing power for other tasks. We have developed a 3D and 2D text rendering engine with a React-friendly API, which provides high performance and efficiency. Simply wrap your scene with a provider component and use the InstancedText component to render text.

It's also quite complicated since it involves shapes and geometry that can have a lot of triangle and vertices which can become expensive to draw. For Flux we needed a solution that was able to scale up to thousands of characters a screen while keeping each character as a 3D extruded shape. For this reason a simple sprite based 2D solution wouldn't have worked. We needed real 3D geometry.

To optimize our text rendering we needed to use instance rendering. But as we said previously instance rendering only works when you want to repeat the same geometry multiple times and this is obviously not the case if you have different characters. We also said though that we can customize how each instance is displayed by providing custom parameters which can encode stuff like position and size. So we asked ourselves what if we represented the desired cliff as an instancing parameter? If that was possible we could encode our string to display into an instancing parameter and have the GPU figure out how to draw each character by itself.

At this point we also needed to pass to the GPU the data about the geometry of each cliff as sequences of 3D points. We did that by using textures. Normally textures are used for attaching images to the surfaces but since they are after all a big array of numbers we can use them also to encode 3D positions so that the GPU can use them to represent characters like those one. We can use the RGB values to encode our XYZ positions so that we can encode geometry into a texture.

The end result is something like this in the example here. We are able to display 20 000 3D characters at the same time with a single draw call and animate each one independently while keeping a steady frame rate. You may ask, well but how would have this behaved without those optimizations? We tried it and even with a fraction of the characters on the screen it was painfully slow, leaving almost no processing power left for other more interesting stuff. At this point I can already hear someone saying maybe have a lot of text to render too And for all you guys, we just published on NPM our 3D and 2D text rendering engine which provides high performance and efficiency with a React-friendly API. Just wrap your scene with a provider component and use the InstancedText component wherever you like. Thank you for following my talk.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Advanced Conference 2021React Advanced Conference 2021
39 min
Don't Solve Problems, Eliminate Them
Humans are natural problem solvers and we're good enough at it that we've survived over the centuries and become the dominant species of the planet. Because we're so good at it, we sometimes become problem seekers too–looking for problems we can solve. Those who most successfully accomplish their goals are the problem eliminators. Let's talk about the distinction between solving and eliminating problems with examples from inside and outside the coding world.
React Advanced Conference 2022React Advanced Conference 2022
30 min
Using useEffect Effectively
Can useEffect affect your codebase negatively? From fetching data to fighting with imperative APIs, side effects are one of the biggest sources of frustration in web app development. And let’s be honest, putting everything in useEffect hooks doesn’t help much. In this talk, we'll demystify the useEffect hook and get a better understanding of when (and when not) to use it, as well as discover how declarative effects can make effect management more maintainable in even the most complex React apps.
React Advanced Conference 2021React Advanced Conference 2021
47 min
Design Systems: Walking the Line Between Flexibility and Consistency
Design systems aim to bring consistency to a brand's design and make the UI development productive. Component libraries with well-thought API can make this a breeze. But, sometimes an API choice can accidentally overstep and slow the team down! There's a balance there... somewhere. Let's explore some of the problems and possible creative solutions.
React Summit 2023React Summit 2023
23 min
React Concurrency, Explained
React 18! Concurrent features! You might’ve already tried the new APIs like useTransition, or you might’ve just heard of them. But do you know how React 18 achieves the performance wins it brings with itself? In this talk, let’s peek under the hood of React 18’s performance features: - How React 18 lowers the time your page stays frozen (aka TBT) - What exactly happens in the main thread when you run useTransition() - What’s the catch with the improvements (there’s no free cake!), and why Vue.js and Preact straight refused to ship anything similar
React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.

Workshops on related topic

React Summit 2023React Summit 2023
170 min
React Performance Debugging Masterclass
Featured WorkshopFree
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
React Summit Remote Edition 2021React Summit Remote Edition 2021
177 min
React Hooks Tips Only the Pros Know
Featured Workshop
The addition of the hooks API to React was quite a major change. Before hooks most components had to be class based. Now, with hooks, these are often much simpler functional components. Hooks can be really simple to use. Almost deceptively simple. Because there are still plenty of ways you can mess up with hooks. And it often turns out there are many ways where you can improve your components a better understanding of how each React hook can be used.You will learn all about the pros and cons of the various hooks. You will learn when to use useState() versus useReducer(). We will look at using useContext() efficiently. You will see when to use useLayoutEffect() and when useEffect() is better.
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
React Summit 2023React Summit 2023
145 min
React at Scale with Nx
Featured WorkshopFree
We're going to be using Nx and some its plugins to accelerate the development of this app.
Some of the things you'll learn:- Generating a pristine Nx workspace- Generating frontend React apps and backend APIs inside your workspace, with pre-configured proxies- Creating shared libs for re-using code- Generating new routed components with all the routes pre-configured by Nx and ready to go- How to organize code in a monorepo- Easily move libs around your folder structure- Creating Storybook stories and e2e Cypress tests for your components
Table of contents: - Lab 1 - Generate an empty workspace- Lab 2 - Generate a React app- Lab 3 - Executors- Lab 3.1 - Migrations- Lab 4 - Generate a component lib- Lab 5 - Generate a utility lib- Lab 6 - Generate a route lib- Lab 7 - Add an Express API- Lab 8 - Displaying a full game in the routed game-detail component- Lab 9 - Generate a type lib that the API and frontend can share- Lab 10 - Generate Storybook stories for the shared ui component- Lab 11 - E2E test the shared component
JS GameDev Summit 2022JS GameDev Summit 2022
165 min
How to make amazing generative art with simple JavaScript code
WorkshopFree
Instead of manually drawing each image like traditional art, generative artists write programs that are capable of producing a variety of results. In this workshop you will learn how to create incredible generative art using only a web browser and text editor. Starting with basic concepts and building towards advanced theory, we will cover everything you need to know.