In this talk we will see how we can bridge React Native app with commonly used game engine Unity3d for two way interaction between the game and React Native app
Run Games Within Your React Native Apps
AI Generated Video Summary
Today's Talk discusses integrating Unity into React Native apps for game development. Unity provides a wide range of capabilities and can be seamlessly integrated with React Native. The integration involves using the Unity View component and the Unity Message callback to enable communication between the two platforms. Native plugins are created to facilitate communication between Unity and React Native. The performance of Unity in React Native apps is comparable to regular React Native apps, and the decision to use Unity or React Native depends on the app's use case and need for advanced UI and gaming capabilities.
1. Introduction to Game Development in React Native
Today we'll talk about how you can run games within your React native apps. Game development is hard and involves rendering, physics, AI, networking, sound engine, concept art, creative assets, level design, gameplay mechanics, animations, UI, and post processing. Game engines handle most of these tasks.
Yeah, thanks for having me here. Really excited finally. Two years of not being at in-person conferences to finally be here, one of my favorite conferences, so like really excited to see the audience.
And today we're going to talk about some cool stuff. We'll talk about how you can run games within your React native apps. So, yeah. Clicker is working. Cool.
So, a bit about myself. So I'm Vladimir Novik, I'm software architect and consultant, and I do a bunch of stuff in all these fields, and beyond that. So if you're interested in like something within these fields, you can reach out on Twitter or my website.
So let's start with like the first statement that game development is pretty hard. So you need to take care of a lot of things there. Rendering, 3D, materials, lighting, only this kind of stuff. Physics, how objects interact. How they collide with each other. Is there a gravity, acceleration, all this kind of stuff. AI, how enemies attack your character. How the pathfinding algorithms, all of that. Networking, sound engine because it's not enough to just have like regular music in the game. It's all sounds of interactions, of like effects and even sounds of the footsteps when your character is walking.
And when you think of a game, you start with a concept art. So, you need to come up with the idea how to do that and you basically need to get lots and lots of creative assets, which is hours and hours of work of 3D artists. Or you need to purchase them. You get into like level design of things. How you structure each level, how they can be like inherently more complex than one another without actually impacting gameplay. Speaking of which, there is gameplay mechanics, which is different from game to game that you need to take account for. Animations, character animations, UI animations, all of that. And obviously post processing, because effects matter if you have like a bloom effect of like things blooming or HDR and stuff like that. And most of these things are taken care of by game engines.
2. Using Unity as a Library in React Native
Today we'll talk about using Unity as a library in React Native. Unity provides a wide range of capabilities, including 2D and 3D world building, characters, mobile, AR, VR, and more. We'll focus on the major feature of using Unity as a library, which allows for seamless integration with React Native. We'll also have a demo to showcase the possibilities and performance of Unity in React Native.
So, we need to have one. There are multiple ones and the top ones in the market, one of them was actually mentioned in last talk. It's Unity.
Now, Unity gives us lots and lots of capabilities, that is, from their website, has like 2D, 3D world building, characters, mobile, AR, VR, lots of stuff. And you can actually use all these capabilities and wrap them into React Native apps nowadays, which is pretty cool, I think.
So, features for mobile that Unity has, there is a lot of them, I will just briefly go through some of them. So, you have like a rendering pipeline with lightness systems and how you present realistic textures on objects and stuff like that. Native 2D tools if you want to create complex 2D games. You have visual scripting. Now, if let's say you don't want to write in C-sharp, which is the language Unity is using for development, you can actually use visual tools to really create your game, like full blown game using visual scripting. Harmonization solutions, AR, which is one of the main reasons to use Unity within React Native because AR within Unity is going like really low level, working with ARKit and ARCore and it's really, really performant. And you can check other features here on the link above on what Unity supplies.
Let's get to the demo. Because I want to show you what you can achieve with Unity and React Native. Focusing also on performance, we will see how performant it is. Load the startup though. Takes a little bit of time to load everything. Let's see.
Okay, cool. So we have regular React Native app. Just blank screen, right? And on the top you have performance profile, it is a little bit small to see, but you can see that we're running with 60 frames per second. Now if I go to Unity side of things, I have two paths. Yeah, sure, thanks. Yeah, thanks. And I will also make it bigger. Okay, that's better.
3. Integrating Unity into React Native
Unity can be integrated into React Native apps by using the Unity View component and the Unity Message callback. By using the postMessage function, data can be sent from React Native to Unity, allowing for seamless communication between the two platforms. To enable this functionality in Unity, two native plugins need to be created. This provides a way to bridge Unity into React Native.
So I have Unity running within React Native app, and on the bottom is React Native. On the top is Unity. So if I change color, you see that immediately the color is changed within the Unity. Now if I write text, I will write, hi Berlin. We can see that this text is rendered within the Unity, and I can obviously clear that.
Now if I click on the message React Native, ta-da, I have the message within my React Native app. Now you can see it's pretty fast. So the whole like, and you can see the frames on the top, if you've noticed, they drop a little bit but not that much. Now I can even have post processing effects here to give some bloom look to the view. But let's get into like more complex games. So the character walking, jumping, and even running, a bunch of physics involved, and more complex interaction. So let's see, somewhere, it has staircase, so these kind of games, and as you can see on the bottom, there is actually React Native rendered. Right, so look at the performance. It's still 60 frames per second. And let me just get to the top. As you can see, we have here our robot working in the outer space. And it's working pretty good. Now I get back to my React Native app to a different view, it's a different scene within Unity, and it works like a charm. So let's see how we can actually achieve that.
So after the talk, you'll have a video of the same demo, so if you want to watch that, and in React Native, it's as simple as just use Unity View, a component that I've created, and we'll see how to create that. And I have on Unity Message callback, which basically sets the state within React, and you have a message posted from Unity within the view. Now, in order to post something to Unity, I have basically, let me just put the cursor here, or rather, do it that way, I have these function, postMessage, and I use the GameObject within Unity Editor, its name, and I use the method name of the script that is attached to that GameObject, and I pass the payload. This payload will be in Unity, basically, parsed back into a serialized object, and then I can do whatever I want with it. In my case, I will set the color and set the text. Now, in Unity, it will look like this. I have, like, our interop object on the left, the one that is marked there. And basically, that's, like, the name of the object I'm calling. Now, in order to enable this functionality fast in Unity, I need to create these two files. These are two native plugins. That's the option how you can bridge Unity into React Native.
4. Integrating Native Plugins in Unity
To integrate native plugins in Unity, you need to create two files: a header file and a main file. The header file contains the register API for native calls, while the main file handles the sendMessageToMobileApp function. To send messages from Unity to React Native, use the allele import internal in Unity and specify the method to be called on the native side. To receive messages from React Native in Unity, create a C sharp script with the desired method. Finally, in Unity, build your iOS project and place the native plugins in the Unity builds iOS folder of your React Native project.
You create these plugins. They have to be in assets, plugins, iOS folder. In this example, we will go through iOS example, but the same possible for Android.
So, the first one is this header file, and it has two functions, basically. It has the register API for native calls, which will be called by Unity framework, and it will register our sendMessageToMobileApp function, and we will be able to call it from Unity. So, we basically use X and C, and we write our message here under sendMessageToMobileApp. We basically make it available for Unity. So, that's it. That's the whole thing we need to do from native plugins perspective in Unity.
Now, sending message to native in Unity will be done in the following way. In Unity, we will use the allele import internal, and we will specify the method to be called on the native side. In that case, Unity framework will search for the method name and will call it. And we'll attach that to a button and call the class NativeAPI, sendMessageToMobileApp, And that's what you've seen in the demo. Now, in Unity, you attach that to the component, and you specify the method button click press to be called, and that's how the interaction from Unity to React Native works on the Unity side of things.
Now, receiving message from React Native in Unity, basically as simple as just creating C sharp script with the method that you will be calling from React Native. You've seen, you just call the name of the object, the method, and it will be passed to Unity. Now, it's up to you what to do there. I just changed the color and the text. But basically, you can pass any data and do whatever you want with that.
So, in order to do this kind of integration, you need to do a few things. And now we will be going through the steps, so you will be able to do that yourself. So first of all, in Unity, you need to build your iOS project. That's the build settings window from Unity that every time you export to iOS, it's just a regular screen. You build as normal. Nothing different except having these two native plugins, two files for the native plugins. Now, I put in my example, I put it under Unity builds iOS folder in the React Native project.
5. Unity with React Native on iOS
Now then on React Native project, I will open an Xcode, my regular workspace React Native, under iOS folder, there is Xcode workspace that I will open. And then just add my Unity project, the build Unity project, into this workspace. So at this point we have two projects. Unity iPhone, which is the default name, and Unity Native, the React Native project.
Now then it's really important to actually add Unity framework to embedded frameworks under Xcode because we need to use the framework for certain interrupt. Another important step to actually remove linkage of, like, linking binary with libraries, you need to remove Unity framework from there, otherwise it will just break and you'll see like wild errors. And another helpful thing is to actually drag and drop the embedded frameworks before the compiled sources because what you want to do, you want to embed framework before actually compiling native files that we'll be writing in a bit.
You also need to change data membership. Unity iPhone project has a data folder which has all the assets and data for the Unity project. So we need to change membership to be Unity framework because when you add the project, it will be, by default assign membership to our React Native project, which is something we don't want. So we need to change that. We need to change build products path of Unity framework to point to our build directory because we want everything to be built into one iOS builds directory. And then we also need to find our like native plugins that were wrote. And we need to change their membership as well, first of all to Unity framework and then to be public because by default they're private. And then we need to build the Unity framework separately.
Now these steps absolutely must, you need to go through them. Everything that on this slide, you need to go through them every time you build your Unity project. And that's a little bit caveat because if you want to change things in Unity, you basically change things, you rebuild, you do the steps, and you get stuff in React Native.
6. Integrating React Native Unity and Helpful Links
So we use this one method queue dispatch getMainQueue. Again that's in official docs, how we integrate stuff. In React Native Unity view, it's not the view manager, it's the view naming a little bit off. But we basically send message, we use this send message to go with name. Now this is the default name of Unity framework API to actually send a message to these game objects within Unity and call the method on it. And then we have these like wrappers for Unity framework to just make sure it loads, and it says the data bundle to be like this data folder, etc.
Now, let's get some really cool news. Now, that can be a little bit complex, right? To write all these, like, native code and stuff. So there is a library for that. It's called React Native Unity. Ta-da! And so no need to write native code, right? So that's a huge part of things. You just use React Native Unity. It has Unity view. It supports Android as well. So I just described how to use it with iOS. You still need to do all the setup and build steps. So it's really good now you know how it actually works under the hood.
So there are some helpful links that you can screenshot and get. There is a demo code in this repo. In order to learn Unity, you can go to learnunity.com. There are amazing paths to actually learn Unity development. There is a Unity assets store where you can purchase assets for a fraction of the cost. And if you have any questions for me after the talk, et cetera, so reach out on Twitter. That's my Twitter on the top, and you can ping me for consulting on my website.
Unity Integration Use Cases and Performance
I will be doing workshops and courses in upcoming years. The real-world use case for integrating Unity in React Native is mostly AR. Unity has an ARFoundation framework which enables you to leverage almost everything you can do with ARKit. Apart from AR, you can use Unity for gamification, fancy ads, 3D views, and UI libraries from the Unity assets store. The performance of UI in fast in-game interaction scenarios is similar to regular React Native apps. To improve performance, consider serializing the data and handing over the logic to Unity.
And I also have these forums. I will be doing workshops and courses in upcoming years, so if you're interested in learning more about when it will be done, you can just fill it in. And I promise not to spam. I will just send you an email whenever any course or workshop will be ready.
And I think at this point the clicker stopped working. And the reason for that is I'm kind of done, and I think we have several minutes for questions here, and thanks a lot for listening. Good working! Yeah, thank you very much Vladimir. Just go ahead. Come come come come! We do have some questions, though. Are we running short on time? Almost, yeah.
So, there's a question. What would be a real-world use case for integrating Unity in React Native? I saw it coming! That's like, yeah. The real-world case, scenario nowadays, mostly it's AR. Because if you want to integrate AR into your React Native apps, right now it's either you use other frameworks such as Viro, for instance, which is a good one as well. But if you want more native capabilities and getting into all the capabilities ARKit has to offer over ARCore, Unity has a ARFoundation framework which enables you to leverage almost everything you can do with ARKit. And now you can bridge that into Unity. That's one of the use cases. There are quite a lot. I mean, you can do gamification, you can do like fancy ads within, I know we all hate ads, but I mean, sometimes it's driving business. I've heard that the Unity ads are pretty good. You can use ads, you can use some 3D views which are really performance oriented. For simple games, the ToggleView before mine was pretty good to just try things in like for 2D games and publish them as a web view. I really like Unity capabilities in general.
That's actually a good answer because you also covered another question which if AR is actually possible. There is another follow up question to the first one. What would be a real use case for Unity inside Reagnative? You showcase fireworks and things like that but what do you think in your perspective it is a real use case of Unity? Apart from AR, what do you think about UI libraries that already exist in Unity? Can we leverage those? Definitely sometimes and that's good you bring it up because Unity assets store has lots of amazing assets that you can purchase and some of those are really fancy UI libraries with bunch of effects and fireworks so if you want to have generally something like physics related and I think like particles are also like physics related like fireworks and stuff you can use these assets from Asset Store and embed them within React Native App and as you've seen you getting pretty good performance on that.
Great speaking of performance how performant is such UI in fast in-game interaction scenarios especially given that on every call the JSON serialization and deserialization is included. So basically you're not introducing another bridge you're basically in the same world as in general like React Native because basically what you're doing is serializing the same data and passing that through the bridge that you call in native methods that go into Unity so there's no like another layer here. So performance will be pretty much the same like talking about like passing through the bridge will pretty much the same as regular React Native apps. With that said if you want to make things more performant I would think of like serializing the data passing to the Unity and handing over the logic to Unity to do like the whole animations and stuff and then passing the data back.
Unity Integration Considerations
Passing too many things through the Unity bridge can cause traffic congestion. The new architecture of React Native aims to address this issue. Expo Native modules can be used to expose the Unity view, but there hasn't been much experimentation with this integration. The decision to use Unity or React Native depends on the app's use case and the need for advanced UI and gaming capabilities. Integrating Unity will increase the bundle size, especially if dealing with 3D assets. Optimization options, such as using asset bundles, can help manage the bundle size. Thank you.
So I would, for instance here I had like lots of renders and passing things through the Unity that's for the sake of the demo but I would advise against that because that's eventually lots of traffic through the bridge, you don't want that. Yeah, the thread will be bloated anyway so I think that's one drawback of React Native but I think that the new architecture is going to solve some parts of that as well.
Speaking about Native modules, have you ever thought about Expo Native modules, like to expose the Unity view using Expo Native modules? What do you mean, like exposing um.. Like Expo has created these Native modules... Ah, yeah, so basically with Expo, yeah, I've thought about that, like something I haven't tried and you need to experiment with that a bit. Just to bring in some context, Unity at some point they had like really like vendor-log sort of thing, they didn't have this feature, Unity as a library is fairly new one, relatively, as things go slower in game development world, it's like for couple of years, but still like from Unity perspective it's a relatively new one, so there is no much experimentation done on that regard, especially with React Native. But yeah, that's definitely something to look into and try it out.
Speaking about integration, have you ever thought about the size of the bundle after integrating Unity? Yeah, the bundle will be absolutely, I wouldn't say huge, it will be way higher because obviously with Unity you need to deal with all the 3D assets if you're dealing with the game. These demos, I think 150 megs bundle which is pretty huge, but if you think of game development in general, usually they have a huge bundle. So that's something to take into account, obviously. Okay, so it's basically something new that is going to enhance over time, but now we should also think about trees and nature, just saying, because 150 megabytes just for a Hello World or a 3D box is quite a lot, but definitely maybe it's going to improve. There is optimization, speaking of that, there is an option with Unity to use asset bundles, so basically you don't download all the assets, you get them through the network on the fly. There is capability to do that, and that's something, obviously, to take into account when you develop your app, especially with AR, for instance, because you have all these 3D models, which is really taking some space on the device.
Thank you. I think we're running out of time, so yeah, let's give a huge shout out to Vladimir. Thank you very much.