AI Generated Video Summary
Unity targets over 25 platforms and technologies, including desktop, mobile, and virtual reality. They use Emscripten to compile the engine and game logic into WebAssembly for web development. Unity can be extended with plugins to access browser features like WebXR's augmented reality mode. The speaker demonstrates intercepting Unity's calls to the browser to modify its behavior. Unity is actively working on mobile support for web export and improving documentation for extending Unity with web plugins.
1. Building for Web and Extending Unity
We target over 25 different platforms and technologies, including desktop, mobile, and virtual reality. When building for the Web, we use Emscripten to compile the engine and game logic into WebAssembly. Unity supports different graphics patterns and can be extended with plugins to access browser features. One example is integrating WebXR's augmented reality mode into Unity, which is not supported by Unity's built-in APIs.
One of the things that we do to support this idea is target over 25 different platforms and technologies from desktop, PC, Mac, Linux, to mobile, iPhone, Android, PlayStation, Xbox, Virtual Reality, Augmented Reality and one my favorites, the Web. When we build for the Web, we build like for any other platform where we compile the engine and the game logic together into a final executable. In this case, the final executable is WebAssembly and we use different tools to achieve this.
For the graphics side of things, Unity has different graphics patterns that it supports from DirectX, Vulkan, Metal, and OpenGL. On the web, graphics are defined by WebGL, which is a variant of OpenGL. So when we build for the web we tell Unity to use the OpenGL graphics device. And when it does this, Emscripten and the compilation processes will generate WebGL calls for all the OpenGL calls that Unity are making. And for shaders, they're typically written in a shading language, a DirectX shading language in Unity, but those aren't directly supported by WebGL so we have to convert those into GLSL to be used by WebGL. To do this, we have shader compilers that will translate HLSL into GLSL.
2. Implementing WebXR and Sharing Data
This is for demo purposes only, and it's a very minimal implementation of WebXR. I'm not implementing all the fun features of WebXR. I'm just showing you how to do this.
All the source code for this project can be found on the GitHub project here, and you're welcome to use it for any purposes.
The JSLib file is where we're declaring the public API for our plugin. And here we'll provide functions that we can call from C sharp to initialize WebXR and get its information in its current state. These functions are callable directly from C sharp, and this is the public API that we're providing.
The JSPRE file is where I put the bulk of the code to keep the JSO file simple, and this is where I define all the main objects and methods for interacting with WebXR, and here you can put all the functions that can be called from your JSO and managing the state of your plugin. Again, there's no requirement to use a JSPRE file. I find it convenient.
3. Intercepting Unity-Browser Interactions
Now I'm copying WebXR's view matrix into the data pointer that I pass to it. I divide the pointer index by two because the heap is aligned by four and so the heap index is the pointer address divided by four.
4. Intercepting Unity's Canvas Rendering
We intercept Unity's calls to the frame buffer and clear function to make WebXR work. By replacing the null frame buffer with WebXR's frame buffer, Unity can render on top of the camera view. We override Unity's requestAnimationFrame function to use WebXR's version. This allows Unity to update its camera view with shared data from WebXR.
So another function that we want to intercept to make WebXR work with Unity is that WebXR has its own frame buffer that it wants you to render into. This has, in the case of augmented reality, the camera feed already into it, and it wants you to render on top of that. Unity is trying to render on top of the canvas that's on your screen, and it doesn't know about the WebXR frame buffer.
So what we want to do is intercept the call that Unity is making when it finds the current frame buffer, and detect when Unity is trying to draw into the canvas frame buffer. We can know that because the null frame buffer is what WebGL uses to draw into the canvas, and when Unity does that we can intercept the bind frame buffer function, detect that it's trying to bind the null frame buffer on the canvas, and replace the null frame buffer with WebXR's frame buffer, and then call the original bind frame buffer function.
Now, when Unity binds the canvas, we intercept and actually bind the WebXR frame buffer, and Unity will then draw into the WebXR frame buffer. One other function we want to replace is WebGL's clear function because Unity will try and clear the canvas frame buffer when it draws into it. We don't want the WebXR frame buffer to be cleared because it already has the camera view in it. We want Unity to draw on top of that, so what we can do is intercept the WebGL clear function. If we know that the back buffer, the canvas is currently bound, or in this case, the WebXR context, we can just skip the clear function. Otherwise, we'd call it normal.
And now, when the WebXR frame buffer is bound, Unity tries to clear it for when it blenders, the clear will be blocked and Unity will just draw on top of what's already there, which is the camera view. One other function we want to intercept is requestAnimationFrame. Unity uses this to control frame rate, and this is what WebGL uses for its rendering, but WebXR has its own version of requestAnimationFrame that it wants you to render into. And so, we want to use that instead. But since Unity is using the built-in requestAnimation function, we can override the built-in requestAnimation function with our own, and route that to the WebXR version of the function.
Now, when Unity calls requestAnimationFunction, it's calling our function, which we then call WebXR requestAnimationFrame, which will call the callback, and then tell Unity to do its rendering from there. Now, that callback has all the WebXR information in it, and we can use that information to update Unity's camera view with that shared data and then call the callback for Unity to do its rendering.
So, if we put all this together, we can see Unity rendering on the webpage. We start with the WebXR context, and Unity is now rendering into the WebXR's frame view. It doesn't clear the back buffer, so it's rendering on top of the camera view, and everything comes together. WebXR's view matrix is being passed into Unity so that it can update its camera view, and it all integrates together.
Now, like I said, this is a very minimal implementation of WebXR. For a more full-featured implementation, other developers have done a great job with that. DeepAnthro has an excellent WebXR package available at his GitHub that you can use to render with virtual reality and augmented reality modes of WebXR, a much more full feature. I highly recommend that. And that's all I have to share with you today. Feel free to reach out for any questions you might have. I'm happy to answer, and thank you very much for taking the time. Awesome.
5. Poll Results and Basis Compression
We asked the audience which web technology they would like to see supported by Unity, and web GPU was the overwhelming majority. Adding support for basis compression would be beneficial for the web, as it reduces download size, load time, and memory usage. Unity already supports mobile texture compression formats, but using basis would simplify the process by allowing developers to use one format for multiple platforms. When publishing without basis, developers can create different asset bundles for different devices and choose the appropriate bundle at runtime.
So we're taking a look at the results of the poll question here we asked before the talk. We asked the web technology would you like to see supported by Unity? It looks like web GPU is the overwhelming majority. Yeah, I'm a little surprised about that, but not really, because it's the hot new thing. I was actually surprised there were. We're actively working on. Oh, that's good to hear. I think a lot of people are happy to hear that then, you're actively working on that, I was surprised that there wasn't more request for basis, because I know that I mean that helps a lot, especially on the web. And I know a couple other speakers, either today or yesterday, mentioned. Because it helps you both in terms of the download size right to load time, but also in terms of using less memory. And that's that's big for a lot of mobile devices, right? Yeah, I agree. We added support for mobile. We've had desktop texture compression for a long time. And then in 2021, we added support for mobile texture, compression formats. But you kind of have to make a design decision and use different asset bundles for which format you want. And if you had basis, then that kind of puts less technical burden on the developer and lets you just use one format for other places. Yeah. And this is something I'm not personally familiar with because most of the stuff I publish has been on the web. But I'm curious, if you were publishing without basis on desktop or otherwise, would you... Because you have all these different compressed texture formats on different GPUs, et cetera. Do you just have these different bundles and you just ship them depending on what the device has support for? Yeah, you have different bundles and then at runtime, you can, depending on your environment, pick the mobile bundle or the desktop bundle. You can have bundles for high fidelity assets and lower quality assets. You can make those choices. It has some technical burden or whatever to make those choices at runtime, but the tools are there to do it. Makes sense. Cool.
I think we can take a look at some of the questions that the audience has been submitting here. The first question I'm looking here, Rick asks, is there a roadmap for supporting WebXR natively in the future? We have the roadmap. I put the link on the poll question, where you can add votes to things or add your own suggestions. And it's definitely something that we're all aware of and excited about. It's really, even though it's been around for a long time, still, the technology that's not supported everywhere.
6. WebKit's Engagement and Unity's Public Roadmap
WebKit is actively engaged in bringing up the feature, although it's not yet available. While it hasn't been a top priority due to limited resources, we are aware of its benefits and excited to include it. Unity has a public roadmap where people can vote on features, although its discoverability has been an issue. Sharing the roadmap in the Discord and on social media will help prioritize features based on community input.
WebKit is engaged in it now and will be bringing it up, but it's not available yet and other places like that. So, it hasn't boiled up in priority for development because there's only so many people we have to work on things and significantly more things to work on than we have time, but it's definitely something that we're aware of and excited about and really want to get in there because it has a lot of benefit for a lot of different use cases.
I'm honored to work with you. And I think this is the first time I was aware of that Unity had a public roadmap where people can vote. How long have you had that? I'm not entirely sure. Okay, but it's not like a new thing like this month or something. It's just been there for a little longer. Yeah, it's been up for a while. Like a lot of things, discoverability is always an issue, putting it up, making it not super obvious of where to find it. Things are... Yeah, we should definitely share that in the Discord and maybe tweet about it too, so other people, everyone can make sure to see that. Cool. I think that's really important for prioritization and things like that. Yeah, absolutely. It's cool that the community can get to voice that in this very direct way, so I think that's cool.
Mobile Support and Internal Emscripten Calls
We started working on mobile support for the web export in the 2021.2 release of Unity. Mobile web development is challenging due to the constantly changing operating systems, but we take mobile-related issues seriously. We work closely with Apple, Google, and web developers to ensure the web is a great platform for everyone. Documentation for internal Emscripten calls needs improvement, and we are working on making the web a first-class citizen platform with better documentation and tooling. The Unity instance send message function is also in the Unity manual, and specific use cases can be discussed on the Discord channel.
We have another question here from Daniel. What is the current progress of mobile support for the web export? So we started working towards web, mobile, back in, I think more seriously in 2021.2 release of Unity, and we had a banner on the page saying that mobile wasn't supported, in 2022 we removed that banner and I think 2023 we'll have put in all the pieces, in 2022 I added the mobile keyboard support. And there's just a lot of different things that go into mobile web that you need to take care of, that desktop doesn't need to deal with, so we've always, you know, any issues that came in that were mobile related, we always took them very seriously and continue to do that and with more emphasis now and we've always had this notion that mobile was not supported because there's just a lot of missing things, but it's always been taken seriously and now we have mobile texture compression supports and all these other features that are going into it that will have the list of supported, you know, browsers for mobile in the upcoming version.
Developing for the web is certainly, I'm super excited about is definitely a challenge because you're working on a shifting sand up operating system that's constantly changing under you. So we had a lot of issues with iOS 15.4 that shipped recently. They kind of broke everything web-related, webGL related and WebAssembly related for us. We work very closely with, you know, Apple and Google and all the web developers to make sure that not only for us, the web is a great platform, but for everybody, because, you know, making the web work for us, you know, and making the web work for everybody else, you know, is equally beneficial.
We have another question here. It says, you mentioned some internal Emscripten calls, like DIN call, a DIN call in heap. Where can we see what the equivalence to the variables like V, I, and the other basic types in those calls? And then just find more info about those internal calls. So these are things that are coming from Emscripten. I think we need to do a better job of documenting from our perspective. You can find this information from the Emscripten documentation, but it's all, you know, it's like discoverability is an issue, how do you find the information? So so it's definitely something that we need to work on documentation wise for, you know, extending unities, you know, with plugin architectures and stuff like that from the web perspective, because we have a lot of documentation, but the web has always been kind of a niche platform, and so the documentation has been skewed towards the other platforms, but we I think that, you know, the shift is being made where and we've been really pushing on it to make web a first-class citizen platform and to make that documentation and the tooling much better for the web platform.
Our next question is from George, the Unity instance send message function is also in the Unity manual, in what cases would you recommend using it? Very specific question that is kind of throwing my brain off topic. We can also get back to it right now that I can answer in a couple of seconds. Yeah, maybe on the discord you can post that. Yeah, exactly. And in a time where I can think more clearly. Sure, no worries. What about...
Improving Documentation for Web Plugins
We need to improve our documentation for extending Unity with web plugins. Currently, the documentation is skewed towards other platforms, but we are making efforts to shift and prioritize the web as a first-class citizen platform.
I think we need to do a better job of documenting from our perspective. You can find this information from the Emscripten documentation, but it's all, you know, it's like discoverability is an issue, how do you find the information? So so it's definitely something that we need to work on documentation wise for, you know, extending unities, you know, with plugin architectures and stuff like that from the web perspective, because we have a lot of documentation, but the web has always been kind of a niche platform, and so the documentation has been skewed towards the other platforms, but we I think that, you know, the shift is being made where and we've been really pushing on it to make web a first-class citizen platform and to make that documentation and the tooling much better for the web platform.
Unity Instance Send Message and Project Tiny
That's awesome. Our next question is from George, the Unity instance send message function is also in the Unity manual, in what cases would you recommend using it? Very specific question that is kind of throwing my brain off topic. We can also get back to it right now that I can answer in a couple of seconds.
Yeah, maybe on the discord you can post that.
Yeah, exactly. And in a time where I can think more clearly. Sure, no worries.
Unity's Investments and WebGPU Development
Unity is investing in making small games by picking specific engine pieces, improving game size, bundle size, and load times. The development of WebGPU is underway, with no public timeline yet. The speaker is actively involved in the Standards Committee to ensure WebGPU meets Unity's needs and benefits others. They are addressing challenges with Unity projects using WebGPU and pushing for improvements in shader compilation. The involvement of individuals like the speaker in the Standards Committee ensures the web platform evolves for the benefit of all.
And so you don't have the giant monolithic engine, but you pick pieces that you want, so you can make very small games. And that was put on hold temporarily while the developers focus on the rendering side of the DOTS technology stack. But it is something that we'll be investing more into in the near future. And so that'll drastically improve things like game size and bundle size and load times and all these other things.
What about as a final question? Are there any features of Unity that you're working on or that you're excited about that you can share with us? Well, the one thing that everybody is interested in, apparently, is WebGPU. And that is something that I am actively working on and developing and we have an implementation of that. We'll not go public. No timeline for that, but it is in the works.
This was really great. Thank you so much for being here, Brandon, and thanks for answering all our questions. Thank you so much.