Extending Unity WebGL With Javascript

One of the things that we do to support this idea is target over 25 different platforms and technologies from desktop, PC, Mac, Linux, to mobile, iPhone, Android, PlayStation, Xbox, Virtual Reality, Augmented Reality and one my favorites, the Web. When we build for the Web, we build like for any other platform where we compile the engine and the game logic together into a final executable. In this case, the final executable is WebAssembly and we use different tools to achieve this.

We use this Emscripten which is a C++ compiler that can generate WebAssembly in JavaScript. All the libraries for the engine and the built-in libraries are written in C++ and all the user code and public APIs are written in .NET C Sharp. We have to compile the .NET C Sharp code into C++ in order for it to be compiled with Emscripten. So we use another tool called iotcp to do this. It takes the .NET assemblies and does some stripping and massaging of those assemblies to reduce the code size and then generates C++ code from there. The C++ code can then be compiled with Emscripten to WebAssembly.

For the graphics side of things, Unity has different graphics patterns that it supports from DirectX, Vulkan, Metal, and OpenGL. On the web, graphics are defined by WebGL, which is a variant of OpenGL. So when we build for the web we tell Unity to use the OpenGL graphics device. And when it does this, Emscripten and the compilation processes will generate WebGL calls for all the OpenGL calls that Unity are making. And for shaders, they're typically written in a shading language, a DirectX shading language in Unity, but those aren't directly supported by WebGL so we have to convert those into GLSL to be used by WebGL. To do this, we have shader compilers that will translate HLSL into GLSL.

We can extend Unity with plugins that provide new APIs to Unity written either in C++ or in this case we'll use JavaScript. With the JavaScript plugins for WebGL, we can extend Unity to access browser features that aren't available in the regular Unity API. We can call these JavaScript functions directly from C-sharp and vice versa you can call C-sharp functions from JavaScript. When you define a JavaScript plugin, you put it into the Assets Plugins folder for WebGL and there's two different types of files from the JavaScript function. The main type of file is a JSLib file. This is where your public APIs from JavaScript will be and define the functions that will be callable from C-sharp. A JSPri file is just arbitrary JavaScript that you can include with your plugin. This will get compiled before the JSLib files so that it can provide JavaScript objects and functions that can be used and shared between your JSLib files. This is just a way to keep your projects clean so that your JSLib files can be your public APIs and you put all the rest of your code in JSPri. There's no requirement to do this. You could just use a single JSLib file. I like to keep things separate. The example of plugin I'll be talking about today is integrating WebXR's augmented reality mode into Unity. Unity currently does not support WebXR with its built-in APIs, but we can extend Unity using plugins to do this. This is not an official plugin.

This is for demo purposes only, and it's a very minimal implementation of WebXR. I'm not implementing all the fun features of WebXR. I'm just showing you how to do this.

All the source code for this project can be found on the GitHub project here, and you're welcome to use it for any purposes.

The JSLib file is where we're declaring the public API for our plugin. And here we'll provide functions that we can call from C sharp to initialize WebXR and get its information in its current state. These functions are callable directly from C sharp, and this is the public API that we're providing.

The merge into function here is part of the script, and what it does is it takes all the functions that we're declaring in the subject and exposing them to C++ or C sharp. To call JavaScript from C sharp, we declare the functions as external static functions in C sharp, and we also tag them to be encoded from the internal DLL in Unity. What this is doing is it's telling C sharp that these functions are not defined in place but they're coming from an external source, and because WebGL does not provide support for external DLLs, everything is bundled together and the internal DLL name defines that they're coming from the built-in DLL.

Here I also implement MTEW versions of the functions for non-WebGL platforms, because non-WebGL platforms do not support JavaScript. We want to call dummy versions of these functions to keep C sharp happy and keep the editor from complaining that you don't have implementations for these functions.

The JSPRE file is where I put the bulk of the code to keep the JSO file simple, and this is where I define all the main objects and methods for interacting with WebXR, and here you can put all the functions that can be called from your JSO and managing the state of your plugin. Again, there's no requirement to use a JSPRE file. I find it convenient.

Sometimes you want to call C sharp from JavaScript. In some cases, you have an asynchronous function in JavaScript that will be called sometime later and you want to call a C sharp function when that asynchronous process has finished. You can do this by declaring a delegate type in C sharp that is then toggleable from JavaScript. It can pass that function pointer of that delegate type to the JavaScript function, which it can hang on to as a pointer, and then it can call that C sharp function when it's done. And then we define the callback function in C sharp as a static function. JavaScript doesn't have any notion of C sharp objects, so we declare it as a static function, and from there you can use global variables or singleton access to whatever C sharp state you want. And then from the JavaScript side, we can use the inscription's dyne call function to call the C sharp callback. This can also be a C++ callback. So the vi argument for dyne call defines the return type as void and it takes a single integer argument and this can define the function declaration that you have. And the state change callback is the function pointer I passed in from C sharp, and these are the arguments I'm passing to the C sharp callback function which is an integer state. You can also share data between C sharp and JavaScript. If we allocate memory on C sharp side it's allocating on the end scripting heap and that end scripting heap is visible from JavaScript. We can read and write to that heap from JavaScript. Here I'm allocating 16 floats that are used to store the view matrix of WebXR so that I can update the camera's view matrix in Unity. I can pass that data to JavaScript as just a function data pointer and then now JavaScript will have access to that data.

And then from the JavaScript side I can write into that function on the data pointer using the inscription's heap variables. In this case, heap 32 is a float 32 view of the heap data.

Now I'm copying WebXR's view matrix into the data pointer that I pass to it. I divide the pointer index by two because the heap is aligned by four and so the heap index is the pointer address divided by four.

I can intercept calls from Unity to the browser to change the behavior of how Unity interacts with the browser. In some cases, the way that Unity is interacting with the browser doesn't work with what we want to do in our plugin. WebXR requires, for example, that the XR compatible attribute is added to the webGL context. When Unity is creating the webGL context, it doesn't have this attribute because it doesn't know about WebXR. So what we can do is actually replace Emscripten's webGL great context function with our own custom function. Now, when Unity calls great context, it's calling our custom function instead of the built-in one. We can then inject the XR-compatible attribute that webXR wants into the attributes for that webGL context and then call the original function to go ahead and create the context now with that custom attribute added to it. This is an example of using private APIs and working around things that Unity is doing by intercepting function calls between Unity and the browser using the dynamic nature of JavaScript. Because we're using private APIs here, there's no warranty that any of this will work on future versions of Unity. But sometimes you have to do things like this to work around limitations.

So another function that we want to intercept to make WebXR work with Unity is that WebXR has its own frame buffer that it wants you to render into. This has, in the case of augmented reality, the camera feed already into it, and it wants you to render on top of that. Unity is trying to render on top of the canvas that's on your screen, and it doesn't know about the WebXR frame buffer.

So what we want to do is intercept the call that Unity is making when it finds the current frame buffer, and detect when Unity is trying to draw into the canvas frame buffer. We can know that because the null frame buffer is what WebGL uses to draw into the canvas, and when Unity does that we can intercept the bind frame buffer function, detect that it's trying to bind the null frame buffer on the canvas, and replace the null frame buffer with WebXR's frame buffer, and then call the original bind frame buffer function.

Now, when Unity binds the canvas, we intercept and actually bind the WebXR frame buffer, and Unity will then draw into the WebXR frame buffer. One other function we want to replace is WebGL's clear function because Unity will try and clear the canvas frame buffer when it draws into it. We don't want the WebXR frame buffer to be cleared because it already has the camera view in it. We want Unity to draw on top of that, so what we can do is intercept the WebGL clear function. If we know that the back buffer, the canvas is currently bound, or in this case, the WebXR context, we can just skip the clear function. Otherwise, we'd call it normal.

And now, when the WebXR frame buffer is bound, Unity tries to clear it for when it blenders, the clear will be blocked and Unity will just draw on top of what's already there, which is the camera view. One other function we want to intercept is requestAnimationFrame. Unity uses this to control frame rate, and this is what WebGL uses for its rendering, but WebXR has its own version of requestAnimationFrame that it wants you to render into. And so, we want to use that instead. But since Unity is using the built-in requestAnimation function, we can override the built-in requestAnimation function with our own, and route that to the WebXR version of the function.

Now, when Unity calls requestAnimationFunction, it's calling our function, which we then call WebXR requestAnimationFrame, which will call the callback, and then tell Unity to do its rendering from there. Now, that callback has all the WebXR information in it, and we can use that information to update Unity's camera view with that shared data and then call the callback for Unity to do its rendering.

So, if we put all this together, we can see Unity rendering on the webpage. We start with the WebXR context, and Unity is now rendering into the WebXR's frame view. It doesn't clear the back buffer, so it's rendering on top of the camera view, and everything comes together. WebXR's view matrix is being passed into Unity so that it can update its camera view, and it all integrates together.

Now, like I said, this is a very minimal implementation of WebXR. For a more full-featured implementation, other developers have done a great job with that. DeepAnthro has an excellent WebXR package available at his GitHub that you can use to render with virtual reality and augmented reality modes of WebXR, a much more full feature. I highly recommend that. And that's all I have to share with you today. Feel free to reach out for any questions you might have. I'm happy to answer, and thank you very much for taking the time. Awesome.

So we're taking a look at the results of the poll question here we asked before the talk. We asked the web technology would you like to see supported by Unity? It looks like web GPU is the overwhelming majority. Yeah, I'm a little surprised about that, but not really, because it's the hot new thing. I was actually surprised there were. We're actively working on. Oh, that's good to hear. I think a lot of people are happy to hear that then, you're actively working on that, I was surprised that there wasn't more request for basis, because I know that I mean that helps a lot, especially on the web. And I know a couple other speakers, either today or yesterday, mentioned. Because it helps you both in terms of the download size right to load time, but also in terms of using less memory. And that's that's big for a lot of mobile devices, right? Yeah, I agree. We added support for mobile. We've had desktop texture compression for a long time. And then in 2021, we added support for mobile texture, compression formats. But you kind of have to make a design decision and use different asset bundles for which format you want. And if you had basis, then that kind of puts less technical burden on the developer and lets you just use one format for other places. Yeah. And this is something I'm not personally familiar with because most of the stuff I publish has been on the web. But I'm curious, if you were publishing without basis on desktop or otherwise, would you... Because you have all these different compressed texture formats on different GPUs, et cetera. Do you just have these different bundles and you just ship them depending on what the device has support for? Yeah, you have different bundles and then at runtime, you can, depending on your environment, pick the mobile bundle or the desktop bundle. You can have bundles for high fidelity assets and lower quality assets. You can make those choices. It has some technical burden or whatever to make those choices at runtime, but the tools are there to do it. Makes sense. Cool.

I think we can take a look at some of the questions that the audience has been submitting here. The first question I'm looking here, Rick asks, is there a roadmap for supporting WebXR natively in the future? We have the roadmap. I put the link on the poll question, where you can add votes to things or add your own suggestions. And it's definitely something that we're all aware of and excited about. It's really, even though it's been around for a long time, still, the technology that's not supported everywhere.

WebKit is engaged in it now and will be bringing it up, but it's not available yet and other places like that. So, it hasn't boiled up in priority for development because there's only so many people we have to work on things and significantly more things to work on than we have time, but it's definitely something that we're aware of and excited about and really want to get in there because it has a lot of benefit for a lot of different use cases.

I'm honored to work with you. And I think this is the first time I was aware of that Unity had a public roadmap where people can vote. How long have you had that? I'm not entirely sure. Okay, but it's not like a new thing like this month or something. It's just been there for a little longer. Yeah, it's been up for a while. Like a lot of things, discoverability is always an issue, putting it up, making it not super obvious of where to find it. Things are... Yeah, we should definitely share that in the Discord and maybe tweet about it too, so other people, everyone can make sure to see that. Cool. I think that's really important for prioritization and things like that. Yeah, absolutely. It's cool that the community can get to voice that in this very direct way, so I think that's cool.

We have another question here from Daniel. What is the current progress of mobile support for the web export? So we started working towards web, mobile, back in, I think more seriously in 2021.2 release of Unity, and we had a banner on the page saying that mobile wasn't supported, in 2022 we removed that banner and I think 2023 we'll have put in all the pieces, in 2022 I added the mobile keyboard support. And there's just a lot of different things that go into mobile web that you need to take care of, that desktop doesn't need to deal with, so we've always, you know, any issues that came in that were mobile related, we always took them very seriously and continue to do that and with more emphasis now and we've always had this notion that mobile was not supported because there's just a lot of missing things, but it's always been taken seriously and now we have mobile texture compression supports and all these other features that are going into it that will have the list of supported, you know, browsers for mobile in the upcoming version.

Developing for the web is certainly, I'm super excited about is definitely a challenge because you're working on a shifting sand up operating system that's constantly changing under you. So we had a lot of issues with iOS 15.4 that shipped recently. They kind of broke everything web-related, webGL related and WebAssembly related for us. We work very closely with, you know, Apple and Google and all the web developers to make sure that not only for us, the web is a great platform, but for everybody, because, you know, making the web work for us, you know, and making the web work for everybody else, you know, is equally beneficial.

We have another question here. It says, you mentioned some internal Emscripten calls, like DIN call, a DIN call in heap. Where can we see what the equivalence to the variables like V, I, and the other basic types in those calls? And then just find more info about those internal calls. So these are things that are coming from Emscripten. I think we need to do a better job of documenting from our perspective. You can find this information from the Emscripten documentation, but it's all, you know, it's like discoverability is an issue, how do you find the information? So so it's definitely something that we need to work on documentation wise for, you know, extending unities, you know, with plugin architectures and stuff like that from the web perspective, because we have a lot of documentation, but the web has always been kind of a niche platform, and so the documentation has been skewed towards the other platforms, but we I think that, you know, the shift is being made where and we've been really pushing on it to make web a first-class citizen platform and to make that documentation and the tooling much better for the web platform.

Our next question is from George, the Unity instance send message function is also in the Unity manual, in what cases would you recommend using it? Very specific question that is kind of throwing my brain off topic. We can also get back to it right now that I can answer in a couple of seconds. Yeah, maybe on the discord you can post that. Yeah, exactly. And in a time where I can think more clearly. Sure, no worries. What about...

I think we need to do a better job of documenting from our perspective. You can find this information from the Emscripten documentation, but it's all, you know, it's like discoverability is an issue, how do you find the information? So so it's definitely something that we need to work on documentation wise for, you know, extending unities, you know, with plugin architectures and stuff like that from the web perspective, because we have a lot of documentation, but the web has always been kind of a niche platform, and so the documentation has been skewed towards the other platforms, but we I think that, you know, the shift is being made where and we've been really pushing on it to make web a first-class citizen platform and to make that documentation and the tooling much better for the web platform.

That's awesome. Our next question is from George, the Unity instance send message function is also in the Unity manual, in what cases would you recommend using it? Very specific question that is kind of throwing my brain off topic. We can also get back to it right now that I can answer in a couple of seconds.

Yeah, maybe on the discord you can post that.

Yeah, exactly. And in a time where I can think more clearly. Sure, no worries.

What about... this is a question for me. What other web functionality examples do you think could be integrated into Unity with a JavaScript plugin? Well, there's a lot of different web APIs out there that have varying levels of support for the browsers, like web transport, web RTC, and things like that that that we don't have direct APIs for in the C Sharp layer of Unity, but you could definitely do this kind of similar technique to expose it to Unity and add it to your game. So, you could do web RTC. One of our developers did a prototype using web transport that was quite successful. So, there's a lot, anything that you can do with a web API, and, you know, it really opens that up. It makes a lot of sense.

And I was curious, if I were making, if I were publishing something with Unity WebGL, where do you see most people hosting their Unity WebGL export schemes or interactive content? So, I mean, like any web project, you can host it on your own server, or Unity has, you know, services like play.unity.com that can host things for you. And, you know, when you host your, you know, do it on your, set up your own server, you can be much more specialized for your game and use, you know, Brotli compression or gzip, and set up your server to be correct for that. But yeah, there's no limitations for how you can host it. We, when you build your game, you can specify the HTML template and scaffolding that goes around it. And so you can customize it for what, whatever your target's going to be and whatever technical needs that you have. That's very cool. So it sounds like you could publish it really anywhere, you could publish any HTML5 game, JavaScript game, like Newgrounds or Itch or really any of those game, any platform like that. Yeah, exactly. And a lot of different platforms and servers do host Unity games, so there's not really a technical limitation for that. Very cool. In some cases, it does require more technical savvy to set things up correctly, but it's all about improving tooling and documentation to simplify those things. And those are all things that we're actively working toward. Awesome. And what about, I remember vaguely a project called a Unity Project Tiny. Can you talk a little bit about that, and what that is? Sure. Project Tiny, we have a technology called DOTS, Data Oriented Technology Stack, and project Tiny was an offshoot of that, which let you build up games much more modularly.

And so you don't have the giant monolithic engine, but you pick pieces that you want, so you can make very small games. And that was put on hold temporarily while the developers focus on the rendering side of the DOTS technology stack. But it is something that we'll be investing more into in the near future. And so that'll drastically improve things like game size and bundle size and load times and all these other things.

What about as a final question? Are there any features of Unity that you're working on or that you're excited about that you can share with us? Well, the one thing that everybody is interested in, apparently, is WebGPU. And that is something that I am actively working on and developing and we have an implementation of that. We'll not go public. No timeline for that, but it is in the works.

That's awesome. Have you found that hard? Because I know the spec for WebGPU has been changing in the last few months or so. Has that been a factor? It's that developing on Quicksand thing. But I've been very involved with the Standards Committee to help ensure that WebGPU has what Unity needs and also can benefit other people and to try and make sure that the spec for that has the APIs that we need and other people need as well. That's awesome. And also, actively, the size of the Unity projects have been building with WebGPU are much larger than a small hand-coded JavaScript contest. So we're pushing things like the shader compilation and things like that. And finding places where it still needs some work and helping uncover bugs and things like that. Yeah, I love that. That makes me really hopeful for the future of the web. I think having folks like you involved with the Standards Committee ensures that, again, this rising tide lifts all boats, the platform evolves for the benefit of everybody.

This was really great. Thank you so much for being here, Brandon, and thanks for answering all our questions. Thank you so much.