Extending Unity WebGL With Javascript

♪♪ Hi, my name is Brandon Duncan. I'm a software engineer for Unity Technologies, where I work on graphics APIs in the web platform. Today I'm going to be talking about extending Unity's webgl logics with javascript plugins. It lets you write APIs to interact with the browser in ways that Unity doesn't provide built-in support for. Quick caveat, everything I'll be talking about here is for informational purposes only. I'm not committing Unity to delivering any functionality or features that I'll be talking about, but I hope it'll be informative and useful. Unity is a real-time engine and editor, and we really believe that the world is a better place with more creators in it. One of the things that we do to support this idea is target over 25 different platforms and technologies, from desktop, PC, Mac, Linux, to mobile, iPhone, Android, PlayStation, Xbox, Virtual Reality, Augmented Reality, and one of my favorites, the web. When we build for the web, we build like for any other platform where we compile the engine and the game logic together into a final executable. In this case, the final executable is webassembly, and we use different tools to do this. We use Emscripten, which is a C++ compiler that can generate webassembly in javascript. All the libraries for the engine and built-in libraries are written in C++, and all the user code and public APIs are written in.NET C Sharp. We have to compile the.NET C Sharp code into C++ in order for it to be compiled with Emscripten, so we use another tool called IL2CP to do this. It takes the.NET assemblies and does some stripping and massaging of those assemblies to reduce the code size and then generates C++ code from there. The C++ code can then be compiled with Emscripten to webassembly. For the graphics side of things, Unity has different graphics backends that it supports from DirectX, Vulkan, Metal, and OpenGL. On the web, graphics are defined by webgl, which is a variant of OpenGL. So when we build for the web, we tell Unity to use the OpenGL graphics device. And when it does this, all Emscripten and the compilation processes will generate webgl calls for all the OpenGL calls that Unity are making. And for shaders, they're typically written in a shading language, DirectX shading language in Unity, but those aren't directly supported by webgl, so we have to convert those into GLSL to be used by webgl. To do this, we have shader compilers that will translate HLSL into GLSL. We can extend Unity with plugins that provide new APIs to Unity written either in C++ or, in this case, we'll use javascript. With the javascript plugins for webgl, you can extend Unity to access browser features that aren't available in the regular Unity api. We can call these javascript functions directly from C Sharp, and vice versa, you can call C Sharp functions from javascript. When you define a javascript plugin, you put it into the assets plugins folder for webgl, and there's two different types of files for a javascript function. The main type of file is a JSLib file. This is where your public APIs for javascript will be, and define the functions that will be callable from C Sharp. A JSPre file is just arbitrary javascript file that you can include with your plugin. This will get compiled before the JSLib files so that it can provide javascript objects and functions that can be used and shared between your JSLib files. This is just a way to keep your projects clean so that your JSLib files can be your public APIs and you put all the rest of your code in JSPre. There's no requirement to do this. You could just use a single JSLib file. I like to keep things separate. The example of a plugin I'll be talking about today is integrating WebXR's augmented reality mode into Unity. Unity currently does not support WebXR with its built-in APIs, but we can extend to Unity using plugins to do this. This is not an official plugin. This is for demo purposes only, and it's a very minimal implementation of WebXR and not implementing all the fun features of WebXR. I'm just showing you how to do this. All the source code for this project can be found on my GitHub project here, and you're welcome to use it for any purposes. The JSLib file is where we're declaring the public api for our plugin, and here we'll provide functions that we can call from C sharp to initialize WebXR and get its information in its current state. These functions are callable directly from C sharp, and this is the public api that we're providing, and the mergeInto function here is part of Emscripten, and what it does is it takes all the functions that we're declaring in the subject and exposing them to C++ or C sharp. To call javascript from C sharp, we declare the functions as external static functions in C sharp, and we also tag them to be imported from the internal DLL in Unity. What this is doing is it's telling C sharp that these functions are not defined in place, but they're coming from an external source, and because webgl does not provide support for external DLLs, everything is bundled together, and the internal DLL name defines that they're coming from the built-in DLL. Here I also implement empty dummy versions of the functions for non-webgl platforms, because non-webgl platforms do not support javascript. We want to call dummy versions of these functions to keep C sharp happy and keep the editor from complaining that you don't have implementations for these functions. The JS prefile is where I put the bulk of the code to keep the JSL file simple, and this is where I define all the main objects and methods for interacting with WebXR, and here you can put all the functions that can be called from your JSL and managing the state of your plugin. Again, there's no requirement to use a JS prefile. I find it convenient. Sometimes you want to call C sharp from javascript. In some cases, you have an asynchronous function in javascript that will be called sometime later, and you want to call a C sharp function when that asynchronous process is finished. You can do this by declaring a delegate type in C sharp that is then callable from javascript. It can pass that function pointer of that delegate type to the javascript function, which it can hang on to as a pointer, and then it can call that C sharp function when it's done. We define the callback function in C sharp as a static function. javascript doesn't have any notion of C sharp objects, so we declare it as a static function, and from there, you can use global variables or singleton access to whatever C sharp state you want. And then from the javascript side, we can use the inscripton's dyn call function to call the C sharp callback. This can also be a C++ callback. So the vi argument for dyn call defines the return type of void, and it takes a single integer argument, and this can define the function declaration that you have. And the state change callback is the function pointer I passed in from C sharp, and these are the arguments I'm passing to the C sharp callback function, which is an integer state. You can also share data between C sharp and javascript. If we allocate memory on C sharp side, it's allocating on the inscripton heap, and that inscripton heap is visible from javascript. You can read and write to that heap from javascript. Here I'm allocating 16 floats that are used to store the view matrix of WebXR so that I can update the canvas view matrix in Unity. I can pass that data to javascript as just a function data pointer, and then now javascript will have access to that data. And then from the javascript side, I can write into that function data pointer using the inscripton's heap variables. In this case, heap of 32 is a float 32 view of the heap data. I'm copying WebXR's view matrix into data pointer that I passed to it. I divide the pointer index by two because the heap is aligned by four, and so the heap index is the pointer address divided by four. I can intercept calls from Unity to the browser to change the behavior of how Unity interacts with the browser. In some cases, the way that Unity is interacting with the browser doesn't work with what we want to do in our plugin. WebXR requires, for example, that the XR-compatible attribute is added to the webgl context. When Unity is creating the webgl context, it doesn't have this attribute because it doesn't know about WebXR. So what we can do is actually replace inscripton's webgl create context function with our own custom function. Now when Unity calls create context, it's calling our custom function instead of the built-in one. We can then inject the XR-compatible attribute that WebXR wants into the attributes for the webgl context and then call the original function to go ahead and create the context now with that custom attribute added to it. This is an example of using private APIs and working around things that Unity is doing by intercepting function calls between Unity and the browser using the dynamic nature of javascript. Because we're using private APIs here, there's no warranty that any of this will work on future versions of Unity, but sometimes you have to do things like this to work around limitations. So another function that we want to intercept to make WebXR work with Unity is that WebXR has its own frame buffer that it wants you to render into. This has, in the case of augmented reality, the camera feed already into it, and it wants you to render on top of that. Unity is trying to render on top of the canvas that's on the screen, and it doesn't know about the WebXR frame buffer. So what we want to do is intercept the call that Unity is making when it finds the current frame buffer and detect when Unity is trying to draw into the canvas frame buffer. We can know that because the null frame buffer is what webgl uses to draw into the canvas. And when Unity does that, we can intercept the bind frame buffer function, detect that it's trying to bind the null frame buffer on the canvas, and replace the null frame buffer with WebXR's frame buffer, and then call the original bind frame buffer function. Now when Unity binds the canvas, we intercept and actually bind the WebXR frame buffer, and Unity will then draw into the WebXR frame buffer. One other function we want to use is webgl's clear function, because Unity will try and clear the canvas frame buffer when it draws into it. We don't want the WebXR frame buffer to be cleared because it already has the camera view in it. We want Unity to draw on top of that. So what we can do is intercept the webgl clear function. We know that the canvas is currently bound, or in this case, the null frame buffer, or in this case, the WebXR context. We can just skip the clear function. Otherwise, we'd call it normal. And now when the WebXR frame buffer is bound and Unity tries to clear it for when it renders, the clear will be blocked, and Unity will just draw on top of what's already there, which is the camera view. One other function we want to intercept is requestAnimationFrame. Unity uses this to control frame rate, and this is what webgl uses for its rendering. But WebXR has its own version of requestAnimationFrame that it wants you to render into. And so we want to use that instead. But since Unity is using the built-in requestAnimation function, we can override the built-in requestAnimation function with our own and route that to the WebXR version of the function. Now when Unity calls requestAnimation function, it's calling our function, which we then call WebXR requestAnimationFrame, which will call the callback, and then tell Unity to do its rendering from there. Now that callback has all the WebXR information in it, and we can use that information to update Unity's camera view with that shared data, and then call the callback for Unity to do its rendering. So if we put all this together, we can see Unity rendering on the web page. We start the WebXR context, and Unity is now rendering into the WebXR's frame view. It doesn't clear the back buffer, so it's rendering on top of the camera view. And everything works and comes together. The WebXR's view matrix is being passed into Unity so that it can update its camera view, and it all integrates together. Now like I said, this is a very minimal implementation of WebXR. For a more full-featured implementation, other developers have done a great job with that. Deep Panther has an excellent WebXR package available at his GitHub that you can use to render with virtual reality and augmented reality modes of WebXR, a much more full feature. I highly recommend that. And that's all I have to share with you today. Feel free to reach out for any questions you might have. I'm happy to answer, and thank you very much for taking the time. Awesome. So we're taking a look at the results of the poll question here we asked before the talk. We asked, well, what technology would you like to see supported by Unity? And it looks like webgpu is the overwhelming majority. Yeah. A little surprised by that, but not really, because that's the hot new thing. I was actually surprised there wasn't. We're actively working on. Oh, that's good to hear. I think that a lot of people are happy to hear that then, that you're actively working on that. I was surprised that there wasn't more requests for basis, because I know that helps a lot, especially on the web. And I know a couple of other speakers, either today or yesterday, have mentioned. Because it helps you both in terms of the download size, right, to load time, but also in terms of using less memory. And that's big for a lot of mobile devices, right? Yeah, I agree. We added support for mobile. We've had desktop texture compression for a long time. And then in 2021, we added support for mobile texture compression formats. But you kind of have to make a design decision and use different asset bundles for which format you want. And if you had a basis, then that kind of puts less technical burden on the developer and lets you just use one format for other places. Absolutely. And this is something I'm not personally familiar with, because most of the stuff I've published has been on the web, but I'm curious if you were publishing without basis on desktop or otherwise, would you – because you have all these different compressed texture formats on different GPUs, et cetera. Do you just have these different bundles and you just ship them depending on what the device has support for? Yeah, you have different bundles. And then at runtime, you can, depending on your environment, pick the mobile bundle or the desktop bundle. You can have bundles for high-fidelity assets and lower-quality assets. You can make those choices. It adds some technical burden or whatever to make those choices at runtime, but the tools are there to do it. Makes sense. Cool. I think we can take a look at some of the questions that the audience has been submitting here. The first question I'm looking at here, Rick asks, is there a roadmap for supporting WebXR natively in the future? We have the roadmap. I put the link on the poll question, where you can add votes to things or add your own suggestions. And it's definitely something that we're all aware of and excited about. It's really, even though it's been around for a long time still, a technology that's not supported everywhere, WebKit is engaged in it now and will be bringing it up, but it's not available yet, and other places like that. So it hasn't boiled up in priority for development, because there's only so many people we have to work on things and significantly more things to work on than we have time. But it's definitely something that we're aware of and excited about and really want to get in there, because it has a lot of benefit for a lot of different use cases. And I'm also here to work with you, Arsene. I think this is the first time I was aware of that Unity had a public roadmap where people can vote. How long have you had that? I'm not entirely sure. Okay, but it's not like a new thing like this month or something. It's been there for a little longer. Yeah, it's been up for a while. Like a lot of things, discoverability is always an issue. Putting it up, making it not super obvious of where to find it. Yeah, we should definitely share that link. Yeah, we should definitely share that in Discord and maybe tweet about it too so other people, everyone can make sure to see that. Cool. I think that's really important for prioritization and things like that. Yeah, absolutely. And it's cool the community can get to voice that in this very direct way. So I think that's cool. We have another question here from Daniel. What is the current progress of mobile support for the web export? So we started working towards web mobile back in, I think more seriously in 2021.2 release of Unity. And we had a banner on the page saying that mobile wasn't supported. In 2022, we removed that banner. And I think 2023, we'll have put in all the pieces. In 2022, I added the mobile keyboard support. And there's just a lot of different things that go into mobile web that you need to take care of that desktop doesn't need to deal with. So we've always, you know, any issues that came in that were mobile-related, we always took them very seriously and continue to do that and with more emphasis now. And we always had this notion that mobile was not supported because there's just a lot of missing things. But it's always been taken seriously. And now we have mobile texture compression supports and all these other features that are going into it that will have the list of supported, you know, browsers for mobile in the upcoming version. That's very cool. It sounds very optimistic, I think. Yeah. Developing for the web is certainly, I'm super excited about it. It's definitely a challenge because you're working on a shifting SAN operating system that's constantly changing under you. So we had a lot of issues with iOS 15.4 that shipped recently. They kind of broke everything. So it's web-related, webgl-related, and webassembly-related for us. We work very closely with Apple and Google and all the web developers to make sure that not only for us the web is a great platform, but for everybody because, you know, making the web work for us, you know, and making the web work for everybody else is equally beneficial. Yeah, that's awesome. And I find this really inspiring because I feel like you mentioned you're working on this shifting SAN, but also part of it is that the platform is changing, but also the Unity engine is evolving and changing it so quickly. And so it just seems insurmountable to get something like that working well on the web, but I think in the process of doing that, everyone's better off for it. It's like, what did they say, the rising tide that lifts all boats? That's kind of how I see this. Yeah, I'm fully on board with that. That's awesome. We have another question here. It says, you mentioned some internal Emscripten calls like din call, dine call, and heap. Where can we see what the equivalence to the variables like vi and the other basic types in those calls and just find more info about those internal calls? So these are things that are coming from Emscripten. I think we need to do a better job of documenting from our perspective. You can find this information from the Emscripten documentation, but it's all, you know, it's like discoverability is an issue. How do you find the information? So it's definitely something that we need to work on documentation wise for, you know, extending Unities, you know, with plugin architectures and stuff like that from the web perspective, because we have a lot of documentation, but the web has always been kind of a niche platform, and so the documentation has been skewed towards the other platforms. But I think the shift is being made where, and we've been really pushing on it, to make web a first class citizen platform and to make that documentation and the tooling much better for the web platform. That's awesome. Our next question is from George. The Unity instance send message function is also in the Unity manual. In what cases would you recommend using it? Very specific question that is kind of throwing my brain off topic. We can also get back to it. I don't know the answer to this right now that I can answer in a couple of seconds. Yeah, maybe on the Discord you can post that. Yeah, exactly, in a time where I can think more clearly. Sure, no worries. What about, this is a question from me. What other web functionality examples do you think of could be integrated into Unity with a javascript plugin? Well, there's a lot of different web APIs out there that have varying levels of support for the browsers like Web Transport and WebRTC and things like that that we don't have direct APIs for in the C-sharp layer of Unity. But you could definitely do this kind of similar technique to expose it to Unity and add it to your game. So you could do WebRTC or one of our developers did a prototype using Web Transport that was quite successful. So there's a lot. Anything that you can do with a web api really opens that up. That makes a lot of sense. I was curious, if I were publishing something with Unity webgl, where do you see most people hosting their Unity webgl export games or interactive content? Like any web project, you can host it on your own server or Unity has services like play.unity.com that can host things for you. When you set up your own server, you can be much more specialized for your game and use Brotli Compression or GSIP and set up your server to be correct for that. But yeah, there's no limitations for how you can host it. When you build your game, you can specify the html template and scaffolding that goes around it, and so you can customize it for whatever your target's going to be and whatever technical needs that you have. That's very cool. It sounds like you could publish it really anywhere. You could publish any HTML5 game, javascript game like Newgrounds or Itch or really any of those game jolts, any platform like that. Yeah, exactly, and a lot of different platforms and servers do host Unity games, so there's not really a technical limitation for that. In some cases, it does require more technical savvy to set things up correctly, but it's all about improving tooling and documentation to simplify those things, and those are all things that we're actively working toward. Awesome. And what about, I remember vaguely, a project called Unity Project Tiny. Can you talk a little bit about that and what that is? Sure. Project Tiny, we had to have a technology called DOTS, data-Oriented Technology Stack, and Project Tiny was an offshoot of that, which let you build up games much more modularly, and so you don't have the giant monolithic engine, but you pick pieces that you want so you can make very small games. That was put on hold temporarily while the developers focused on the rendering side of the DOTS technology stack, but it is something that we will be investing more into in the near future. And so that will drastically improve things like game size and bundle size and load times and all these other things. What about, as a final question, are there any features of Unity that you're working on or that you're excited about that you can share with us? Well, the one thing that everybody is interested in, apparently, is webgpu, and that is something that I am actively working on and developing, and we have an implementation of that. We'll not go public in the timeline for that, but it is in the works. That's awesome. Have you found that hard? Because I know the spec for webgpu has been changing in the last few months or so. Has that been a factor? It's the developing on quicksand thing. But I've been very involved with the standards Committee too to help ensure that webgpu has what Unity needs and also can benefit other people and to try and make sure that the spec for that has the APIs that we need and other people need as well. That's awesome. And also, actively, the size of the Unity projects we've been building with webgpu are much larger than a small hand-coded javascript test, and so we're pushing things like the shader compilation and things like that and finding places where it still needs some work and helping uncover bugs and things like that. Yeah, I love that. That makes me really hopeful for the future of the web. I think having folks like you involved with the standards Committee and ensures that, again, this rising tide lifts all boats, the platform evolves for the benefit of everybody. This was really great. Thank you so much for being here, Brandon, and thanks for answering all our questions. Thank you so much. Thank you very much. ♪♪♪