JS Nation

JSNation 2022

Makepad - Leveraging Rust + Wasm + WebGL to Build Amazing Cross-platform Applications

All right, welcome to my talk. I'm going to talk about Makepad. It's a new way to build UI for web and native using a WebAssembly and Rust. So I want to start this journey long, long ago, maybe before some of you were still in primary school even. I founded Cloud9 IDE. It's a company to build a web IDE. We had a HTML-based code editor called Ace. It was one of the two at the time. And so I was in the middle of HTML world building what was considered a complex application, an IDE. Everyone was making web pages, some web apps, but the IDE with the code editor was considered a complex web application. So at the time, I was working on the editor. And I remember we were working on code folding. And I wanted the code folding to animate. It's a very simple thing. You know, computers are fast. They can animate it. But no matter what I tried, I couldn't get it done with HTML and CSS. Whatever we did, it just was choppy and it sucked. So at the time, WebGL was just released. And I started investigating using WebGL to render UI, render editors. And I realized that this was a huge task ahead of me to reinvent the UI in WebGL. So I left Cloud9. And I started on the journey of redoing the entire render stack of the browser in WebGL and JavaScript. And I spent six years on that. I probably built as many, six or seven, complete file new iterations of trying to do this. And I want to show you the result of those six years here. And that is MakePad JavaScript. This is MakePad JavaScript. It is a complete do-over of the render stack in WebGL. And you can do, let's see, you can live code. All of this is live codable. So let's see if I can change the color of the text here. I can also, let's see, I have a bit of feedback on the audio. Yeah. So it was really fast. It was really, really fast. This is 50,000 interactive circles in a browser. It was using multi-threading. So this application running here was running in a worker thread and sent over the display list to the main browser thread. So you could have any number of applications running at the same time as long as you had workers. And you could do very, very cool interactive things. This all runs also on an iPhone 6, I think, still, if you load it up in Safari. And at some point, I even managed to run itself in itself. And this is great. This is all fine. But it had a big problem. It had a big problem. At the time in Chrome, when I was working in the application for more than 30 minutes, Chrome would just crash. It would just crash, and I didn't know why. And having dealt with browser bugs before, sometimes they take years. Sometimes they never get resolved, especially when you're like, yeah, I live code in Chrome for half an hour, and I hot reload a worker. That's a bug. They go, OK, you're probably the only person in the world that does that. So good luck with that. So that was very depressing. And hold on. So it wasn't stable. That was really a problem, because I wanted to use it myself. If I can't use my own developer tooling for my own stuff, then the developer tooling sucks. However, there was a real problem, because maybe I could have gotten this fixed. I could have fixed it myself or lobbied forever to get someone else to do it. But the real problem is that you don't have CPU power left if you do all of this in JavaScript. A worker thread just caps at 100% if you use the editor, the syntax highlighting. All of that stuff is just too heavy to do in real time with JavaScript. And that meant that I was looking ahead. I see AR, VR user interfaces. I see high frame rate screens. The latest max are 120 hertz. So you don't have that 16 milliseconds anymore. You're down to eight or lower. So it's a dead end. It's a dead end. The JavaScript was wrong. Essentially, it's the wrong technology to try to do this, especially because I wanted to build complex applications like IDEs, design tools, 3D modelers. These kind of tools I wanted to build on the web. But if you try to do it in JavaScript, just the data structures that you need to build anything complicated will drag down or hiccup your application to no end. It's sort of a limited domain. And dynamic typing also doesn't scale. TypeScript sort of alleviates that. But with JavaScript, if you scale your code base and you try to build on things that you did before, at some point it's going to start to fall apart. So this was six years in. Was this the end? Did my life have no purpose? It's a bit much to sort of throw away your work at this time, even though it looked great. It looked nice. So yeah. And then Rust came along. Rust is a relatively new programming language developed out of Mozilla. It's a borrow check language. It doesn't have a GC. This is a very fundamental difference, which is also why it takes time to learn. You can't just randomly create objects and magically have them go away. This is also why it's fast. It doesn't have GC hiccups. It's compiled native, and it compiles to WebAssembly. And it's both high and low level, which is different because JavaScript is kind of a mid level language. It's not very high level, but it's also not low level. And that is very limiting because sometimes you want really high level constructs, like high performance iterators or kind of trade systems. And sometimes you want low level because you need the performance. So I had a new chance at trying this out with a new technology. And I joined forces with Eddie, who is an ex-Mozilla developer who was there when Rust got invented. So he knew Rust, and he could teach it to me as well. And Sebastian, who is one of the most brilliant UI designers I know. And because if you're building UI, you better try to make it look good. And I just play games. So make that Rust. This is a complete reimagining. So first off, this is make that JavaScript, and here's make that Rust. It's the same idea. It has a code editor. And now I could finally do animated code folding. This is a feature that shouldn't be hard. But considering that you're clapping, people do consider it hard. Because this is what I wanted. I wanted to have an overview of my code that just folds back. And I keep my spatial memory. I don't want it to code to fold to nothing. I want to see that there's body of work there. And you can just skip your cursor around. And it also still has the live editing. Like here's the live thing going on with the changing code. I'll tell a little bit how that works later. But it even worked in VR until the web APIs changed a bit. And I just didn't feel like updating it. But I'll fix that later. This whole UI runs in AR, VR just fine. You can position it anywhere in 3D space. It has depth in the UI, all that stuff. And I finally got it. And also, this is my own desktop version of the build. I use it every day. This is the same code base. You just compile it as a native desktop application that runs directly on Metal on Mac, directly on DirectX on Windows, OpenGL on Linux. And yeah, no browser sandbox, no Electron bullshit. This binary is one megabyte in a zip file. We're back to 2,000 in terms of file size. So could this be it? Could this be finally what I wanted, a hyper-fast application stack that works on native and on web? So one of the things that I said is that I didn't have CPU time left, right? So let me show you Chrome trace, a Chrome profile. So it's on performance. I'm going to record. Do this. Stop. So as you can see, I can't even see my CPU code almost. It's like half a millisecond to do a color pick in this UI, as opposed to this is the, let's see. Let's see if I can get the same. No. I mean, the tech stack is completely different, right? I'm profiling functionality here. Because that is what, in the end, matters. It is not about comparing JavaScript versus WebAssembly in an inner loop. It's about comparing the whole stack. So here we go. I'm going to color pick this. And I'm going to see. Yeah, so this is what the, I don't know, it's a bit small, but this is what the JavaScript profile looked like. And these slash things are my worker thread. And now this is an M1 Mac, so it's gotten faster since I abandoned this project. So now it's only half. But back in the day, this was capping at 100% CPU. So yeah, I finally got to the point where I can do useful things with my UI, instead of just doing the UI. So what is it that we built? It's a UI stack. So it's a widget set. It has a React-like render model. It is its own IDE and code editor built on top of that stack. And our final piece to build this into something useful that I can offer to you all is going to be a design tool. Because hand coding UI is something that I would love to delegate to the past. You want to use a proper visual designer. So this whole system is designed to support a visual designer. And that means that we have a DSL. And a DSL is a domain-specific language. HTML is a DSL. CSS is a DSL. We have to have something like that as well, because I don't want to do it all in Rust. If you build a visual design tool, the visual design tool manipulates the data structure that is your UI. And in this case, that data structure is something custom that has properties of HTML, CSS. It has inheritance. It's actually my attempt to finally fix the crap. But we'll see how far that went. Here's, for instance, the color picker in line now. This kind of stuff where you have a code editor that's extendable with all sorts of visual editors is one of the reasons I want a fast render model. Like the code folding just folds everything with it still as well. But why? Originally, my original motivation of getting into programming was having fun with computers. I wanted to make it draw graphics, make sounds, do fun things, and not fight with webpack or I don't know what kind of deferred problem you have as a web developer. And that's why I want to show you our first example application, because it's fun that the IDE is an example. But it's a bit of a domain-specific example. And that's why I built FunAudio. And FunAudio is a post-to-logic audio. It's an audio application that uses our audio API on Mac. However, I have accessed, because in Rust, you can access all native APIs. I accessed the audio unit API. That means that I can load native instrument. I can load native instrument components straight into the UI. This will also build to web, but then you don't get all these nice features, because this is what the web sandbox limits you to. You cannot do this. There's just no way. And with Rust, you get the power to be limited on the web, but get to deploy and the power on desktop. And that is why I get excited by this tech stack. It's a very different approach to putting the web on desktop. It's putting the desktop on web. About four minutes left. So what is the future? When can you guys use this? We're having an open source release to crates, which is NPM for Rust, so that you can very easily install the stack in about three months, including some documentation. And we're ramping up to release the design tool ASAP. And for that, you can open makepad.dev. That was the web thing that you just saw. You can open it right now on your phone. This is the GitHub repo. Everything that you've seen so far is MIT in the GitHub repo. And Twitter for myself and the product. So I have about four minutes left. Let's see if I can add some interesting bits. So yeah, makepad on web, right? So if you deploy these kind of applications to web, size matters, right? You don't want it to be 16 megabytes over the wire. That's why I'm very excited that this thing, as you see it here, is 500 kilobytes over the wire, including all dependencies. So I mean, that's broadly compressed. But still, 500 kilobytes to deploy this whole thing, and it loads in under a second time to interactive. So it's very, very viable to do things this way. It's not like compiling a ginormous amount of code to the web, as a lot of C++ applications are doing. And similarly, if you look to the future of AR and VR, this demo had a collaborative VR. So you could put your quest on, open this URL, and you'd be collaboratively in this IDE space whilst live editing an object in between. And I was very impressed with that as well for web VR, because I was like, ah, it's going to be janky, right? Because VR is so performance sensitive, if your garbage collector hiccups up, or you're going to get sick, because the frame is going to start to judder. But if you're super disciplined about it, and you use WebAssembly only, I could get it completely rock solid on web VR. And this is also when that one millisecond of CPU time and doing stuff really starts to matter, because this is a millisecond of CPU time on an M1 Max, not on a crappy Android chip that is in a VR headset. So that one millisecond that I showed you, you should multiply by 10 to have your full application domain for other devices. And so don't think that I'm over-optimizing that one millisecond. It's actually the space you have to reach your entire audience. So yeah, I think that is the end of my talk. I think we can move to questions. Thank you. I really love the question that is, oh, did it go away now? Ah, yeah. So do you get to work on this full time, or is there a sponsor or company behind? So for the first many years, I did consulting, and I did this as an iterative cycle every time, because I restarted six times. More recently, when I started working with my co-founders, we have an angel-funded startup situation. But it's, yes, full time. Full time. Wow, that's really cool. Any plans to donate it to a foundation at one point? Absolutely. I mean, this is an open source stack, right? The only thing that we're going to commercialize is the design tool that sits on top of it. All of the libraries, all the dependencies are going to be unencumbered MIT, so that everything you build with this, you can pick any license you want. So as this becomes popular and successful, hopefully, then we can definitely use one of those constructs that many open source projects use. Super, very interesting. I hope that answers the question from the person who asked. Is it also possible to import 3D models or textures from Blender, for instance? That's definitely a higher level tool. Yes, absolutely. But someone has to write the importer. It's a custom render stack with a shader model that is unified across all platforms. So it's not GLSL. It's something that looks like GLSL, semantically GLSL, but it cross compiles to all the backends, Metal, HLSL, and WebGL, and in the future, also WSL. So it's a custom engine. It's geared for UI, but it can technically do all the things you could do in 3JS. But we're going to need to develop the tool chains around that. Very cool, thank you. Did you start with building an MVC for the Rust implementation to know if it would really outperform JavaScript? Or was this just sort of fingers crossed, hoping that it would be better? This was fingers crossed. Essentially, this is not something I can benchmark dry very much. I just had to build it. And as we got further, I started getting more excited about half a millisecond of CPU time doing anything. And this means that the amount of bandwidth I have for this UI, I could render. The startup time for this IDE, in terms of spawning up the UI, is a millisecond. Like, I could draw tens of these things at the same time on your phone, and it wouldn't matter. So yeah, it is definitely something I hoped. But I would say it's better than I thought. Awesome. I see your question whether it will only continue to run in the browser. You did show the desktop version that you used yourself. Is that going to be available too? The idea is that we want the power of both. You want to easily deploy to web. And if you're constrained, use the desktop compile. And maybe he means mobile, or they mean mobile. Mobile support will come later, because I think focusing on the web deploy target first is, I think, a good start. Because it's a very big space, especially because we're bridging the APIs to Rust. That's a lot of work. But mobile will come later. Very cool. Do you see a future for Rust-based UIs? Do you think it will be? I think it's absolute garbage. No. That was a very unexpected answer. All right. Thank you very much. I think that's about the time that we have. And we're going to move to lightning talks. Thank you so much for your time. All right. Thank you. Woo! Thank you.


Rik Arends

JS GameDev Summit

JS GameDev Summit 2022

Unreal Engine in WebAssembly/WebGPU

Alex St. Louis

Extending Unity WebGL With Javascript

Brendan Duncan

What Can you Do with WebGPU?

Hi everyone, and thanks for joining me on this What Can You Do With WebGPU session? My name is Omar. I work as a graphics engineer at Snapchat. I'm currently based in Ithaca, New York, and I used to make Flash games back in the day. So I want to talk about WebGPU, what it is, what its current state, and when it will be widely available. And I also want to show a few of the things that are new in WebGPU that you can do with it that you couldn't do today. And I want to share a bunch of links and resources to help you, point you in the right direction. And I'll share a link to these slides, because they have a lot of links, so you'll have those available. So, what is WebGPU? It's an upcoming web API for low-level access to GPU. It's basically the next step of the successor to WebGL. So kind of at a glance, the big difference is that WebGPU gives you more direct access to the GPU. So unlike WebGL, WebGL was more of like an abstraction around OpenGL. It wasn't so much about how the hardware works. WebGPU is more of a modern graphics API, so similar to how Vulkan, Metal, and DirectX12 work, they give you control over what the hardware can do more directly. And so generally what that means for us building applications and games with it is it means better performance. So you can use, you can also use new techniques that otherwise weren't available, which I'll talk a bit more about this. It also kind of makes it more future-proof, because it means as the hardware does more things like ray tracing, those things can be exposed directly instead of having to create kind of a whole new API layer wrapper around that for developers to use. So this is very exciting. So the current state is you can use it. You can enable it in Chrome or Firefox behind a flag. It can also be used through an origin trial, which means you sign up for a token and then your users can, WebGPU will be enabled for those users on your URL or domain, so they don't have to flip a flag. And it'll be widely available for everyone in Chrome in September. Some of the engines that support it, Babylon was the most mature that I've seen. And all of these link to pages that explain how to turn on WebGPU in these engines. 3GS, I think it has a lot of support, but it's still not officially released as part of it, but you can still use it today. And then Play Canvas is currently in development for WebGPU support. And so compute shaders are kind of the most significant new feature. And what that means is you have the ability to essentially write like general, like GPGPU, general computation on a GPU, instead of before, if you wanted to run any computation on, you know, on the GPU on the web, you'd have to run it through a fragment shader and you have to run the results on a texture and then something else would have to read that texture. And so that was kind of a hassle. So this is just easier, but also it could be much faster because doing this gives you much more control over memory synchronization and threading here. So one example here is rendering point clouds. So point clouds are just instead of a 3D mesh made of triangles, a point cloud is just made of millions or sometimes billions of points. And this is a paper that's talking about getting significantly faster rendering using a compute shader to render instead of traditional pipeline. So you can see in the bottom left here, like with the vertex and frag shader, you get 10 FPS on this scene and with the compute, you get like 300 FPS. That's a huge, obviously, difference. And basically what they're doing here is they're skipping the vertex and frag pipeline, their compute shader takes in the vertices as a buffer, and then it writes the pixels to another, the color data to a buffer, and then that then gets rendered to the screen with a frag shader. And one cool kind of technique that you can do here that you couldn't otherwise do, like with an aggregate frag shader, is because they have a lot of points that render to the same point, a lot of points that render to the same pixel, instead of just showing the most closest one to the camera, they average the colors to just make it, in a way it kind of shows you what's inside if you have a complex point cloud that describes a volume. And so they kind of average all the points. And this is something that we can do because we have full control over how the threads synchronize when they're all writing to the same pixel, but not something we have a lot of control over in a frag shader. And the other reason it's really fast is because generally if you're writing very small things like points or very small quads, a compute rasterizer approach is going to be much faster, and this is actually something that Unreal Engine 5's Nanite uses to speed up rendering like really tiny triangles. And I also wrote this, how to build a compute rasterizer of WebGPU if you're curious how this works, and it's a good way to get started with learning both WebGPU, but also with this technique, how it works and why it can be faster. There's a few kind of teaser here, videos, just to show you how much control we have once we build something like this. Here and it may not be very clear on the screen here, but what I'm doing is I'm changing between smooth shading and flat shading on individual triangles as the model moves back and forth, which normally this is something you can change on the whole model, but because in this compute rasterizer we control the full pipeline, so I can change individual triangles from smooth to flat dynamically, or one cool effect you could even have it so as you mouse over it or as you touch different parts of the model, they turn from smooth to flat, which creates like a really unique effect I think. And this is another one kind of visualizing the order in which triangles are drawn, which again isn't really something you can normally do, but with a compute pipeline, because we have full control over it, we can kind of just see that. And more commonly you'll have like particle simulation, so again, this is something today maybe you'll do on the CPU, or if you do it on GPUs through a frag shader, but here you can do it on a compute shader directly, which will be easier but potentially faster as well. And finally, this one is really exciting because there's a lot of games that use these kind of techniques, not just for speed, but also to create very unique effects like here, this is a game called Claybook, which everything is rendered assigned distance fields, which makes everything deformable in real time, which is really cool and very unique. And as of today you can't really do this on the web, but with WebGPU, stuff like this will be possible, which I'm very excited about. And there's a link to the talk where they talk about Claybook and how it was made using these techniques here, not using WebGPU but the general compute technique. So to summarize, what WebGPU promises is mainly better performance, but also the ability to kind of explore new rendering techniques, and maybe in the future, not today, not supported today, but in the future, something like support for ray tracing may come to the browser, which would be huge. Thank you so much. And here's links and resources. The big one I want to point out is the WebGPU official GitHub forum. This is where they write the specification and the developers working on it are very nice by answering questions and they're available to answer things. So it's a good place to learn. I've learned a lot from that. And I've also linked the tutorial I wrote about creating a compute rasterizer here. Thank you so much.


Omar Shehata

#webgpu