Makepad - Leveraging Rust + Wasm + WebGL to Build Amazing Cross-platform Applications

Rate this content
Bookmark

In this talk I will show Makepad, a new UI stack that uses Rust, Wasm, and WebGL. Unlike other UI stacks, which use a hybrid approach, all rendering in Makepad takes place on the GPU. This allows for highly polished and visually impressive applications that have not been possible on the web so far. Because Makepad uses Rust, applications run both natively and on the Web via wasm. Makepad applications can be very small, on the order of just a few hundred kilobytes for wasm, to a few megabytes with native. Our goal is to develop Makepad into the UI stack of choice for lightweight and performant cross-platform applications. We intend to ship with our own design application and IDE.

22 min
16 Jun, 2022

AI Generated Video Summary

Welcome to MakePad, a new way to build UI for web and native using WebAssembly and Rust. JavaScript is not suitable for complex applications like IDEs and design tools. Rust, a new programming language, was used to reimagine MakePad, resulting in a fast and efficient platform. MakePad offers live editing, high CPU performance, and the ability to load native instrument components. The future of MakePad includes an open-source release, a design tool, and support for importing 3D models.

1. Introduction to MakePad

Short description:

Welcome to my talk about MakePad, a new way to build UI for web and native using WebAssembly and Rust. I founded cloud9 IDE, a company to build a web IDE. I spent six years redoing the entire render stack of the browser in WebGL and JavaScript. The result is Makepad JavaScript, a complete do-over of the render stack in WebGL, allowing for live coding and high-performance interactive applications.

All right, welcome to my talk. I'm going to talk about MakePad. It's a new way to build UI for web and native using WebAssembly and Rust.

So, I want to start this journey long, long ago, maybe before some of you were still in primary school even. I founded cloud9 IDE, it's a company to build a web IDE. We had an HTML based code editor called Ace, it was one of the two at the time. And, so I was in the middle of the HTML world building what was considered a complex application, an IDE, right? Everyone was making web pages, some web apps, but the IDE with the code editor was considered a complex web application.

So, at the time I was working on the editor and I remember we were working on code folding and I wanted the code folding to animate. It's a very simple thing, you know, computers are fast, they can animate it, but no matter what I tried I couldn't get it done with HTML and CSS. It just, whatever we did, it just was choppy and it sucked. So at the time WebGL was just released and I started investigating using WebGL to render UI, render editors. And I realized that this was a huge task ahead of me to reinvent the UI in WebGL. So, I left Cloud9 and I started on the journey of redoing the entire render stack of the browser in WebGL and JavaScript.

And I spent six years on that. I probably built as many, six or seven complete file new iterations of trying to do this. And I want to show you the result of those six years here. And that is Makepad JavaScript. This is Makepad JavaScript. It is a complete do-over of the render stack in WebGL. And you can do, let's see, you can live code. All of this is live codable. So let's see if I can change the color of the text here. I can also, let's see, I have a bit of feedback on the audio. Yeah. So it was really fast. You could have, this is 50,000 interactive circles in a browser. It was using multi-threading. So this application running here was running in a worker thread and sent over the display list to the main browser thread. So you could have any number of applications running at the same time, as long as you had workers. And you could do very, very cool interactive things.

2. Challenges with Stability and CPU Power

Short description:

This all runs on an iPhone 6, and at some point, I even managed to run itself in itself. However, there was a big problem. When working in the application for more than 30 minutes, Chrome would crash. It was unstable, and I couldn't use my own developer tooling. The real problem is that JavaScript doesn't have enough CPU power for all the tasks. This is a dead end.

This all runs also on an iPhone 6, I think, still, if you load it up in Safari. And at some point, I even managed to run itself in itself. And this is great. This is all fine.

But it had a big problem. It had a big problem. At the time, in Chrome, when I was working in the application for more than 30 minutes, Chrome would just crash. It would just crash, and I didn't know why. And having dealt with browser bugs before, sometimes they take years. Sometimes they never get resolved. Usually when you're like, yeah, I LiveCode in Chrome for half an hour, and I hot reload a worker, that's a bug. They go, OK, you're probably the only person in the world that does that. So good luck with that. So that was very depressing. And hold on.

So it wasn't stable. That was really a problem because I wanted to use it myself. If I can't use my own developer tooling for my own stuff, then the developer tooling sucks. However, there was a real problem, because maybe I could have gotten this fixed. I could have fixed it myself or lobbied forever to get someone else to do it. But the real problem is that you don't have CPU power left if you do all of this in JavaScript. Worker thread caps at 100% if you use the editor, the syntax highlighting. All of that stuff is just too heavy to do in real time with JavaScript. And that meant that I was looking ahead. I see AR, VR, user interfaces. I see high frame rate screens. The latest max are 120 hertz. So you don't have that 16 milliseconds anymore. You're down to 8 or lower. So it's a dead end.

3. Challenges with JavaScript

Short description:

JavaScript was the wrong technology for building complex applications like IDEs and design tools on the web. It lacks the necessary data structures and scalability, leading to performance issues and limitations. TypeScript helps to some extent, but at some point, the code base will start to fall apart. Despite the initial appeal, it was time to explore new technologies.

It's a dead end. The JavaScript was wrong. Essentially, it's the wrong technology to try to do this, especially because I wanted to build complex applications like IDEs, design tools, 3D modelers. These kind of tools I wanted to build on the web. But if you try to do that in JavaScript, just the data structures that you need to build anything complicated will drag down or hiccup your application to no end. It's sort of a limited domain. And dynamic typing also doesn't scale. TypeScript sort of alleviates that. But with JavaScript, if you scale your code base and you try to build on things that you did before, at some point, it's going to start to fall apart. So this was six years in. Was this the end? Did my life have no purpose? It's a bit much to sort of throw away your work at this time, even though it looked great, right? It looked nice.

4. Introduction to Rust and Reimagining Makepad

Short description:

Rust is a new programming language developed out of Mozilla. It's a borrow check language that doesn't have a GC, making it fast and without hiccups. It compiles to WebAssembly and is both high and low level. The speaker joined forces with developers experienced in Rust and UI design to reimagine makepad. The new version includes a code editor and animated code folding for better code overview and spatial memory.

So yeah. And then Rust came along. Rust is a relatively new programming language developed out of Mozilla. It's a borrow check language. It doesn't have a GC. This is a very fundamental difference, which is also why it takes time to learn. You can't just randomly create objects and magically have them go away. This is also why it's fast. It doesn't have GC hiccups.

It's compiled native and it compiles to WebAssembly. And it's both high and low level, which is different, because JavaScript is kind of a mid-level language. It's not very high level, but it's also not low level. And that is very limiting, because you sometimes you want really high level constructs like, high performance iterators or kind of trade systems. And sometimes you want low level, because you need the performance.

So I had a new chance at trying this out with a new technology. And I joined forces with Eddie, who is an Exmozilla developer, who was there when Rust got invented. So he knew Rust and he could teach it to me as well. And Sebastian, who is one of the most brilliant UI designers I know. And because, you know, if you're building UI, you better try to make it look good. And I just play games. So makepad Rust, this is a complete reimagining.

So first off, this is makepad JavaScript and here's makepad Rust. It's the same idea, it has a code editor. And now I could finally do animated code folding. This is a feature that shouldn't be hard. But, you know, considering that you're clapping, people do consider it hard. Because this is what I wanted. I wanted to have an overview of my code that just folds back and I keep my spatial memory, right? I don't want the code to fold to nothing. I want to see the body of work there. You can just skip your cursor around.

5. MakePad's Live Editing and CPU Performance

Short description:

MakePad has live editing and works in AR and VR. It can be compiled as a native desktop application without browser sandbox or Electron. The binary file size is one megabyte. The speaker demonstrates the difference in CPU performance between MakePad and JavaScript by using Chrome trace and profiling functionality.

And it also still has the live editing. Like here's the live thing going on with changing the code. I'll tell a little bit how that works later. But it even worked in VR until the web APIs changed a bit and I just didn't feel like updating it. But I'll fix that later.

This whole UI runs in AR and VR just fine. You can position it anywhere in 3D space, it has depth in the UI, all that stuff. And I finally got it. This is my own desktop version of the build, I use it every day. It's the same code base, you just compile it as a native desktop application that runs directly on Metal on Mac, directly on DirectX on Windows, OpenGL on Linux. And yeah, no browser sandbox, no Electron bullshit. This binary is one megabyte in a zip file. We're back to 2,000 in terms of file size. So could this be it? Could this be finally what I wanted? You know, a hyper fast application stack that works on native and on web.

So one of the things that I said is that I didn't have CPU time left, right? So let me show you Chrome trace, a Chrome profile. So it's on performance. I'm going to record. Do this. Stop. So, as you can see, I can't even see my CPU code almost. It's like half a millisecond to do a color pick in this UI. As opposed to, this is the, let's see, let's see if I can get the same. No. The text stack is completely different, right? I'm comparing, I'm profiling functionality here. So, because that is what, in the end, matters. It is not about comparing JavaScript versus WebAssembly in an inner loop. It's about comparing the whole stack. So, here we go. I'm going to color pick this and I'm going to see. Yeah.

6. MakePad's UIStack, Design Tool, and FunAudio

Short description:

This is what the JavaScript profile looked like on my M1 Mac, capping at 100% CPU. We built a UIStack, a widget set with a React-like render model. It includes its own IDE and code editor. Our final piece will be a design tool to support a visual designer. The data structure of the UI has properties of HTML, CSS, and inheritance. Our first example application is FunAudio, an audio application that uses our audio API on Mac.

So, this is what the, I don't know, it's a bit small, but this is what the JavaScript profile looked like. And the slash things are my worker thread. And now this is a M1 Mac, so, you know, it's gotten faster since I abandoned this project, so now it's only half. But back in the day, this was capping at 100% CPU. So, yeah.

I finally got to the point where I can do useful things with my UI instead of just doing the UI. So what is it that we built? It's a UIStack, so it's a widget set. It has a React-like render model. It is its own IDE and code editor built on top of that stack. And our final piece to build this into something useful that I can offer to you all is going to be a design tool. Because, you know, hand coding UI is something that I would love to delegate to the past. You want to use a proper visual designer. So this whole system is designed to support a visual designer. And that means that we have a DSL, and a DSL is a domain specific language. HTML is a DSL, CSS is a DSL. We have to have something like that as well because I don't want to do it all in Rust. If you build a visual design tool, the visual design tool manipulates the data structure that is your UI. And in this case, the data structure is something custom, that has properties of HTML, CSS, it has inheritance. It's actually my attempt to finally fix the crap. But we'll see how far that went. Here's for instance the color picker in line now. This kind of stuff, where you have a code editor that's extendable with all sorts of visual editors, is one of the reasons I want a fast render model. The code folding just folds everything with it still as well.

But why? Originally, my original motivation of getting into programming was having fun with computers. I wanted to make it draw graphics, make sounds, do fun things, and not fight with Webpack or I don't know what kind of deferred problem you have as a web developer. And that's why I want to show you our first example application because it's fun that the IDE is an example, but it's a bit of a domain-specific example. And that's why I built FunAudio. And FunAudio as opposed to LogicAudio, it's an audio application that uses our audio API on Mac. I have accessed, because in Rust you can access all native APIs, I access the audio unit API. That means that I can load native instrument...

7. MakePad's Future and Performance

Short description:

I can load native instrument components into the UI. Rust allows for limited web deployment and powerful desktop performance. The future includes an open source release in three months and the upcoming release of a design tool. MakePad on the web is lightweight and loads quickly. The demo showed collaboration in WebVR, which performed well with WebAssembly. The one millisecond of CPU time shown is significant for performance on various devices.

I can load native instrument components straight into the UI. This will also build to web, but then you don't get all these nice features because this is what the web sandbox limits you to. You cannot do this. There's just no way. And with Rust, you get the power to be limited on the web, but get to deploy, and the power on desktop. And that is why I get excited by this tech stack. It's a very different approach to putting the web on desktop, it's putting the desktop on web.

Okay. About four minutes left. So what is the future? When can you guys use this? We're having an open source release to crates, which is NPM for Rust, so that you can very easily install the stack in about three months, including some documentation. And we're ramping up to release the design tool ASAP. And for that, I... You can open makepad.dev, that was the web thing that you just saw, you can open it right now on your phone. This is the GitHub repo, everything that you've seen so far is MIT in the GitHub repo. And Twitter, for myself and the product.

So yeah. So yeah, makePad on web, right? If you deploy these kind of applications to web, size matters. You don't want it to be 16 megabytes over the wire. That's why I'm very excited that this thing, as you see it here, is 500 kilobytes over the wire, including all dependencies. So I mean, that's broadly compressed. But still, 500 kilobytes to deploy this whole thing, and it loads in under a second time to interactive. So it's very, very viable to do things this way. It's not like compiling a ginormous amount of code to the web as a lot of C++ applications are doing. And similarly, if you look to the future of AR and VR, this demo had a collaborative put your quest on, open this URL and you'd be collaboratively in this IDE space whilst live editing an object in between. And I was very impressed with that as well for WebVR because I was like, ah, it's going to be janky, right? Because VR is so performance sensitive, if your garbage collector hicks up, you're going to get sick, because the frame is going to start to jutter. But if you're super disciplined about it and you use WebAssembly only, I could get it completely rock solid on WebVR. And this is also when that one millisecond of CPU time when doing stuff really starts to matter, because this is a millisecond of CPU time on an M1 Max, not on a crappy Android chip that is in a VR headset. So that one millisecond that I showed you, you should multiply by 10 to have your full application domain for other devices.

QnA

Importing 3D Models and Donating to Foundation

Short description:

And this is also when that one millisecond of CPU time when doing stuff really starts to matter. So yeah, I think that is the end of my talk. Any plans to donate it to a foundation at one point? Absolutely. Is it also possible to import 3D models or textures from Blender, for instance? That's definitely like a higher-level tool. Yes, absolutely, but someone has to write the importer. It's a custom render stack with a shader model that is unified across all platforms.

And this is also when that one millisecond of CPU time when doing stuff really starts to matter, because this is a millisecond of CPU time on an M1 Max, not on a crappy Android chip that is in a VR headset. So that one millisecond that I showed you, you should multiply by 10 to have your full application domain for other devices. And so, you know, don't think that I'm over-optimizing that one millisecond. It's actually the space you have to reach your entire audience.

So yeah, I think that is the end of my talk. I think we can move to questions. Yay, questions. We have a few, so that's good. Okay. People also can still, you know, send their questions. If you have burning ones, and you were just waiting for the talk to end, to type in your questions.

I really love the question that is... Oh, did it go away now? Ah, yeah. So do you get to work on this full-time, or is there a sponsor or company behind? So for the first many years, I did consulting, and I did this as an iterative cycle every I restarted six times. More recently, when I started working with my co-founders, we have an angel-funded startup situation. But it's, yes, full-time. Full-time? Wow. Yeah. That's really cool. Any plans to donate it to a foundation at one point? Absolutely. I mean, this is an open-source stack, right? The only thing that we're going to commercialize is the design tool that sits on top of it. All of the libraries, all the dependencies, are going to be unencumbered MIT so that everything you build with this, you can pick any license you want, you know? So as this becomes popular and successful, hopefully, then we can definitely use one of those constructs that many open-source projects use. Super. Very interesting. I hope that answers the question from the person who asked. Is it also possible to import 3D models or textures from Blender, for instance? That's definitely like a higher-level tool. Yes, absolutely, but someone has to write the importer. It's a custom render stack with a shader model that is unified across all platforms. So it's not GLSL. It's something that looks like GLSL, semantically GLSL, but it cross-compiles to all the backends, Metal, HLSL, and WebGL, and in the future also WSL.

MakePad's Custom Engine and Future

Short description:

So it's a custom engine that can do all the things you could do in Three.js. The Rust implementation outperforms JavaScript. The startup time for the IDE is a millisecond, and it can handle rendering tens of UIs simultaneously. The desktop version will be available, and mobile support will come later. The speaker does not see a future for Rust-based UIs.

So it's a custom engine. It's geared for UI, but it can technically do all the things you could do in Three.js, but we're going to need to develop the tool chains around that.

Very cool, thank you! Did you start with building an MVC for the Rust implementation to know if it would really outperform JavaScript, or was this just sort of fingers crossed hoping that it would be better?

This was fingers crossed. Essentially, you know, this is not something I can benchmark dry very much. I just had to build it, and as we got further, I started getting more excited about half a millisecond of CPU time doing anything, and this means that the amount of bandwidth I have for this UI, I could render, the startup time for this IDE in terms of spawning up the UI is a millisecond. Like, I could draw tens of these things at the same time on your phone, and it wouldn't matter. So, yeah. It is definitely something I hoped, but I would say it's better than I thought.

Awesome. I see here a question whether it will only continue to run in the browser. You did show the desktop version that you used yourself. Is that going to be available too?

The idea is that we want the power of both. You want the easy deploy to web, and if you're constrained, use the desktop compile. And maybe it means mobile or they mean mobile. Mobile support will come later, because I think focusing on the web deploy target first is a good start, because it's a very big space, especially because we're bridging the APIs to Rust. That's a lot of work. But mobile will come later.

Very cool. Do you see a future for Rust-based UIs? No. I think it's absolute garbage. No. That was a very unexpected answer. All right. Thank you very much. I think that's about the time that we have, and we're going to move to lightning talks.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Vue.js London Live 2021Vue.js London Live 2021
8 min
Utilising Rust from Vue with WebAssembly
Rust is a new language for writing high-performance code, that can be compiled to WebAssembly, and run within the browser. In this talk you will be taken through how you can integrate Rust, within a Vue application, in a way that's painless and easy. With examples on how to interact with Rust from JavaScript, and some of the gotchas to be aware of.
JSNation Live 2021JSNation Live 2021
27 min
Building Brain-controlled Interfaces in JavaScript
Neurotechnology is the use of technological tools to understand more about the brain and enable a direct connection with the nervous system. Research in this space is not new, however, its accessibility to JavaScript developers is.Over the past few years, brain sensors have become available to the public, with tooling that makes it possible for web developers to experiment building brain-controlled interfaces.As this technology is evolving and unlocking new opportunities, let's look into one of the latest devices available, how it works, the possibilities it opens up, and how to get started building your first mind-controlled app using JavaScript.
ML conf EU 2020ML conf EU 2020
41 min
TensorFlow.js 101: ML in the Browser and Beyond
Discover how to embrace machine learning in JavaScript using TensorFlow.js in the browser and beyond in this speedy talk. Get inspired through a whole bunch of creative prototypes that push the boundaries of what is possible in the modern web browser (things have come a long way) and then take your own first steps with machine learning in minutes. By the end of the talk everyone will understand how to recognize an object of their choice which could then be used in any creative way you can imagine. Familiarity with JavaScript is assumed, but no background in machine learning is required. Come take your first steps with TensorFlow.js!
JSNation 2022JSNation 2022
21 min
Crafting the Impossible: X86 Virtualization in the Browser with WebAssembly
WebAssembly is a browser feature designed to bring predictable high performance to web applications, but its capabilities are often misunderstood.
This talk will explore how WebAssembly is different from JavaScript, from the point of view of both the developer and the browser engine, with a particular focus on the V8/Chrome implementation.
WebVM is our solution to efficiently run unmodified x86 binaries in the browser and showcases what can be done with WebAssembly today. A high level overview of the project components, including the JIT engine, the Linux emulation layer and the storage backend will be discussed, followed by live demos.
JS GameDev Summit 2022JS GameDev Summit 2022
33 min
Unreal Engine in WebAssembly/WebGPU
Traditionally, browser games haven't been taken seriously. If you want to target the web, that traditionally has meant compromising on your vision as a game developer. Our team at Wonder Interactive is on a mission to change that, bringing one of the world's premiere native game engines to the browser - Unreal Engine. In our talk, we'll dive into our efforts porting the engine to the browser and carrying on the pioneering unfinished work started at Epic Games nearly a decade ago in collaboration with Mozilla. We'll dive into what this means for the future of games in the browser, and the open metaverse on the web.