Building the Interconnected, Traversable Metaverse

Yeah, so another title for this room might have been import bracket bracket of the metaverse. I kind of have a few goals that I wanted to hopefully accomplish here. One is to just open your eyes maybe to what's possible for open virtual worlds on the web. A challenge goal is to kind of convince you that the metaverse can be good. I know there's a lot of skepticism and controversy about that these days. And my personal goal is, I don't know, I hope I learned something today. I wrote some script notes that I kind of wanted to cover, but I hope that anybody who has any questions about the crazy things that I'm going to be talking about, just kind of chime in. I'd love to kind of deep dive into whatever parts of this people find most interesting. So some background for me, I've been making basically metaverse adjacent technologies for about eight years now. And all of that has been on the web. I basically started on the web and I never left. And I've been constantly surprised what I was able to do. I'm just using basically what's available in the browser, the open source tooling that's already out there, just to make experiences, worlds, and even my own browsers. I wrote a browser called ExoKit. I was really surprised that all of this is possible on an open technology like the web that basically anybody can host, anybody can experience on any device. And it seems like I was the only person that was doing that. Every single time I would show off one of the cool new hacks that I made, whether that was the browser running on Magic Leap, or some immersive N64 emulators that I wrote, which basically let you play Legend of Zelda in your HTC Vive in the browser, no less, using a web VR. It was called at the time. So I basically started hanging around all the people who thought that, like, wow, this is really cool. I didn't know that this was possible. And yeah, we kind of started building up this technology toolkit of all these different things that we've been releasing basically open source from day one and working together on. And eventually what it started looking like was that we basically had the ingredients to quote unquote import the metaverse, meaning you can use a standard web api/talks">browser api to load in assets from all over the web, from all over different places, including, for example, IPFS, GitHub hosting, basically any website, or even things like NFTs. And it turns out that this actually isn't that hard to do. All you really have to do is add a translation engine that can basically take a URL and then write some javascript code, which will be able to load that asset. So what you're looking at here, basically everything here is just an import statement in javascript. So this is actually a GLB file. This I'm not sure if it's a GLB file or it's procedurally generated. This is a JSON file that's actually describing that pistol and referencing all of the assets. And the character itself is a VRM file that we just imported into the world. And so the way that this works is we actually just have a system on the backend where it takes those URLs and basically uses templating and some intelligent code generation to give you a loader for the frontend that will give you the 3d version of that asset. And that gets hooked in through some APIs to the rest of a quote-unquote game engine, which provides the services that you'd need for something like this, where you can walk around, you can interact with the environment, you can have things follow you, and you can have an inventory that you can kind of equip on yourself. And so we basically started plugging all of this stuff together and started realizing that, hey, we're actually making possibly the world's first distributed video game where all of these assets are just kind of plugged together using a client that we're all kind of agreeing on using. So that technology that we're using to basically make all these translations happen and plug into the game engine, we ended up calling that Totem. It's called a Totem because we're basically trying to develop a culture of creating art from the raw ingredients of the web. And we're basically sticking things together. Things are being imported, they're interacting with each other, and we're basically stacking art together. We're creating these virtual Totems, in a sense. Everything in the game is an asset with a URL, and that can be even a local file if you're hosting locally, I am, right now. It could be something on the web, right now, it could be something on a GitHub repo, IPFS, or like I mentioned, even NFTs, if you just want to point it out in the ethereum address. Yeah, and in the future, there's a lot of really crazy things that we'll be able to do once we get this baseline game working. One of the things is just providing simple services to recreate the software that a lot of people love, like VRChat, where you basically have your avatar, which can animate, it has facial expressions, you have mirrors, and we'll have soon AR as well as VR support, as well as integration with KaleidoKit. So that's basically a way where you can use your webcam to Vtube your avatar here on the side. And another thing that this allows is for us to build a world so that we can traverse across and share, in the sense that I have some sort of way to identify where I am in this quote unquote metaverse, and I can go over to your place and we can have a spatial relationship between those two places. So this is still actually kind of buggy, but I'm going to demo that as well. Because we're loading everything through these structured import statements, it allows us to basically have one scene be contained within another scene, and for scenes to be a first class concept that can be, for example, captured. So you can draw previews of the scene and basically get level of detail from a distant asset without actually having to go there. So right now what's going to happen is, this is a place that we call the street. It's basically supposed to be a directional representation of virtual worlds, and each section of the street has different content. So as I approach, what this should actually do is it should sideload that scene, take a 3d virtual screenshot of it, and then basically give me a low res version that's LOD'd before I even go there. So I actually have to turn off the fog there, but what that actually did is all of that is javascript code that ran on the side and basically gave you a level of detail. But if you turned off the fog, you'd even see that that is fully textured. And what we're also going to do to improve that is use marching cubes to give basically a full 3d mesh with texture of the virtual world that will be there when you go there. And of course, you can actually just go there. And now you're in somebody's virtual world. Another thing that this allows us to do is use metadata to define the kinds of things that you'd want in a video game. Like, for example, quests. There's actually a quest in this world. It says, destroy all Reavers in the area, and there's a reward of some virtual asset. So if I can find it, I'm not sure where it is. This might actually be broken, but we also have a pathfinding system to kind of geolocate you in the virtual world so you can locate your objective. And then if there's some Reavers, you can go and slay them and get your virtual silk, whatever that means. So all of this is actually open source. And our goal is to essentially build the first distributed kind of video game. Anybody can fork this stuff. And one of my priorities right now is actually to plug in all the different virtual communities that have already been starting to build on top of this. So technology-wise on the renderer, in terms of engine, a lot of the stuff is both stuff that I built, and a lot of it is just plain open source code. Like, for example, everything is 3JS. There's a zine graph. There's really nothing crazy about what we're doing other than we do use Unreal engines, Bloom shaders, and SSAO. Sorry, not Unreal. This one's a CryEngine. But that's what kind of gives worlds, if you define the render settings properly, this really nice, misty effect. We use PhysX, which is the same, basically, PhysEngine that everybody's using, including Unity. It's built into webassembly, and we're loading it that way. For the UI, we're just using react, although there's a lot of webgl, for example, to render these kinds of panels. And these different shaders, it's all just actually react code under the hood. The avatar IK stuff is actually an open source Unity project that we ported over to javascript, and that's how you get all the different smooth animations and basically the ability to animate any character, regardless of what asset they are. In fact, let me show you some retargeting of animations. We're going to change characters. This is just using the same exact animations, but the character still works, just the same. And we also support drag and drop in this game. So if you have your own virtual assets, you can drag them in. That actually gets uploaded to a server and then downloaded back again. And if you want to drop them into the world, you can. And if it's an avatar, you can wear it. And essentially, we're trying to build the world's best toolkit that we know of to make these kinds of things work together in the context of a video game. The animations actually for the avatars are from Mixamo. So it's also basically just open stuff that we're reusing. And we even have AI support for basically everything in the world. So this kind of shows one of the coolest reasons why we want to build things in this way. It's not only that it lets us work together, which it definitely does, but it's also because if you combine this with AI, you can build some really amazing experiences. And the way that AI works is you need all the data to be structured in the sense of open data that you can download, things that you can import, code that you can run, and metadata that you can parse in order to generate, for example, a script between multiple characters talking. And we actually have that system integrated as well. You basically plug in any api key for any AI engine that you want, whether that's Goose AI, which is the open one, if it's your own self-hosted or even open AI. And essentially, it will write character scripts for things that happen in the world, which then actually happens in the world. We parse it out, which basically allows you avatars that are true AIs in your open to virtual world. I actually can't show you too much of that because it's still not approved to be deployed by open AI, but I'll give you a sense of what is possible. By the way, if there's any questions, I'd be glad to hear them, or if I'm going too fast. So here again, we're loading in all these avatars, they're just different VRM assets. Every object here is basically an interactive JSON file. I can even show you what this hat looks like, in fact, in terms of the code. It's something we call a .metaverse file, but it's really just JSON. Essentially, the engine sees this and renders it into the world. And you have your basic start URL, which is referencing the GLB file. In this case, it's going to be relative to the location of the file. And this file can be hosted absolutely anywhere. Right now, it's on GitHub pages. And then you have the components for defining how that hat can be worn. In this case, it will be attached to the head, and it will be attached to 0.1 meters above the head. And that looks about right. And basically, that gets completely synced with the avatar. We also have a mode of wearing where it's conformant to the avatar, but there's still work that we need to do to make sure the displacement of clothing is proper for what you're doing, or proper for the shape of whatever avatar you're wearing. And there's different abilities that objects can be defined to have. In this case, we're working on just kind of a spell system, magic combat type stuff, as well as things that will actually make sense in the context of an action game, because we probably want to explore that angle. So you probably want your slurped juice or whatever this is supposed to be. And yeah, the different objects can, for example, be editable. And the avatars basically do have pathfinding implemented. So they know to follow you. They know to essentially interact with the rest of the objects. This is the part that I can't show you, because OpenAI is actually pretty strict about what you're allowed to publicly show. But there's ways where we can literally talk to the avatar, have it go and fetch the object for us, and then, for example, help us in a fight. That's actually just relatively simple integration between AI and traditional action-based character systems. We also have features that are, once again, inspired by VRChat, like there is a speech of the avatar in the sense that... Sorry, I actually can't do that because of the echo effect. But basically, when you talk, we run the audio through a speech phoneme engine, and then that's mapped to the avatar. Another cool thing, though, that you can do is you can do speech to text. So then you can basically turn your mic messages into chat messages. But we can even do one better than that, which is the opposite. So I'll give you a demo. Hey, Drake, what's up? And that's about all I can show you in terms of AI. But yeah, basically, our idea is to allow all of these different assets to come together through the powers of AI and just basically using standard tools, standard open source projects to kind of build out the best quote-unquote version of the metaverse. And it's an open one. I think that's the only way that we get any sort of good version of the metaverse. And I'm not talking about building the kinds of things that Mark Zuckerberg wants to build. I'm more interested in kind of building open systems and basically the next version of the internet. And I think we can honestly get there. I think the web has evolved far too quickly I think the web has evolved far past the point where we can't do it. And it seems like the future is actually pretty bright for us because first of all, webgl and WebXR has great support on Oculus. And it seems that Apple is also on the web GPU and WebXR bandwagon. So everything that we're building here is going to be accessible in everybody's VR devices. In fact, I'm mostly coming from a VR background. And the only reason I'm not doing this in VR is because there's a lot of other stuff that I have to do. But we're actually going to fix up the VR here. And basically all of this is going to be also fully immersive. And what we're basically trying to do here is kind of show the way forward for a version of the metaverse that can be good, that can be open. And that's actually forward thinking where whatever we're building, it's going to be accessible. A year from now in whatever form we want, whether it's on desktop, it's just content that we're viewing on our phone or like we're just meeting in VR headsets. So another thing that we're working on is of course multiplayer to make sure that everything here can just be done in a synchronized fashion with other clients. I hope that's not used to create crazy bots, but you never know. But actually the technologies behind that are again pretty simple. We're using CRDTs to replicate basically the object state in the world. So essentially the positions, the transforms, and all the components of each object are just in a global store. And then we get updates from that store and that store is synchronized back and forth with eventual consistency with the server. And that's essentially how we do multiplayer. It's just synchronizing those states. And for voice, actually we don't use WebRTC. We're going to use WebCodecs, which is essentially just doing the transcoding ourselves and doing it over a web socket, which tends to be more reliable than some of the crazy hoops you have to jump through with WebRTC. It's also more secure in the sense that this is something we want somebody to be able to host on their Raspberry Pi if they want. You should be able to set up your own private server that has your own private encrypted voice connections and runs the same content. And what I hope for the future is that we start building on more open systems like this because I think it's truly the way that we're going to get to whatever a good version of the metaverse is. The more content there is, the better we're going to make this kind of crazy game that we're making. If anybody wanted me to deep dive into any particular technologies, I know there's a lot here. Then just let me know. Somebody in the chat had asked earlier, what about performance? So right now, the main problem that we have is our avatars are so high quality, there's 55 draw calls mostly in the avatars. And we're not only rendering it once, but we're rendering a depth pass over top of all of this. So the way that we're rendering is actually extremely unoptimized right now. We're working on that, hopefully within the next week or two. But in terms of just forward thinking performance, actually, we have systems for configuring basically the graphics fidelity automatically. So for example, for the avatar, you can reduce the quality to medium, which actually changes what I said. It's no longer going to be 55 draw calls for the avatar. It's going to be one. Actually, I even disabled lighting on here. It didn't have to be disabled. But now we have the entire avatar basically automatically at list to a single texture and a single draw call and a single geometry. So we can actually make this super fast. And this actually basically fixes all the performance problems that we ever had. But we are planning on going a bit further actually than just standardizing the avatar because we also have a sprite sheet system where it will take screenshots of your avatar from all the necessary angles and basically render you as a single plane. And this is something that can allow for connection with thousands of people. So I hope that answers the question. And despite being a sprite avatar, you still actually have full interactivity with the world. There's a few angles that we need to fix, but overall, this has worked well for us. This is also a way or a technological path where we can be interoperable with other types of virtual worlds. There are other virtual worlds where everything is 2D. And if you standardize on your formats, it's pretty easy to create something like a sprite generator. If somebody asks a question about, is react rendering the scene? Are the components just buttons or the UI with webgl? Okay, so basically everything is react. It's actually using vite.js for things like styling and importing that. But there are Canvas elements in the react. And then we just use react hooks to hook into our engine. That Canvas for the character here is the same as the render as the main scene. So you actually basically get this for free. It doesn't even take any more memory to do it this way. I can show that code. I think that's on the character. There's our Canvas. And we're hooking it into our engine here, game.players.roma.addCanvas. And this pushes us onto the render loop. And then we're just copying frames over to the Canvas. Where's the copy? There it is, copy frame. This Canvas draws from the main scene where we actually temporarily drew on that window. Actually, there's an interesting quirk of the browser that we're taking advantage of to be able to essentially render items in your inventory. These are, once again, standard JSON files, standard GLB files that we're importing. But the way that we do the previews is actually on a separate render thread. It's a browser trick where, as long as you change your domain name entirely, and by domain name, I mean top plus one. So that means you have to be on .xyz rather than .com. Otherwise, this doesn't work. But essentially, as long as you iframe over to a URL on a completely different origin structure this way, then you'll get a different render thread. And as long as you run a second copy of the engine, which in this case we do, you can do drawing completely asynchronously and then post the image buffers back to the main thread. So that allows you to have basically the aspect where you're exploring the world and things continue to load. And you can load the previews for it asynchronously. You're not actually hitching, which is the common problem that you get with a lot of 3JS-based things. When you add something to the scene, there's a compilation process. But in this case, a lot of that can just be done off thread, including on a separate GPU thread. And then as long as you have a fast GPU, you won't even drop any frames. And usually you don't. And it can also be throttled. Can you zoom in the code a little bit? It's hard for people to read. In fact, if anybody wants to talk about any of the code that I just mentioned, I can probably find it and explain it. Somebody asked, sounds interesting. Somebody asked, the routing is only instructing the scene's source, correct? The routing? I guess in react, the routing is only instructing the scene's source, correct? I'm not sure where the routing starts at the moment. We're not actually using any standard routing framework. It's kind of hand-rolled. But it does eventually hook into the react. Actually, I would have loved to show a lot more scene traversal stuff, but we're still working on shader compilation issues there. But the idea being that as you're traversing the virtual world, the URL does actually change. And the back button does actually work. It will just actually teleport you back to the previous URL. Most likely modulo some coordinate. What kind of an auto progress save in your URL bar? Yeah, so I was asking about the routing because I saw that you changed the different scenes, like darkness to the scene's grass. And that's what I noticed. But I was also interested, how do you render another scene when a level changes, like when you're going into a dungeon or you're going in another area? Or are you doing that still manually? Sure, I can show you. This isn't hooked up yet, but I can explain how it works. So a scene is an abstraction. I can even show you what a scene file looks like. I'll show you this file. It might help to explain. Awesome, thank you. A scene is just another file type that you can import in WebReverse. The interesting thing about it is that it's a file type that can import other file types. It's a scene and the scene contains other URLs. So in addition to URLs, like for example, the street base, which I think is this object right here, the blue stuff, you also have the lights defined for the scene. Basically, this is inline content, which is going to be translated into a data URL. It's the same as if you loaded a data URL, but it's basically just all these different assets in the scene. And so with scenes being a first class concept, you can actually have two scenes at the same time. So I have a system here where if you get close to the edge of where the next scene should be, it's actually going to add it, and then it's going to preview the scene by basically taking a 3d volumetric snapshot of it and then basically rendering it in front of you. And this is just one draw call. There's zero performance hit because that entire scene is paused, but it's still there. And now we basically have two scenes in the world at the same time. And when we cross the threshold, we can magically switch one to the other or switch, there's actually a glitch here. This should have actually rendered the full preview of the entire scene before. But basically we can walk backwards and forwards, and these are just different scenes that have entirely different data and they could have entirely different multiplayer rooms, for example. So if I'm over here, I can walk over to my friend's place and say, oh, hi. So the most expensive operation that is happening right now is just the skeleton mesh, right? It's just the character itself. Everything else is kind of like dynamically loaded, correct? That's right. Actually, everything is dynamically loaded, but there's plenty of opportunities to actually optimize a lot more. Like, for example, we will probably rewrite our avatar system, at least the kinematics part in webassembly. We're already using a lot of webassembly, by the way. Like, for example, this procedural generation is a webassembly algorithm, the physics are webassembly. And we're probably also going to take the bones code, which is the real only computationally heavy part of this app. The rest is just matrix multiplication. And we're probably just going to write that in webassembly, as well as probably making the default of the characters just the single draw call version, which means we do an offline pass to, once again, generate the clean texture for the character and then swap that out directly for the 55 material version. And once you've actually fixed the lighting, you almost can't even notice the difference, but it's 55 times faster. Actually, it's more than that. It's like 110 times faster, because there's also a secondary depth pass that we do for some effects, like SSAO. But yeah, this isn't mostly the only expensive part of anything that we're rendering here. Everything else is just basic shaders. It's three JS shaders. It can be as fast as any shader you can write. So eventually, are you going to rely on, like later in the project, I guess, are you going to rely on something like real-time, like in a real engine or something more powerful, or are you going to just do more geometry tricks, so to speak? Can you explain what you mean by geometry tricks? Like how, for example, you can have some blockers in the light to kind of cheat the behavior with the environment, rather than actually do it properly and see where the light spots off, and kind of just use something like Lumen, for example, and being able to tackle that problem just with the Alpana reel, for example. No, we're experimenting with essentially the best shader techniques that we can implement. Most of this research has already been done, and most of it isn't open source in some form, whether that's GLSL or some other language. It's just open source Unity code. We've actually already implemented a lot of that kind of stuff, like, for example, for SSAO. And then there's a whole bunch of different ways that you can do shadows, you could do voxel-based lighting. That's all stuff that we're experimenting with. It's actually one of my favorite things to do is to just kind of see all the algorithms that are possible and see how awesome that looks. I'm always surprised with the results, and then everybody says, like, oh, wow, is this Unreal or whatever? Well, I'm like, no, it's not Unreal. It's not Unreal, but it's using the same technology as Unreal, so is there a difference? And my last question, sorry, I don't mean to take time from the other participants, but I also was curious, are you using a store to be able to keep the state in the general application itself? Or is it just like a one page, so you're just, okay. So you are using something like Redux, or I don't know. It's not Redux. On the UI side, it's mostly just use state, and there's some context in there. Oh, it's context api, okay. Yes, but that's only for the UI stuff. The core of the engine content, like the actual, what's happening in the scene, the state of the characters, all of that is in YJS state, the CRDT that I mentioned. And that is the thing that's actually replicated on the network, and that's why even though everything here looks like it's single player, it's already basically running the same multiplayer code, because it's just updating the state engine. The engine is responding to state changes, and that's how you can have these characters follow you. You're updating the position on the server pretty much, so two people are able to see the position on the plane. Yeah, except in this case, the server and the client are the same thing. Yeah, okay. Yeah, thank you so much. This is awesome. Can't wait to participate. So one of the coolest things that we've been doing is kind of working with artists to build out whatever this virtual world is, and just exploring with all the things that we can do with avatars. We're basically adding combat systems for the different items. I can show you, for example, bows and pistols, and just all these kind of cool gadgets that you want to play with in any sort of virtual world. The avatars also have a full emotion system, so if you're in a multiplayer room and you want to say that you're angry, then you can just increase the angriness, as well as the fun, so you can be fun and angry. As well as doing things like dances. And I wanted to show some items. So we also have basically inverse kinematics implemented for a bow. And you can shoot, and the arrows stick to things. We're also treating a lot of this as just game design problems, like how do you make the avatars move in interesting ways? What are the kinds of interactions that you want to have in a virtual world that will actually make you want to be there and spend time with your friends there? And I think the best virtual world for testing the pistol might be darkness. In fact, one of the features that we're hopefully going to enable soon is your entire inventory and loadout and all the things that you earn. There's just kind of stored on a database that we're running, and that allows for things like to be on Discord and kind of interact with all these virtual objects from other interfaces, besides just the game. But yeah, there's basic lighting effects for essentially everything. In fact, this pistol is a good example of multiple applications working together because the pistol and the explosion are actually not the same app. The pistol imports the explosion. And that allows for a degree of isolation. So if you ever need this really cool explosion somewhere else, then you can just import it. And are you doing more like the bonus injection type of thing? Yep, essentially. Awesome. The backend compiler is just VJS. So you can think of it as a dynamic import. And it is recursive. So it'll go through. It'll figure out what type of thing you're trying to load. In this case, I'm referencing a GitHub repo. I can even show you how that works and why. But you really just have to specify the GitHub repo. Is it a pixel explosion working with some kind of object pooling because of the animations and how expensive the particles are for the shooting action, so to speak? It's event-driven. I'll show you the code in a second. But it's event-driven. And on the firing action, it's all based on state changes. It's a state machine. So on the firing action happening, that's when we, for example, add the light and then we initialize a new particle. And then every kind of frame, we go through the particle's animation. And then eventually, when the particle realizes that it's dead, it'll kill itself. It'll remove itself from the geometry. And then the mesh becomes invisible. So it's actually quite efficient. And that also allows multiple particles to be done in one draw call because we're just using instancing for a bunch of cubes in this case. But these effects are going to get way more advanced once we put some actual time into them, even though I think that actually looks pretty cool already. So the reason that pixel explosion works is anywhere you put any directory, HTTP directory that has something called .metaverse file in it will be loadable in WebReverse, regardless of everything else. And what that allows you to do is essentially specify what your entry point is and some components that you're describing about your entry point. Actually, this is a bug. Ignore that. This shouldn't have any components. But for example, this has an entry point of index.js. And then here ultimately defines some shaders, some constants, some functions for itself. And then ultimately, you're just exporting a default which is going to specify what your object is. In this case, it's going to do a dependency injection type thing for a use app, which is basically calling up to our engine. Here's where you import the engine, import metaverse file from metaverse file. So it's getting basically these engine functions that allow it to access what application am I. The engine has already kind of set me up with components, set me up into the scene. And then all I have to do is create some additional meshes, animate them every frame. Here it is. ExplosionMesh.update. This might be... I guess using the custom hook and the class itself. So kind of like separation of concern because then you can do like for the garbage collection that it needs to be done by the explosion, it's not going to be tied to any other asset, I guess. That's right. And applications can set components on each other and they can send messages to each other. And that's basically the level of isolation. If you know which kind of object that you're interacting with, then of course you could do more advanced things like expose your own api. But otherwise, these are just basically open objects on the web. Then it's up to you to define how you want to use them. And we just provide the primitives. For example, use frame is just a convenient thing for... Every single time I should be rendering myself, call this function and then we do the update. Yeah. And another thing that the pistol, for example, imports. It can just directly import GLBs. So it's the explosion plus the GLB equals the pistol. And the rest is just javascript code for making these things kind of work together. So when you're shooting or you're hitting something, you also using a class just for that to handle the collision? Yes. So you have basically rocks access to the PhysX api if you want it. So if you want to do a raycast, then there is a function for that. I wonder if it's here in the pistol repo. Here we go. We have a raycast from the gun's position, gun quaternion. I guess this is correcting for some error in the GLB where we have to flip it 180. But you're essentially taking the position, the direction that it's facing, you get a result out of that. And it will tell you actually not only like the point that you hit, but also the application that you hit. So this is how you can target a thing and then see, oh, this is the thing that I hit. And then I can call the .hit function on it. And that actually allows for damage tracking systems. This building here is not invincible. You can actually destroy it if you shoot it enough. Let's see how long that takes. There you go. It's dead. It also works for avatars where you can define raycasts for where are you going to hit. And if that hits some sort of avatar inside of their bounding box, inside of their character controller, then they will take damage. And that allows for automatically defining like a lifetime of character gameplay mechanics. So is there any other applications that somebody want me to cover or deep dive into? There's vehicles as well. And mounts. This is actually a bug. He's not supposed to be going this crazy, but you can hop on. He's only supposed to be going crazy when you're riding him. I guess you would have like some kind of problems with frame rates sometimes or not at all, like based on the object that you're interacting with. No, that's usually not the problem. In fact, the assets are mostly just quite optimized. And it's pretty easy to get optimized assets these days with just KTX2 and compressed textures and just kind of crunching everything into single meshes. The main problem that we actually have right now is almost exclusively related to animating the avatars because we don't do any optimization and they're just really high quality. We basically need to do a runtime swap of the geometry to make sure that's a single draw call. And that will probably mostly make us limited by whatever other content there is inside the scene because the avatar will be a relatively cheap operation. And it's just a matter of what is the expensive, all the shaders that you're using. Are you using fog? Do you want to set up your render effects? Another thing that, by the way, scenes do is they define the render settings for how you want the scene to look, which is honored for every scene that you go to. I'll show that as well. So you get easy access to things like, we have your lighting, but then you have SSO, Screen Space, Ambient Occlusion, Depth of Field, HDR, Bloom, Fog, type of fog. And as you go from world to world, these render settings just will recompile your shaders essentially and basically make sure that every world looks the way that it looks and it can even have its own skybox. You could also work with something like a procedural mesh, right? You don't have just to work with like a static one. Absolutely. Oh, we have a creature here. Do you mean for avatars or for something else? Just for assets in the world in general, I think for avatars it's pretty expensive. So I didn't look into the code. I'm not sure how the actual work sort of handles it. I'm not sure how the actual work sort of handles it. I'm not sure how the actual work sort of handles it. In terms of procedural generation, we're already doing a lot of that. This is actually a shader that's using noise functions inside the shader to draw all of this. Basically, this geometry is entirely done on the GPU. Also, just in one of our main scenes, we have these buildings that we were trying to build it out of Fortnite blocks. That kind of generate together the geometry. It doesn't look good yet because we haven't replaced all the different wall sections. But all of the different wooden structures that you see here are just procedurally generated. And it's properly built. It has an interior. We also have some kind of container system up here, which is again procedurally generated. You can kind of go inside and have different sections of some sort of virtual living habitat. And some of this is using similar techniques where we're just using noise functions to generate a terrain. And we have much better stuff in the works as well, where it's infinitely generated. It has biomes. We apply some really great textures and shaders to it. And we also sprinkle it with some other geometries that we've baked. So trees with a canopy and just a bunch of different items that you can interact with in the world. The techniques for doing this are well known. This was basically Minecraft's claim to fame. And we're just doing the exact same thing. We're using noise functions, biome maps, and basically efficient updating of the geometry in chunks as you progress through the world to make sure there's no frame drops. I've actually been doing that for years. So it's really fun for me to kind of go back to it again. But we're going to be doing a lot of that for the actual gameplay. Hi. Can you elaborate on how you can integrate a 3d NFT or any other type of metadata from NFT? Sure. So we're still hooking up, for example, the login system. But once we have the logins working, then it'll just be a metamask plugin, which allows us to basically know your address. And then once we know your address, we can pull what your entire list of NFTs is in. But the way that we render all of this is once again through the system that I kind of talked about earlier. It's Totem. Because we actually have done this for other NFTs. So Totem is a system where you request a URL. And from that URL, it's supposed to give you back an importable javascript file that can load your content in the world. So that URL can be an ethereum URL in the form of either ethereum and then your address or then ethereum contract address and then token ID. And then we figure out the standard on our backend. And Totem's job is to figure all this out and generate the javascript file that you want. We have actually done that for... I'm actually going to bring up Loot here. We've done that for some other projects already. So essentially, all you have to do is use the standard metadata that's coming from the NFT and then figure out how to write the javascript file for it. And in many cases, this is actually really simple because the VRM file or whatever asset or the picture is just there directly. And then all you have to do is use our system to import the picture. And that's actually how some of these things are being loaded. For example, this is just importing a GIF and then adding it to the world and then Totem handles that. Same thing here. This one is actually a shader. It's on chain. And yeah, essentially all you have to do. And we'll accept pull requests for anybody who wants to integrate their token because it's based on the contract address. But anybody who kind of wants to plug in their NFTs, we would love a pull request. Or if you just really want that, and you want to plug in whatever content you have on chain, we'd love to kind of help you out. I hope that answers your question. Anybody can actually do it. Last question. Can I have my end user customization opportunity? Let's say if my end user wants to scale, modify it, does only the developer have the option? Or you can... No, no, no. I wish I could show you more tools. There's bugs, which I'm fixing. But if you know what I mean by the Fortnite build system, that's kind of what we have, as well as just kind of in-world tools for inspecting everything in the world. And if this was working, you'd see the transforms here. So there is a bit of a native experience. We're also working on tooling, for example, a Unity Explorer that as long as you haven't done anything too crazy, that will generate all the files, including the scene file. And then it'll just work that way. But it's not only necessarily for developers. We want to make sure that anybody can essentially have access to basic creation tools. So we're probably going to be integrating all of that, including the Fortnite build tools. And actually, we're probably also going to be providing a hosting service for a lot of this content for people's virtual worlds. So it actually could even be simpler. In addition to being drag and drop, it's really just a matter of logging in, then you can kind of play around without even using any external tools. But if you want to use external tools, then of course, we encourage that because in most cases, they're better. Like we actually use Blender for everything. Awesome. Are you thinking of like integrating some like, maybe like Hashgraph or any other kind of like web3 technology in order to improve the processing of, you know, like meshes and assets and everything that you have like in your quote unquote pipeline? We're probably. Okay, so I mentioned kind of the technique of preloading worlds. We're probably actually going to be doing that offline, as well as providing cool tools for people to kind of explore each other's worlds without necessarily going to them, just kind of exploring the previews or even creating topographical maps that you can kind of share and say like, oh, meet me over here. So a lot of that is in the pipeline and we're going to be building a lot of backend infrastructure for doing that. But if you have any suggestions of specific technologies that we should be using, I'd love to dig into them. We want to use the best of what's available. If you actually have any ideas, probably the best way is to open an issue on GitHub. Some of the best feature requests and suggestions that we've gotten have come through that. Yeah, we'll do that. We'll send you any numbers as well. Thank you. As you just mentioned that people can drop their artworks. So like, do you charge any gas fees like many other NFT websites like they charge gas fees and all on the blockchain? We'll probably, I can't make any promises because I don't know how the market, I don't, I can't predict the market, but we don't really have any plans for charging gas fees for like that. Actually, we just want to fund ourselves mostly by building the best game possible and giving people reasons to kind of get first party stuff or get integrated into the rest of the game. Because most of the costs for doing things like minting are actually quite low. So the way that we might restrict it is just like you need to have bought a copy of the game, quote unquote, which might be an NFT. And then that allows you, for example, infinite minting or something like that. But if you kind of want to make your pitch for what the best solution is, again, I think the GitHub and just joining the Discord, joining the discussion is probably the best way. If you don't mind, can I share your Discord link in the chat box? Yeah, that's cool. Yeah, thank you. So you mentioned about creating your own virtual world and your hosting service for the next steps. Like if someone wants to create their own quest and gaming in their virtual world, like do you envision that they'll set up their own database and infrastructure for it or do you provide that too? We're providing it if only because that's how our first party game content is going to run. But I think the case that you're talking about is a very advanced one. I would hope that we're able to keep up with feature requests and demands from users that they'll just continue to say like, okay, like it's better if we work together and just kind of connect to the service and then it's all handled. And then you're trusting us, right, to run it. But the thing that kind of keeps us honest, I think the best hope is that the thing that keeps us honest is the fact that at any point anybody can run the more advanced version which is figure this out yourself. I'm coming from a background where I know how hard it is to make these things because I've made them. So I'm actually trying to spare a lot of people a horrible fate by saying that, yeah, we'll probably try to get you to host with us. But yeah, at any point, anybody who has a better idea for Webiverse and what it can be, I encourage Forks because that's how we actually make everything better. Awesome. Hello. Yep. Hello. So I'm just curious about the performance capabilities of such a large scale browser-based app like this. So how does it compare with like traditional desktop applications like, for example, Unity or Unreal apps? Are there any notable advantages or disadvantages building Webiverse into a browser-based application like this? Building it this way allows basically everything else to happen. It allows the technology to move at the pace that users actually want it rather than at the pace that some whatever game engine company is doing. I actually don't think there is much of a difference at all these days because things like Chrome's implementation of webgl even is actually highly optimized, especially if you know the power trick where in Chrome you select the OpenGL drivers for Nvidia cards. It's basically native at that point. In addition, we're going to be moving to webgpu, which is kind of even more low level than webgl, which is what we're currently doing. I actually don't think there's going to be any appreciable, like in the next couple of years, there's probably going to be no appreciable difference between a Unity game and a game on the Web because ultimately this comes down to the computing power and the APIs available and somebody actually having the passion to take the leap and make something good. I think that's pretty much the only thing that the Web has been missing in terms of being basically the best gaming platform. It's just for somebody to take it seriously rather than constantly saying, well, yeah, well, Unity sucks for all these reasons, but we have a deadline to meet, so I'm going to use Unity. And then if everybody does that, the Web can stagnate. I think one of my goals is to kind of show people that you don't need to use Unity to make something good. And in many cases, you can actually just make it way better with a drag and drop interface in the browser. Thank you. Honestly, this kind of app is honestly impressive. We're just getting started. Yeah. So you might have mentioned this. What are the things you integrated which makes this way more performant than other browser-based games like from html file, webgl? Yeah. There's not much difference other than we actually... I consider myself a pretty hardcore engineer with all this stuff. And the way that I started with all of this was compilers and optimizing code. So I think with 3JS especially and with the culture of the Web where it's mostly like copy-paste code that you're just trying to get to work and then you paste something and then it kind of works, so then you ship it. I think that has kind of built up a lot of code bases that they just didn't take performance seriously at all because they don't necessarily understand the underlying game engine. They don't understand shader recompilation. They don't understand why compressed textures are necessary and so on. And if you pile on enough of these mistakes, you can build kind of this monstrosity of an application that appears to work despite all of the mistakes that you've made. I think with Webiverse, we've just taken a very first principles approach. Like what does it mean to load an avatar? What kind of shaders is it going to have? Are the shaders going to depend on the context of the world? Do we need to preload things? Do we need to offload this processing over to some sort of worker? I think a lot of people just haven't really had the willpower to ask those kinds of questions. But when you do ask yourself those kinds of questions, the same questions that the people who made Unity and people like my heroes like Tim Sweeney, they're the people who actually did take that seriously. They thought about these things. And I think what I would hope that Webiverse ultimately is considered something that like, okay, well, once somebody took the web seriously, you actually have a pretty damn good game engine. Great job. Do you plan to publish tutorials on how you engineered it? Could you repeat the question? Publish to studios? Publish tutorials on like how you engineer? Oh, tutorials. I wish I had more time to do that, honestly. I'm going to say yes, because I know a lot of people want me to do that. Hi, guys. Can I ask you a question? Is it mobile-friendly? Is it working on mobile or on the desktop? I don't know if it's currently working on iOS. Like there might be bugs, but there is no reason that it actually wouldn't. If there is, it's probably some small bug. Because right now we haven't reached beta state. Beta state being all the features we need are kind of there in the first version. And then we're going to basically iron out all these things like, oh, is it working on iOS? Set up the testing pipelines and so on. It's definitely in the works. We also want to integrate hopefully just things like ARKit. Because we actually have a system already that's using webcams and just tensorflow, I think. Is the one, PoseNet with KaleidoKit to animate the avatars. So even if I don't have a VR device, I just have a webcam. I can join multiplayer and basically have an animated avatar in the world. So that's also kind of coming in. So can you use your cam and make your own avatar? Like use your real face and make one? That's not what we're using that for right now. But that sounds like a cool idea. I know other people are doing it, like Ready Player Me. Like it might be more good if you implement it, I guess. It's a lot of work. Yeah, I understand. I have more of a game design question. So a very common problem I've seen with virtual worlds is there's usually not all that much to do. Like you can walk around, you can insert images, you can shoot. But what else is there going to be to do? How are we going to entertain ourselves with it? That's an excellent question. The answer, I think, is procedural generation. That's actually kind of my specialty. And like I said, I'm really glad to kind of go back to it. But we're basically creating a Minecraft combined with Fortnite where everything is generated, everything is destructible using all of the physics systems that we already have and using a whole bunch of optimization techniques. So there's biomes that you can explore with both NPCs, AI-controlled NPCs that are either going to help or hinder your quest based on what the AI is returning. And actual just procedurally generated quests like talk to this NPC. And then people can kind of share interesting seeds like, hey, did you know if you go over to this place, there's an NPC that offers you a really good deal on this quest and you can really grind that. All of this stuff is actually pretty easy to hook up with all of the infrastructure that we've built because NPCs are just objects. Mobs are just objects. They're just things that you set animations on. It has its own hit box. And basically everything has stats as well. So it allows combat mechanics to exist. I actually think people are going to be really surprised by what we come up with here because I basically love doing procedural generation stuff. But I think that's all you really need in order to create that amazing game loop. And on top of that, of course, if you want to drop in your object to help you in your quest, that makes it even more interesting. Yeah, I like that. Thank you for your answer. Okay. So you kind of caught my attention when you were saying that you have a pretty heavy background on compilers and more memory-based problems. Sure. Right now there is a matchmaking server or is there something that you are using off another third-party game server to be able to synchronize positions in like almost real-time? That's just using CRDTs. So I might be able to show you that code. It's basically a replicated data type that the server reflects to all the clients. It's probably WSRTC. And this can run on any server. I don't recommend running it on a home network or whatever, but you theoretically could. It's actually not data-intensive at all because even things like the audio are compressed using WebCodecs. But yeah, it's mostly just a quick data exchange. There is, of course, latency. We don't do any prediction right now, but we could. We do actually interpolate, though. So regardless of what the player's different frame rates are, everybody still sees smooth animations for each other's games. That's actually, surprisingly, not that hard of a problem. So the heaviest operation in terms of building, I guess, you don't really have any building bridge to javascript. javascript is just running clean in react. react is just rendering components. It doesn't have anything that it's doing in terms of converting code or being able to read from a language-agnostic file, like a Perlbox or anything like that. Correct? Yeah, that's mostly correct. You're dealing with JSON, essentially. Yeah, but that reading of JSON, is that the fastest that you can go? Isn't there, for example, something to work with, I don't know, buffers or serial data that could be even more performant? Yes, if our profiling tools ever say that this JSON stuff is a problem, we're going to move it to webassembly. Because I think that's an excellent idea, and I think it'd be really cool to kind of explore. Can we take all of these CRDTs, just write it in rust or whatever, compile it to webassembly, and then you have this awesome state engine, and then you just have a clean interface of the events. Because it looks like a lot of the work is done from the front end, so there is not any server render being done or anything like that, correct? Because it's just a static site, so to speak. Yeah, mostly. It's not a static site in that we do use react State in order to draw, for example, the UI. But for example, when everything's closed, none of the react render loop is running. This is just the game engine, and everything is just painted on top of that by the browser. But when the component is closed, it's pretty much like component on destroy for react, so it just gets garbage collected. In some cases, yes. In other cases where we don't want to reinitialize things, then we keep it around and we just hide it. Okay, it's just a model, so to speak. Yep. But it really depends. We just make sure that it's efficient, whatever it is. For example, we don't need to render the avatar when it's off-screen. Like that. It also allows us to do cool things like have a minimap. It's kind of hard to see, probably, but there is a minimap that real-time renders from the top view, and it kind of caches the images as you're going through. So at all times, you have a perfect view of where you are. And in VR, we hope that this could actually be a physical map that you can pull out. It's kind of rendered. Awesome. In terms of the backend code, do you think you can show some interesting problems that you came across and kind of explain how you solve that? Because I guess not being a super backend-intensive application shouldn't have so much load, but I see a lot of interesting type of quaternions, operations done. So I was just wondering if you could give us a quick overview. Do you mean backend code literally because it's running on the backend, or are you more interested in technical geometry problems? Because most of the geometry stuff is handled entirely on the frontend. Yeah, and that's something that, for example, isn't that less performant than having it on a backend? Yes, in some cases. So that's why right now, one of the things that we're going to do after we make sure that this is all interoperable and we can start kind of traversing worlds to better be able to talk to each other, we're probably going to be optimizing a lot of this stuff and moving it from the frontend to the backend. Because it's just javascript, it's isolated javascript, it can basically be run anywhere. In the future, actually, right now, we're using Node, but in the future, we hope to use deno so that it's even more browser-like on the backend. deno already has webgpu, so if we had webgpu, we could probably already do that and then be essentially a native application. But then with deno, you would have to cross through the build to JS, because it's only typescript-based if I'm not... I think it supports any javascript, too. Okay. But we already do have our own build system in Totem. I guess some of the functions here can be also serverless, right? Because you want to have a little bit more monitoring, a little bit more power associated to them. Yep, absolutely. But the way that I did wrote all of the different code, for example, the code that turns your avatar into a sprite is just a javascript file that defines the 3JS interface for generating that and the mesh for rendering it. This is what's staying in the store, right? On the general settings of the application, so to speak. Yeah, yeah, yeah. Got it. And then the engine, based on that, will respond by turning you into the sprite. But the code for generating the sprite is something that we could actually ship right off to the back end. You saw there was a pause there. That's because it's doing it all on the front end, but this could just as easily be done off-thread. I just haven't had time to optimize it. That's the nice thing, though, about the web, is that you can basically move code across computers and it totally just works. It's just a buffer. Yeah, I mean, if you have, like, node, yeah. If you had, like, a little bit of, like, a different backend framework, I guess it would have been a little bit more complicated. Yeah, but from the start, we've tried to keep it as bare bones and compatible with everything. That's how we've been able to kind of glue this all together with pure open source components. This is the avatar cruncher code, because it basically takes your avatar and crunches it. In this case, it's the code for atlasing. And then we have another function, which is the avatar spriter, which also does the math of positioning the cameras in the different places, animating the avatar because you get the full walk cycles, you get the jump cycles, then you get the different actions, like, for example, crawling. So it poses the avatar in all these different positions, positions the camera, snaps the snapshot, and then it actually generates the full sprite sheet image. And then we have a shader, which takes the camera normal to the plane. This is actually just a single plane drawing, but it takes the camera normal, and then it finds the correct frame from the sprite sheet, from the texture, and then it just draws it. It's actually a super cheap operation. It's one texture lookup. And basically that lets you create characters, which are as simple as you can possibly get. Which is hopefully going to be the temporary solution for how we can get thousands of people into these kinds of worlds, because the bottleneck at that point tends to not even be the graphics, it's just the network volume, and how do you pipe that correctly? So it doesn't really feel like at this point there are really many limitations to what we'll be able to do. I wonder if that works. We also have a system for characters to kind of be transformed by additional shaders. Somebody has a question though, I'll take it. There we go, it worked. Like basically you can detect when it's a character's hair, and then just change their hair programmatically. So this is one application interacting with another, just due to the fact that the standard file format is being used and you can detect what is hair and what is not. That's also what allows us to do things like when you zoom into the character, you actually don't see their face, you don't see their hair, we just kind of clip that out using the vertex shader. We just kind of zero out those positions that were in that morph. And then when you scroll back out, you can see the avatar's full head and it kind of makes the experience relatively seamless. It works for most avatars, you can just kind of drag and drop them in and you get either first person or third person. That's tied to the action of like the mouth wheel or something that switches the camera pretty much. That's right. So in terms of animation, I saw that there is like a switch case in there. So you were able to play different animations at the same time, or there is some kind of synchronicity between those kind of actions? Multiple actions can happen at the same time. So for example, aiming is an action and it makes your avatar face a certain direction, but I'm also kind of moving. So it's the intersection of these two different actions that create the animation, which is that they're kind of like walking sideways. And when you are wearing an object, that can change the functionality for what does aiming mean. These animations are actually still kind of rough, we're going to do a clean transition, but our entire animation system is aware of all the different actions that you can have on your avatar and we specify what the list of those is. Like for example, this is the back sword type of aim. And then the sword just defines that I am the kind of object that when you aim me, I should basically have this. And then when you click me or when you trigger me, then I should attack in this manner with these combo animations. And the avatar just performs that. The binding between the object that defines the animation is mostly just an offset of one of the character's bones. Like in this case, I think it's the wrist because that's the closest thing to the hand and then we just offset it so that the character is holding it. You have some kind of like player stage that says, hey, I'm in vehicle, hey, I'm handling a sword and all that. That's exactly right. And that's actually, I can show that as well. Actions, probably game. So here's the aim action. I think there's a better one. Yeah, the aim action. Most likely when the user, yeah, when the user right clicks, that's handled by the IO manager. And then the IO manager is gonna call into the game engine kind of to say, well, what should I do? In this case, it means add the aim action. And then later the avatar system itself, it recognizes that, hey, this player has this aim action. I can show that code as well. Which will then include its animation into the animation stack for the character. And there is a bunch of rules for how those animations stack and compose. And there's also a bunch of bugs that we need to fix. But the idea is that no matter what your character is doing, and no matter how you're kind of moving around, we want there to be a believable animation. In some cases, that means you have to break kind of the avatar into two parts, a top half and a bottom half. Because you want the legs to go one way, but they're holding something in some other way. So in a multiplayer environment, would the other player know that you are in a certain player state? Yes, because I forgot to mention, this is actually using the same CRDT when you're connected to a multiplayer room. It's also your own avatar that has its own kind of state. So your avatar has its own inventory, and that's not related to the world inventory. It's bound to your avatar. And in fact, when you drop something into the world and pick it back up, it's moving between those different app managers, which also allows the possibility for when you go to a different world, all the things that you're wearing also come with you. And all of this is replicated using the CRDT. And basically the entire game engine is just responding to these changes in the CRDT. Like this object disappeared from over here and it appeared over here. So that means I need to move it. I need to reanimate it on the avatar because they just grabbed it. But in the shared world, so to speak, like when you do multiplayer, the other people can also see what kind of item you're dropping for that reason. Yes, because the single player and the multiplayer code is exactly the same. It's using the same data store. The only difference with multiplayer is that that state is replicated. And then in addition to that, we of course do the actual voice transfer, so you can speak. In fact, the more efficient way to do this once we fix VR support is probably going to be to do the animation once on one client side and then push the actual bone transforms to the network. And then the rest of the clients won't actually need to animate it because as I mentioned, this is actually one of the most expensive parts of the entire engine is just animating avatars. It's actually a known problem. This is how VRChat works as well. You basically just want to animate it once on the client and then not on other remotes. And then the remotes are just playing back animations. So that's kind of an optimization that we can do on top of this. But the way that it currently works is just the actions that are replicated. The engine, whether that's single player or that's some remote client is just picking up those transforms, interpolating them and then passing them to the application so the users essentially see the same exact thing. I guess there is something that I like. Maybe you're going to expire to more like the newly Adobe-obtained product or the one that pretty much it's able to understand what kind of mesh and skeletal mesh it has to apply. Sounds cool. I haven't heard of that. I don't remember the name of it. I'll definitely send it to you through email, but as soon as I remember. But it's going to be pretty much easier kind of like problem because as I was mentioning before, I guess like the skeletal mesh is like the most expensive one. So if you have like a boss that has kind of like more intricate movement or some more kind of like floating movement or anything like that, I feel like that could be like really expensive and she have like thousand players at the same time. Oh man, you need like a T630 for that. So you mentioned that like when you were talking about animations that the animation that the avatar has sort of depend on the sword or the weapon that they're carrying. Does like the weapon send signals to the avatar, so to say, or how does that work? So the weapon is a bunch of metadata. I can actually show it, which is then interpreted. In fact, this is the code for this exact sword. So first of all, there's just, it's saying that it's index.js and this is just because the sword has additional features like it can actually carve things into the world. I don't know if that's working right now, but it basically has that feature. So that's why in addition to being the GLB, it also has a script, but it also specifies, for example, that for the aim action, the app should play a certain animation and the player should also play a certain animation. So in this case, when you aim, the player should go into the animation of sword side idle, which is exactly what happens when we right click. And this is all kind of defined and parsed into a javascript file that then you get on the front end, which implements this behavior of adding this action to the avatar, removing it when they un-aimed and basically making sure that the avatar animates in accordance to the thing that they're currently wearing and activated. And we do have the option of, for example, having multiple items. So you can either hold the bow or you can hold the sword at the same time, just based on the keys that you're pressing. And this just changes the state of the avatar accordingly. Is the system aware if an NPC is person controlled, but if it's AI controlled? Yes, so there are different classes in the code. There's a local player and then there's a remote player and then there's an NPC player, I believe. I mean, that's an easy check. I can even show that. Well, based on that, you're able to kind of like understand how much rendering you have to make for an object to be placed in the world. This isn't generally how we estimate that. Like the rendering costs for an object tends to be more a property of how many materials it has and how big the geometries are. Those are the two big factors. So that generally is not related to the actual players. We handle that at a much more lower level. But for example, the difference between a local player and a remote player is most important for actually the AI features that we have because we need to know when we're basically chatting with the system, is this something that's from the player? Is there an NPC that should move, like perform some sort of action? One thing that I haven't even shown is that we actually have AI-generated selectable voices for all these different characters. So in addition to some of the characters that we're creating in this list, we're going to be adding a whole bunch more. And they all have their own kind of unique AI-generated voices, which right now it's actually easy to test that you can go into the audio settings and just change the voice to somebody else and then your character has an entirely different voice. Is that a neural map that understands what's going on and kind of like learns from the fighting style or adapts the boss difficulty or anything like that? Not at the moment. We're getting started mostly with just procedural generation of quests using the AI system. In the future, we'll probably, once we figure out what's the most fun thing to do and what do people enjoy the most, then we'll probably find ways to how do we make this even better and maybe give these certain characters memory. That actually sounds like a really cool idea. Right now it's just mostly how can we make something that's compelling really fun to play? And then the sky's the limit from there. I really wish I could show a lot more of the AI features because it blows my mind. In terms of AI engines, it's actually just a selectable option in the application. If you have your own api key, for example, for OpenAI, you can enter it. We don't actually store that. That's entirely local. And then you can basically use OpenAI with the app. We're hoping to be able to expand that to basically everybody for free. So you don't even have to do that. But that's the thing that requires OpenAI's approval before we can quote unquote ship it. We want this to essentially just be an awesome experience for when you just kind of jump in. You feel that for the first time. You opened a website and you're confused. Is this a website or is this a serious video game? Are you going to keep it at low poly or are you actually going to apply it like a little bit higher graphics? No, we're probably not really trying to be realistic, but we're probably not going to be low poly quote unquote. It's probably going to be closer to Fortnite. So like a cartoony style? Probably. We found that a lot of people want to build a lot of different things on these kinds of technologies. And it feels like the cartoony style is the thing that's most compatible. So when one character goes from one world to another, they don't necessarily feel out of place if it's a cartoon. So that's kind of what we're probably going to be doing for our first party stuff. But we encourage people to basically build whatever. There's been other stuff that's actually trying to be realistic. And it's all really pretty easy to set up if you can make it in Blender, then it's just a drag and drop. I love that feature. I guess you can either help or confuse so many people during a quest by duplicating the quest item or something that doesn't have any interaction. I'm actually really interested in what the game is going to come up with because the techniques that we're using, we're just learning from the best. The techniques that we're using are inspired by the games that we grew up on, like the Minecrafts and No Man's Skies and so on. We're just re-implementing the best of that stuff. And if you've ever played something like No Man's Sky, you know how crazy it is, like the feeling when you first drop onto an alien planet and then there's so many things to explore and you feel like there's a living, breathing world. Another one that I really enjoyed was Subnautica. So if you didn't check it out. Yeah, absolutely. I wonder where the cave entrance is. Oh, there it is. It turns out that things like web audio are also quite good. So we can get convolutions in the game engine, essentially for free, just with metadata defining that this is an area with this much reverb. And again, you can get positional sound for the things that your characters say. I actually don't know if the boss is enabled for this. Looks like not yet, so we might be safe. Another thing that we're experimenting with, since we're doing quests, procedurally generated stuff like that is, for example, a party system, where sometimes you don't just want to be one character, you want to be a group of characters. So how do you kind of switch controls between them? That's mostly just a matter of tweaking a bit of code. That's mostly just a matter of tweaking a bit of code. I'm curious to know, how did you architect a procedurally generated quest? This is really early days, but right now, just based on the lore mostly. So the way that the OpenAI system works is there's a virtual scene. In addition to the real scene, which you're rendering, there's two more scenes, I guess. There's the physics scene, which is kind of the view of the physics engine, which is not the one that you're rendering. But there's also the third kind, which is the AI scene. And so what the AI scene is doing is it's understanding all the different descriptions, even textual descriptions, and the parsed out descriptions of the data that you've loaded. So whatever this thing is called, and whatever additional descriptions you said for the scene get fed into the AI algorithm, which then outputs what it thinks would be the best quest the most fun quest that we could have in this territory. And then that's once again, parsed out by the engine based on a pre-made set of quests. We're expanding it. And there's tons of people who have many ideas, but one simple one is like go and slay this boss because it understands that this is boss.glb and the world says, oh, this is a cavern which has a dangerous creature living in it. When you plug something like that into a system like OpenAI and you just do proper prompt generation, it will actually output a reasonable approximation of, hey, the best quest would be, go over here to the quest entrance, then follow this path to go and slay this boss and your reward will be such and such. Once you parse that out by the engine, you have basically the procedurally generated quest. And we're constantly exploring with like what the limits of that is. That's super interesting. I wanted to learn more about it. Any reference you can point to me? I feel like I wish the answer that other people had for me was yes, but like a lot of this stuff is just new and I've mostly had to learn it all myself. I don't know. But what I really encourage anybody, especially like on the AI side, if you're really interested, just go and play with OpenAI. I think it's open now for everybody. You can just try some prompts and I think you're gonna be surprised. And if you're an engineer, it's pretty easy to kind of convert people and make them realize what this can actually do for something like a medium like video games. Is there any part of that on your GitHub which I can look for reference? Yeah, there's a lore or sorry, there's an AI folder in our repo. You're saying AI, but like the main theme of this is the fact that you build a game a pretty good game with just JS that runs is going to run on web. But what does AI have to do with that? Isn't AI like a smaller thing in this? Isn't the bigger thing like the engine, the shaders and the performance optimization, et cetera? I think it's the combination of everything because if we talk about something like Minecraft, which by the way, like a lot of people ask for examples of like what are some games on the web that like actually have had success? Well, Minecraft actually started as one of those things. It was a Java applet initially and it certainly like didn't even have the best graphics at all. So it's actually really hard to say what the focus for the future of like the next Minecraft or Fortnite should actually look like that's using all these open technologies. I think the AI part of it though is one of the key differentiators that is only possible on something like the web where all of the metadata has been precisely defined and structured in the ways that basically make the web just such an awesome platform. You can just parse it out. Once you combine that with something like AI, that's what allows, for example, mostly a one-person dev team, which for most of this project's lifetime, that's all it's been. It allows a one-person dev team to basically make an entire universe. And I kind of want to convince people that like that's probably the way that we should be creating and kind of self-empowering ourselves. And I'm just kind of trying to show people and inspire them, I guess. You probably talked about this in the beginning, but I was a little late to the workshop. So are you working on this all on your own? No, no, no, we have a team. We're actually... No, we're hiring as well. I hope that one of the people on the team can actually share some links if you're interested. No, we need a lot of help to build this. Despite how far we've come, there's still a long ways to go. Yeah, we can get lost here. So the AI is doing like matchmaking for Quest or it's very like zero-pronged where it's making new quests. Hasn't we seen before? Two different things. There's different systems that are mixed together here. Some of it is just procedural generation, which I don't know if that counts as AI, but that's the kind of thing that can, for example, place different classes of characters in the world. But then on top of that, you have the entire chat system and the personalities of all the different avatars. So that's where we're going to be generating, for example, lores and bios and descriptions for these different avatars, which gives them, for example, different stats, different objects and different behaviors. So actually, despite the system being AI controlled, it's kind of interesting because you can't directly tell the game to do anything. The game is you have to convince the AIs to do it. And some of the AIs that you're going to meet are not going to do what you want them to do. Like their personality might be really stubborn or like that they're going to rip you off. So basically, the AI layer allows us to mix our procedural generation techniques in entirely new ways that kind of build a livable, breathable world. Despite the fact that this is all just web assets, what you really have behind it is like the knowledge of the entire internet, like the entire history of the internet trained into an AI that's kind of telling you an immersive story with all of these assets. And then it's our job to make that entire experience really fun, interesting, and to add in the game mechanics that we just kind of want to play. Like, hey, this is a really fun game. It's a really cool way to kill time. It's a really cool place to make art. Maybe make some friends, I don't know. I'm yet to see a really good game which uses like multimodal AI models like text to image, text to 3d assets. I think that's all. All that stuff is in its infancy, but I think once a lot more people start waking up to the different combinations that you can have there, it's probably going to be transformative for a lot of industries, especially things like games. I know this is exactly what Mark Zuckerberg wants to do. Like, I think Mark Zuckerberg and I probably would agree on a lot of things, but the problem with what Meta is doing is they're trying to build all this stuff on the wrong foundations, essentially. Like, it's all built on top of advertising and all of these closed-source platforms that are each extracting taxes from the user and sometimes like taxes that we don't even understand, like, taxes on free speech. And I think we have the opportunity to actually make something way better by basically just using the web, the web that's always been there. The web that everybody is already using, just know we really took it seriously as a platform for Metaverse. I was actually debugging html rendering features. Since we're in the middle of a pandemic, html rendering features. Since we're in the browser, we can actually just drop html pages in here, but I still have to fix a bug there. That's just using iframe transforms. Are there any other interoperability which is in your roadmap? Can you expand on that? There is interoperability on avatar level, assets level, people's identity, let's say on blockchain or NFT level. What amount of things I can carry from one game to another game in various Metaverses? We're probably going to cryptographically sign all these things just on blockchain. One of the things that we're going to do, one of the most asked for features actually is to be able to in the URL just specify somebody use ENS. That will just kind of load their own gallery that they've set up, which is just going to be kind of written to our contract or stored on some system like Ceramic. There is actually login with both MetaMask and Discord. With the MetaMask login, we'll just pull all those tokens. We also have a minting system in the works so that anything that you're dragging and dropping will kind of give you a wizard for defining all the different metadata for the object that you're dropping, and then maybe you can even mint it right here, which will once again just be added to your MetaMask, and there's no fees for that at all, as long as I think we're probably going to limit it to people who actually own one of our tokens, like a copy of the game, so that we can actually provide you reasonable hosting service without breaking our own bank. But it's also possible since the stuff that we're doing is just open source contracts, anybody can kind of load up whatever they want and mint their own things. It doesn't have to go through us. Interesting. And have you, just out of curiosity, have you seen any algorithm or any workflow where people are able to make a character out of their 2D NFT and generate avatar or like whole storytelling? I've played around with it. There's some easy wins that you could do there, but I haven't seen anything that was really compelling for me. I think that's probably kind of going to be the future where we have some sort of assets and it's transformed through this AI process or something into some other form. The problem is that it requires a lot of guesswork most of the time because if you have a 2D PFP, that really doesn't tell you everything you need to know. It doesn't have even a full body that you could generate. But if anybody, by the way, has any interesting techniques, I'd love to hear about them and test them out. Because ultimately, we do want to make sure that anybody who has some sort of data, whether that's on chain or not, is able to have a good 3d experience and they don't just have to be a 2D plain profile pick. All right, well, if that's all, thanks for having me. You can either follow us, I guess, it's WebReverse on Twitter or if you want to follow me, I'm WebMixedReality. I haven't been tweeting lately. I might actually be doing streaming sessions though, but that's probably going to be announced on the WebReverse Twitter.