JS GameDev Summit

JS GameDev Summit 2022

Building Fun Experiments with WebXR & Babylon.js

David Rousset

React Summit

React Summit 2022

Exploring AR Try-On with React Native

Wow, hello. Thank you all for coming. I'm really excited to be here today, and I'm going to be talking about exploring AR try-on with React Native. So, here's the mandatory intro slide. Hi, my name is Kadi. As Jani said, amongst his very kind words, I'm currently the head of mobile development at Formidable. I've been an engineer for about ten years, but for the past five years, since 2017, I've been building things in React Native. I've been very fortunate to be able to work on some really exciting projects in React Native, and I'm very excited to share one of them with you today. So, in this talk, it's going to be in two parts. So, first, we're going to talk in general about what's happening in the AR VR space, and in the second part, I'm going to go through a bit of a case study of how we actually implemented a virtual try-on feature in a React Native application. So, to kick us off, what's currently happening in the AR and VR space? So, I will do a quick clarification on the difference between AR and VR, because we tend to use those terms together. VR stands for virtual reality, and it is a fully simulated experience, so it may look like the real world, or it may look completely different. So, the distinguishing feature in a VR experience is that you will have some sort of a VR headset, so it will be an Oculus, or a Google Cardboard, or an Index VR, but the point is you need to wear something in order to have this immersive experience. An example of that is, for example, Beat Saber, which you will see on the screen. AR stands for augmented reality, and it is a process in which the real world is enhanced by some computer-aided graphics. So, usually, for an AR experience, you will be looking through a screen, so it could be your phone screen, it could be your laptop screen that has a camera that's recording the real world, and then your device will add some kind of computer-aided enhancements. So, for example, you could use it to place furniture in your space. If you've followed the news recently, you might have seen that last week there was a news story making rounds that Amazon has launched an AR-powered virtual shoe try-on experience in their iOS app. Now, this in itself is not particularly groundbreaking. Amazon are not, like, they're nowhere near the first, and they're not going to be the last company that launches something like that. But the reason that I found it significant is that it's speaking to us of a trend that is happening more and more, which is that in e-commerce, AR is going to stop being a gimmick, it's not stopping the cool thing that maybe 5 per cent of the people would use, and it's more and more starting to be an expected part of our shopping experience. And just to give you a couple of examples of myself using AR in my shopping experience, so these were a couple of things that I used before I even knew that I was going to be giving this talk. So, Bailey Nelson is a company that sells glasses, and I like their glasses quite a lot, and on their website, they have an AR try-on where you can try on glasses before you purchase them. I actually got this pair, so that was very pleasant. Another thing, so Gucci have an app that's just for experience features. They have a couple of different AR features. They have a shoe try-on, they have a make-up try-on, and in this example, you can also try on different nail polishes. I recently moved house, well, I say recently, it was the beginning of the year, and it was my first time moving into a completely unfurnished house. It was my first time moving into, sorry, I can hear some echo. It was my first time moving into a completely unfurnished house, so I needed to furnish it from scratch. Here was me trying to figure out how to place a sofa into my very weirdly-sleeping living room. As you can see, that corner sofa was wishful thinking. I did end up going with a smaller version of this two-seater sofa. Now you might be wondering how do I do this in React Native? Well, with everything in React Native, we're going to start by looking at the underlying platforms. Both iOS and Android provide an AR platform. In iOS, it's called ARKit. It was launched in 2017, and it's available from iOS 11 onwards. Android, not to be far behind, launched the equivalent on the Android platform, it's called ARCore, and it was launched in 2018, and it's supported on Android 7 and onwards. As ever with React Native, someone somewhere has written the React Native bridge for the native libraries, and in this case, it's called Duvero, which builds the bridge for ARKit and ARCore. It's an open source library. I think they've moved it into the community space now. They're not actively maintaining it. If you want to try out AR in React Native with no native code for yourself to write, then this would be a good place to start. Now, let's look at how we actually implemented a virtual try-on experience in a real-world React Native app. As always, let's start with the requirements. So, what our client needed was an AR shoe try-on experience. So, we are in the e-commerce space, we are selling shoes, amongst other things, and on our product details page, we want the users to be able to try the shoe before they purchase it, and hopefully improve conversion. They wanted this to be part of the main shopping app, so some companies have made the decision to have a separate app specifically for experience features, so Gucci is an example of that, but for this client, they wanted it to be part of the application that all the users will have, regardless of whether they actually use the AR feature. They also wanted this to be a bespoke UI, so the UI was designed way before we even got to finding a provider or building any of the code. And in terms of resources, we had in the order of weeks rather than months or years to build this, meaning that it's unlikely that we will be able to start completely from scratch. So based on these requirements, we found the solution that will work for us. So, because it's part of the main shopping app, the bundle size and the API level matter, so we can't add anything that will add hundreds of megabytes of space because every single user will have to download this app regardless of whether they use the feature. Also, with the API level, we need to make sure that our minimum API level is as low as possible, again, to make sure that the most amount of users are able to install this app. And finally, because we have weeks and not months to build it, our only real option is to use an off-the-shelf solution for the native platform and build the React Native integration with it. And just as a side note at this point, to really take your React Native engineering to the next level, I very much recommend that you start learning native code. So, if you're someone that came from the web, like me, you might find and your first experience with iOS development was a bit of Objective C. Sorry. Yeah. A bit of Objective C. You might find it terrifying. What are all these brackets? However, on iOS, you can use Swift, and on Android, you can use Kotlin, and both of these languages are much more lightweight, they're much more functional, and much more easier to grasp from a JavaScript developer. And once you get more acquainted with the native platform, native languages, it will really enhance the amount of features you're able to add to your React Native application because it opens the door of being able to install native only SDKs and write your own bindings comfortably. So, for our AR try-on, we looked at two different companies. The first company that we had used is called Wana. They are a company that specializes in virtual try-on. So, they have an SDK where they've implemented a virtual try-on experience for shoes, and you can use their SDK to basically build your own integration. You can actually try this out yourself. So, this is their example app, which is available on both the iOS and Android stores. So, this is an app that uses their own SDK, and you can see what the try-on experience would look like. So, from an implementation point of view, we got in touch with them. They sent us a demo SDK so we could have a look at integrating it and seeing what it would look like in our app. So, here are some screenshots of when we built the integration. This was obviously a proof of concept, so the UI isn't nothing to write home about. But in general, the API itself was pretty straightforward. There wasn't a huge amount of native code that we had to write in order to build the integration. It was fully customizable, so we were able to just render the virtual try-on video and then do everything else in JavaScript. The thing that was kind of a deal-breaker for us is that it had quite higher minimum API levels than what we were aiming for. So, it requires Android API 26, whereas we were at 24, and it requires iOS 13, whereas we were at 11. Now, this is a trade-off for some clients, for some customers, for some use cases. This doesn't matter, and you can just aim for higher APIs. But in our use case, our client was quite sensitive to make sure that no one gets missed out because of their older API levels, so we didn't want to upgrade. Did you know that Snap, the company that builds Snapchat, has a developer portal, and you can use a lot of their AR technology in your own application? You can check out developers.snap.com. They actually do more than just AR try-on, but the point is that they provide a couple of libraries, most notably for us, a library called CameraKit, which is basically the SDK that powers Snap itself. It does a lot more than just virtual try-on for shoes. It does makeup, it does clothes, it does front camera, back camera, everything you see in Snap, so it's very, very powerful. And under certain circumstances, you can use it in your own application. And something new that they've recently built to enhance it that's currently in beta is the AR Shopping Extension. So, this is something that Snap is providing. It's alongside their CameraKit, and it's specifically built to create an integrated experience for AR on the e-commerce product details page, so within the shopping experience. And using the Snap's SDK, this is what we ended up with. So, starting from the left, we have the product details page with the AR badge on the products that have an AR. Then we have the product details page with the AR badge, and finally on the right, we have the module with the actual AR experience. I realize this is a technical talk and I haven't written, I haven't shown you any code, which is basically a crime, so I've added some code. I'm not going to show you the native side of things, so I'm just going to show you what we actually import and use on the JavaScript side. So, on the product listing page, we have these AR badges, and what we need to know from the Snap SDKs, which is the source of truth on whether something has an AR experience, we pass in the product IDs, and it does its magic and tells us whether or not this product has an AR experience. If it does, we'll show the AR badge. We use the exact same method here on the product details page. And this is the interesting part. This is the actual AR experience. Being the React native developers we are, we made the conscious choice to have as much of the code as possible on the React native side. So, the attribution, the product name, the camera button, and the variant picker, these are all rendered using React native. So, the only thing actually rendered natively using Snap SDK is the actual camera experience. So, the way that we've exposed this is as a native component. And the way it works under the scene is that we pass in the product ID to this native component. Then using the Snap SDK, we fetch the products. So, this is from the shop kit. We fetch the products for that variant. Then we call the on products load at callback, which will then populate the products underneath the screen. Then next, what the SDK does is it fetches the lens for the selected product. So, a lens is kind of a Snap terminology, but it's basically the virtual reality experience with something inside. In this case, the pair of shoes. And then when the lens is fetched, we're going to call on lens loaded callback, which will then tell us that it's time to stop the loading spinner and the user is good to go. And let's look at this experience end to end. So, here I'm on a product details page. I'm gonna scroll down to the launch AR. So, in this case, this is the first time we're launching it. So, we have to say yes to camera, agree to terms, which we've definitely read. So, here we can also look at the 3D model of the shoe, which is actually built into the lens, which is pretty cool. So, you can change colors and see what it looks like from either side as if you were holding it. And then if you swap to the AR experience, obviously, you can look down and you can see the shoes on your feet, on your friend's feet. You can also mix and match and show one of your feet, one of your foot to one of your friend's foot. That works as well. And finally, you can take a picture and share it with your friends. So, in summary, the snap SDK, I will say that there was quite a lot of native code that we had to write. So, the previous slide that I wrote about embrace native code, it was there for a reason. I definitely had to embrace the native code for this project. But on the upside, they did provide us with a fully working example app. So, obviously, the UI didn't look like anything like us, anything like us. We had to change things quite a bit and build it from scratch, but it was a really nice help to get started. On the upside, it was a very, very smooth experience. You can really tell that this is an SDK that is powering a really powerful app, the Snapchat app itself. They've had lots of engineers and years and years of practice of making this as smooth and as nice as possible. So, it was a very nice end experience with kind of a minimal effort on our part. And very importantly for us, it works from Android API 24 and iOS 11 onwards. So, in summary, in this talk, the three main things I talked about was Vero, which is the React Native bridge for AI if you just want to try it, try out AI in React Native. And then for virtual trial in particular, we talked about Wana and we talked about Snap for developers. So, if you're interested in either of these things, I would definitely check them out. And just a shout out to Formidable, the company I work for, thanks to whom I am here, who are also with you lots of open source. Check us out at formidable.com. Thank you very much. Thank you so much. Oh, hello. Thank you so much, Catty. Now, we're going to do a brief Q&A. We're a little behind schedule, so we're going to keep this in with a few minutes. And then if you have more questions for Catty, you can find her at the speaker Q&A booth afterwards. So, the first question from the audience is, how did you get the 3D model for the shoes that you're selling? That is a great question. Actually, so there is another company called Vertibray that we work with and they are the ones that actually, well, they get shoes shipped to them, they take the 3D model of the shoe and then they work with Snap to get it integrated. So, could anybody just go and get their shoes modeled or is this like a sort of like an enterprise deal type of situation? I think this particular one is an enterprise deal type of situation, but I'm sure there are ways to do it as an individual person as well. I think the easiest way to actually do it would be to create a 3D model in like some kind of 3D modeling software and use that rather than the real shoe. That's cool. And by the way, I think we do have a talk about that. Sarah is talking about, Sarah Viera is talking about modeling in Blender later. So, that might be a good time. Perfect. I have a question for you. This is not from Slido, but you were given a brief to do this within a few weeks or within the timeline of weeks. That sounds scary and difficult. How did that actually go in the end? I think there was always, I think people tend to figure out things based on the time frames that we have. I think if we were told that we had two years to build this, we would have probably started from a much lower level and we're like, yes, we're going to build our own custom AR solution. But knowing that we had a couple of weeks, then that made that option go out the door straight away. So, we were like, okay, we need to find the best SDK, best tool for this job. Nice. Thank you. So now, if you do have more questions for Kati, that was all that we have in Slido. So, feel free to go and talk to Kati in the speaker Q&A booth. We're going to be moving on to our next talk. Thanks so much, Kati. Once more for Kati Krammer.


Kadi Kraman

DevOps.js

DevOps.js Conf 2022

Gaming the System: How Video Games Can Help us Create More Effective Virtual Teams

♪♪ Hello, everyone. I'm Jessica Craig, and I'm a developer advocate with LaunchDarkly. Now, I'm here today to talk to you about video games and what they can teach us about building better virtual teams. Now, when something goes wrong, what is your first instinct? Do you ask for help? Because you'd be forgiven for thinking that collaboration is indeed the answer to your problems. Between the idioms of two heads being better than one and a problem shared as a problem halved, it's understandable to think that bringing another person into the mix could stand to improve your chances of getting towards the solution and getting you there faster. I mean, it works with pair programming, right? But this isn't always the case. Now, if you've played computer games and remember Age of Empires 2, you'll recall that the best way to get a structure built quickly is to assign as many villagers onto that task as possible. But when it comes to digital infrastructure outside of a video game or indeed video games themselves, we need to factor in a few key differences. Now, the Mythical Man Month, as many of us know, is a collection of essays on the craft of software development that discusses the idea that the number of people and the amount of time required for something to get done can be interchangeable as commodities only when a task can be partitioned among many workers with no communication among them. But when a task cannot be partitioned, either due to its structure or indeed its complexity, the application of more effort has no impact on the schedule at hand. Now, software development falls in between these two aforementioned categories. Tasks can, of course, be partitioned. We can break something down into smaller batch sizes, but communication is required in between each subtask, either for problem solving or for handing off for status. Now, in these situations, the best possible outcome doesn't come close to an even trade-off between people and hours. Now, there are a few schools of thought around how the cognitive load towards communication is weighted, but it's widely accepted that intercommunication is where the majority of the work lies. Now, if each part of the task that you're working on needs to be separately coordinated, the effort required to complete that section of work increases at a near exponential rate with the more people that you get involved. Now, three people will require three times as much interaction as two people to get the same amount of work done, and that just increases, as we can see with the line here. Now, things only get compounded when group meetings come into the mix. This projected timeline, it really shows Brook's law in action, that adding people power to late software projects only stands to make them even later. But to tackle this, let's talk about the concept of collective intelligence. Now, collective intelligence is described by researchers as a group's capability to perform well. It's essentially an index of group-level competence that despite being widely recognised, is still being actively studied and understood. Now, one study used the vehicle of massive online battle arena game, League of Legends, as a method of testing whether or not virtual teams paired based on their abilities, rather than an in-depth understanding of their working habits and one another, could outperform the latter. Now, what the hell has it got to do with work? Well, massive online battle arena games are often characterised by their intensity, their fast decision making, and their competitiveness, which is not quite unlike the real-world organisations that we work in every day. And what did these researchers find? Well, when investigating the measure of collective intelligence, researchers found that a group's ability to perform well at a task was largely transferable, meaning that if they could do one thing well, they're likely to be able to do new tasks well in addition. Now, the things that can make us collectively intelligent are social perceptiveness, a moderate level of cognitive diversity, and large amounts and the equal distribution of communication. But on this last point, researchers noticed that the verbal communication didn't equate to, or high levels of verbal communication, rather, didn't equate to a high level of collective intelligence. In fact, researchers measured the standard deviation both in the lines of chat of successful players and the word count within those chat messages. And they found that higher chat lines or more chat communication didn't equate to a high level of collective intelligence or, indeed, performance. One of the early observations of the study, in fact, found that the most skilled players tend to spend less time communicating by preference. They're good at the game, and that's where they want to spend their time. They minimise context switching. They engage in non-verbal cues almost instinctively. Does that sound familiar? A taste of communication mentioned here is the unexpressed recognition of the position of others. And most importantly, it leads to actions for common goals. In the context of a company, this looks like this is evidence of advocating for observability measures, automation, but of course opting for metrics that everyone genuinely understands. If we don't understand the software that we're testing, we're likely to restrict the number of times we deploy. And as anyone who's read the book Accelerate or followed the annual state of DevOps reports will know, optimising for team performance can be achieved by tracking these four key metrics, all of which you want to lower aside from deployment frequency, which you want to be as high as possible. Now, if the software that we're working with doesn't fit neatly within our heads, these metrics will have a near finite value. We're likely to distrust our tests in the first instance. Our confidence to deploy will plummet. And if we don't understand the services that we're working with, we won't be able to restore them when they inevitably do indeed go down. So as a quick recap, keep your team small. Aim for no more than nine on your squad if possible. Keeping teams cross-functional with one team member intentionally chosen to help you enforce this final principle to take on people with the reputation of specialists, but with the capabilities of generalists is a great way to get your virtual teams off to a good start. Enable taster communication by making systems easy to understand and easy to traverse. Lower cognitive loads. Our cognitive load needs to be kept low so that everyone knows what to do. This helps us strengthen our systems. And when we communicate and we do communicate, it becomes far more meaningful. Now here's the list of resources. And finally, thank you so much. I've been Jessica Craig, and I hope you have a wonderful DevOpsJS.


Jessica Cregg

React Summit Remote Edition

React Summit Remote Edition 2020

Virtual Reality with React

Hello, everyone. I'm Hamza Hawi, mobile and web VR developer and founder of HYO. As you know, I'm going to talk about virtual reality with React and how can we create new VR experiences using React 360 framework and run it into our web browser and all the VR devices. To get started, what is React 360? React 360 is JavaScript framework created by Facebook for the creation of 3D and VR user interfaces built on the top of React. This library is designed to simplify the creation of complex UI. And as you know, React 360 allows us to use the familiar concepts and tools of React and React Native. To get started, all what you have to do is install the React 360 CLI via npm, then init a new project and do an npm start to make sure that everything is going fine and you will have your default project into your web browser. For the folder structure, we have the static assets folder, which contains our 3D objects files and the material files, sounds, images, videos, all the media files should be here. And we have the client.js file and here where we create our locations and surfaces. And we have the index file, which is our main file. For the components, we have four main components. First, the views, which is kind of like divs, and text component and the image component and the VR button, which is a normal button, but on VR. The most important thing on VR is surfaces. We use surfaces to render the 2D UI components and we have two types of surfaces, cylinder surface and the flat surface. We have two angles, the yaw angle and the pitch angle. And every surface should have a width and height that should be given by the developer. For the location, locations are used to render 3D UI components. And we use the Cartesian coordinate, x, y and z. And at the end, we create a new route, which will help us to connect our surface to the upper registry on the index file. As you see here, we have the z-axis and here if z is negative, the object will be on the front. If z is positive, the object will be on the back. And we have the x for the pitch and y for the yaw. Here is an example. So here we like to add 3D objects into our VR application. All what we have to do is import the entity from entity. Entity is used to render the 3D objects on React360. We use here the, if you like, you will use the ambient light for the light intensity. And we use the source to give the link to our object. Here our object have two extensions, the object and the material, and we can style it as you see here. We have transforms and too many options to style our 3D object. If you like to run this application on your web browser, just scan the following QR code and you will have this application on your phone or your web browser, on computer, on any VR device. And now we will move to the code. Let's see what we have. Here is what this looks like. It's similar to React Native and React as you see here. We use the flexbox for the styling and we have the client file. As you see here, we have a new surface and we give it a root, which is a slide root. And if you go to the index file, you will see here we have the app registry with the slide root and the class name, which is slides. Here it is. And for the 3D objects, we have a model, the model class, and as you see here, we have the entity object with an asset, which is a Porsche car, and we give it texture, which is a default image that we have here. So let's run it into our web browser. Let's do localhost and run it and wait until it's fresh. And here we are in our React 360 application. As you see here, we have the slides and we can change between them. We have video here and we can change our background image. I think it's all what you have to know about React 360 to get started. For further questions or help, just contact me on Twitter or GitHub, Facebook, email, and thank you for watching. I hope that you enjoyed the talk and thank you for everything. Bye.


Haoui Hamza

React Summit 2020

Create Collaborative VR Environment in the Browser with React and GraphQL

♪ Hi everyone, really excited to be here at React Summit and today we'll talk about how you can create collaborative VR environments using React and GraphQL. So let's dive in. A little bit about myself, my name is Vladimir Novik, I'm a software architect, author, entrepreneur and I work in web, mobile, VR, AR, IoT and AI industries. You can follow me on Twitter or check out my website for more information there. I'm also CTO and co-founder of Eventloop and we create a robust, performant, feature-rich online conference experience that you really want to be a part of. We have our own conference running on Eventloop in March next year and it's called Modern Web Summit and you can check out more info about us on eventloop.app or follow us on Twitter for updates. So let's start. VR in the browser. So how it started? We started with WebGL, it was so-called like Canvas 3D back in the time, 2006, then continued into WebGL 1.0 spec and now WebVR, WebGL 2, and eventually it turned to be WebXR when WebVR was deprecated in favor of having both AR and VR in one API. As you can see on the Can I Use website, there is not much browsers that are using WebVR or ready for that, so mostly just Firefox. So you cannot rely on WebVR or WebXR on Chrome. WebXR device API is also not that good, as you can see here, but it's a work in progress, so we can envision that in upcoming years it will be better, hopefully. So what are our solutions today if we want to create some kind of VR experience? I'm using Chrome here and will still be using VR. So with WebVR and WebXR, if you don't have WebXR enabled in the browser, it defaults to 3D environment and game-like experiences and so on, which is also good and nice, we'll see it today. So with WebVR and WebXR, you can actually get your sensors positioned, you can get your headset positioned, and based on that, you can do interactions with the UI. For that, you can use different tools. One is for developing that. One is A-Frame, which is an amazing library and huge community that basically wraps around 3GS and gives you lots of capabilities like physics, particle effects, movement, controllers, break-casting, you name it, lots of things. And you can actually create really complicated experiences with that just by using its so-called NT component system and using JavaScript. Now, if it uses JavaScript, obviously we can embed this in React as well, and we'll see today how it's done. Another option is just to fall back to best practices in terms of game engines. We can use Unity game engine and develop something in C Sharp, or we can use Unreal Engine and develop in C++, or their so-called feature, feature so-called Blueprints. But we are not here to talk about Unreal Unity, so we are here to talk about both A-Frame and React 360, mostly about React 360. So let's continue. Let's take a look at the A-Frame, and let's look at the examples of experience that you can get. I don't have VR enabled here because I'm on Chrome, but I still can navigate the environment by pressing arrow keys and by just looking around this wall of basic shapes. So this is a nice experience, but it's just a really simple one. I want to see something more complex, so for more complex, let's get into a directory of different examples. And one of such examples is Minecraft-like world, VR blocks, you can go around it, navigate, and it's pretty awesome what you can do just with the power of A-Frame and JavaScript. You can get all these cool stuff here. So apart from that, let's take a look at something more complex, something more like Beat Saber type of thing, if you played on Quest or other devices. So we can get something like this also developed in A-Frame. And I think it's pretty nice, also music is kind of nice here. And you absolutely cannot create something like that in React 360. So with A-Frame, you can do really amazing and hypnotizing experiences, right? So pretty cool what you can do with that. Now, what A-Frame uses? A-Frame uses coordinate systems, and not just A-Frame, basically every 3D environment uses coordinate systems. And one of these, like this system, is basically either left-handed or right-handed. And the idea is if you take your hand and you point middle finger at you, so Z-axis will be the point at you. So it means that positive X will be to the right side, and the negative X to the left side. If you need to position something in front of the viewer, it will be minus Z, and so on. So we use right-handed coordinate systems. So when you see coordinates, take in mind that, give in mind that we'll have these coordinate systems both in A-Frame, in React 360, and so on. So axis direction also changed when applied rotation. So if we rotate something to the right, then all the axis change, and we need to take this into account. A-Frame uses entity component system, which means it consists of entities, components, and systems. Entities are containers into which components can be attached. So you will see, when you will see it in the code, it looks more like entities are like React components, components are like props, and systems are the global systems like services and stuff. So the reason why it's so, because in game engines, entity component system is pretty common pattern, and A-Frame kind of mimics that system. Unity uses also entity component system, so it's pretty close. So if we look at the A-Frame code, we'll see something like that. So this just loaded, so I can navigate around, I have these basic shapes. As you can see, I need to just import A-Frame, bring a bunch of styles, and then this is the scene component, these are assets component, entity. I call it component, but in A-Frame, this is called entities. So these are entities, and like background, fog, and scale, roughness, width segments, all of that are components. So it's kind of misleading because in React, we're used to the whole thing is component, and these are props, but in A-Frame, in documentation, you will see these are entities and these are components. The main point of showing that is that you can easily see that you can just take this and abstract that into React component and start using it. So there is React A-Frame library, but I'm not sure it's maintained. I think it's not maintained for a while now. So you can just create wrappers yourself, and as you can see, it's just fairly straightforward here to pass different data to them and so on. So what about React 360? So React 360 is built for 360 user interfaces, hence the name. The main idea is that you don't create these fancy experience, you don't walk around, you just look at the curved screens and interact with curved screens in VR, or in general, in 3D environment on the web. So the architecture of React 360 is actually based on React Native architecture. We have React running in a Web Worker. When you bootstrap a new project, you will see an index.js. This code doesn't have any access to browser APIs. It's just running in a Web Worker. We have a React Native bridge that uses MessageQueue to pass all the asynchronous messaging to the browser, to the client.js. Now on client.js, you need to define native modules, which basically have access to browser APIs. So to get started, you run npx React 360 CLI init, you get your demo up, and then you get these kind of components. So this is a client.js, the one that has access to the browser APIs. So you have the init function, you have a bunch of React instance creation, and you pass a bunch of stuff, such as getting default surfaces, setting background to the whole environment, and so on. We'll see more complicated examples here, but this is the thing that you will get once you create a new project. Now what you can do, you can render something on different surfaces. And there are several types of surfaces. The most important ones are the cylinder one and the flat one. Actually, it's supposed to be... this one's supposed to be cylinder, but anyways. Cylinder is a cylinder that surrounds you. And rendering something on top of this cylinder will have something curved in front of you. So it will look better in UI. It's the best practice in VR user experience, in VR design, to put flat UIs on curved surfaces so it will look more natural, so to speak. And if you put a flat surface, then it's just a flat surface that you can position however you want, like set an angle, and that's it. So to interact with the browser, you need to define different native modules. This is how we can define them. We import module from React 3.16. We just define a class here, and we have the game state and game ID and user ID. You understand in a bit what is this game state because it relates to the demo I will show in the end of the talk. Then you basically register these modules, and you register them by adding native modules array here, and adding a function will get a context. And you can return a new game module passing the context back. So you can use the context to pass callbacks around and so on. So this is pretty cool thing that you can create different surfaces, can create a game module to basically share your logic between different surfaces. Because they are rendered in different places, basically. So on the React side, you use regular React components, and you use different components provided by React 360. For example, you have VR button, and you have different callbacks for that, like on enter, on exit, on click. Now, in VR, you have a concept of, in general, in 3D environments, you have concept of ray casting. When you project an invisible ray going into specific direction, when it hits the target, it will return back the target that it hit. When dealing with VR controllers or VR headsets in general, there is a ray cast projected from the camera or projected from the controller. And it hits the button or hits the target or hits something like that. So in React 360, VR button does that under the hood. So if you just look at the button in VR, depends on the headset or you point with the controller and enter the button space, basically with the reticle of controller, then it will trigger on enter or trigger on exit if you leave that. So it's like on mouse enter or mouse leave, but in VR. And we have our component here, game tile, and that's about it. So it's pretty similar to React Native. We have views. We have different styles we can provide, like so, for the buttons. There are specific styles for positioning and so on. We can animate using animated similar to React Native and basically use practices we are aware of and using for several years. So what about the data? How we can bring in the data into our VR environment? And in addition to that, how we can bring it real time? Because we're talking about collaborative in VR environments, we need to bring some data, we need to have data updated constantly and so on. So for that, we'll use GraphQL. And GraphQL, as you all probably know, it's a query language for API. And it's like magic that just works. And yeah, I'm really passionate about GraphQL. I talk a lot about GraphQL. So this talk is no different. We'll talk about that as well. So the idea with the GraphQL is that we can send one request to the server and get all the data we need. And the shape of the data will be the same as we requested. So it gives us a lot of benefits of having the data when we need and in the shape that we need and having all the introspection of what is happening on the server. So this is how GraphQL looks like in like sandbox environment, pretty simple one. And under the hood, we send in the query and getting the data back. The query sent in stringified format with variables and we're getting back the data in JSON. Now, this is more like realistic format, how it will look like in production. This is like Star Wars API. We have different directives, we have variables, we have different types. GraphQL is all about types. So yeah, GraphQL is amazing and we will use it for our talk as well. So what about architecture and especially how about architecting something collaborative, right? So lots of things happening asynchronously. We have different third parties APIs, we have lots of things, like lots of moving parts. And obviously, architecture wise, we need to think about more like microservices architecture. So having different microservices here and third party APIs and everything combined into one GraphQL facade that will return everything back. And the best practice to my experience is to use something like called the three-factor app architecture. And the three-factor architecture is something similar to that. You have an app which talks with the real-time GraphQL API. It updates some kind of state in the database or wherever you like and we trigger events. Now, an event will trigger a serverless function that will update the state and basically do the processing that it needs. And when state is updated, everything is propagated back to the client using subscriptions. For our example, we will use Hasura, which is an open source and free engine, gives you auto-generated real-time API. And on top of new existing Postgres and it has a bunch of features that I won't go through right now, but you can read that after the talk in the slides, there are lots and lots of features. This is just like the tip of the iceberg. To see in action our environment, we can go to Hasura console and see something like this. We have a game with a bunch of moves happening. We can see these moves, like subscription is running, and we get a bunch of data back. This is the subscription that I'm calling and I always can insert a new move and this will be updated. Now, what this backend serves is this demo. Basically, I have a tic-tac-toe game and I can change my background, I can change it from being a grid to being a Skyrim background. I can have a chat, I can scroll in this chat. The coolest thing here is that we have subscriptions in place and it's all real-time. For example, if I will click on something here, you see this one was updated. Let me click here. Again, I get another move updated here. Now, in turn, if I will go and try to stop that, we'll execute a move. Let's say we have a row number zero and column two. Let's write move here, zero, two. We insert a move. What will happen? Oh, yeah, it's like a race. It starts from zero. We have this one added or let's add another one. Insert move. You see everything is updated in VR and in real-time. This is pretty cool how we can do this with React 360 and Hasura together and create this kind of environment. Also, we can chat here. The code is in GitHub, so you're welcome to look through the code. We will briefly look through that. But you're welcome to look more closely at the code and comment on that and so on. How we add things to VR, how we add things, for example, here in A-Frame, we basically add Apollo, Apollo provider. We have our HTTP link, we have a WebSocket link, and we wrap everything with Apollo provider. Now, here in A-Frame example, it looks pretty ugly because A-Frame is not built for something like this. It's built more for games and interaction. I did the other things around. The Tic-Tac-Toe game was a game in React 360 and A-Frame is like a blog post list in A-Frame. Maybe it's a bad example. But anyway, you can go into posts here and get a subscription, getting all the data back, and you can see that you can update lots of stuff. It's different endpoints, so you will see different things here. You're welcome to play around with that after the talk. How to add stuff to React 360, you add Apollo provider, you add our game board, and basically, you execute subscriptions, you execute queries. Now, before finalizing this talk, let's look briefly through the examples here, through the code here. Here we see our client.js. In client.js, we'll have multiple things. We'll have cylinder surfaces rendering the whole game board, and we have our VR chat panel rendering on the right side of things, and game gallery rendering on the left side where you can change the background. The whole logic is done through the native modules here. We have one native module called game module, and it will be shared later on. Not now, but it can be shared between this instance and other instances here. We have ‑‑ yeah, it can be shared across the whole thing. We can also look into index.js. Index.js is registering multiple components. Now, you have to register a component, VR chat panel component to use that, or game gallery, you have to register that, because these are different routes. You basically wrap each one of them with Apollo wrapper, and you can use subscriptions inside. Here I can render the board, and the whole idea with the board rendering is that I have the matrix that I go through, and I basically execute mutations whenever I click. One caveat, though, React 360, unfortunately, is not updated to the recent version of React, so unfortunately we cannot use hooks, so we still use the old method of render props with mutation, and you have a component mount and all of this stuff. I wish the hooks will be there soon, but it doesn't depend on me, so whenever it will be ready, it will be ready. Meanwhile, you can use A-frame easily with hooks and context and all the new React by just wrapping A-frame components and so on. So let's get back here, and before wrapping up, I just want to first of all thank you, thanks for React Summit organizer for inviting me, and I want to basically encourage you to get into VR and AR and try things and try to create these amazing demos that were created for A-frame, create amazing demos for React 360, experiment, play around with the code. VR is just starting, it's just getting into the browser, it's still not supported everywhere, but it will be, so get into this stuff, try to learn new things, and thanks for having me, and enjoy the conference. Bye-bye. See you. ♪♪♪


Vladimir Novick

JSNation Live

JSNation Live 2021

Let's Expand the Reality!

I work as a front-end developer, but actually at school I dreamed to become a game developer. And I was lucky and I got a job in an artec 3D company where I had different interesting 3D tasks with WebGL and 3js framework. And last two years I organized a skill-up school in Luxembourg and I teach people web basics here. And it's also an online school. So virtual reality. We talked today about this. And how long have we heard about these words? I think it's quite a long time. And the first movie which I found, it was in 1982, it was with very funny computer graphics 3D effects. And now it's much more interesting. Let's see what we have now in browsers and let's meet with WebXR. Virtual reality. How to understand what it is. It's when you wear some glasses, for example, or use your phone with Google Cardboard special place where you can put your phone. And you have absolutely full immersion in digital environment, you're in virtual world with your eyes and see only virtual world. In all 3D and in WebXR and WebGL, you have three axes, three dimensions. And you have six degrees of freedom. What does it mean? It means that you have firstly movements along each axis. So three axes, three movements. Forward and backward. And then you have ability to rotate around each of axes. So you have three plus three. It means six degrees of freedom. And if we want to see how we, the example scene of VR, look at this scene. Firstly it was one, like WebGL scene, but if we want to use it with VR, we need to have the same scene but with small difference in position of camera. So we are camera. We look at the scene, but for each eye, it should be small difference in position. So in the bottom of this slide, you see the special Mozilla plugin for emulation of WebXR devices. It could be quite useful for development. So what we can do in VR, let's summarize. We can see only virtual world. We can move around in this virtual world. We can rotate in this virtual world. We see different objects. And we can interact with all these virtual objects. So example of also very old example, when we, I think, saw the virtual reality, it's when you put sunglasses in attractions and different parks, you can see some special videos. And on YouTube, you also can check this video and you can use your glasses and check this video with glasses and it will be like full immersion in such a trip. Also an indication system. We use it with different educations. It could be like a cinema where you interact with all this stuff with your hands. Look at this guy. He has devices in his arms and he can do something with this tool. So there are quite a lot of different videos with this education and it's like a game, but to educate. And AR, augmented reality, what is it? What's the difference? The main difference that in AR, you see actually the real world. So you use another glasses, they're transparent, and you can use also your phone, but you look at your phone and with camera, you see something special, some special objects from digital world through your phone or through these glasses. So with AR, we see real world. We see, we can move in this world, in our real world. We can rotate in this world and we can see objects, some additional objects of virtual world. We can interact with these objects of virtual world, so virtual objects. And also we can get data from some objects from the real world. So from table, I don't know, coach, et cetera, what you want. And example of AR, it's, you can also in education, you can see different objects as close as you never can see them in real life. For example, you can look at this Mexican thing, very ancient thing, and check very close it, or you can see Jaconde in AR, very close. So it could be, you can use it in education, in also in business. For example, I'm a visual person. I like to see everything, what I want to eat, like menu with pictures, but you can also use it like menu with AR objects. It could be really more visual, more understandable way to show the dishes. So and IKEA has some special application where you can put into, even to have sometimes different size of objects because you want to build something and to understand how big size you need, you can check it with this application. And so why XR reality? Because VR, AR doesn't matter, just XR, some reality. And that's why we use X. And first you need to know that it's already standard. There is some specification about the devices which support this technology. And also we have very good support in different browsers. Yes, it's with plugs, but actually 71% of all browsers is quite good. Only Safari and iPhone doesn't support WebXR. Some special tools, just usually we use for testing Android and for emulation and browser, you can use this plugin, which I mentioned before. And for iPhone development, you can use 8-wall service and build with it already now for iPhone some application. And there is application WebXR of your as well. Good news that when you need to do something with WebXR technology, you also need to know WebGL and we already knew something about WebGL. You can check on my website, my presentation about WebGL and 3GS framework. So some basic things about 3D and WebGL. The main thing in this theme that there is a scene, it's like a container. You can put everything on this scene and then to look at it from your browser, from glasses, doesn't matter from where. So scene is the basic stuff. Then you put all objects there. And then you need the camera, because camera actually it's your eyes. It's the place from which you look at this scene. And camera can move when you move, for example. And also there is one important thing, it's render. It's what we called, what makes all this digital scene to pixels and draw everything on our screen. So it could be interesting. You can check the website to know more about it. If we talk about XR, then here actually the main what you need that firstly you need to check do you have support of XR devices? And then you can start XR session. It could be VR session or AR session. Then you run render. So you draw your scene and you always do it in loop because you need to see animated scene. And then you need to end XR session. About framework. So if you want to start, it's better always to use framework, the most famous framework, because it's much easier for you to start. Here's some small example of my WebXR scene. It's augmented reality in my house. There is a dancing guy and now I started to throw tomatoes through him. So just funny example, small, a little bit caught and you can check it by code, by QR code. If you want to learn more about WebXR, go to mixedrealitymozilla.org website. They have a lot of examples and you can check how it works. It's funny. But don't forget that real world is maybe not the best place, but very interesting place to live and to be happy with our friends and families. And check this course if you are interested in WebXR. Just try. It's interesting. So thank you very much. And I hope you enjoyed it. See you next time. Bye. Bye.


Vasilika Klimova

#VR and AR