React Summit Remote Edition

React Summit Remote Edition 2020

Controlling Apps with Your Mind and AI

Hi everyone. Really excited to be here at React Summit Remote Edition streaming from outer space. And today I want to talk about controlling apps with your mind. My name is Vladimir Novik. I'm a software architect and consultant in Vladimir Novik Labs. I'm a Google Developer Expert, author, engineer, and on a daily basis I work in web, mobile, VR, AR, IoT, and AI fields. I'm also CTO and co-founder of Event Loop, and we are creating robust, performant, feature-rich conference experience, online conference experience. And basically instead of having Zoom for your virtual conference, we bring you the set of conference tools and different plugins and widgets and so on that helps you to organize the conference, attend the conference. So if your organizer or speaker or attendee reach out to us and sign up for our alpha product, it will be an open source product. So if you want to collaborate, you're welcome. You can find us at eventloop.ai or on Twitter at eventloopHQ. Today I want to talk about the future. And I think we are in kind of transition phase technologically wise. When we're exploring new horizons, we're exploring new medium, we have VR, AR, mixed reality, web, mobile, everything is constantly changing. And that's why we are sort of the ones who break the rules to make the new ones. And let's think about what medium we will use in the future. What dimensions we will use. Maybe VR, maybe AR, maybe something different. I mean, I'm just thinking here, but there are like quantum computing is getting traction and VR is getting traction, AR, and everything is kind of changing, right? So which dimensions we will use. Will there be web as it is today or mobile or we completely switch to different medium? And how we should prepare for that transition, right? How we should adapt. Should we adapt or should we actually shatter foundations that we have and maybe invent new techniques to manipulate things, to interact differently, right? Maybe new UX patterns, maybe new best practices. So we have this ability right now to change things, to adapt and transform things to something totally different. And let's talk what are the very foundations of UI and what are the very foundations of UX. And I think it's 2D medium. And if you think about that, we've gone a long way from cave drawings to the mobile apps. But if you think about all the color theory and lines and how history of art and how everything kind of created the foundation of design and everything in our screens are basically 2D medium. Because we fake 3D, third dimension. It's not really like third dimension. We have 3D models in the browser or in headset or wherever, but all the like shapes and depth is based on shaders, which is basically a function of how light is reflected. So it's kind of fake, right? And we have things on our phones, we have screens. Everything is 2D. And the XR is adding a new dimension to that. So how we've adapted. We took different forms. Let's say you have a sign up form. So we have this form hovering in thin air in VR. Realistic? Not really. Something that we've adapted from 2D medium, right? Or we have in AR, we have arrows pointing to different directions that completely break the notion, but we have it because we've adapted and we haven't invented something new. So I think it's more crucial to create some sort of like reality based interactions in VR and AR. And if you think about that, if you need to log in inside VR. So currently you have a login form, you have this hovering keyboard, you type on this keyboard and then you get into, you type your username and password, you get inside. But is it something that you will see in the reality? Not really, right? So it's more realistic to have some kind of passport or key or whatever that you just put on the spot or turn the key inside the door and it will let you through. It's a sort of reality based interactions, right? So we need to understand our reality in order to create these interactions. The medium is completely different. Now in VR, there is another dimension of something happening behind the viewer, right? So I'm looking at the camera, but something is happening behind. So I cannot use color theory to do this amazing call to action buttons or animation. So I need to use different things like haptics, like sounds, like maybe slowing time and so on. There is also an adaptive UI. Adaptive UI is something that is used on the web and the idea is UI is kind of learning from what are you doing with it. So like forms are learning and adapting. So you can Google that. It's a kind of new trend-ish. Another thing that I propose is actual mind reading. And yeah, obviously I cannot read your thoughts, right? But to some degree. And I want to ask you what is it, right? And obviously we are online, so you can answer in chat. I'll pause. So it's the known universe. And all these dots are clusters of galaxies and it looks amazing. But what is this? It looks pretty much the same, right? But it's actually neurons in our brains. So we are the universe and act accordingly. So how do neurons in our brain, how do they work? So the neurons come in pairs and there is excitatory neuron and it releases glutamate and creates dipole mechanism. Basically have plus and minus and it sort of acts like a battery. So you have potential change between different neurons. So it creates a potential change that can be measured by electrodes on our skull. And it looks like this if you measure your brain. So this is the awake state. This is a sleep state. And you see it's kind of different, right? But it's sort of random-ish, right? So we need to analyze that and to get what does all this mean, right? So in order to do so, we will use EEG headsets. And there are lots of consumer and research versions of EEG headsets. And the idea is to put electrodes on your skull. And based on that, measure the potential change under our skull. So research EEGs look like this. And they are quite costly. But there are consumer ones. And I actually have one here. It's called Muse. It's a nice product that helps you with meditation. So if you're doing meditation, it kind of helps you to focus and so on. And yeah, I will read my brain now, my brainwaves, and you will see how it looks like. So it's fairly cheap. It has only like five electrodes and that's about it. But it's good for our example. So how do we connect this thing to our brain? Now I don't need to open my skull and plug it in. I can connect it using Bluetooth. And specifically, we will use web Bluetooth. I can use Bluetooth by accessing Navigator Bluetooth object. And I call request device. I will filter the Muse service. And I will just connect to and get some attributes from the Bluetooth. Now the support is not quite there, right? So we see it in Chrome and for some reason Opera. But the rest is kind of, no, right? It doesn't support. But we're talking about the experiment, right? It's sort of thoughts experiment, literally. But also like experiment where technology will lead us. And let's look at the demo. So I have this playground here and I can pair to my headset and you will see the brainwaves coming through. And this is my actual brainwaves. As you can see, when I talk and the waves kind of change, when I blink, you see these tiny spikes of voltage spikes. If I do something like this, you see the higher spikes. So I can measure to some degree what's happening on my brain, right? So this is pretty nice. But what do I do with this data, right? So I created here a little app. And before connecting to Muse, actually, yeah, before connecting to Muse, what I want to do, I want to see all the readings of this headset. I will subscribe to readings and I will just console log them. Now I use Muse JS library and this library exposes the readings as RxStream that I can subscribe to and just console log whatever Muse shows me. So let's connect. And let's see what we have here. And as you can see, similar as the graph, we don't really have, like the data is quite weird, right? So we cannot do pretty much anything with that. So the question is how we measure things more precisely. And to measure more precisely, we will use bandpass filter. So we cut out the frequencies on, this is like the graph, right? We cut out these and get only the spikes. Now then we need to cut all these into epochs, which is basically a timeframe because we want like time reference. If in specific amount of time there is a spike, that's probably a blink, right? So we cut these into epochs and also we need to pass it through Fast Fourier Transform. So that means that we take the data that we get is a micron volts and we want to convert that to frequency domain. So we use Fast Fourier Transform and we convert that to frequency domain. So it looks from like the raw data that we get, we will see different frequencies. Now we can recognize different brain waves based on these frequencies and the differentiation is gamma, beta, alpha, theta and delta. And each one of them is different to our state of mind. So for an instance in delta, this is like a sleep, a loss of body awareness, repair and so on. And like gamma is like heightened perception, learning, problem solving tasks, quality processing. As you can see, they are not like super distinctive, there's like a broad range. Beta is generally awake, right? And alpha is like relaxed. So I can measure the alpha state and see if I'm relaxed. So we can react basically to brain wave spikes. And we also can feed this data into machine learning. But before doing some amazing stuff with machine learning, I want to show you something. I want to subscribe to the focus. And basically it will give me the alpha wave readings. So as far as code goes, if... And also I want to subscribe to blinking. So in order to differentiate the blink, basically what I do, I get the readings. I filter them, I get the reading only for the electrode above my left eye, get the maximum of it, like the spike, right? And then I use RX operator switch map to differentiate the spike, because I don't really care about the rest of the data, just the spike. If there is a spike, then this is something that I will return. So how it will look like. These are... okay, I need to also remove this one. Let's connect it again. And what we will see. If I blink, you can see here, me blinking. So as you can see, sometimes it works, sometimes it's not, because it's like a threshold and maybe I put it not really close to my skull. So yeah, so this is the blinking. So now we want to feed this data to machine learning. In order to do so, what I will do, I will go to my app.js and I will add my prediction panel. I will predict some stuff. And here I have three cards. And these three cards, one is web and mobile, other is VRXR and another is IoT AI. So what I will try to do, I will connect to my use headset, get all the data, pass through all the filters that I need, and then add this as a sample to KNNClassifier, which is machine learning algorithm. And we'll start classifying which card I'm thinking about. So let me record these waves really quick. So click on this button while looking at the web and mobile. Now VRXR. And now IoT AI. So now if I click on classify, I will be able to switch by just looking, hands are here, so I can just look at different cards and switch between them. And yeah, as you can see, the main topic that I'm interested in is VRXR and IoT AI. So this is quite cool, but if it will work, I will bring another level of coolness here. So I have MobX store here with enable drone flag. And yeah, I have a drone here, this tiny fellow. So it occasionally works. So let's see if it will work this time. So what I will do, I will first connect to my Muse headset. I need not to blink. And then I will connect to my drone. Okay, so here we have the drone. And hopefully you see it. I will record brainwaves and start classifying. Now I hope it's in the camera view. Now I will try to move it by just looking at it. And let's land it. And it fell. I'm not sure if it was in the view. So let me try to put it in the view again. Now it's probably here and again I'm just moving that with the power of my thought. So that was quite cool. And the main takeaway here is what is the purpose of all of this, right? Like why are we flying drones with our mind, why we're using these devices, they are not that reliable. We have web Bluetooth support only in Chrome. The main reason for that is that we are the ones who made the rules and we break the rules, we invent new things, right? And my main takeaway from this talk is that the future is already here and you are the ones to build it. Thank you. All right. That was incredible. I mean I didn't think that we were going to see somebody fly a drone with their mind this early in the morning. Unfortunately, we don't have much time but we're going to bring Vlad back to the stage for one quick question and then we're going to move on to our next session. So Vlad, I think after seeing something like this, the mind control with the AR, where does somebody even start if they want to start learning this stuff for themselves? I mean the whole point of this talk was like, right, you can change the world basically, right? You're the one who can change everything, right? And then you need to decide what you want to learn like technology wise. You want to learn, be more experienced just in React or you want to broaden your horizons, right? So for instance, if you want to have a React native and AR or have React and web Bluetooth and drones, you can get on different resources. There's egghead.io which is an amazing website where I will be also doing courses there. Actually, I will be doing a workshop on React native and AR pretty soon. So I actually created this sound for React Summit so I will send the link in the community channel. So if you want to get to this workshop, you can do so. And VR wise, I also had a bunch of talks. So I have a YouTube channel streamed about VR, but I'll be also recording a bunch of stuff on this topic because I mean, I like to teach these things, right? So that's why I also started a Twitch channel. And yeah, that's like one of the places, but obviously there are lots of places where to learn and if you really want to get into this, I would say just DM me on Twitter. I will let you know, like, let me know what you want to do as far as technology is concerned. But I probably will be able to direct you to the learning materials and free sources somewhere and so on. Excellent. Well, Vlad, thank you so much. We really appreciate it. I wish we had more time for Q&A, but to get us back on track, we are going to jump now to a panel discussion.


Vladimir Novick

ML conf EU

ML conf EU 2020

Broadening AI Adoption with AutoML

Hello everyone, my name is Bernhard Zum, I am product manager for machine learning with the Mathworks. Let me motivate my topic with some questions to you. Where do you want to apply AI? Are you concerned with the lack of your experience in AI or with black box models? The community widely recognizes these as challenges and barriers to broader adoption of AI across many industries. Today I will focus on AutoML, automation that takes the routine iterative effort and most of the science out of building machine learning models. So what exactly is AutoML? To understand that you need to know a bit about the typical workflow for building machine learning models. The focus of this talk, but building deep neural networks isn't that different. First, you need to process your raw data, deal with its messiness and get it into a shape that's suitable for later stages, like dealing with missing data and outliers. Next, you need to engineer features, extract a few variables from your data that serve as input to your model and capture the majority of the variability. That's fairly easy for numeric data, but a lot harder for signals. Next, you're faced with the choice of different machine learning models. And even to experts, it's not clear which model performs best on any given problem. So you have to try multiple, which leads to the model tuning stage where you assess the performance of some initial models, optimize their hyper parameters, maybe select a subset of features to avoid overfitting, but that may not be enough to get really good performance. You may have to go back, replace some features with others and do this all over again. If you're familiar with machine learning, you will know the most difficult and time consuming stages are the feature engineering and the optimization. If your head is spinning now, don't despair because you don't have to know all this complexity. The point of AutoML is to simplify it. Ideally, to go directly from your initial data and your machine learning problem to a model you can deploy. However, really taking it seriously, that is not a realistic expectation in a single step machine learning is not possible. However, what is realistic is freeing up engineers like yourself to focus on the hard machine learning problems and on your application. Otherwise, without AutoML, you'll have to find that AI expertise either inside your team and organization or outside. And those data scientists are hard to find and expensive. So as the first barrier that AutoML removes, it overcomes the lack of machine learning expertise. But even if you have that expertise, you are increased in productivity because AutoML takes away those time consuming and iterative steps. Finally, AutoML allows you to solve problems that otherwise wouldn't be feasible, like use cases where you need to build many different models representing different variations or different environmental stages. So how do you apply AutoML for engineering? Most of those engineering applications are based on signal image data. And that's where the feature engineering becomes critical for good performance. And that's notoriously difficult. We at MathWorks brought our signal processing knowledge to bear and came up with the following three step AutoML. First, you apply wavelet scattering. These wavelets are very suitable in that time bounded shape to represent spikes and irregularities in your signal. Therefore, you get very good features. Many engineering applications, however, require deployment to memory and power limited embedded systems. For those, you cannot deploy large models. So second, we apply automated feature selection to reduce the maybe hundreds of wavelet features to just a few very performant features and reduce the model size. Finally, and key is the model selection hyper parameter tuning step. You have a choice of different models and for the model to perform well, the hyper parameters need to be set just right. Let's look at that stage in a little more detail. How does that simultaneous optimization of model and hyper parameters work? Well, you can perform random search, but that's not efficient either because the search space is very large. We employ Bayesian optimization that builds a model of the search space. And here you can see how that Bayesian optimization switches between different types of models and optimizes the error over in the course of the iterations. How do we know that AutoML works? We compared AutoML to the traditional manual process on two classification problems. First, we look at human activity recognition, where you take auxiliary meter data from mobile phones. We have about 7K observations in a set we collected and we manually engineered 66 features using various signal processing functions. Second, we look at heart sound classification. Think about being in your doctor's office with a stethoscope and listening to your heart sound. So those phonograms, we could have a data set of 10K observations that's publicly available and engineered less than 30 features. So what results did we get? We can see here with the manual process, we achieved accuracies in the high 90s as you would want to have for such an important application. For AutoML, for one application, slightly lower, but the point is without all that expertise and without the time consuming iterative process, you get very good models in a few steps. So AutoML empowers engineers without AI expertise to build optimized models, including for signal applications where the feature extraction is notoriously difficult. You can apply AutoML to signal applications in a few steps, automated feature generation with wavelets, automated feature selection to reduce model size and make it fit on your hardware, and model selection along with hyperparameter tuning in an efficient way using Bayesian optimization. Finally, to deploy your AI model to the edge and embedded systems, you need low level code like C. MATLAB, you can translate automatically to C, C++ code that can be deployed directly, and thus another barrier to broader adoption of AI removed. Thank you for your attention. And if you want to know more, Monday afternoon or evening, I'll have a longer session on AutoML and interpretability, a seminar on those two topics, one hour and two hours, a hands-on workshop on machine and deep learning using MATLAB online. And now I'll head back to the moderator for questions.


Bernhard Suhm

Teaching ML and AI to Coders

Laurence Moroney

React Summit Remote Edition 2021

Build a UI that Learns - Intelligent Prefetching with React and TensorFlow.js

Hello all and welcome to the session. My name is Eliran Atan and I'm happy to be here in React Summit. Today I will talk about intelligent prefetching in React. So we start by talking about prefetching of code, later we show how we can predict the user behavior using machine learning, and finally we combine those solutions to form fast React applications. So I want to start by talking about the context. Usually we are building those amazing single page applications and eventually we turn up with this huge bundle size that eventually causes problems in loading times, slowness, and potentially harming the user experience. So what we can do is to use the React code splitting API in order to import components on demand. So here instead of just importing the chart component, increasing my bundle size, I'm lazy loading the chart component and the actual fetching of code would happen only when we render it. But that does not solve the problem entirely, right? Because we are just shifting the fetching time to somewhere else. Whenever the user would actually want to render this chart component, then it will have to wait for the fetching to occur and that can affect the loading time and harm the user experience. So what about this crazy idea? We could prefetch, we could break this tradeoff by prefetching the chart component. So using the time that the user is just staring at the screen and before it ever reaches the chart, we could prepare this content. We could use the dynamic import feature to dynamically import the component and then overriding this variable. Now we could do that if we had some indication or some educated guess about the user's next move. We could use this trick to further increase the performance of our applications if we are doing that for entire routes. So if I have this routing between dashboards and products, I could use a lazy loading to lazy load those routes. And then conditionally, if I have some indication about the user's next move, I could dynamically import those routes, saving a lot of time. So that brings us to the question, and that's a very interesting question, of how we can predict the next move of user. So let's just analyze this question for a bit. We have this complicated single page application that is combined from a lot of different components. And we can list those different triggers, those actions, links, buttons that the user can interact with, or at least those that are interesting in the sense that they would cause a rendering of other large components. Now we could keep track on the user's behavior and make this, perform this ordered sequence of interactions that the user is performing. And the question is how we can base, based on this sequence, how we can predict the next element in that sequence. So what we need is this intelligent mechanism that receives a sequence and returns the prediction or estimation of the next item in that sequence. And in machine learning, we call this problem a sequence prediction. So we can use supervised learning, specifically neural network, in order to figure out this estimation. So a neural network will receive an encoding of that sequence as this series of numbers. Each number is the identification of a certain trigger and returns this probability distribution across all the possible triggers that the user can interact with. So each output would represent the chance that that corresponding element will be the next element in the sequence. So this is a supervised learning and the neural network acts as a function approximator. The function that receives a sequence and returns a probability distribution. Now once that we have this output, we could just take the maximal argument and derive from that the next UI element that the user is about to interact with. Now of course those predictions will be meaningless unless we train the network. So in supervised learning, we have to supply the network with examples. And the best source for those examples is the actual behavior of the user. So we could sample the behavior of the user, taking those sequences and feed the neural network with those examples. Basically telling the neural network, this is the actual behavior of the user, please adjust your predictions accordingly. So I think it's cool that we are using the user in order to train the application. So the more that the user uses the application, then it trains it accordingly. And when the user changes its behavior over time, then the application can adapt. Now specifically speaking about implementation, this predictor can be implemented as a neural network. And what we are seeing here is that the input of that neural network will be, we are using here one-hot encoding instead of just pure numbers. And this is because we want to break any numerical correlation between those elements. There is no meaning to say that a dashboard is smaller than products because it is represented by the number three and not by the number 21. So this is the way to break this correlation. Now once that we have, the most important thing in the architecture of your neural network is that we have to use this LSTM type of neural network. This is the best type that can support sequence predictions. It is also important to notice that the input layer is here, the number of units in the input layer should match the length of the sequences that we are working with. And the number of units in the output layer should match the number, all the possible triggers that the user can interact with. Now once that we have the output from that network, we can attach it to the corresponding element in the user experience and assume that the user is about to hit this element. Now implementation wise, we have to implement this mechanism somewhere in the browser environment. And we can do that using TensorFlow.js library. So TensorFlow.js library is based on WebGL and it allows us to implement machine learning algorithms within the browser environment. So we are using the sequential command in order to stack layers in the network one after the another, and we are specifying this LSTM type layer, specifying all the shape of the input as a matrix of binaric numbers. And eventually, once we have this network, we can use the dot predict command supplying a sequence and asking to predict the probability for each element. Now those predictions will be meaningless unless we train the network, as we said. And for the training part, we would use the dot fit command. So we are supplying this batch of sequences and the correlated labels. Those labels represent elements. So we are taking the actual behavior of the user and supply it as examples to the fitting mechanism. Now training the network could take time, that's a heavy operation, and we don't want to disturb the user experience. So we are using the fact that the fitting command is asychronical and we can basically do it in the background without disturbing the user. Now the next challenge that we need to solve here is how we should link between those predictions that we are doing and the prefetching that we discussed earlier. So what we want to do is to prefetch content whenever we know that the user is about to hit a certain button, basically making the application faster and faster. And the more that the user uses the app, the faster it gets. So we have this button somewhere in the application that is called show chart. And we can wrap it with this custom component that I call prediction link. And this prediction link will be connected to a React context. The purpose of React context is to supply this global state that is available for the entire app. And here I'm using a React context to encapsulate all the prediction logic. So it will hold all the implementation of the neural network together with the TensorFlow code and so on. And it will be available through the entire app. So those prediction link components can query the predictor for the current prediction, the next move of the user. And they can expose this on prediction handler that will be triggered upon an associated prediction. So on prediction, we could use this trigger to communicate with another React context, which is responsible for the prefetching. So it basically acts as a messages bus, announcing that the content that is associated with this button should be prefetched. Now, any consumer that is interested in this information can use it and use dynamic imports to dynamically import the content, the necessary content. So what we're doing here is connecting between predictions and prefetching using two React contexts. Now, let's see how that works in code wise. So we have this button somewhere in the application, this show chart. And we can wrap it with a prediction link, a custom component. Now, this prediction link exposes the on prediction trigger. Whenever there is a prediction for that key chart, then we could do something with this prediction. Now, let's take a closer look on how this prediction link is implemented. We are using the use context hook to connect to the predictor context. And then we are querying this predictor context for the current prediction. If the current prediction matches this specific prediction link, then we are triggering the on prediction handler. Now, in order to have the predictor available through the entire app, we have to wrap the entire app with this higher level provider. Now that we have this prediction link, we can use it to call the prefetch component context using this trigger, basically saying, please prefetch the necessary content that is associated with this button. Now, going back to the component that actually fetch code and render the necessary content, we could again use the use context hook to connect to the prefetcher context, query that context and basically asking if the component that we should prefetch is the chart component. And if so, we're using the dynamic imports to do that. And of course, we can do the same thing with entire routes. So if we have this routing between dashboard and products, we could dynamically import those routes conditionally based on the context, the prefetcher context. So I want to summarize everything that we have discussed here. The first thing that we did was to use the React code splitting API in order to improve the performance of our application. Later, we have combined the lazy loading and the prefetching of content in order to further improve the performance of our application, basically using the time that the user is staring at the screen in order to prepare the next content. Then we showed how we can predict the user behavior using a machine learning based click predictor in the browser environment using TensorFlow.js. Of course, predicting the user behavior have more applications that are very interesting to think of. Now, we showed that we can use React context to link between UI handlers and predictions. And finally, we could use another React context to link between predictions and prefetch. So that's the entire overview. That's a personal project that I thought of. And I hope that that will bring you some inspiration about how we can combine AI and UI and basically using machine learning in order to leverage UI development. Thank you. It was fun.


Eliran Natan

#artificial intelligence