ML conf EU

ML conf EU 2020

TensorFlow.js 101: ML in the Browser and Beyond

Jason Mayes

JS Nation

JSNation 2022

JavaScript Beats Cancer

Hi, my name is Karol Pustelski and I will tell you more today about how to use JavaScript to beat skin cancer. My experience is about 15 years in machine learning. So my background is machine learning, it's computer science. I did a PhD degree in artificial intelligence, how to use it in medical imaging and dermatoscopy as well. You can find some of my papers, research papers, in this topic on Google Scholar, for example. So feel free, here's one of the articles I have published, actually this is around five years ago, about analyzing of skin cancer on multispectral images. Actually, in that case, I used Python, but because JavaScript became more and more popular in the recent years, and also the usage of JavaScript specifically for this topic, I decided to also prepare a presentation on our solution app for skin cancer analysis. So my background is not only scientific, I also have founded in 2010, so 12 years ago, a company, a service company, working for Fortune 500 companies, building also data science machine learning solutions. And yeah, before that, I did also some other commercial work, for example, at ABM. So as I said, I have 15 years of experience in machine learning, and specifically in medical imaging, I mean, in applications in medical imaging. So why I decided to actually cover this topic and to build some solutions in this area? Well, as you can see, I'm not really in the risk group when it comes to skin cancer, because the biggest group of risk group is actually the blonde people with blue eyes. So this is the phototype number one with the highest risk of having skin cancer, especially if you're becoming kind of a... Your skin doesn't become brown when you're exposed to the sun, but actually it's more going in the direction of red. And actually also the risk of actually having, getting skin cancer is higher in this group. So the darker the skin is, and how it reacts to the sun, the lower the probability is to get a skin cancer. So there are six type, phototypes of skin. I'm more or less in the third group because of my hair color, eye color, and so on. But yeah, that's why the biggest... Well, the problem actually, it is the biggest, the countries like Germany, Scandinavia, and the Nordic countries, the US, Australia, especially Australia. So this is actually where this problem is even more and more important. So in the meantime, I also have done some partnership with some dermatoscopy companies, I mean, the companies who actually develop the hardware. So as you can see here, here's one of the device. This is a dermatoscope here. That's something, that is a device that is actually used by a dermatologist. In this case, I have also used an iPhone here on the front, because this is actually an extension. So it's not a typical dermatoscope. Usually, it doesn't come with an iPhone or any kind of mobile phone. It comes alone. It's a standalone device. Some dermatologists use also this kind of extension case just to make the pictures, to take the pictures in an easier way. And obviously, it's quite small, right? So we can take it to your pocket and actually visit even your patient to take a look on the mole like this, right? So this is how, where actually my solution is used. And well, it is combined with the special lenses, right? Special lenses, special light to get the best possible image of the skin mole. When it comes to the data set, because any kind of machine learning topic, model should be fed by some data. And when I started my research, I actually started with 50, 53 images or less. So as you can imagine, that's not a big enough data set to do any kind of research. So what I did, I did a lot of, I met, I guess, almost every company in public or private that do anything with dermatology in the city where I actually live, where I live in Krakow and in Poland. And well, actually, most of them actually declined to collaborate and actually build some models. It was in 2007, 2008. So the way how people thought about machine learning was totally different than actually compared to what's happening now. Machine learning is AI became a buzzword and everyone want to do AI. In the past, I mean, 15 years ago, when I said AI, most people said, oh, no, thank you. That's not, I'm not interested. Now it's totally opposite. I need to explain people why not to use machine learning rather than actually use machine learning. So it changed dramatically and the COVID pandemic even increased the hype on AI. So when I reached to the companies, I obtained a data set of around 5,000 images. Now you can easily download a data set of about 26,000 images of skin moles. It's available on the ISICarchive.com website and you can use it for your research. So now it's even easier to develop algorithms to find different kinds of skin ills. It's not only cancer, which is technically, I mean, it's not the most popular. That's good illness when it comes to skin. So for all of you that want to use some of my code samples that I have prepared for you, you can always download my Docker image that contains JupyterLab, JupyterHub, together with some kernels for JavaScript as well, and also some libraries, JavaScript libraries. It's a bit old because I am doing it for many years already. So it might be that I will update it soon, but it's still working. So you can easily use it with the notebooks that I will show you next. So the architecture, how I started actually to use machine learning, well, how I combined machine learning with JavaScript. So because of this device, I decided obviously to use one of the JavaScript solutions to build a mobile app because the mobile devices are changing every year. So as you can see, this is an iPhone 6S, so quite old, and probably I will need to change it also soon to a newer one. Still it has a good, good, I'm able still to use this phone to make good quality pictures because the quality is here, not here, right? So it's not in the phone, but it's actually in the lens here. So that's why I decided to use JavaScript instead of building a native application. And in the past, I had to use some different kind of solutions, starting from Cordova, PhoneGap, now actually working on React Native. And for the machine learning part, I use TensorFlow.js. So you might ask, okay, why TensorFlow.js and not, I don't know, Keras or actually Torch, for example. Well, there was one reason why I use TensorFlow.js because it's the most robust library when it comes to JavaScript, obviously. So why I use, or why I actually choose JavaScript in this case, that's not exactly that I said, okay, let's do everything in JavaScript. And that's not true. I mean, the truth is that actually the model is trained in Python, but it is used with JavaScript. So I use TensorFlow.js, not really for the training. And to be honest, I don't know anyone who actually do that. I mean, maybe because I'm into the data science field for a long time, and I mean, I know many people in this area, and they actually do mostly in Python or, I don't know, maybe Scala, some of them, especially related to big data. So in this case, in this specific case, I use JavaScript exactly because of the possibility actually to use TensorFlow.js to load the model, use the model, retrain the model, and use it on a phone, mobile phone. So in production, it is used together with some Kubeflow and TensorFlow service. I mean, there are two models, one that you see on the left, actually, you can combine it together with the main app. And that's how I did it here on the phone. But actually, and in many cases, it also calls the service that actually is trying to find similar models, similar lesions. And actually, it is a web app. So the app is actually connecting to the web app, and then also use that for retraining as well. So how does it look like? Let me just shortly move to some examples. Here we go. Here we go. That's partially, I did it also in the past for another conference at Harky for Ukraine. So I will just use part of the notebook. You can easily find it on my GitHub account, on my company's GitHub account. And a few repositories that are about machine learning, AI, and JavaScript. So when you start to do any kind of research related to data, when you do it in Python, you might use probably the first library that you will think about is Pandas. And there is a fork of this library in JavaScript. It's called Pandas JS, obviously. And this is, well, to some degree, it's very similar to Pandas, to the original one from Python. But actually, it's quite limited compared to the original Pandas. Still, many, many features are not yet implemented. So to some degree, you can use it in JavaScript, but still, there are a lack of many, many, many advanced features from the original Pandas, especially the one that actually are already just statistics. So another library that you can find or you can use in JavaScript for data analysis or data manipulation, DataFrame.js, Reclaim, DataForge, and so on. So there are plenty of actually such libraries to use. This is just a few examples how to work with series, how to use with DataFrames. Typically, that's something normal for people that work on data, probably for JavaScript engineers, not so typical, but still easy to manipulate, easy to export to JSON, for example. And so this is just a notebook on how to use that. Just a few short examples. When it comes to the visualization, in my opinion, JavaScript do a better job than Python. Actually, there are some libraries like Matlib, Seaborn, and some other ones in Python. When you compare it to the ones that actually are available in JavaScript, I think here, JavaScript has a huge advantage because there are good libraries for visualization, for printing the charts, that I think in many cases are just doing better jobs than the Python ones. So that's good. That's one one, let's say. And then when it comes to Scikit-Learn, the most popular library for machine learning for the shallow methods. So TensorFlow, I mean, that's for building neural networks, where actually Scikit-Learn is more about shallow methods. Actually, most of the cases that you want to deal with when it comes to machine learning can be solved or should be solved using shallow methods. So Scikit-Learn, that's the first library we will use and easy to use in Python. In JavaScript, you have JSKitLearn and Scikit-Learn as a fork of Scikit-Learn, but not updated in the recent years. So it looks like someone did that, started something, but then dropped the maintenance and they are gone. I mean, not really a good library here to use. There are plenty of libraries that actually implement some specific shallow method like SVM, KNN, and so on. But actually, if you want to have it in one place, like it is in Python, then you don't have such a library in JavaScript, unfortunately. So when it comes to TensorFlow, here's a very easy example of how actually you can build a linear regression. Here are some examples, how to import, how to use, how to implement a linear regression. And here's an example, here's also an image from the documentation of TensorFlow.js. You have many kinds of APIs. The most popular is in Python, but actually back is written in C++. And you have also some other forks like Java, Go, JavaScript, and so on. So JavaScript is not special here. It's just another fork to get to the core of TensorFlow, really. Here again, example of how to build a last function, combine it together, how to use it, import it, and train it. When it comes to neural networks, how we do it actually in Python. So mostly we do it with Keras, PyTorch. Actually, you can easily combine the network, build it from kind of a blocks, actually have layers, connect the layers together, and you can easily build a very huge network shortly. In JavaScript, we have Keras.js, but to be honest, it's far, far away from the one that is developed in Python, unfortunately. So that's the one thing. Let's just move back to the slides. When it comes to the skin cancer, as you can see here, you have three images, and only two of them actually are cancers. The one on the left and one on the right. Here on the left, you can see some patterns, I mean the dots here and the stripes, some vessels here visible, that all of that is not very symmetric, the borders are quite smooth really. Here on the right, you have this kind of a pattern that is called the blue veil pattern. You can see kind of a white, blue colors here. It means that actually, well, it's going deep. I mean, the cancer is actually going deeper into the skin, actually trying to get to the vessels. So that's bad for the patient, but that's also a pattern that actually tell us, oh, that's really bad. Here in the middle, you have an example of a suspicious care, but not confirmed to be cancer, because all of them actually are pathology confirmed or not. I mean, confirmed that they are cancer or not. So how medical doctors do that? They use some kind of a scoring method like ABCD, seven-point checklist, seven-point score, Hunter's score, three-point checklist, and so on and so on. So they use some patterns like asymmetry, border sharpness, number of colors. I mean, in ABCD, you have six colors that actually really counts, and they check how many colors there are. When you combine it together, I mean, you write down one by one how many different patterns are available, you can count easily more than 30 patterns that actually the doctor, the medicals can find out on the image. On the image or just using the loop directly. What you can also do, and what I did in my research is that I used different kind of wavelengths of light to get not only what you see with the visible light, but also what's actually deeper in the skin using, for example, the infrared light. Actually, I used four different wavelengths of light in total, but actually here you can see one of the infrared, UV to get the melamine, the vessels more visible. And that's how you can actually do a better research because you have more details, more information. Or what you also do, you can also do some image processing like binarization in this case to find out, oh, this here border is not smooth. You can, for example, use fractal methods to analyze that. All right. So the demo, another one, because I would like to show you also how you can also use that, what kind of methods you can use. Really, here's one of the examples. You can find more actually using, going to my GitHub repository. Here's a way how to extract the asymmetry, actually extract the region here first of all. So we could take this part, this region. I use here the ISIC as you can see there. And next, that's the next step to calculate. In this case, I can see it's Python because I have developed the modern Python that I actually exported to JavaScript to be used in JavaScript. And here you can use it to divide the picture into some regions and calculate the symmetry of the opposite sides of the blocks. So this is how you count easy the asymmetry. And this is how you can do it in many patterns. I mean, do some basic image processing. It allows us to find out some of the patterns. Some of them actually are more difficult. You need to build some sophisticated models based on neural networks. Please feel free to get more about that. But what is important because I mentioned about the neural networks and also the shallow models. So just to give you a better understanding, here you have, because we have the black box and white box battles. So the black box are neural networks. In most cases, you can see here a very short network with three layers here. It's easily trained with an accuracy of 98%. Very hard. But if you actually draw the weights, you can see printed weights of just one layer. It looks like that. So if you try to explain that, it's very hard, even not possible to explain each of this number. What does it mean? So there are some explainability, explainable methods to explain the numbers in the weights in the neural network, but it's more complex. When it comes to shallow methods, so the white box methods, it's easy to interpret, easy to explain, because as you can see here, this is a decision tree. Anyway, if you don't know what's a Gini index, if you don't know what X2 is, it's a feature, right? But if you don't know that, just looking on this simple chart, this simple tree, you can easily convert it to the set of if statements. It is easy, understandable for anyone, especially for software engineers, right? Just going back to the slides. Again, because of the limitation, please find more examples on the GitHub repository. This is how the first example, the first application looked like. So we check, I mean, the algorithm check automatically. Here you have the ABC scoring method, take a photo, and we find out the algorithm was able to find the specific patterns. So what are the pros of actually using JavaScript for skin cancer analysis, or in general for machine learning? It can be easily used for prototyping and actually using it also for mobile phones, right? Especially it works with all of them or most of them. You have a huge community, JavaScript community, that actually can support and actually learning JavaScript. The entry level to JavaScript is quite low, so not a big deal. And also what can be done actually, if your app also can be used with a camera in your laptop, then actually you can easily convert to a web app as well. What is not so good? Well, there is no really machine learning MLOps support when it comes to JavaScript. So still most of that rely on the Robots Python, mostly Python solutions when it comes to MLOps. I mean here, Kubeflow, obviously you have the cloud solutions like on AWS, Google Cloud, Azure that actually provide it with some MLOps tools. But when it comes to JavaScript, it's still limited. When it comes to the community, huge community, still not so huge when it comes to the machine learning related topics. And many, many libraries, I mean the jokes about JavaScript libraries that actually every week the new JavaScript library pops up and technically we can use this library. You can see it actually in JavaScript very well when it comes to the machine learning, because if you type machine learning, you will find more actually libraries, machine learning libraries in JavaScript than in Python. But the thing is that, well, even if there are more libraries in JavaScript for machine learning, there is less Robots libraries that you can really use in production. In JavaScript compared to Python or R or Scala or Java and probably some more languages. Thank you for your attention. I hope you will not get a skin cancer and everyone will be healthy. In any case, please find some of the notebooks with some examples on my GitHub repository. Thank you.


Karol Przystalski

Build a 3D Solar System with Hand Recognition and Three.js

Liad Yosef

Boost Productivity with Keras Ecosystem

Haifeng Jin

Node Congress

Node Congress 2021

Machine Learning in Node.js using Tensorflow.js

Hello, everyone. The topic of my lightning talk is machine learning in Node.js using TensorFlow.js. Hi, I'm Shivalamba. I'm currently a TensorFlow.js SIG member and also a Google Code-in mentor at TensorFlow. So the first question that comes to everyone's mind is what exactly is TensorFlow.js? So TensorFlow.js is an open source library that enables easy integration of machine learning models directly in JavaScript. Hence, it reduces the need to learn a separate language like Python to host machine learning models. And it allows very easy integration of some kind of pre-trained models or writing your own models from scratch directly in JavaScript. And as we know that JavaScript is a really versatile language, it can be run across different platforms like on the browser, on the mobile, or let's say on even IoT devices running Raspberry Pi. So all of these different platforms now get easy access to machine learning based applications without any need of plugins. And machine learning that can be used in TensorFlow.js allows from not just pre-existing models, but you can also use a transfer learning to retrain some of the existing models based on your own dataset. And you can actually also write machine learning models directly in JavaScript using JavaScript and without the need of any other language like Python. Now this is the architecture of TensorFlow.js. There are three different APIs that are provided. So the top layer that you see are the TensorFlow.js pre-trained models that are completely ready to use JavaScript classes. The next one is the layers API that enables to easily build and train models using high level building blocks. Similarly to how Keras has been built on top of TensorFlow. And finally, we have the ops or the core API that helps to give you fine tune control of model architecture, or let's say for doing mathematical calculations like linear algebra. And this can be run either on the client side and on the server side. So on the client side, we have basically the browser or let's say the mobile based applications that use the CPU or let's say the web accelerated graphics that is WebGL or WebAssembly. And on the server side, there is support for TensorFlow CPU that are run via the C bindings. And we also have the support for TensorFlow GPU that help get acceleration with the help of CUDA that is also being run on Python based machine learning models. And finally, you know, what are some of the main advantages of utilizing Node.js to actually run TensorFlow.js. So the first one is, you know, we can write the machine learning models directly from scratch in JavaScript. And we are only coding in one language, there's no need to use any other language to run and train and also test out these machine learning models. Now, by using the TensorFlow.js on Node.js, we are also giving the ability to run much larger models that require a lot more power. If they are, for example, let's say if you're running some kind of neural networks that are having a lot of epochs and will require graphics based acceleration. So with the help of the server hardware, like in Python, we can utilize the power of these servers to accelerate our larger models that cannot be done in the client side. And also we have the support for the NPM ecosystem, because the NPM ecosystem is such a large that the Node modules are so heavily used and are there, so a wide variety of Node modules can be used directly with your TensorFlow.js based code as well to help assist with your TensorFlow.js application. And most importantly, we get much better performance, because Node.js uses the just-in-time compiler. And the models that are written in TensorFlow.js utilizing the Node.js can also get this performance boost. And that is quite evident with a few examples. For example, this chart shows you the comparison of TensorFlow running on Python based models and also on the TensorFlow.js for a mobile net, which is a convolutional neural network model. And as you can see that the comparison for the time that it actually takes for the model to actually run is not a lot different. So that means that TensorFlow.js itself is quite optimized for running very industrial standard models as well, and also for newer models at the same time. And in some cases, the TensorFlow.js model actually provides a lot more better performance as compared to the Python versions. For example, BERT, which is a state-of-the-art language model for natural language processing, we can see that over here, there's actually a two times better performance boost by actually using Node.js as compared to a Python based model that is running this BERT classification. So now coming on to the most important part, that is how can we start utilizing the TensorFlow.js in Node.js. So we get namely three different packages that you can install, like the npm packages. So the first one is the TensorFlow CPU. The TensorFlow CPU is, whenever we are importing this package, the module that we get is basically accelerated by the TensorFlow C binary, and it runs on the CPU. And the TensorFlow on the CPU uses the hardware acceleration to let's say, accelerate any kind of mathematical computations, for example, linear algebra. Now, the extension to that is the TensorFlow GPU package. Like the CPU package, this particular GPU package will be accelerated via the TensorFlow C binary, but it also runs the tensor operations on the GPU with the help of CUDA that heavily accelerates your performance. And this binding is definitely the fastest as compared to any other package that we have. And now finally, we also have the vanilla package. This package is very similar to the one that we run on the browser. And in this package, the operations are run in vanilla JavaScript and on the CPU. Now, as compared to the other modules, that is the TensorFlow CPU and the TensorFlow GPU, this does not have like a support for the TensorFlow binary, which actually makes the overall package very smaller. And because of this, since it does not rely on TensorFlow, it can also be actually used on a lot of other devices that support Node.js. Now, one of the important considerations that need to be made is that the Node.js bindings run on the backend for TensorFlow.js that implements them synchronously. That means that whenever we are running the Node.js bindings on a production application, like a web server, we should actually set up a job queue or a server or like worker threads so that it does not block your main thread. Now, there are also support for APIs, because once you have imported the package, all of the normal TensorFlow.js symbols that we use can be appeared in once we have imported the specific module. Like, for example, one of them is like the tf.node that contains Node-specific APIs and TensorBoard is actually a notable example of the Node.js-specific APIs. Now, this is a sample code that should give you an example where we have defined like a model and we are training it by utilizing the tf.node.TensorBoard that helps in your model training. With that, that finishes off my presentation. I hope that you have liked it and you can connect with me on these social platforms and to ask any questions regarding TensorFlow.js. Again, do follow any kind of TensorFlow.js models using the hashtag madewithtensorflowjs on Twitter and on LinkedIn. And again, feel free to connect with me for any queries regarding TensorFlow.js. I hope you have liked it. Thank you for watching.


Shivay Lamba

JSNation Live

JSNation Live 2021

TensorFlow.JS 101: ML in the Browser and Beyond

React Summit Remote Edition

React Summit Remote Edition 2021

Build a UI that Learns - Intelligent Prefetching with React and TensorFlow.js

Hello all and welcome to the session. My name is Eliran Atan and I'm happy to be here in React Summit. Today I will talk about intelligent prefetching in React. So we start by talking about prefetching of code, later we show how we can predict the user behavior using machine learning, and finally we combine those solutions to form fast React applications. So I want to start by talking about the context. Usually we are building those amazing single page applications and eventually we turn up with this huge bundle size that eventually causes problems in loading times, slowness, and potentially harming the user experience. So what we can do is to use the React code splitting API in order to import components on demand. So here instead of just importing the chart component, increasing my bundle size, I'm lazy loading the chart component and the actual fetching of code would happen only when we render it. But that does not solve the problem entirely, right? Because we are just shifting the fetching time to somewhere else. Whenever the user would actually want to render this chart component, then it will have to wait for the fetching to occur and that can affect the loading time and harm the user experience. So what about this crazy idea? We could prefetch, we could break this tradeoff by prefetching the chart component. So using the time that the user is just staring at the screen and before it ever reaches the chart, we could prepare this content. We could use the dynamic import feature to dynamically import the component and then overriding this variable. Now we could do that if we had some indication or some educated guess about the user's next move. We could use this trick to further increase the performance of our applications if we are doing that for entire routes. So if I have this routing between dashboards and products, I could use a lazy loading to lazy load those routes. And then conditionally, if I have some indication about the user's next move, I could dynamically import those routes, saving a lot of time. So that brings us to the question, and that's a very interesting question, of how we can predict the next move of user. So let's just analyze this question for a bit. We have this complicated single page application that is combined from a lot of different components. And we can list those different triggers, those actions, links, buttons that the user can interact with, or at least those that are interesting in the sense that they would cause a rendering of other large components. Now we could keep track on the user's behavior and make this, perform this ordered sequence of interactions that the user is performing. And the question is how we can base, based on this sequence, how we can predict the next element in that sequence. So what we need is this intelligent mechanism that receives a sequence and returns the prediction or estimation of the next item in that sequence. And in machine learning, we call this problem a sequence prediction. So we can use supervised learning, specifically neural network, in order to figure out this estimation. So a neural network will receive an encoding of that sequence as this series of numbers. Each number is the identification of a certain trigger and returns this probability distribution across all the possible triggers that the user can interact with. So each output would represent the chance that that corresponding element will be the next element in the sequence. So this is a supervised learning and the neural network acts as a function approximator. The function that receives a sequence and returns a probability distribution. Now once that we have this output, we could just take the maximal argument and derive from that the next UI element that the user is about to interact with. Now of course those predictions will be meaningless unless we train the network. So in supervised learning, we have to supply the network with examples. And the best source for those examples is the actual behavior of the user. So we could sample the behavior of the user, taking those sequences and feed the neural network with those examples. Basically telling the neural network, this is the actual behavior of the user, please adjust your predictions accordingly. So I think it's cool that we are using the user in order to train the application. So the more that the user uses the application, then it trains it accordingly. And when the user changes its behavior over time, then the application can adapt. Now specifically speaking about implementation, this predictor can be implemented as a neural network. And what we are seeing here is that the input of that neural network will be, we are using here one-hot encoding instead of just pure numbers. And this is because we want to break any numerical correlation between those elements. There is no meaning to say that a dashboard is smaller than products because it is represented by the number three and not by the number 21. So this is the way to break this correlation. Now once that we have, the most important thing in the architecture of your neural network is that we have to use this LSTM type of neural network. This is the best type that can support sequence predictions. It is also important to notice that the input layer is here, the number of units in the input layer should match the length of the sequences that we are working with. And the number of units in the output layer should match the number, all the possible triggers that the user can interact with. Now once that we have the output from that network, we can attach it to the corresponding element in the user experience and assume that the user is about to hit this element. Now implementation wise, we have to implement this mechanism somewhere in the browser environment. And we can do that using TensorFlow.js library. So TensorFlow.js library is based on WebGL and it allows us to implement machine learning algorithms within the browser environment. So we are using the sequential command in order to stack layers in the network one after the another, and we are specifying this LSTM type layer, specifying all the shape of the input as a matrix of binaric numbers. And eventually, once we have this network, we can use the dot predict command supplying a sequence and asking to predict the probability for each element. Now those predictions will be meaningless unless we train the network, as we said. And for the training part, we would use the dot fit command. So we are supplying this batch of sequences and the correlated labels. Those labels represent elements. So we are taking the actual behavior of the user and supply it as examples to the fitting mechanism. Now training the network could take time, that's a heavy operation, and we don't want to disturb the user experience. So we are using the fact that the fitting command is asychronical and we can basically do it in the background without disturbing the user. Now the next challenge that we need to solve here is how we should link between those predictions that we are doing and the prefetching that we discussed earlier. So what we want to do is to prefetch content whenever we know that the user is about to hit a certain button, basically making the application faster and faster. And the more that the user uses the app, the faster it gets. So we have this button somewhere in the application that is called show chart. And we can wrap it with this custom component that I call prediction link. And this prediction link will be connected to a React context. The purpose of React context is to supply this global state that is available for the entire app. And here I'm using a React context to encapsulate all the prediction logic. So it will hold all the implementation of the neural network together with the TensorFlow code and so on. And it will be available through the entire app. So those prediction link components can query the predictor for the current prediction, the next move of the user. And they can expose this on prediction handler that will be triggered upon an associated prediction. So on prediction, we could use this trigger to communicate with another React context, which is responsible for the prefetching. So it basically acts as a messages bus, announcing that the content that is associated with this button should be prefetched. Now, any consumer that is interested in this information can use it and use dynamic imports to dynamically import the content, the necessary content. So what we're doing here is connecting between predictions and prefetching using two React contexts. Now, let's see how that works in code wise. So we have this button somewhere in the application, this show chart. And we can wrap it with a prediction link, a custom component. Now, this prediction link exposes the on prediction trigger. Whenever there is a prediction for that key chart, then we could do something with this prediction. Now, let's take a closer look on how this prediction link is implemented. We are using the use context hook to connect to the predictor context. And then we are querying this predictor context for the current prediction. If the current prediction matches this specific prediction link, then we are triggering the on prediction handler. Now, in order to have the predictor available through the entire app, we have to wrap the entire app with this higher level provider. Now that we have this prediction link, we can use it to call the prefetch component context using this trigger, basically saying, please prefetch the necessary content that is associated with this button. Now, going back to the component that actually fetch code and render the necessary content, we could again use the use context hook to connect to the prefetcher context, query that context and basically asking if the component that we should prefetch is the chart component. And if so, we're using the dynamic imports to do that. And of course, we can do the same thing with entire routes. So if we have this routing between dashboard and products, we could dynamically import those routes conditionally based on the context, the prefetcher context. So I want to summarize everything that we have discussed here. The first thing that we did was to use the React code splitting API in order to improve the performance of our application. Later, we have combined the lazy loading and the prefetching of content in order to further improve the performance of our application, basically using the time that the user is staring at the screen in order to prepare the next content. Then we showed how we can predict the user behavior using a machine learning based click predictor in the browser environment using TensorFlow.js. Of course, predicting the user behavior have more applications that are very interesting to think of. Now, we showed that we can use React context to link between UI handlers and predictions. And finally, we could use another React context to link between predictions and prefetch. So that's the entire overview. That's a personal project that I thought of. And I hope that that will bring you some inspiration about how we can combine AI and UI and basically using machine learning in order to leverage UI development. Thank you. It was fun.


Eliran Natan

#tensorflow