Giving Superpowers to Your React Apps with Machine Learning

Well, hello everyone. Welcome to my lightning talk at React Summit 2023. It's giving superpowers to your React apps with machine learning. I'm Shailamba. I'm a TensorFlow.js SIG and working group lead from India. And I'm also Google Code Mentor at TensorFlow.

Now, we're learning about the hype of machine learning. And of course, with tools such as Chats ABT and large language models, we're seeing machine learning everywhere. So whether it's text iterative or it's iterative AI. And as a front end engineer, you might be very curious to also be able to leverage the use of machine learning in your own applications.

Now typically, if we look at any new web developer who might want to build a very cool web-based machine learning based application, they tend to directly jump into React. And we know that React itself can be very confusing and has a lot of advanced topics. So rather than jumping directly from HTML, Javascript, and learning about Git and GitHub and then finally learning about React, newer developers tend to jump into React. And now think of as a new front-end developer where you're learning the basics of Javascript, if you want to integrate machine learning capability in your Javascript or your React app, traditionally you will have to use a programming language like Python or R, which are more generally focused towards machine learning.

But that is not the case. You yourself being a machine learning...a front-end engineer can integrate machine learning capabilities without having to go in the loop or learning a new language stack altogether. And how you can accomplish that is with the help of a plethora of open source machine learning libraries for Javascript that are available for you to use with your React application.

So whether it's one of the most popular open source Javascript based machine learning libraries TensorFlow.js or ONNX.js or even some of the more upcoming frameworks and libraries such as Transformers.js that allow you to add the capability of very large machine learning models directly in your browser with the help of Javascript, which you can of course include in your React app.

And there are five different reasons why you should actually run machine learning model inference on React and in your front end apps. And primarily that's due to five things. So the first one is the privacy. When you use these machine learning based models and you run these inference on the browser itself, that is a lot more private and it's a lot more better privacy. Because you're not doing the inference of machine learning on a dedicated server. That also is a lot less costly because you don't have to maintain dedicated servers. So you can do all of these particular inference, machine learning inference directly in the browser and you also end up getting lower cost and lower latency as well.

And the reach and scale with which you can actually build and serve these applications it's a lot more now. Also one important thing to consider over here is that the way that we are able to actually do this inference on the browser is with the help of being able to leverage the use of your system hardware such as CPU, GPUs, because all these different libraries that we discussed come with support for technologies such as WebAssembly, WebGPU, which allow to improve the performance of your browser based applications with the help of different technologies such as WebGL or of course with your system hardware being utilized to be able to run these inference or training directly in the browser. So the idea is that, with the help of these particular libraries, you can use them without having to learn about something like Python and in fact, in a lot of use cases you get better performance with JavaScript as compared to even something with Python in a lot of these particular models, because they are more optimized for JavaScript and in some cases JavaScript is actually faster.

And one demo that I'll be showing to all of you is built with the help of TensorFlow.js, which is of course a completely open source machine learning library that is available for anyone to use, and we'll be using a pre-built machine learning model that we'll be utilizing for our app. So there are a lot of pre-built applications or models that are already available with TensorFlow and similarly for other libraries that we have already covered, including for human-based, text-based, sound-based, and others as well. And the one that we'll use is the CocoaSSIST model. So the CocoaSSIST model is a very famous, well-known model for being able to classify your images. So it's going to be a very straightforward step. We'll load an image, load the actual machine learning model, and get predictions. For example, over here we are predicting some dogs inside of an image that we upload, and we get a bounding box with a result of the classified image and the tag that it gets classified with. So for our demo, we'll be installing these particular packages. So it's going to be the TensorFlow.js itself and the CocoaSSIST model. And you can import these models directly in React by using this particular way. Then we will be loading these models in our React app and wait for the model to load completely. So in this case, we are going to be loading the CocoaSSIST model. Now once the model has been loaded, we can actually go ahead and write the predictions for being able to predict based on a particular video source. So let's say I open my webcam and I try to detect what is in the video source and I will be able to very easily do that with the help of the predictions function. Now in order to basically get started with it, let's see how that actually looks. So I'll go to my VS Code and this is my React app.

So I'm using White React and CocoaSSIST. So if you take a look at my packets.json, here are my two main dependencies for my React application.

Now if I go to my main file, that's my app.jsx. So this is where I have imported both of my JavaScript framework ML framework that stands for JS and the model itself. I've created some states for ensuring that my model is loaded. Here is the function that we covered in the slide where I'm loading my CocoaSSIST model.

So I first upload or get my dataset. So this dataset is a set of images which have been trained. And once you actually open up your video source, if one of the frames contains any image that is in the dataset, it will be able to detect and classify that particular image. Then we have our async function where we are using the model loaded.detect. So the dot detect function will first look at the frame from your video source and then it will basically go ahead and predict a particular class and the prediction score. So how confident is the particular machine learning algorithm in determining whether like let's say an image contains it all, whether it's like 90% confident or 100% confident. So you're getting those detections for your setup. And of course over here we are using our webcam component to get the webcam footage and then here from line number 56 and 57 we're actually going ahead and detecting these results.

So let's go ahead and open up a terminal and just quickly do npm run dev. So I'll go ahead and do npm run dev and start to run in my browser. So if I refresh my screen, this is my simple object detection and if I go ahead and click on predict object it will see that this is a person with an accuracy of 58%. Let's try to probably bring a phone. So I have my phone with me in front of me and let's try to see if it is able to detect this as well. So, of course, right now it is trying to detect the person and now we can see that it's a cellphone. So I'll go back and if I try to click again, it detects the person's name. So very easily what we have done is we have created a front-end app where we have essentially deployed a pre-built model. We didn't have to train the model. We didn't have to know what is going on behind the scenes with this particular model. But we were able to understand and see and create a very simple to use machine learning application in less than 70 lines of actual React code.

So similarly, you could also use some of the other libraries such as Transforms.js, this is also a very popular framework, a really great example that actually being utilized and that I would like to quickly showcase is this one called Ermine.ai that actually does client side live audio transcription directly powered by Transforms.js and it's also a React app so you can take a look over here. So if you load, wait for the model to load, this does live audio recording and transcription for you. For example, let's say, hey, I'm Shivaay, and I'm speaking at React Summit virtually. So if I stop this now, it will do the live transcribing for you. And it's using the Transforms.js, and you can see that it says I'm speaking at React Summit virtually, so within a matter of two seconds, it did live audio transcription.

And similarly, you can of course use a TensorFlow.js. You can also use Onyx.js. And these are all the different libraries that you can directly use, so you just need to know or need to be aware of at least just the very core basics of machine learning, that's what is training, what is prediction, and you can build such an application yourself as well.

So here are some of the resources that you can actually leverage. And finally, you can connect with me on Twitter. That's where I'm the most active, if you have any questions related to how to integrate machine learning capabilities in your React app. But I'd like to conclude with the statement that as a React developer, you don't have to worry too much about learning the basics of machine learning or learning another framework like Python.

There are enough JavaScript frameworks and libraries that include that are available for you to use in your React app, and a lot of them are actually being used in production as well. So whether you want to create, like you have a really nice startup idea, and you want to create a React app that leverages the use of machine learning, then go ahead and use all these different libraries that I shared with you today and build your next startup idea without having to learn about another machine learning language or about the core machine learning capabilities. So that's it. Thank you so much, and I hope that you have a wonderful React Summit.