Giving Superpowers to Your React Apps with Machine Learning

Rate this content
Bookmark
Slides

Have you ever questioned whether Javascript is a viable alternative to Python or R for creating machine learning models? After all, a 2019 survey by Stack Overflow found that Javascript is the language that developers use the most. Given that machine learning models like neural networks require a lot of computational power and that javascript was not intended for high-speed computing, this approach seems unreasonable. But hold on, this not entirely true, as javascript libraries like Onnx.js and Tensorflow.js are here to save the day! I'll be going into further detail on how to create intuitive and innovative machine learning applications with React in this talk.

11 min
06 Jun, 2023

AI Generated Video Summary

Welcome to my lightning talk at React Summit 2023 where I discuss integrating machine learning capabilities in React apps using JavaScript libraries like TensorFlow.js and ONNX.js. These libraries allow for better privacy, lower cost, and lower latency by leveraging system hardware. Examples include using TensorFlow.js and CocoaSSIST to classify images and Ermine.ai for live audio transcription. React developers can now integrate machine learning without needing extensive knowledge of Python or other frameworks.

1. Introduction to Machine Learning in React

Short description:

Welcome to my lightning talk at React Summit 2023. It's giving superpowers to your React apps with machine learning. As a front-end engineer, you can leverage machine learning in your own applications. Typically, new web developers jump into React, but integrating machine learning usually requires Python or R. However, you can use machine learning libraries for Javascript to integrate machine learning capabilities in your React app.

Well, hello everyone. Welcome to my lightning talk at React Summit 2023. It's giving superpowers to your React apps with machine learning. I'm Shailamba. I'm a TensorFlow.js SIG and working group lead from India. And I'm also Google Code Mentor at TensorFlow.

Now, we're learning about the hype of machine learning. And of course, with tools such as Chats ABT and large language models, we're seeing machine learning everywhere. So whether it's text iterative or it's iterative AI. And as a front end engineer, you might be very curious to also be able to leverage the use of machine learning in your own applications.

Now typically, if we look at any new web developer who might want to build a very cool web-based machine learning based application, they tend to directly jump into React. And we know that React itself can be very confusing and has a lot of advanced topics. So rather than jumping directly from HTML, Javascript, and learning about Git and GitHub and then finally learning about React, newer developers tend to jump into React. And now think of as a new front-end developer where you're learning the basics of Javascript, if you want to integrate machine learning capability in your Javascript or your React app, traditionally you will have to use a programming language like Python or R, which are more generally focused towards machine learning.

2. Integrating Machine Learning in React

Short description:

You can integrate machine learning capabilities in your React app using open source machine learning libraries for JavaScript. These libraries, such as TensorFlow.js and ONNX.js, allow you to add machine learning models directly in your browser, providing better privacy, lower cost, and lower latency. They leverage system hardware like CPUs and GPUs, improving performance. JavaScript can outperform Python in many cases. We'll use TensorFlow.js and the CocoaSSIST model to classify images in our demo.

But that is not the case. You yourself being a machine learning...a front-end engineer can integrate machine learning capabilities without having to go in the loop or learning a new language stack altogether. And how you can accomplish that is with the help of a plethora of open source machine learning libraries for Javascript that are available for you to use with your React application.

So whether it's one of the most popular open source Javascript based machine learning libraries TensorFlow.js or ONNX.js or even some of the more upcoming frameworks and libraries such as Transformers.js that allow you to add the capability of very large machine learning models directly in your browser with the help of Javascript, which you can of course include in your React app.

And there are five different reasons why you should actually run machine learning model inference on React and in your front end apps. And primarily that's due to five things. So the first one is the privacy. When you use these machine learning based models and you run these inference on the browser itself, that is a lot more private and it's a lot more better privacy. Because you're not doing the inference of machine learning on a dedicated server. That also is a lot less costly because you don't have to maintain dedicated servers. So you can do all of these particular inference, machine learning inference directly in the browser and you also end up getting lower cost and lower latency as well.

And the reach and scale with which you can actually build and serve these applications it's a lot more now. Also one important thing to consider over here is that the way that we are able to actually do this inference on the browser is with the help of being able to leverage the use of your system hardware such as CPU, GPUs, because all these different libraries that we discussed come with support for technologies such as WebAssembly, WebGPU, which allow to improve the performance of your browser based applications with the help of different technologies such as WebGL or of course with your system hardware being utilized to be able to run these inference or training directly in the browser. So the idea is that, with the help of these particular libraries, you can use them without having to learn about something like Python and in fact, in a lot of use cases you get better performance with JavaScript as compared to even something with Python in a lot of these particular models, because they are more optimized for JavaScript and in some cases JavaScript is actually faster.

And one demo that I'll be showing to all of you is built with the help of TensorFlow.js, which is of course a completely open source machine learning library that is available for anyone to use, and we'll be using a pre-built machine learning model that we'll be utilizing for our app. So there are a lot of pre-built applications or models that are already available with TensorFlow and similarly for other libraries that we have already covered, including for human-based, text-based, sound-based, and others as well. And the one that we'll use is the CocoaSSIST model. So the CocoaSSIST model is a very famous, well-known model for being able to classify your images. So it's going to be a very straightforward step. We'll load an image, load the actual machine learning model, and get predictions. For example, over here we are predicting some dogs inside of an image that we upload, and we get a bounding box with a result of the classified image and the tag that it gets classified with. So for our demo, we'll be installing these particular packages. So it's going to be the TensorFlow.js itself and the CocoaSSIST model. And you can import these models directly in React by using this particular way. Then we will be loading these models in our React app and wait for the model to load completely. So in this case, we are going to be loading the CocoaSSIST model. Now once the model has been loaded, we can actually go ahead and write the predictions for being able to predict based on a particular video source. So let's say I open my webcam and I try to detect what is in the video source and I will be able to very easily do that with the help of the predictions function. Now in order to basically get started with it, let's see how that actually looks. So I'll go to my VS Code and this is my React app.

3. Integrating Machine Learning in React

Short description:

I'm using White React and CocoaSSIST for my React application. I've created states to ensure the model is loaded. The detect function predicts the class and score of an image. We created an object detection app with a pre-built model in less than 70 lines of React code. Another example is Ermine.ai, a React app powered by Transforms.js for live audio transcription.

So I'm using White React and CocoaSSIST. So if you take a look at my packets.json, here are my two main dependencies for my React application.

Now if I go to my main file, that's my app.jsx. So this is where I have imported both of my JavaScript framework ML framework that stands for JS and the model itself. I've created some states for ensuring that my model is loaded. Here is the function that we covered in the slide where I'm loading my CocoaSSIST model.

So I first upload or get my dataset. So this dataset is a set of images which have been trained. And once you actually open up your video source, if one of the frames contains any image that is in the dataset, it will be able to detect and classify that particular image. Then we have our async function where we are using the model loaded.detect. So the dot detect function will first look at the frame from your video source and then it will basically go ahead and predict a particular class and the prediction score. So how confident is the particular machine learning algorithm in determining whether like let's say an image contains it all, whether it's like 90% confident or 100% confident. So you're getting those detections for your setup. And of course over here we are using our webcam component to get the webcam footage and then here from line number 56 and 57 we're actually going ahead and detecting these results.

So let's go ahead and open up a terminal and just quickly do npm run dev. So I'll go ahead and do npm run dev and start to run in my browser. So if I refresh my screen, this is my simple object detection and if I go ahead and click on predict object it will see that this is a person with an accuracy of 58%. Let's try to probably bring a phone. So I have my phone with me in front of me and let's try to see if it is able to detect this as well. So, of course, right now it is trying to detect the person and now we can see that it's a cellphone. So I'll go back and if I try to click again, it detects the person's name. So very easily what we have done is we have created a front-end app where we have essentially deployed a pre-built model. We didn't have to train the model. We didn't have to know what is going on behind the scenes with this particular model. But we were able to understand and see and create a very simple to use machine learning application in less than 70 lines of actual React code.

So similarly, you could also use some of the other libraries such as Transforms.js, this is also a very popular framework, a really great example that actually being utilized and that I would like to quickly showcase is this one called Ermine.ai that actually does client side live audio transcription directly powered by Transforms.js and it's also a React app so you can take a look over here. So if you load, wait for the model to load, this does live audio recording and transcription for you. For example, let's say, hey, I'm Shivaay, and I'm speaking at React Summit virtually. So if I stop this now, it will do the live transcribing for you. And it's using the Transforms.js, and you can see that it says I'm speaking at React Summit virtually, so within a matter of two seconds, it did live audio transcription.

4. Integrating Machine Learning in React

Short description:

You can directly use TensorFlow.js and Onyx.js to integrate machine learning capabilities in your React app. Just knowing the basics of machine learning, such as training and prediction, is enough to build such an application. There are plenty of JavaScript frameworks and libraries available for React developers, eliminating the need to learn Python or other machine learning frameworks. Leverage these libraries to create your next startup idea without the need for extensive machine learning knowledge. Connect with me on Twitter for any questions related to integrating machine learning in React. Thank you and have a wonderful React Summit.

And similarly, you can of course use a TensorFlow.js. You can also use Onyx.js. And these are all the different libraries that you can directly use, so you just need to know or need to be aware of at least just the very core basics of machine learning, that's what is training, what is prediction, and you can build such an application yourself as well.

So here are some of the resources that you can actually leverage. And finally, you can connect with me on Twitter. That's where I'm the most active, if you have any questions related to how to integrate machine learning capabilities in your React app. But I'd like to conclude with the statement that as a React developer, you don't have to worry too much about learning the basics of machine learning or learning another framework like Python.

There are enough JavaScript frameworks and libraries that include that are available for you to use in your React app, and a lot of them are actually being used in production as well. So whether you want to create, like you have a really nice startup idea, and you want to create a React app that leverages the use of machine learning, then go ahead and use all these different libraries that I shared with you today and build your next startup idea without having to learn about another machine learning language or about the core machine learning capabilities. So that's it. Thank you so much, and I hope that you have a wonderful React Summit.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Advanced Conference 2022React Advanced Conference 2022
25 min
A Guide to React Rendering Behavior
React is a library for "rendering" UI from components, but many users find themselves confused about how React rendering actually works. What do terms like "rendering", "reconciliation", "Fibers", and "committing" actually mean? When do renders happen? How does Context affect rendering, and how do libraries like Redux cause updates? In this talk, we'll clear up the confusion and provide a solid foundation for understanding when, why, and how React renders. We'll look at: - What "rendering" actually is - How React queues renders and the standard rendering behavior - How keys and component types are used in rendering - Techniques for optimizing render performance - How context usage affects rendering behavior| - How external libraries tie into React rendering
React Summit Remote Edition 2021React Summit Remote Edition 2021
33 min
Building Better Websites with Remix
Remix is a new web framework from the creators of React Router that helps you build better, faster websites through a solid understanding of web fundamentals. Remix takes care of the heavy lifting like server rendering, code splitting, prefetching, and navigation and leaves you with the fun part: building something awesome!
React Advanced Conference 2022React Advanced Conference 2022
30 min
Using useEffect Effectively
Can useEffect affect your codebase negatively? From fetching data to fighting with imperative APIs, side effects are one of the biggest sources of frustration in web app development. And let’s be honest, putting everything in useEffect hooks doesn’t help much. In this talk, we'll demystify the useEffect hook and get a better understanding of when (and when not) to use it, as well as discover how declarative effects can make effect management more maintainable in even the most complex React apps.
React Summit 2022React Summit 2022
20 min
Routing in React 18 and Beyond
Concurrent React and Server Components are changing the way we think about routing, rendering, and fetching in web applications. Next.js recently shared part of its vision to help developers adopt these new React features and take advantage of the benefits they unlock.In this talk, we’ll explore the past, present and future of routing in front-end applications and discuss how new features in React and Next.js can help us architect more performant and feature-rich applications.
React Advanced Conference 2021React Advanced Conference 2021
27 min
(Easier) Interactive Data Visualization in React
If you’re building a dashboard, analytics platform, or any web app where you need to give your users insight into their data, you need beautiful, custom, interactive data visualizations in your React app. But building visualizations hand with a low-level library like D3 can be a huge headache, involving lots of wheel-reinventing. In this talk, we’ll see how data viz development can get so much easier thanks to tools like Plot, a high-level dataviz library for quick & easy charting, and Observable, a reactive dataviz prototyping environment, both from the creator of D3. Through live coding examples we’ll explore how React refs let us delegate DOM manipulation for our data visualizations, and how Observable’s embedding functionality lets us easily repurpose community-built visualizations for our own data & use cases. By the end of this talk we’ll know how to get a beautiful, customized, interactive data visualization into our apps with a fraction of the time & effort!

Workshops on related topic

React Summit 2023React Summit 2023
170 min
React Performance Debugging Masterclass
Featured WorkshopFree
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
React Advanced Conference 2021React Advanced Conference 2021
132 min
Concurrent Rendering Adventures in React 18
Featured WorkshopFree
With the release of React 18 we finally get the long awaited concurrent rendering. But how is that going to affect your application? What are the benefits of concurrent rendering in React? What do you need to do to switch to concurrent rendering when you upgrade to React 18? And what if you don’t want or can’t use concurrent rendering yet?

There are some behavior changes you need to be aware of! In this workshop we will cover all of those subjects and more.

Join me with your laptop in this interactive workshop. You will see how easy it is to switch to concurrent rendering in your React application. You will learn all about concurrent rendering, SuspenseList, the startTransition API and more.
React Summit Remote Edition 2021React Summit Remote Edition 2021
177 min
React Hooks Tips Only the Pros Know
Featured Workshop
The addition of the hooks API to React was quite a major change. Before hooks most components had to be class based. Now, with hooks, these are often much simpler functional components. Hooks can be really simple to use. Almost deceptively simple. Because there are still plenty of ways you can mess up with hooks. And it often turns out there are many ways where you can improve your components a better understanding of how each React hook can be used.You will learn all about the pros and cons of the various hooks. You will learn when to use useState() versus useReducer(). We will look at using useContext() efficiently. You will see when to use useLayoutEffect() and when useEffect() is better.
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2021React Advanced Conference 2021
145 min
Web3 Workshop - Building Your First Dapp
Featured WorkshopFree
In this workshop, you'll learn how to build your first full stack dapp on the Ethereum blockchain, reading and writing data to the network, and connecting a front end application to the contract you've deployed. By the end of the workshop, you'll understand how to set up a full stack development environment, run a local node, and interact with any smart contract using React, HardHat, and Ethers.js.
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn