Visualize your music with React, a sound design experiment

Rate this content

React is a great library to design user interfaces, but what about rendering sounds? In this talk we'll dive deeper in handling sounds, creating a transformation to convert any input vibration into images.

9 min
22 Oct, 2021

Video Summary and Transcription

Claudia Bressi, a senior front-end and mobile developer, combines music and computer science in her sound design project using React and the Web Audio API. The project supports different audio file formats and utilizes powerful methods for analyzing and visualizing sounds. Custom hooks are used for modularity and code reuse. The project is divided into four areas: shape, color, animation, and playback. Future improvements include making it a progressive web app, adding ML algorithms, server-side rendering, and optimizing performance with workers.

Available in Español

1. Introduction to the Sound Design Project

Short description:

I'm Claudia Bressi, a senior front-end and mobile developer from Italy. I'm passionate about music and computer science, and I've combined them in my sound design project using React. The project visualizes music using the Web Audio API and supports different audio file formats. We use React, TypeScript, Redux Toolkit, Framer Motion, and Material UI for development. The Web Audio API provides powerful methods for analyzing, filtering, and visualizing sounds, and we utilize the Analyzer node and Audio Worklet for this project. Browser compatibility is a consideration, and we have built custom hooks for modularity and code reuse.

Hi everyone. I'm really grateful to join react advanced today for presenting you my sound design project. I'm Claudia Bressi, and I am a senior front-end and mobile developer. I live near Venice in Italy, and I work for a consultant agency in Milan, Italy, where we build applications using react and react native for mobile.

I'm really passionate about both music and computer science. And since I was a child, I started learning piano and also I started using the computer in my father's office. At present time, I still love playing instruments like electronic keyboard and guitar, listen to my favorite playlists and love going to jigs and festivals. That's why I try to combine a front-end project with music.

Let's talk about the project. The main goal is to visualize music using React library. And the overall application can handle any type of audio file, for example MP3 or WAV files. And as a result, we generate a visual component for each sound spectrum. Let's dive deeper in the project details. The code is React and TypeScript in order to have easier maintainability. For handling a global state throughout the application, I've set up Redux toolkit For animating components, I've opted for Framer Motion Library, which is a quite modern and straightforward package. Then, on global UI components, like buttons, inputs and typography, I've chosen the well-known material UI, which probably many of you already use in your React applications.

The Web Audio API is an amazing library that lets you deal with audio data. It gives us developers thousands of methods to analyze, filtering and visualizing sounds. Also, there are multiple available options to add 3D special effects, panning, multichannel splitting and much more. For this project, we mainly use two interfaces, the Analyzer node, which lets us handle with specific audio information as for frequencies, and the Audio Worklet, that lets us developers add new modules to be executed of the main thread. High level speaking, every time you work with Web Audio API, the generated result will be a direct acyclic graph, where each node is either an audio source, a filter effect or a destination target. Through the Web Audio API, visualizations are achieved by assessing audio parameters ranging over time. These variables are gain, pitch and frequencies most of the times. The Web Audio API has indeed the already mentioned analyzer node that leaves unaltered the audio signal passing through it. Instead it outputs audio data that can be passed to a visualization technology such as HTML Canvas or, in our case, React Functional Components. An expected fun fact is the browser compatibility. As you can see, Internet Explorer is not supported at all. Luckily for us developers, this browser will be retired next year on June 15. In order to improve our project, I've built up some custom hooks to improve modularity and reuse throughout the codebase.

2. Improving the Project and Final Thoughts

Short description:

To improve our project, I've built custom hooks for modularity and reuse. UseFrequency extracts frequency values, UseWaveform gets sound spectrum data, UsePitch reads pitch values, and useGain pulls out volume. We split the project into four areas: shape with Pitch Audio, color with UseFrequency, animation with FramerMotion, and playback with AudioHTML. Tools like AudioOnChrome and WebAudioDevToolsExtension are useful for debugging and inspecting the audio graph. Next, we'll make the project a progressive web app, add ML algorithms, server-side rendering, and optimize performance with workers. Experiment with React, AudioData, and the Web Audio API in your projects. Share your thoughts with the hashtag I've created. Thanks for listening and enjoy the conference!

In order to improve our project, I've built up some custom hooks to improve modularity and reuse throughout the codebase. UseFrequency extract any frequency value from the audio file. UseWaveform gets the whole sound spectrum in terms of time domain data. UsePitch reads the current sound pitch value. And finally, useGain pulls out volume from input.

Having a rich data visualization, it's fundamental and to do this, I split up in four areas. For giving specific shape, I've used the Pitch Audio parameter. For coloring components, I've applied the UseFrequency custom hook. While for animating the UI, I've opted for FramerMotion API. And I've used the AudioHTML tag for providing the user playback functionality on the already uploaded audio files.

Having tools is a crucial part in code development to let us inspect and troubleshooting our code. AudioOnChrome is a useful Chrome plugin for debugging any audio project. Then, we have WebAudioDevToolsExtension that comes handy for inspecting the generated audio graph.

As next step, I would improve the whole project, enable it to perform as a progressive web application, thus having a universal accessible tool. I would also enhance the features by adding some Machine Learning algorithms, server-side rendering, and the ability to use workers for performance optimizations.

Finally, I'll leave with you some reflections. React is a great library to express web development in a creative way. So I suggest you to try and experiment as much as possible in your personal project, in particular coding with AudioData and the Web Audio API. which is very simple to apply. If you'll ever try to code on this topic, please share your thoughts using the hashtag I've created below. It would be awesome to see your creation. So yeah, thanks for listening, I hope you found this topic interesting and enjoy the rest of the conference.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Advanced Conference 2022React Advanced Conference 2022
25 min
A Guide to React Rendering Behavior
Top Content
React is a library for "rendering" UI from components, but many users find themselves confused about how React rendering actually works. What do terms like "rendering", "reconciliation", "Fibers", and "committing" actually mean? When do renders happen? How does Context affect rendering, and how do libraries like Redux cause updates? In this talk, we'll clear up the confusion and provide a solid foundation for understanding when, why, and how React renders. We'll look at: - What "rendering" actually is - How React queues renders and the standard rendering behavior - How keys and component types are used in rendering - Techniques for optimizing render performance - How context usage affects rendering behavior| - How external libraries tie into React rendering
React Summit Remote Edition 2021React Summit Remote Edition 2021
33 min
Building Better Websites with Remix
Top Content
Remix is a new web framework from the creators of React Router that helps you build better, faster websites through a solid understanding of web fundamentals. Remix takes care of the heavy lifting like server rendering, code splitting, prefetching, and navigation and leaves you with the fun part: building something awesome!
React Advanced Conference 2023React Advanced Conference 2023
33 min
React Compiler - Understanding Idiomatic React (React Forget)
React provides a contract to developers- uphold certain rules, and React can efficiently and correctly update the UI. In this talk we'll explore these rules in depth, understanding the reasoning behind them and how they unlock new directions such as automatic memoization. 
React Advanced Conference 2022React Advanced Conference 2022
30 min
Using useEffect Effectively
Top Content
Can useEffect affect your codebase negatively? From fetching data to fighting with imperative APIs, side effects are one of the biggest sources of frustration in web app development. And let’s be honest, putting everything in useEffect hooks doesn’t help much. In this talk, we'll demystify the useEffect hook and get a better understanding of when (and when not) to use it, as well as discover how declarative effects can make effect management more maintainable in even the most complex React apps.
React Summit 2022React Summit 2022
20 min
Routing in React 18 and Beyond
Top Content
Concurrent React and Server Components are changing the way we think about routing, rendering, and fetching in web applications. Next.js recently shared part of its vision to help developers adopt these new React features and take advantage of the benefits they unlock.In this talk, we’ll explore the past, present and future of routing in front-end applications and discuss how new features in React and Next.js can help us architect more performant and feature-rich applications.
React Advanced Conference 2021React Advanced Conference 2021
27 min
(Easier) Interactive Data Visualization in React
Top Content
If you’re building a dashboard, analytics platform, or any web app where you need to give your users insight into their data, you need beautiful, custom, interactive data visualizations in your React app. But building visualizations hand with a low-level library like D3 can be a huge headache, involving lots of wheel-reinventing. In this talk, we’ll see how data viz development can get so much easier thanks to tools like Plot, a high-level dataviz library for quick & easy charting, and Observable, a reactive dataviz prototyping environment, both from the creator of D3. Through live coding examples we’ll explore how React refs let us delegate DOM manipulation for our data visualizations, and how Observable’s embedding functionality lets us easily repurpose community-built visualizations for our own data & use cases. By the end of this talk we’ll know how to get a beautiful, customized, interactive data visualization into our apps with a fraction of the time & effort!

Workshops on related topic

React Summit 2023React Summit 2023
170 min
React Performance Debugging Masterclass
Featured WorkshopFree
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
React Advanced Conference 2021React Advanced Conference 2021
132 min
Concurrent Rendering Adventures in React 18
Top Content
Featured WorkshopFree
With the release of React 18 we finally get the long awaited concurrent rendering. But how is that going to affect your application? What are the benefits of concurrent rendering in React? What do you need to do to switch to concurrent rendering when you upgrade to React 18? And what if you don’t want or can’t use concurrent rendering yet?

There are some behavior changes you need to be aware of! In this workshop we will cover all of those subjects and more.

Join me with your laptop in this interactive workshop. You will see how easy it is to switch to concurrent rendering in your React application. You will learn all about concurrent rendering, SuspenseList, the startTransition API and more.
React Summit Remote Edition 2021React Summit Remote Edition 2021
177 min
React Hooks Tips Only the Pros Know
Top Content
Featured Workshop
The addition of the hooks API to React was quite a major change. Before hooks most components had to be class based. Now, with hooks, these are often much simpler functional components. Hooks can be really simple to use. Almost deceptively simple. Because there are still plenty of ways you can mess up with hooks. And it often turns out there are many ways where you can improve your components a better understanding of how each React hook can be used.You will learn all about the pros and cons of the various hooks. You will learn when to use useState() versus useReducer(). We will look at using useContext() efficiently. You will see when to use useLayoutEffect() and when useEffect() is better.
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2021React Advanced Conference 2021
145 min
Web3 Workshop - Building Your First Dapp
Top Content
Featured WorkshopFree
In this workshop, you'll learn how to build your first full stack dapp on the Ethereum blockchain, reading and writing data to the network, and connecting a front end application to the contract you've deployed. By the end of the workshop, you'll understand how to set up a full stack development environment, run a local node, and interact with any smart contract using React, HardHat, and Ethers.js.
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn