Video Editing in the Browser

Rate this content
Bookmark

Video editing is a booming market with influencers being all the rage with Reels, TikTok, Youtube. Did you know that browsers now have all the APIs to do video editing in the browser? In this talk I'm going to give you a primer on how video encoding works and how to make it work within the browser. Spoiler, it's not trivial!

Christopher Chedeau
Christopher Chedeau
24 min
06 Jun, 2023

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This Talk discusses the challenges of video editing in the browser and the limitations of existing tools. It explores image compression techniques, including Fourier transform and Huffman encoding, to reduce file sizes. The video codec and frame decoding process are explained, highlighting the importance of keyframes and delta frames. The performance bottleneck is identified as the codec, and the need for specialized hardware for efficient video editing is emphasized. The Talk concludes with a call to create a simplified API for video editing in the browser and the potential for AI-powered video editing.

Available in Español

1. Introduction to Video Editing in the Browser

Short description:

Hey, everyone. Today, I want to talk about video editing in the browser. I spent a lot of time doing video editing during the pandemic. However, I realized that the existing tools didn't have the AI advancements I needed. I wanted to remove the green screen and shadows, and cut based on spoken words. On the other hand, I saw exciting developments in JavaScript, such as WebCodecs, TensorFlow.js, and Whisper. This talk will explain why I couldn't fully achieve a good video editor powered by AI. Let's start with thinking about making a video.

Hey, everyone. My name is Christophe Archido, also known as Vegeux on the Internet. And I've done a few things for the React community. I co-created React Native, Prettier, Excalibur, CSS in JS, but today I want to talk about something different. I want to talk about video editing in the browser.

So during the pandemic, I spent a lot of time doing video editing. And I was even thinking maybe I should go like become a YouTuber full-time. But then I realized that with this number of views, I should probably keep my job as a software engineer for a bit longer.

So what does it mean to edit videos? So I used a tool called Final Cut Pro. And I felt that it was built like many, many years ago and didn't have all of the AI advancements that we've seen recently. So for example, I bought a $20 green screen. And I need to pick the green color and the range in order to remove it. And as you can see, there's some shadows behind me in the picture. And it wasn't properly removed. Then in order to cut, I want to know what am I actually saying to know which part I should be cutting. But I only got the sound waves and not the actual words spoken. On the other side, I was looking at the JavaScript, like the browser news, and I saw a lot of super exciting stuff happening. So we can start doing encoding and decoding with WebCodecs. TensorFlow.js lets you remove the background from the video. And then, Whisper is letting you take what I'm saying into actual words. So we had seemingly all of the building blocks in order to be able to do a really good video editor powered by AI, but unfortunately, I wasn't able to get all the way there. And this talk is going to be the story of why.

So usually when I walk into some new product like this, there's some things that I think are true I'm going to use to base all of the things I'm doing upon. But there were three things in this case that were not true. So the first one is that time only travels forward. The second is that when you encode one frame, you're getting one frame back. And finally that WASM is faster than JavaScript for video decoding. So if you want to know why this is not true, buckle up. We're getting to it. So let's start with thinking about making a video.

2. Video Editing API and Image Compression

Short description:

Unfortunately, the desired API for video editing in the browser is not possible due to the large file sizes involved. A single image of a thousand by thousand pixels can already be around four megabytes in size. With 60 frames per second, a one-second video would be around 200 megabytes. This is too big for current browsers and computers. However, image compression techniques have been developed to address this issue, which will be discussed in the following minutes.

And unfortunately I cannot be here in person today, so what I decided to do was to bring some of the sunny California to Amsterdam. And for this I put a palm tree in all of the pictures. So in this case, we have React summit in the background and then moving to the foreground and the palm tree fading away. So what would be the API that I would expect to be able to do that? So I initially wanted a load video kind of API. That takes a file path and returns me a list of images. And then I'm going to modify the images, remove the background, like cut and paste and a bunch of stuff. And then like a save video that would take the file path and render. And like a list of images and like actually save the video.

So unfortunately, this API cannot exist. So let's see why. So let's go into like one image of this whole video. And not too big, not too small. Like a thousand by thousand image. And how large is it actually to represent this? So it's going to be like one thousand by one thousand pixels. About one megabyte. And then there's red, green and blue. And so we are about like four megabytes in size. And this is just for one image. Now, if you want like 60 fps, like one second, you're going to be at like 200 megabytes for every single second. So this talk right now is around 20 minutes. So this is going to be big. And this is actually going to be too big for the browser or like any computer right now. And what do we do? So fortunately, a lot of very smart people have worked on this for years. And what they built is a shrinking machine. Well, not exactly. What people have been doing is image compression. And so I'm going to talk for like the next few minutes around like different types of image compression. And not because I find interesting, which I do, but because they actually have a big factor into the actual API used for video encoding. So let's see the main ideas around video encoding. Sorry, about image compression.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

A Framework for Managing Technical Debt
TechLead Conference 2023TechLead Conference 2023
35 min
A Framework for Managing Technical Debt
Top Content
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt.In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.

Debugging JS
React Summit 2023React Summit 2023
24 min
Debugging JS
Top Content
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.
Power Fixing React Performance Woes
React Advanced Conference 2023React Advanced Conference 2023
22 min
Power Fixing React Performance Woes
Top Content
Next.js and other wrapping React frameworks provide great power in building larger applications. But with great power comes great performance responsibility - and if you don’t pay attention, it’s easy to add multiple seconds of loading penalty on all of your pages. Eek! Let’s walk through a case study of how a few hours of performance debugging improved both load and parse times for the Centered app by several hundred percent each. We’ll learn not just why those performance problems happen, but how to diagnose and fix them. Hooray, performance! ⚡️
Monolith to Micro-Frontends
React Advanced Conference 2022React Advanced Conference 2022
22 min
Monolith to Micro-Frontends
Top Content
Many companies worldwide are considering adopting Micro-Frontends to improve business agility and scale, however, there are many unknowns when it comes to what the migration path looks like in practice. In this talk, I will discuss the steps required to successfully migrate a monolithic React Application into a more modular decoupled frontend architecture.
Building a Voice-Enabled AI Assistant With Javascript
JSNation 2023JSNation 2023
21 min
Building a Voice-Enabled AI Assistant With Javascript
Top Content
In this talk, we'll build our own Jarvis using Web APIs and langchain. There will be live coding.
A Practical Guide for Migrating to Server Components
React Advanced Conference 2023React Advanced Conference 2023
28 min
A Practical Guide for Migrating to Server Components
Server Components are the hot new thing, but so far much of the discourse around them has been abstract. Let's change that. This talk will focus on the practical side of things, providing a roadmap to navigate the migration journey. Starting from an app using the older Next.js pages router and React Query, we’ll break this journey down into a set of actionable, incremental steps, stopping only when we have something shippable that’s clearly superior to what we began with. We’ll also discuss next steps and strategies for gradually embracing more aspects of this transformative paradigm.

Workshops on related topic

Building a Shopify App with React & Node
React Summit Remote Edition 2021React Summit Remote Edition 2021
87 min
Building a Shopify App with React & Node
Top Content
WorkshopFree
Jennifer Gray
Hanna Chen
2 authors
Shopify merchants have a diverse set of needs, and developers have a unique opportunity to meet those needs building apps. Building an app can be tough work but Shopify has created a set of tools and resources to help you build out a seamless app experience as quickly as possible. Get hands on experience building an embedded Shopify app using the Shopify App CLI, Polaris and Shopify App Bridge.We’ll show you how to create an app that accesses information from a development store and can run in your local environment.
Build a chat room with Appwrite and React
JSNation 2022JSNation 2022
41 min
Build a chat room with Appwrite and React
WorkshopFree
Wess Cope
Wess Cope
API's/Backends are difficult and we need websockets. You will be using VS Code as your editor, Parcel.js, Chakra-ui, React, React Icons, and Appwrite. By the end of this workshop, you will have the knowledge to build a real-time app using Appwrite and zero API development. Follow along and you'll have an awesome chat app to show off!
Hard GraphQL Problems at Shopify
GraphQL Galaxy 2021GraphQL Galaxy 2021
164 min
Hard GraphQL Problems at Shopify
WorkshopFree
Rebecca Friedman
Jonathan Baker
Alex Ackerman
Théo Ben Hassen
 Greg MacWilliam
5 authors
At Shopify scale, we solve some pretty hard problems. In this workshop, five different speakers will outline some of the challenges we’ve faced, and how we’ve overcome them.

Table of contents:
1 - The infamous "N+1" problem: Jonathan Baker - Let's talk about what it is, why it is a problem, and how Shopify handles it at scale across several GraphQL APIs.
2 - Contextualizing GraphQL APIs: Alex Ackerman - How and why we decided to use directives. I’ll share what directives are, which directives are available out of the box, and how to create custom directives.
3 - Faster GraphQL queries for mobile clients: Theo Ben Hassen - As your mobile app grows, so will your GraphQL queries. In this talk, I will go over diverse strategies to make your queries faster and more effective.
4 - Building tomorrow’s product today: Greg MacWilliam - How Shopify adopts future features in today’s code.
5 - Managing large APIs effectively: Rebecca Friedman - We have thousands of developers at Shopify. Let’s take a look at how we’re ensuring the quality and consistency of our GraphQL APIs with so many contributors.
0 To Auth In An Hour For Your JavaScript App
JSNation 2023JSNation 2023
57 min
0 To Auth In An Hour For Your JavaScript App
WorkshopFree
Asaf Shen
Asaf Shen
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.js backend + Vanilla JS frontend) to authenticate users with One Time Passwords (email) and OAuth, including:
- User authentication – Managing user interactions, returning session / refresh JWTs- Session management and validation – Storing the session securely for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Build a Collaborative Notion-Like Product in 2H
JSNation 2023JSNation 2023
87 min
Build a Collaborative Notion-Like Product in 2H
WorkshopFree
Witek Socha
Witek Socha
You have been tasked with creating a collaborative text editing feature within your company’s product. Something along the lines of Notion or Google Docs.
CK 5 is a feature-rich framework and ecosystem of ready-to-use features targeting a wide range of use cases. It offers a cloud infrastructure to support the real-time collaboration system needs. During this workshop, you will learn how to set up and integrate CK 5. We will go over the very basics of embedding the editor on a page, through configuration, to enabling real-time collaboration features. Key learnings: How to embed, set up, and configure CK 5 to best fit a document editing system supporting real-time collaboration.
Table of contents:- Introduction to the CK 5 ecosystem.- Introduction to a “Notion-like” project template.- Embedding CK 5 on a page.- Basic CK 5 configuration.- Tuning up CK 5 for a specific use case.- Enabling real-time editing features.