Practical talk showing how to go from a local state to an on-the-edge offline capable persisted state to make an app like Figma, Notion, and more.
Real-Time Collaborative State on the Edge
AI Generated Video Summary
Steven, a design tool nerd, worked on InVision Studio and now focuses on LiveBlocks, providing APIs and tools for multiplayer collaboration. Collaborative design tools like Figma, Canva, Mural, Notion, Mirror, Linear, and Pitch use conflict resolution algorithms such as OTs and CRDTs. Data migration and collaboration patterns are important considerations for building production applications. Figma stands out for its seamless assembly of collaboration features. LiveBlocks enables front-end developers to build collaborative experiences without complex backend infrastructure, using fractional indexing for consistent ordering.
1. Introduction to Design Tools and Experience
Hey, everybody. My name is Steven, and I've lived all over the place. I am a massive design tool nerd. I worked at Facebook Messenger and Campaign Monitor. I created Easy and joined InVision. At InVision, I worked on InVision Studio, a UI design tool.
Hey, everybody. My name is Steven, and I've lived all over the place. So, I'm doing this talk from Paris today, but I lived in many different places. I spent, you know, 7 ½ years in Australia, in Sydney, even lived in Quebec before that for a little bit, and also spent a few years in New York City.
So, yeah, great to be here today. And what I want to tell you guys is that I am a massive design tool nerd. I love building. I love designing design tools. I love creating apps and things that people use as a return and create something with that tool that I helped make. And so, this is the angle for my talk. I'm a big design tool nerd and I love making this kind of creative tools.
So, I worked at Facebook Messenger for a little while on the design system slash Messenger team to help them with their tooling needs internally. I also spent some time at a company called Campaign Monitor in Australia back when I was in Sydney. And, my main project there was to work on this email builder. So, it was a drag-and-drop tool where you could design beautiful emails that were like responsive, so they worked on mobile and desktop. And we helped brands kind of create those emails just in a drag-and-drop way very easily. That was the first time really that I was really introduced to creative tools. That eventually led me to create Easy. And Easy, I don't know if some of you have heard of it, it's pretty old now, was a tool to help designers create animations for the web, and then to do that without having to code. And then you just could click that and get the CSS from it. It was pretty cool. I enjoyed working on that. And that eventually led me to join InVision.
And so at InVision I worked on the team responsible for creating InVision Studio. I worked on that for four and a half years. And I don't know if you guys are familiar with InVision Studio. But that was essentially a UI design tool where you could, you know, obviously design apps, websites, but also create animations for them and prototype them, make them very interactive. And this sounds pretty obvious, but back in the day there weren't very many tools that could do that sort of stuff. And really, that experience really led me to the path I'm on today. We had a, you know, it was a desktop application working on the file system, and Figma was this amazing browser based tool to do UI design and that sort of stuff was starting to become really popular.
2. Challenges of Building Collaborative Products
Figma's multiplayer feature and the challenges of converting InVision Studio to a browser-based collaborative tool led me to start LiveBlocks. At LiveBlocks, we provide APIs and tools for developers to build multiplayer and collaborative experiences. Building a collaborative product involves more than just multiplayer functionality. It includes features like comments, annotations, shared dialogues, document browsing, and audio/video capabilities. Google Docs played a significant role in shaping the collaborative SaaS landscape with its browser-based document editing capabilities.
And one thing that Figma did really well is that, as I said, the product worked in the browser, meaning you don't have to download an app to get it to work, and it was what they called multiplayer. And so, we eventually had this project at InVision to convert InVision Studio from a desktop based file based application, excuse me, to an application that was browser based and collaborative by default. And so I led design for the project, and this is the first time I really got exposed to the problem of like, damn, this is freaking hard to do. We had a dedicated squad of, I think, at the peak, seven engineers, one point manager, and I was the designer of kind of heading up that initiative. And it took over it took about a year and a half before we could have a working version of that desktop based, sorry, browser based multiplayer application a year and a half. That's a very long time. And, yeah, I was eventually a little bit frustrated with this because I was like this, I just don't understand how this could take that long. But that is that's just how it is. Like it takes very long to build those kind of experiences when you do all that stuff yourself. And so being exposed to that I didn't realize, but that eventually led me to the path I'm on today. I run a company called LiveBlocks and at LiveBlocks what we do, we provide APIs and tools to developers like you guys to build multiplayer experiences, but also more broadly collaborative experiences like comments and pretty much anything you want to do that's collaborative and requires multiple people to work together on some type of digital experience. But don't don't be worried. This is not a sales pitch. I promise I'm only here to share my learnings and dive into the behind the scenes of those kind of infrastructures, the kind of infrastructure needed to build those kind of experiences. But yeah, just to show that this problem that I personally faced with my co-founder Guillaume at InVision has become I mean, it's always been an issue, but most recent years there's been a lot of technologies and frameworks to do to work on this problem. And so a lot of those frameworks and tools when they talk about this often refer to multiplayer. But in the world of collaboration, when you build a collaborative SAS product, multiplayer is only the tip of the iceberg. There are a lot more things to think about when building a collaborative product, right? You need to think about, okay, well, when my users are in my product, how can they leave comments? How can they mention their co-workers or collaborators in comments, in specific annotations within the document, so that those people end up collaborating asynchronously and can join the document and be notified elsewhere? There's also shared dialogues, like how do you share an artifact or something with somebody else? How do you add those people? How do you take care of the permissions for those people? And then, obviously, once you have those documents and those collaborative artifacts working, how do you browse them? How do you find them? How do you find all those documents? How do you share those set of documents? How do you put them into a folder and that kind of stuff? So document browsing is a big part of that. And then, obviously, things like audio and video are becoming more and more present. So if you want to do an audio huddle, you can do that sort of stuff. Share like a video as you're working on something. Having that experience directly in the tool is becoming more and more relevant for a collaborative experience. And so if we take a step back and think about how we got to the state of where we are today in terms of collaborative SaaS products. To me, it really started with Google Docs. I think that was 15 or 20 years ago at this point. They introduced this is not the screenshot. You see, this is a lot more recent, but they introduced Google Docs, a browser-based document or similar to Microsoft Word. And it was mind blowing. Like you could just open a URL and that URL was a Word document pretty much. And you could edit it.
3. Collaborative Design Tools and Conflict Resolution
Figma revolutionized the design industry by making design tools browser-based, allowing a wider range of users to collaborate. Other companies like Canva, Mural, Notion, Mirror, Linear, and Pitch also provide multiplayer collaboration experiences. Conflict resolution is essential for multiplayer collaboration, and two main algorithms, operational transforms (OTs) and CRDTs, are used. CRDT frameworks like Yjs and Auto merge handle conflict management and provide easy-to-use data types. However, CRDTs can be challenging to use and scale in production.
You could copy that URL, paste it to somebody else. Pasted it into somebody else, and that other person would be in the same document with you collaborating. And then probably a decade later, Figma started working on their product. They were creating this design tool, made it browser-based and what they did with Figma is that they revolutionized the design industry so that people could work directly in the browser. And now not only designers could actually design files or have access to those design files, they made the design process available to a lot more than designers.
Now you have developers coming into Figma to inspect the elements in order to help them build those UIs that designers put together. You have stakeholders, founders, you know, C-level people that can go into files, review and experiment with design. And you even have, like, product people or pretty much anybody that wish to do so can just go into a file and play around with some of the components that are pre-built by the design team and start actually designing some of that stuff themselves. Figma has been really, really successful. And I think what they showed with their product is you can actually make a high quality, performant tool that typically was only desktop based. You can actually put that in the browser and make it collaborative. And all of the sudden you get so many benefits organization to seamlessly collaborate together in one space. And there are obviously more companies nowadays, you know, that have those type of multiplayer kind of collaborative experiences by default.
Canva is one of them, Mural, obviously for whiteboarding, Notion, Mirror, Linear has been really great at that for the ticket management software and obviously Pitch to do presentations like this one. And to enable multiplayer collaboration, you need to think about conflict resolution. And what is conflict resolution? Essentially, when you are in this world where multiple people can manipulate the same document state, you need to be able to resolve. Conflicting actions from different people that are being processed at the same time. And so if you have somebody requesting to make that rectangle red and somebody requesting to make it a circle, like what do you do at that point, you know, do you make it just red or do you combine them make it red and make it a circle. You need to think about those things when you're in the multiplayer world.
4. Data Migration and Collaboration Patterns
You have to think about data migration and schema validation when building a serious production application. CRDTs can lead to large document sizes due to storing all changes over time. Collaboration tools like Google Notion, Figma, and Pitch have similar patterns, such as shared dialogues, permissions, and comments.
You have to think about data migration, like, you know, if you're a serious production application, you want to be able to say, like, my data is going to look like this and you want to be able to evolve that over time. So that can be pretty hard to do. Schema validation, you know, very similar to data migration that kind of linked with one another, but being able to update the schema of a document over time as you're as your product evolves, it's pretty pretty important as well.
And so one of the big things as well with CRDTs is that the data can be very, very complex because most of the like all the changes typically get stored. And so instead of having one JSON representation of of a document, typically you have all the changes that happen over time. And that often leads to a potential, very large document size, even if the visually like the actual JSON representation can be small, that JSON representation is made of all the changes that have happened. And that can lead to like a pretty big file, especially if that file has been around for years and many people interacted with it.
And so what we just looked at is just the tip of the iceberg. It's only the multiplayer part. There are a lot more things to think about, as I said earlier. And so if you look at I just selected like four top kind of SaaS products that do collaboration really well, they all have similar patterns, right? If you look at Google Notion, Figma and Pitch, they all have some kind of shared dialogue where you can invite other people, copy the link, and also set specific permissions for those people. Like, can this person edit the file? Is that person the owner of a file? Can they only view that file? Those are the type of things that that multiplayer products need to enable as well. Comments and mentions. Like great collaboration needs comments.".
5. Collaboration Features and Figma
This is just a small sample, but that list goes on and on. It's a DevTools browser extension to inspect your Yjs experience and understand how the document has been constructed. The multiplayer part is just the tip of the iceberg. SaaS products like Google Notion, Figma, and Pitch have similar patterns, enabling shared dialogue, permissions, comments, mentions, document browsing, and audio/video collaboration. Figma stands out for its seamless assembly of collaboration features.
This is just a small sample, but that list goes on and on and on and on. And you can scroll for probably a couple of minutes without reaching the end of those changes. And if you look at kind of how those changes get assembled, they look very much like this.
This is a tool that we built. It's a DevTools browser extension to enable you to inspect your Yjs experience to understand how the document has been constructed. You can see kind of the dependencies between the changes and what leads to that representation of the data. And so, yeah, pretty simple data at the end. But the actual representation itself is pretty simple. But the actual data making that representation can be can be pretty complex and lead to pretty big, pretty big file size in the end.
And so what we just looked at is just the tip of the iceberg. It's only the multiplayer part. There are a lot more things to think about, as I said earlier. And so if you look at I just selected like four top kind of SaaS products that do collaboration really well, they all have similar patterns, right? If you look at Google Notion, Figma and Pitch, they all have some kind of some kind of shared dialogue where you can invite other people, copy the link, and also set specific permissions for those people. Like, can this person edit the file? Is that person the owner of a file? Can they only view that file? Those are the type of things that that multiplayer products need to enable as well.
Comments and mentions. Like great collaboration needs comments. Like if you want to be able to collaborate, chances are you're not always all going to be in the same file at the same time. You want to be able to comment and collaborate asynchronously as well. So Google, you can comment, you can mention people and that's the same across the board. And then document browsing. All of them have some type of workspace view, you know, where you can browse your documents together with other people. You can create folders and those people can access those those folders and browse the documents. And then not all of them, but I think this is a pattern that we're going to see emerge in more and more collaborative tools. Audio and video plays a big part into some of the tools like pitch, for instance, enables you to have video right in your presentation so that as you move around your cursor, your face is essentially attached to the cursor, giving all the people in the document a sense of presence and a sense of being together in the same room with you. And then if you look at Figma, Figma has this audio huddle feature, enabling you to quickly get into an audio call with somebody else that's looking at the same document as you. Making it very easy and seamless to collaborate with those people in real time. And so if I look at all those companies, they all do a great job. But I think there is one that definitely stands out for me and has been super inspiring for what we do at Liveblocks. It's definitely Figma. I think Figma has been great at assembling all the required collaboration features in a very nice way so that they seamlessly work together.
6. Building Collaborative Experiences with LiveBlocks
As a user, you have a great collaborative experience. Figma's multiplayer technology is explained in a brilliant blog post by Evan Wallace. This led us to the idea of creating LiveBlocks, enabling front-end developers to build collaborative experiences without complex backend infrastructure. LiveBlocks focuses on the practical side of multiple experiences and enables collaboration on various artifacts, providing presence, broadcast, data store, and comments functionalities.
And as a user, you end up having a great collaborative experience, almost as if you were in the same physical space with somebody else. And so if there is one blog post that I would recommend from them and especially from Evan Wallace, who was the co-founder of Figma and the CTO for most of the for multiple years there, he explains how the multiplayer technology works. And it's a brilliant blog post with like a lot of interactive illustrations that makes it very easy to understand how that stuff works.
And so the work that Evan and Figma did eventually led Guillaume, my co-founder and myself to this question. What would developers build if they could access Figma's infrastructure through React hooks and components? This is something that I really wish we had back at Invision when we were converting that desktop based application to a browser based multiplayer collaborative application. And so eventually we were like, damn like that would be... This is so hard to build. Like, why don't we why don't we have that as React, some React hooks and components and make it very easy for like front end developers to build those kinds of experiences without having to worry about all that complex back end infrastructure? And so that's what that's what it is to create live blocks. We kind of stumbled upon this idea as we were working on a video presentation tool. And as we worked on that tool, we realized, damn, this feels very much like what we experienced at Invision, like we're spending awful time trying to figure out the real time aspect of things and very little time actually focusing on the tool. And so at that point, that that's when we decided to create live blocks.
And so once again, I promise this is not a self-speech. I just I just want to show you a bit more. The practical side of this, like to show you some of those kind of multiple experiences, how the infrastructure works behind the scenes and how we enable that at live blocks. And if you think about the collaborative experiences that I highlighted before, they all work in the same way. There is one central artifact on which multiple people collaborate and that artifact can be a text document. It can be a whiteboard. It can be a creative tool. It can be pretty much anything you want. And to build that central artifact, there are multiple things that you need to make it collaborative. You need a concept of presence so that the people looking at that artifact know who is currently currently looking at this document, showing the live cursors, like showing the even eventually like showing the video of the person next to the cursor like pitch does, showing like a live avatar stack, showing what item people have selected in the document, that kind of stuff. Then you need the concept of broadcast. Right. So when you are in a room, you want to be able to broadcast events to the other people that are currently in the same room as you. So if you want to do things like live reactions, for instance, a fire emoji on this element, whatever you click it, you broadcast an event and other people can then show that fire emoji at the same time. And then this is the core of the document itself. You need to you need a data store to store the document of the data and that data store needs to enable people to make edits to the same data at the same time. And that's what CRDTs enable like Yjs or like auto merge or some other frameworks out there. And then lastly, you need comments. So comments is more for async collaboration, but you want to be able to comment and annotate on specific parts of that document so that when you mention somebody there, that other person receives a notification and they're incentivized to then go into the product and collaborate and see what the other people have done.
7. Document Storage and Fractional Indexing
But this is a pretty short talk, so I will only focus on the document part and how that works. So at LiveBlocks, we have two ways to store documents, so we have our own proprietary solution called LiveBlocks Storage, and we also recently introduced a way to store Yjs documents right into the LiveBlocks infrastructure. And so if we look at storage, we have three data types for now. So LiveList, LiveMap and LiveObject. And I wanted to dive into the list use case because I think this is one that's nice to highlight visually.
8. Infrastructure and Scaling Challenges
The infrastructure powering the real-time side of things is complex and requires careful scaling. WebSocket servers are difficult to maintain and scale, with many edge cases to consider. We use a state machine to handle WebSocket connections at Liveblocks, ensuring users don't have to worry about the connection. Collaborative documents, or rooms, are located close to users for optimal performance and horizontal scalability.
But then there's another obviously big part of this, which is pretty, pretty complex and that's something we spend a lot of time on is the infrastructure powering the real time side of things and making sure that all that stuff scales very nicely. And if you've worked on WebSocket servers before, you would know that maintaining them and scaling them in production is very difficult. They're really hard to scale, hard to manage, managing the connection can be can be challenging and there are a lot of edge cases.
And so this is just an example of our, this is a simplified state machine that we put together with Stately, which I highly recommend. It's a fantastic tool. And this is the state machine we use, simplified again, to handle WebSocket connections at Liveblocks. And there are a ton of edge cases to think about. For instance, you know, when you finish your day at work, you're going to close your laptop and then in the morning you open it back up again. You actually need to do to restart the WebSocket connection, you need to do a ping pong mechanism to check if the connection is still active. There are a lot of edge cases like this that were pretty hard to map out. And we used this state machine to create kind of all the edge cases and make sure that as a user using LiveBlocks, you don't have to worry about thinking about the WebSocket connection, we handle all that stuff.
And in order to scale this, you know, and not have a single point of failure, each collaborative document, we call them rooms, they live milliseconds away from people on the edge. And what's great about this is that it gives people an absurd performance, like, you know, in most cases you're 10 to 60, 80 milliseconds away from the server, meaning you can get really fast real-time performance. But not only that, it enables us to scale this horizontally, essentially, because there's not one single point of failure and we can have as many rooms as we want. It's just a central server on which people collaborate.
And that was it. I hope this talk was enjoyable. I hope you learned something, and yeah, thank you so much for listening. Cheers. Bye-bye. Bye-bye.