Solve 100% Of Your Errors: How to Root Cause Issues Faster With Session Replay

But anyway, my name is Ryan Albrecht. I work at Century as a lead front-end engineer on our session replay product. And so this talk or workshop is about how to solve 100% of your errors and root cause your issues faster. Basically it's an introduction. But if there's one thing you learn today, it's that there's a new tool available called Replay. There's a bunch of companies that make the same kind of tool, but ours is one of the only ones that is sort of developer focused and something that you can add to your toolbox in order to actually solve problems faster. So that's the whole gist of the talk. If you get anything out of this, it's that this thing exists and I hope you look into it more.

What I want to do is tell you about it and get you excited. So what we're going to talk about is the problem, which is just bugs, and then describe what is a replay. And then I want to show you actually how it works for real. So there will be a live demo. Not going to be so much code, but because there was a question, I want to show you guys the documentation. So we'll just put that up here. If you want to set this up on your website, you can go to the docs and there are links in the discord, I believe. And all you have to do is copy-paste this kind of a thing. So this is probably the only code that we need to see. And the rest is just using the tool and getting some value out of it. So the docs are there, links are there, you guys can ask more questions later. But I just want to show this so you have some context.

Okay, so what is the problem? The problem is bugs. You will never solve all of your bugs, but they do exist and we have to solve the most important ones. So this is a bug, it's a real bug that I think I solved. I probably solved it. But just looking at it the way it is, it's pretty impossible to actually do anything with this. We can see at the top it says S printf expecting number but found array. So this is and this is the only useful part in this full page here. It's a minified JavaScript stack trace. Which means it's a snapshot in time where something happened, something unexpected happened. And the JavaScript threw an error.

And this is sort of like where we're starting from when we when we think about bugs in JavaScript and think about what do we need to do to solve them? This is this is the basics, and it's basically impossible. Sometimes you say okay, that's great, we have a stack trace, wouldn't it be great if it was a ticket? Somebody could put it into our our JIRA or something like that. If they do that, you might get like an unobfuscated stack trace. So here's one that's a little bit more readable. We can see that there's like error boundary.tsx is here, message formatter.tsx. So that makes this stack trace a little bit easier to read. But it's actually still very, very difficult to solve this bug. We're expecting a number, we found an array, it's happening inside message formatter. It happened once anyway. It could have happened many times. I don't know what the array was. I don't know where it came from. I could put an if statement in there and just be like, if is array, stop doing what you're doing. That'll prevent this bug from happening. But I don't know what the user experience is going to be. It's not great. This is where we're coming from.

And what I'm trying to say is it's not great. So this is the story of debugging, the classic game where you're the detective, the victim and also the killer or the murderer. Because I wrote this code, and now I'm here thinking about what could possibly be going wrong.

So this is where Sentry comes in. So a little plug for Sentry. Sentry is great because it shows you not just what we were looking at, but a lot more context to help you figure out what this bug is and whether or not it's important. Try to get you moving along to solve it.

So we still have that same message. Expecting number, but found an array. It's not the most important. But Sentry has gone and looked at my GitHub repo and it says it's probably this commit because the commit appeared to touch those lines. It might have appeared recently. Related to when the, like, recently, just before the first instance of this error happened. You can see here in the last 30 days, it started up about a week ago, or two weeks ago. It's first seen 13 days ago. All this context as well as the stack trace helps you to debug these problems.

So if you think about the progression of solving bugs, we've gone from this terrible stack trace that tells you basically one piece of information about one statement that was executed in a moment in time. And we've been able to update that and upgrade that and have all this other extra information. So this is pretty cool. And actually, it's really useful and it does help solve a lot of bugs. But what if we take this single snapshot and think about, OK, what's next? So stack traces are still a snapshot in time. This error happened once. Wouldn't it be cool if we could see a bunch of errors over time? What if we were able to see, like, not just that this happened at this moment, but what was leading up to it? Kind of like a video. So there's a lot of steps that the user took. And it'd be nice if we could know what those were in order to reproduce this case and then see the error happen ourselves. Because it's not enough just to fix the problem just by throwing in that if statement. We've actually got, we should actually be reproducing it so that we understand why it happened, what the data was going into it, and maybe there's an upstream fix or a user experience that could just be better overall.

So this is part two now. What is a replay? A replay is not just a video like I alluded to, but it's actually a way to do time travel debugging, a way to think about the bug in context of what did the user do leading into a bug and what happened after a bug was triggered. So here is the session replay page in Sentry.

There's a lot of other products like this and they all have the same fundamental parts. This is one I'm very familiar with because I had a chance to work on it. We're going to see it live, but before we do that I want to walk through five insanely cool features and just point some things out to you here.

So there's a video player here. But it's not just a video like you might see on YouTube, this is the actual DOM of your website. So session replay helps you debug bugs that are happening on your website by showing you the HTML and letting you inspect and basically look over the shoulder of your customers who are having problems.

You've got access to full console logs. So if you were debugging something on your computer, you would open up the inspector and you would see your own console and maybe give yourself some debugging hints. When it's a remote computer you can't do that unless you have something like session replay capturing those DOM messages and showing them back to you here. So we can see that in this sample application, there's a bunch of console logs, some type errors, which we probably need to be debugging, and some like just error messages that cannot read properties of undefined that maybe we added ourselves.

Network request and response. So this is a very cool feature where you're able to see, again, just like in the dev tools, what was happening on the network. So we can see that there was a get request or a bunch of get requests that all returned that's great. And this one was a fetch, and we can see here what were the results. So in this app, this is my Pokemon app, which we'll look at in a little bit, it looks like it's loading all the Pokemon here, 1,200 of them. And so everything's working great. Replays are cool, and finding errors and fixing errors is great, but you might also want to find out interesting replays based on what the user was doing. So there's a cool feature we have to be able to search for replays based on the DOM nodes that were clicked. You can look for DOM nodes based on the labels that they have, the classes, the IDs, any of these, I think, nine selectors. And that's just an easy way to zoom in and say, okay, we've got some replays, we've got some users taking some actions, I want to know like who's opening this modal? Who's clicking save? Who's clicking buy now? And what are the errors or what are the behaviors that people are doing around those features? And finally, the last thing is that session replay, even though it's a UI focused feature, recording that DOM, it also can link backend errors and transactions. So these are things that might happen in your Python server, it can link them to the front end. And so you can see, hey, my database query had a problem and that returned to 500. What did the user see when that 500 showed up? Was it handled properly or was it, did it just kill the page? So these are some of the features that we have for session replay.

And so what I want to talk about next is how it works under the hood. So session replay is not just a video, it's actually a recording of the HTML DOM on your website. And so when people encounter an error, we can have that, use that recording to say, to show you this is what the user was looking at on the website, this is what was clicked on, and here's when the error happened. And if you're familiar at all with keywords like virtual DOM, or like React has been very popular with their use of virtual DOM and a lot of other frameworks, the implementation of this is quite similar. What we do is if we have an HTML page like this that's loading in, right now there's just the main div and h1 and a paragraph. What we can do is serialize this entire page into JSON when the page loads up and when the replay SDK loads up. And so, all this JSON when we first tried to do it, we thought this is going to be really expensive and really bad.

But it turns out it's actually not that much content. This is slightly simplified. It's missing things like styles and stuff like that. But it's really not a lot of data. And then the magic of this is when the page changes, so something else loads into the page or maybe the user clicks on a menu, or you've got a single page application and a lot of things get re-rendered. Whatever triggers something new to be added to the page like this, all we have to do is using that virtual DOM and the diffing algorithm is create new JSON for the new HTML content. And we can use the parent IDs and all the IDs that we have within our virtual DOM implementation to just have the new, whether it's inserted or removed nodes, saved on our list. And then we can play it back as if it was a series of commits where we have our initial JSON, and then all of these mutations applied upon it. This is all done with the mutation observer, which is widely supported in all kinds of browsers. And it turns out it's really, really efficient. Maybe even more efficient because we can press it on the front end.

So that's sort of a quick, quick overview of some of the things I want to show. But this is a workshop, so we're going to do live demos. I think it's way more fun. If there's any questions you can ask now or at any time. But first, I'm going to show you, if there are none, I'm going to show you my Pokemon app, which is going to be what we're going to base the demo on.

Let's just reload this page here. So this is my Pokemon app. It's super popular, as you probably expect. Pokemon is the most popular thing in the world right now. The app is pretty basic right now, but it's getting more and more complicated every day. Some of the features you have is, obviously, there's a list of Pokemon here, which is what we're looking at. So you can scroll through, and you can load more and find your favorite Pokemons. When you click on one, let's try this Sandshrew. You can see all of his data. He's got some moves here, different angles, a lot of fun. And then of course, catching it. The main reason we're all here, to catch them all. So of course, it's success. I've set it to be like about 90%. So this is Pokemon number seven. What is he? He's the Shrewd Shrew. Good enough. So once we catch a Pokemon, we can go to our list of Pokemon and see all of the things that we've caught. This is pretty much the basics of the app. We've also got a search feature. So we can see here, Pikachu, very popular. So we search for him, load him up, we can catch Pikachu as well. Hopefully this works. Been practicing my throwing arm. Yeah.

So this guy, it should be Detective Pikachu. And so while I'm browsing around the site with session replay enabled, what I can do is actually see replays of all the users as they're happening. So if I switch over here to my Sentry instance, go to the right spot. We can see here the replay that I just started two minutes ago of myself catching a bunch of Pokemon. And so the first thing you notice is that everything is starred out. This is a privacy feature because a lot of websites might have PII, very important in Europe and around the world. We don't want to have ingested. We don't want to know what people's credit card numbers are, what their emails are, what their addresses are, anything like that. So by default, Sentry has a privacy first stance, and we replace all text. All text with stars. But let's keep watching here. So remember, we started looking at the page, list page. Oh, there we go. And now you can see actually the user scrolling through the page. This was me a minute ago, looking at more Pokemon and then navigating to our Santry and catching him. So the images, everything is all blocked out for privacy reasons. But we do give controls to turn those back on again. So here the Santry was caught. The name is getting typed in. With all my spelling mistakes. There we go. And on and on. So this is the essence of a session replay. If you take nothing else from this, from this talk, this workshop, session replay is here. It's a tool that you guys can use. You can see what is actually happening on the screen when users are using your site. When there are errors, use it to debug.

So how do we debug things? Well, we've got the HTML here, as I talked about. And I'm going to show you that this is the real HTML of the real webpage that I was looking at. So there's a table you can sort of see in the background here.

Everything has been started out on the client side, so nothing is saved to the server here. And where's our nicknames? I'm looking for the image. Everything is here, the second column, all the moves. It's all real HTML with real CSS attached to it. So if we got rid of paddings, the site actually changes.

The console is also here, so this is a great feature for debugging what's going on. You can leave yourself notes. So I can see here a throw was made, some random value was generated, which is good enough to be a catch. And then the new Pokemon was saved.

We also got the network tab, which I mentioned earlier. So here early on in our replay we can see that all the Pokemon were loaded. So this is just like in the screenshot, but I'm able to dig in and look at exactly what was returned in this response. So this replay doesn't have any errors in it. So it's not, there's really nothing to bring us in, and nothing for us to do.

So what we can do actually is go back and see if we can find a bug or trigger a bug in our app, and I happen to know of one. So let's look for Pikachu again, but this time we'll spell him wrong. So you'll notice in the URL that I searched for Pikachu and I got something and I already caught a Pikachu. So if I try to catch him again, everything seems like it's gonna work fine. Oh, it failed. Let's try. I have a question here in the chat. Session replays are added with Sentry default setup, or is there something I need to enable it? So it does need to be enabled. But it is part of you can enable it with any Sentry plan that you have. If I have the documentation again. So in order to enable it you do need to add the session replay integration. So that's down here, the default. If you have Sentry set up for errors, you would already start with this kind of thing, just the Sentry dot init call. So enabling replay requires this new integration and these are just sample configurations, you don't have to set anything else up if you don't want to. Oh, and Discord. I see as well.

If we use self-hosted, we need to evaluate the amount of resources the Sentry server will capture, how much more memory disk space should we expect to need if we use a feature. So, the memory and disk space, the disk space for sure depends on your retention policies and how much load you generate. But we found for our own uses that each replay is actually pretty small. I think that some of our like our P75 averages are in the single megabyte range for the, for the size of the replay itself. The replay can be up to an hour long. And we have seen like some really exceptional cases where there's like a very complicated website, and people are using it for hours and hours and hours and it's it's taking up more than a couple of megabytes. But it depends on the sample rates and the type of site you have, for sure.

Memory wise, it's very efficient on the server, because we're just ingesting all the data and saving it out to disk basically right away. Another related question. Documentation says this feature is using a distributed system to send the replays. But not be the case with self with the self hosted solution. Yeah, so I don't know exactly what part of the documentation that's talking about. But it's distributed in that... What could be distributed? Any user on your website, who opens the browser, they're going to send the data directly to Sentry, and that'll they'll go through different point of presences we have all around the world. So maybe that's the distributed part. But then it all gets funneled into and it's asynchronously saved into the, the database. If there's any privacy blocking. If there's ever a problem with the SDK. It won't break your website it should turn itself off. Just like with the base SDK. Here we go performance overhead. Thank you for that. What's distributed here. Oh yes, so it, it does send it around the world to our distributor point of presence. So these are. These are basically servers that are around the world that are entryways into our internal network. That way you have a lower latency is between your clients or your clients and our system. I hope that answers the question.

Okay, I have demo back to Pokemon. So, we've saved the Pokemon.

Everything looks fine. Now if we go back to my list. There's a problem. This is a white screen, white screen of death. If we look at the inspector here, we've triggered an error, cannot read properties of undefined this is terrible cannot read properties, reading map, reading length. Some other stuff here front default.

So, because I'm doing this on my own machine I can see here in the console what what's going on and have a little bit of a hint. But if it was happening to a customer's machine, you wouldn't be able to see this console messages. Let's go to our replays, close that and see if we've captured this error. And indeed we have. So the newest error here is was started 10 minutes ago, it's a nine minute long replay which makes a lot of sense. And there's three errors inside of it. So now when we load up the page. We've got some errors to debug here.

And so there's, we can see in the top right there's three hours, and they're all react errors, in this case. So, if I'm looking for these errors, one great way that I can do it is to start at the top and use the timeline component. This is one of my favorite ways to get an overview of what's in this replay. And so we can see at the start of the replay that I was talking and clicking through there's about a minute and a half minute and 45 seconds of different click events, and it tells me what was clicked on, and the green dots here are navigations. And then after I was done talking, we triggered some errors. And so we can use this timeline, and these errors here to, or the use a timeline to jump right into these errors and see at five minutes and 25 seconds. And we can see what was on the page. When this error happened. And so here it is here's the first error. In the bottom of the breadcrumbs we can see again cannot read properties of undefined reading front default. So that was the first one in our console logs. It's the first one in our console on the right hand side here. And so this is what we have to debug. And we also see in the timeline that there were other errors cannot read properties of undefined reading map. So we can jump there to the similar but not quite the same error. And this is happening where the catch Pokemon model has appeared.

So how might we go about debugging this. Well, we know that reading properties of undefined. It's probably a statement like undefined dot where it's probably a statement like some variable dot map, and the variable is undefined. So we can't access a map property on this undefined. It's the same problem as our front default. But front default happened first. So this is a big clue when we're going to debug these errors. If we explore around a little more.

Oh, another question. Even though the values in the DOM are hidden for privacy concerns, the network call responses are displayed. Wouldn't that be a privacy breach, if we are certain that we get calls returning data about a user after logging in, for example.

Yes. So for this example in this demo I've turned off some of the privacy that scrubbing that we do for the requests and responses and request response data itself is an opt in feature. So there's a little message here that you can capture more headers. But we're not talking about headers, we're talking about the bodies themselves. So if we go to the back to the documentation I can show you a bit. So configuration. Allowing network details is fully opt in so you can pick which URLs you want to allow my demo I've added them all. But no matter what you allow all of the bodies are going to go through our regular PII stripping. So if you have any fields called password or Social Security number or credit card number or any, anything like that, any of those kind of keywords that we they do get filtered out and that's not optional. So capture network bodies is opt in. And those will get scrubbed. So, in our case the pokemon names and things like that are not going to be scrubbed because those are not exactly sensitive. And you can also set custom filters for this kind of thing too, which is the same system that we have for normal data ingestion. So the configuration for that is here, allowing URLs capturing bodies, and also capturing request headers and response headers. So sometimes these can be sensitive, but sometimes you might want to track trace headers distributed trace headers or headers related to like different proxies and things like that so all of this is opt in, in a privacy aware way. Where's my page. Okay, so going back, we're gonna debug some errors and use the replace to help us narrowing on it. So, we've got our air properties of undefined front default. One thing that we can do that we've always been able to do with Century is to see the issue itself.

Oops. I think the zoom is not helping here. Okay. We can see the issue itself and all of the useful error debugging that we've always had. So this is only happened twice in the last couple days, that's because I'm just doing this now. We've got our stack trace here and I know that that's not going to be helpful in this case. And the breadcrumbs that we've had before. So in this case, the big difference between the session replay. And the issues that we've been able to debug in the past is that the breadcrumbs from the past of only ever showed us what happens before the issue happened. So we're able to see here that like a bunch of clicks happen some console messages were printed out going back in time even further. There was the fetch call with the 404, so that's kind of standing out. We know that that happened because I intentionally misspelled the name of the Pokemon. But all of this information is just leading up to the error. When we look at the replay itself. We can find out a little bit more information. So let's jump back to there. Or let's look at the second error. Because usually when one error happens, the second one happens or more errors can happen afterwards. So the replay in addition to the breadcrumbs that we've had before it also shows us what happens next. And so just by playing it from that error onwards, I can see that the white screen has happened. And this is a serious error to fix.

What other tools do we have here. We have a DOM events tab. So this is a wonderful tool to figure out, at a glance, instead of looking through the breadcrumbs but like what has the user clicked on and in what order. So if I'm trying to reproduce a bug here, I can use the DOM events to follow the user and all of the clicks that they've made. So in this case, a lot of it's been stripped out. But if we click on one of these DOM events and highlight it, we can see where in the page the user clicked. So even with the privacy enabled, we can tell the user clicked here and some heading, or the body tag. They clicked here on an nav item, and because the nav item is static I can guess what it said. But by hovering over this, and going right to that moment in time. I can see where on the page this lives and if I'm following along in my own browser, I can click it to follow what the user did and reproduce this bug.

We already looked at the issues tab. Memory is not going to be so useful in this case, but every once in a while I find that if you have a long lived single page application, memory and a memory leak is happening. I'm going to show you and could be the source of maybe that one page load that just crashed for no reason. It's because memory was exhausted. I finally tracing, so I don't have any traces in this demo. But if I did control the back end that we could see all of the API requests and the databases requests that were made, afterwards, but because I'm using just a generic Pokemon API today. There's nothing here. So, it's a little overview of the errors and things.

Now we actually have to go and solve this bug. So we've talked about or I've mentioned the bug is an undefined value. We're trying to read a property. It's happening more than once, so this one we're trying to read a property called front defaults. Later on we're trying to read map, and we're trying to read length. So, let's jump into the first property. So what was the reason here? So we have some hints. We know that Pikachu is an incorrectly spelled name. If we go to our network requests and responses here, we can scroll and find this point in time in the network table. So here is about five and a half minutes. And we can see there was a 404, so this is a good hint that something has gone wrong. And clicking on it, there's no response body.

The request was for that Pikachu data set, and nothing came back. So this is a really good reason why there would be an undefined getting entered into our system. And yes, we could have seen it by looking back at our back at our issue. Find our issue here. But the real value in the replay is seeing that not only did this was this error triggered. But it triggered three other errors on the page. So these errors are all related, because once we had that one undefined on the page. The next two errors were bound to happen just by clicking around.

And it was really strange to that this error. When it did happen, when I was looking through just now. Nothing looks broken on the page, we found our some kind of a Pikachu. We saw the image I was able to catch it. And we can even see in the replay that the user was able to catch it. When we're talking about solving problems it's not always enough just to say, Okay, there was a 404, let's throw our if statement in there and just do something so that it, it's okay. Sometimes the errors, the second ones and third ones here are because of that earlier step, where we don't always know it's related.

And so if we were starting on starting by looking at this first error, this secondary here on the page. We might not have any context would be like okay something's undefined. We're doing a catch operation, how could the Pokemon be undefined if we're trying to catch it like it's, it's here the pages here, but with the context of a replay. We're able to say, Oh, something else broke first. And then that's what caused the second error which is very similar to the first one, and then even the third error, which happened on a completely different page. So this is the power of session replays.

We've got our network error here which is a pretty much a smoking gun in this case. And that was easy to find out for the first error, even when we're looking on our error issues details page, but for the other related errors. It would almost be impossible to find. And so, yeah, this is the the power of the session replay tool. And if there are more questions I'm happy to answer them. But yeah, I think, I think this is pretty much it. You guys want to talk about it. I'm happy to talk about it. Aware about client device performance.

Specifically how does Sentry know whether to record a replay or not. Want to put load on the phones for nothing, even if no bugs happened. This is a good question. So this is about configuring replays and whether and the performance impact of having a replay on the page. So if we go back to the configuration here, there are two ways to configure a replay.

So one of them, I think is the, the default way that people think about it, and it's called the session sample rate. And so when you configure a session sample rate. Basically what you're doing is you're setting up a percentage of whether or not the replay will start when the page load starts. And so in the demo that's what I had set up. So that way we could see the user loading the page, all of the things that they do, whether or not an error has happened.

The other way to set up a session replay is to have the on error sample rate enabled. So what this does is it starts to record the page but it doesn't send any data. It keeps a buffer of all the activity that the user was doing, and when an error happens or if an error happens, because it's not guaranteed. But when an error appears, it'll take that buffer, which is about a minute of data, and it'll start to save it and start to record the session from that point on. So if you're clicking around for 20 minutes, we're going to have a buffer that's just a minute on average of data. If the error happens we're going to turn on the replay and so that first error and all the other errors afterwards are going to be caught and tracked. So you get a little bit of context, leading up to the first error, and then you can definitely see what happens after the error. And then, you know if there's a second error or third error, and so on.

In terms of performance. Both of these are. It depends on the website I would say, but we've noticed for centuries website. And a lot of our customers that the load is not so bad. It's especially not so bad if you have a desktop application for mobile phones. It's going to be a little more dependent on like where your target market is, because you're going to be thinking about memory CPU and also upload. But some of the things we do to mitigate this are these problems are compressing the data on the phone so that upload is a lot, a lot lighter. This saves people's phone plans, upload is also generally a lot slower than download. So compression on the phone. Really, really saves a lot of a lot of space. We do compression in order to maintain the performance of your web page. We do that compression in a web worker.

And so that happens off the main thread and it doesn't impact the rendering of your page. Also, for a lot of websites that operate like sentry or I think like a lot of websites on the page, or on the internet that are not games. Because the DOM is changing after in response to a user action and then maybe a network response. Usually what was what we'll see is that there'll be like some user interaction like a scroll, or like clicking on a modal or an X like I just did. And then nothing happens on the page. And so I've noticed, and I, and with our customers as well, is that a lot of times, the compression and the work from the SDK is happening while the users reading something, not while the pages being interacted with. For games and things where there's like maybe a lot of confetti on the page a lot of things are moving around, it might be a little bit worse. And those are things that you can disable tracking of because they're not actually content.

Another question in Discord. So even though the values in the DOM are hidden for private oh I've already read that. Yeah, so performance is a big concern for us. We don't want it to be slow on people's pages because then you're just going to turn it off. It needs to be useful and non impactful, but like any kind of monitoring solution or a profiling solution, there is a small overhead. Including the JavaScript bytes that you have to download. So we're going to switch back to the slides, because there's a couple more. Oh, there's a question. Replay is included in the Sentry slash browser package. So even if we don't use replay, it will be included in my sources. Um, so there's a few different ways to include the Sentry browser package. If you're using NPM and a build system, then, and you don't use the replay integration, then it is very likely that it will be eliminated from the bundle. So, tree shaking will work for the Sentry bundle. Yeah, tree shaking. But if you're using CDN, or if you're using other ways to install it. Not really. But on the CDN there are different versions of the CDN bundles, so you would want to make sure that you're using, using the right version of the CDN bundle that has the packages that you want. The things to look out for is the core browser bundle. There's a the replay integration and then also tracing integration. So you can pick which combination of those that you want. We also have a new configuration. That's much more automatic, it's called our loader.

And so with this, you'll get a custom script tag that you can install just paste onto your screen and you can control whether replays are enabled or not. In your sentry organization so it's not like a custom CDN you don't have to worry about what the URL is. And then when you load the page in your on load you would enable the integration or not, which is optional I think you can, you would just change the default settings here. And that's what the loader configuration.

Okay, so like I said a couple times, replay is sort of a new product category that I don't think a lot of people have heard about, and we're not the only ones building it, but we are making it different. And I think a lot of the early versions of this were sort of geared for PMs, product managers, people who want to know, like, how people are using their products, and I think that they've sort of geared their other products have sort of geared their, their designs around, making people sit down and watch videos, like hours on end. So, at Century, everything that we do we think about developers first, and so this Century Replay product is made for developers. That means that we try to be very tactical and specific about saying, here's an error, here's what you need to know to jump in and see just that error, little bit of context on either side, but then get on with the job of fixing it. Because remember we are the victims and the murderers here. And so fixing that our errors is the most important thing. Privacy first is also very important to us. So that's why by default, everything has privacy enabled, and you can roll it back slowly. As, as you feel comfortable or depending on your needs. The two ways to sample this was a good question and so thank you for that. But sampling on error versus generic sampling. This again is another way that we try to be developer centric. If you just want to do generic sampling and see a lot of examples, then that's one way to go about it. But I think for a lot of developers, we want to know, we want to know what we need to know and so sampling on error is the best way to get the most bang for your buck and see replays that are actionable and require your attention. It's also embedded into your century workflow. So I showed in the demo a little bit going back and forth between replays and issues. So if you're already looking at a list of issues and a list of problems, you can jump into a replay that shows an example of how that problem came about, and what happened afterwards. And the setup like we've talked about takes just minutes, there's different snippets you can copy and paste depending on whether you have a build system or if you're using NPM, or if you just want to use the CDN and the loader, you can literally just copy and paste one line, and you'll be capturing replays, and you'll be able to see them live. Like we did. So there's a lot more docs here and places you can get some help. Some of these should be in the discord channel. But the docs are this is the docs that we've looked at. There's also marketing pages and places on GitHub, Twitter and discord where you can reach out and get help from me and people on my team. All of this is open source and so if you find us on GitHub and you find a problem, we're going to work together in open source and fix it. So that's the talk, that's the workshop. I hope you guys learned a little bit and get curious about Session Replay. There's a lot more ways you can use it other than fixing errors. I've seen people install it in their CI and run it against their integration tests. So they might have had a flaky integration test that runs headless and with Session Replay they can see it. The sky's the limit. It's a tool, and I hope you guys get some value out of it.