How I Test a Million UI States with Every Merge — Visual Testing with Storybook

What's up everybody? How you doing today? Okay, it's okay, it's okay. I need more coffee. Man, my flight over here. I'm working on like a couple hours of sleep, so I apologize if things are a little bit rough today. But super excited to be here, super excited to be hanging out with you. I'm a little bit worried about the timeline because like 20 minutes, I mean like I could talk about the subtle differences between number two pencils for 20 minutes. So I don't know how how much of this we'll be able to thoroughly cover, but I'm excited to get through as much of it as we possibly can. Now my goal today is just to convince you that just snapshots are not enough for what we're doing today, and that we need more robust tools for testing our UI. So we're gonna go through that. My name is Chan, Michael Chan, Chantastic, whatever you feel comfortable with. This is how you can find me online. Earlier this year I gave a talk called Taming the UI Multiverse, which was a little bit more of like a feely, squishy version of this talk. We're gonna take a little excerpt of that and then jump into a little bit more code. But if you want more like stuff that's just talking about the feeling of UI development, how hard it is, and some of the challenges that we feel but don't really talk about a lot of times, this is gonna be a good follow-up talk for you. So jumping right in to what we're talking about today, who here is familiar with the testing trophy in React? Okay, only a handful. Okay, okay, hands are slowly starting to go up, so I think that probably a good half of you. The idea behind the testing trophy is there's this kind of like shape of like how many tests we should write in any regard, and the biggest of those is in that integration slice of the pie. This is kind of a distribution of the types of tools that we use for the different types of testing, but integration is supposedly like the biggest piece of that. Now I propose that there's this big piece of integration tests that we haven't actually integrated into our development workflows, which is visual testing. So that's what we're gonna talk about today, and the tools that I've been using for this are Storybook with Chromatic. Those are the tools we're going to talk about. Now both of these tools kind of derive from this mindset of being component driven, and so what's the idea of component driven? Like how does it work with visual testing? Well the idea behind it is really boils down to like how much value can we extract from one component? We have this really cool isolated nugget of user interface. How can we, I guess to borrow a violent metaphor, like how many birds can we kill with this stone of a component? Sorry if you love birds. I love birds. I actually had a podcast for a little bit where we just like reviewed bird songs. A little different than a React podcast. So it's one of the things that we haven't, like so components have really become like every part of our life, but something we haven't really like explored a lot is like how we integrate components with browsers in isolation. Up to this point, in visual testing has often been bringing the full stack of our application into the browser, testing it, and that can be really hard to test, getting the environments right, getting all the personas right. Components allow you to just kind of like inject a state and go really fast. So I want to describe the problem a little bit and I call this the UI multiverse. It's a description of the multi-dimensional challenge that we have when we're building UIs. And yes, like all of us, I was watching a lot of Marvel at the time that I was thinking through this. I also call this the ten-ish dimensions of web UI or 35,000 perfect states. So every UI starts with the thing that everyone thinks that they're building, the very clear vision of what we're going to make. But we all know as modern web developers that any view is potentially three views, an error state, a loading state, and then the successful loaded state. And then it goes even further because like we have like different variants of those views. We could have a spinner, we could have a skeleton, the errors could be a 404, which is distinctly different from a 500. And then for our successful views, like these multiply like rabbits, right? Because maybe we have six breakpoints or so that we actually think about and care about. But like we could be supporting any number of breakpoints in reality. This gets a little bit more complicated multiplied by the number of browser engines that you have to support. Fortunately, these are getting better year over year, but there are still subtle differences. Now I would argue that the subtle differences aren't necessarily important to account for as long as you have a consistent experience inside of those browser engines. And then we have usabilities, right? So are you using like touch, are you using sight, are you using your ears? At least in America, the CDC suggests that at least 4.6% of Americans have blindness or low vision. Now it's really interesting because like I've spent a huge amount of time testing and reworking views in Internet Explorer to save our app for like one and a half, two percent of users. But we dedicate so little time to the 4.6 plus percent of users who aren't able to use our sites with their eyes. Now we also have like device capabilities. I'm not going to talk about that a lot, but it is part of the like dimensionality of UI development. And then these views get multiplied by all the complexity in our app. So it could just be like, you know, authorization, that's like one metric, but then you have like logic and we all know like how quickly props and state and the authorization of an application can really like balloon out of proportion. And if you have the privilege of having an app that's so popular that you need to localize it across different locales, well now you have at least two versions that you have to worry about. So just take all of those views and multiply them by two. And if you've ever had to do this, you'll realize like how poorly your apps are probably suited to like just make that switch because anytime you use like a margin right like for anything, like all of that just is something you got to fix now. And then docs. Docs are always a huge problem, right? Like we create these component libraries and we expect people to use them diligently, but we need to put in the work to make good documentation. So these are the real, the problems that we have to face. Now let's talk about my personal problems. I've been working for the last like 12 years in like design systems, component libraries types of work, and so this is what the challenges that I was facing in the last handful of years looked like when I came to these solutions. So for every view we would have six viewports, three browser engines, three device capabilities, and four user abilities. So that's 216 perfect states per view. So at scale let's say it's two authorization types like user types. It was actually like four or five, but I want to like keep these numbers as low as possible. We weren't doing right to left so we just have the one, so I'm gonna cut that. And we had two UI frameworks, so we had, it needed, our stuff needed to work in React and on Rails. Wait, React and yeah, I got that right the first time. So that's 864 perfect states per view. Okay, let's take this a little bit further. So now we're gonna talk about style. Style, we have color schemes. Color schemes are very important these days. People want their dark mode. We had two contrast preferences, so you could have it be a low contrast or more contrast, and we also supported 10 discrete applications. So we had this kind of like white label type of design system that could be themed for individual applications. If you multiply all those out you get this rough number of 34,560 ideal states per view, which is kind of nuts. I think that we look at this we kind of like we're accustomed to not thinking about this because if we did think about it too much we would just kind of not do the work. And I think that that's the approach that we've taken to testing a lot of this stuff. It's really easy to write unit tests for certain types of code and it's really easy to, you know, do all kinds of testing except visual testing across all of these various dimensions. Now just to like show you that I'm not like way out on left field, this is a Figma file. Someone wrote a blog post talking about all the, you know, documenting all the variants in Figma and they landed on just in these two color modes 1134 variants. This isn't multiple contrast support, we're not talking about viewports or like logic or authorization, so just buttons alone. So it adds up really fast. Now again we're not gonna have a ton of time today so I want to leave two links with you. We're gonna cover as much as we can and then I'm gonna leave you with these. If you go to chan.dev slash 1m that's gonna be the github repo that we're working from in the exact state that I have right now. And see is this counting up or down? Oh my gosh, down. Okay so and then youtube.com slash at storybook.js where I've been covering a lot of these topics in depth so anything that we covered today will also be available there. And you can see me doing some like really debasing poses as well, so enjoy. Okay let's jump into demos. Okay so we're going to look at storybook real quick. I'm sure many of you have seen storybook in its very like basic state. This is the storybook that gets generated for you when you run NPX storybook in it. I've added a couple libraries just so I don't have to NPM installed today but like this is really what you're looking at. Now at its base it's a component catalog and like many people when I first saw storybook I was like I don't need that I know how to put a component on a page like I'm gonna be fine. So I would just like spin up Gatsby dump out all my components on there or whatever. However if you go a little bit deeper you see there's actually a lot of really cool things that we start extracting from components. Remember our goal is to like get as many things out of a component as we can. So if we go if we jump over to this docs page we can get code examples which is pretty sweet. Copy and pasteable but then we also have this really cool thing where if you are adding type annotations using prop types or type scripts well we can actually just pull all of that out for you and give you a really nice really nice interface documentation for this component. But we can take that a little bit further as well. You shouldn't have to be a developer to be able to see how all of these things can it how all of these interfaces interact with each other. So since we know all of the type interfaces we create a playground for you by default. So I can say I actually don't want this to be a primary button I can kind of make this text really long if I want change the color we have all kinds of stuff in here that you can use. So really cool. So I'm gonna go back to Canvas that's available for all of these components if I click on this add-ons pane right here. Now another cool thing about stories is that I can just kind of inject certain state into this. So if my components are well-designed they're pure function of props well then I can just kind of throw in the data that I want and so that I'll render with the data that I want I could pass in objects all that kind of stuff. Now another cool thing is is that components aren't just a function of their props all the time. They're also a function of the interactions that can be done to them. So we also have answers for that as well. Let's look at this page component that we have that's a composition of all of the buttons and the header that we have right here. And we're gonna click on this logged in state. So we have logged out you'll see that there's you know the the login and setup button and we're going logged in and we see this logged in state. Well how did we get that? We actually have an interaction that we can run using testing library on top of these components which is really cool. That means that we can derive stories from the components themselves but also the interactions that we expect people to take on them. Now let's jump into code a little bit. I have never before done live coding in front of a crew so I've tried to do as much as I can to kind of make it go well but bear with me. So we have our storybook running. I have a bunch of examples if you decide to look at this repo later. I have a bunch of to-dos over here that you can jump to various parts. I want to show you just the ergonomics of a story first. We import the component that we want to document. We just export default a little bit of metadata, the title where we want that story to render, the component under test to see why that's important and then we can pass in kind of default arguments if we want. Args, think of these as props in React land. It's just kind of a disambiguated term for other libraries. Okay so what do we have? So these are what the test looks like. So we could just like write out JSX like this but we have this object format where we just say like hey we want to do a primary test we're going to assume that we're testing this component we're going to throw these args on there. We can run it without args. We can pass in whatever args we want and we actually compose args. So down here we can say hey I want to use all the arguments from the large and the primary button to compose this new story. So we'll just spread those out and then we have a new story. That's what stories look like. Super easy so far. Now let's go to the next thing right here. So one of the values of storybook is that we can capture these states that are kind of unideal. So up to this point we have this like Jane Doe story over here right and this is like a very like ideal name but if you've been doing web development for a really long time you know that like these these views are kind of sent with these ideal states not the extreme cases. So let's do an extreme case right here. We're going to do John Jacob Jingleheimer Schmidt so with a long name and I'm going to show you one more thing. We can do some really cool stuff. So I'm going to do another test right here or another UI state. We'll create a story using the long name using the args from the long name but we're also going to set a viewport. So one cool thing about this object syntax that we have is it allows us to do parameters of all kinds. So I don't have to kind of imperatively put this story into a certain state I can just say hey the viewport that I want is going to be this mobile one viewport. So did I save that? No. I'm gonna save it. I'm gonna get this new story and then here we go. So as I jump to that I can see that like well this isn't like exactly ideal and then we can kind of make some decisions that based on that but we're just effectively like parameterizing all of the things that we care about in this specific story. How much time we got? Two and a half minutes. Sweet. Okay we're gonna go a little bit long so we've you know yeah. Okay so cool so we have that so these stories are really cool. Now let's jump into play functions real quick. So we saw those really cool interactions. I want to jump over here into play functions. Now how do we start doing those interactions? Well this is what that test looks like. So we have again we're gonna go through this we have a page so we're importing that whole page. That page is the subject of all of these stories and we're just actually here let me show you one more thing I have done just a little bit this is all this is all just like regular code you can write your react in here. I have a page and this is just kind of passing in those functions and I've done a very very small little like mock where I have a tiny component that I insert that you know catches these actions and then kind of returns a resulting state so I just didn't want to skip over that. Really easy to just kind of like make the component put the components in the states that you want. Now from there we just use testing library to actually like interact with this story. So we take our canvas we grab that log in button so we just use our our queries to grab that and then we just say use event click on that button. Super easy. Now we'd actually compose these stories as well so we saw a composition of like all of the arguments and parameters but we can also test these compose these stories so I can say hey everything from the logged in play function I want to run that but then I also want to do this other thing so I'm gonna write a test for log in and then log out. Okay super cool we have let's see where is it log in and then log in and then log out. Now this looks on the surface to be the same as the logged out but you can see that we've run both of our interactions over it to get into that state. Now we also run just assertions in here so I'm going to just import expect storybook jest and then write some assertions here to make sure that it is in fact getting into those states. Did I save? Yep I did. Okay so it goes we click the log in button we assert that Jane Doe is there and then we click the log out button and then assert that the log in button is there again. Pretty neat pretty neat pretty neat. Okay let's see and then check this out this is super cool so I installed the storybook test runner library and we can just run yarn test all right test storybook and that's gonna run tests ingest using headless chromium for all of our and just running all these tests. Now I want to show you something really cool now this goes back to our like dimensions of UI we really haven't had a really good way to like test accessibility. I want to show you something really cool that we're doing with the test runner. So I'm gonna jump in here and say I'm gonna make a couple of changes to the test runner. Now I'm not a super good coder so I literally everything you're seeing here I just copy and pasted. So this is on our blog we augment the or we give some configuration to the test runner to say that we want to run the axe playwright library on during that phase. So I'm gonna run let's see yarn test storybook again it's gonna run all of our tests oh shoot what I do and then save there we go again not a good coder. Okay we're gonna run that again and we're gonna get a few errors. Now I already know what these errors are but you can see that it ran axe playwright on all of our stories and now we have the ability to say like okay so we're not hitting contrast on a couple of things and then make some fixes. So these are all CSS super easy I'm gonna make some changes real quick that color was a little low we're gonna change some contrast for all of these run our tests again. Oh I got a reload I got a reload because I changed CSS there we go. Run these again. Oh what I do did I not I probably didn't save a file again. Remove a comment. Oh thank you. Yeah mob programming this is yeah now we're now we're getting into it. Okay cool now we have it now all this has passed super awesome. Oh yeah yeah let's do it. Now this is really cool because this protects the user experience of something that if you're a visually oriented user of the web it protects the experience of something that you don't always like get to see or like kind of is not always in your mind because it's this thing that's kind of like invisible to you but is the way that people interact with the web a lot of the time. So that was super easy like just wildly easy and now we have the benefit of all that stuff. So that is okay so that's all of those things I want to show you one last thing check this out we can even do this and I said in the I had that slide up in the beginning burn all snapshots just snapshots with fire. Well this is probably like the one I think acceptable use case of snapshots which is to generate check this out I'm generating snapshots for the what is it the accessibility tree and now we can know for certain if something that we changed actually resulted in a regression in the accessibility tree. So super super super super super cool okay now for this part there's a lot of NPM installing and this will probably be the last thing that we covered today and so I wanted to fast-forward through that but I wanted to show you how you can get visual regression testing in just less than two minutes. So I'm gonna run through this real quick we got a video so first of all we take this library this is the one we're looking at today we're gonna put it on github we go to chromatic.com sign in with an account there we go add a project find our project on github super easy so far we're gonna add our chromatic library to our git repository quick and easy we're gonna run this script npx chromatic with our project token it's gonna take longer than this to build but you know then we see our build online there it is so we can go to our project and see all of the things that it did for us so it has it shows all these snapshots and then also it shows these stories in the post interacted state as well as DOM snapshots as well so we can know when something just in the DOM changed we get a free hosted storybook right out of the box and we can test across multiple browsers so that's that thing that I was talking about we want to keep things consistent across browsers since we have a little bit more time I'm gonna show you how to install github action again all copy and paste here I ain't trying to write any code so we copy and paste that we're gonna throw that into our github action paste create a secret for our github action settings secrets action again copy paste all the way easy easy we add that secret and then commit and send it go over to github actions and then we watch these things go the first one failed that's my fault but this one's good and we go and see our build so now every time we push something it's gonna run all these tests for us we have an army of robots just testing across all of our browsers all of our viewports like everything super easy now I think I'm pretty sure that I'm well over so I think that we're just gonna leave it at that I want you to go check out this rebook because there are some really cool additional steps in the what is it I think we have what is it 50 14 through 17 some really cool stuff I'm talking about integrating with chromatic testing your themes all that kind of stuff so anyway thank you so much for your time today I hope that you learned something new about storybook and chromatic and start visually testing your apps thanks for having me thank you thank you Michael yeah why don't you step into a chamber yeah you indeed did go a little bit over which is a shame because I've been looking forward to interviewing you how much over did I go on about seven minutes we only have three minutes oh dang it that's all right here but you are now going to be safe from you know from all the questions I'm all the questions yes sorry well we have the live Q&A right here this is for you oh thank you so much yeah with your third hand do you want me to oh shoot oh no I'm okay yeah go ahead no thanks all right let's get into the questions I'm not sure do we see the questions yes I think the most upvoted question is what are the most common mistakes we should avoid when using storybook I know this is a big question but what is the one big thing that people tend to screw up oh man that's really that's a hard question I don't know if I have a really good answer for that I think like something that that I think is really important is like finding ways to not write a story for every exact state so one of the examples that we didn't get to was I'm testing themes and so it's really easy to say like oh I want you know I'm gonna have this story and I want to present it in this theme and that theme in this contrast mode but those kind of balloon out of proportion right and so now you have like seven stories to just see the various themes one of the examples that we didn't get to is setting setting all of that as a global parameter you can make new like controls or add-ons that you put in the top bar really easily and now you can just like change those parameters like via the like top bar and then in your testing environment in chromatic you can just say like hey just use this snapshot which is like a grid of themes and so now you just kind of like you don't have to think about themes of the story it's something you can apply to any story so I think finding strategies like that where you can keep the stories nice and simple but then use the robots to do all the work for you is the dream yeah I think that was kind of like the question there as well is how does it how does storybook help you check thousands of states and I think that's kind of answered yeah they're in yeah exactly cuz like I don't want to write stories for all my for every single viewport I just want to know that when I push it like that nothing changed and so yeah cool all right let's do one more question this is the most uploaded one with this testing library just integration should we now all do all our unit tests directly on storybook is that way the way we're going oh well so the goal of this like the test runner is is that you can integrate it in whichever way you like feels best to you so if you have you know so if you have a team that's you know a lot more like engineering and they're comfortable in that just environment well now you can bring some of those into the just environment and use that however like maybe your line is a little bit different you want to do a lot more stuff on the visual side put a lot of it on on chromatic and then just have some of these things where you're like again like testing the accessibility tree for regressions like that kind of stuff I'm run that and just but like I I as a person tend to like stay away from like hard lines or like recommendations and just say like hey whatever supports the team that you have and like their comfort like you should go towards that yeah I think that's that's a good note to finish on we have so many questions for you I think this might be the most question oh well well then I'm sorry it's so much time but there will be the live Q&A so if you do want to ask any questions from Michael you can do it on the next break in the speaker Q&A room yeah unfortunately now we need to thank you Michael for your time. Thank you for having me.