Visual Regression with Puppeteer, Playwright and Cypress

Rate this content

Visual Regression tests components via screenshot matching. I'll show how you do that in three different libraries/frameworks. Additionally, I will use Storybook to extract the components from your SPA choice.

9 min
19 Nov, 2021

Video Summary and Transcription

Hello, I'm Rainer Haneckamp, a trainer and consultant at Angular Architects. In this talk, we'll explore visual regression testing using tools like Puppeteer, Playwright, and Cypress. We'll learn how to use Storybook and Puppeteer with Jest for visual regression testing. We'll also see how Jest and Playwright can be used together for visual regression testing. Finally, we'll discover how to use Cypress for visual regression testing. Thank you for watching!

Available in Español

1. Introduction to Visual Regression Testing

Short description:

Hello, I'm Rainer Haneckamp, a trainer and consultant at Angular Architects. In this talk, we'll explore visual regression, which involves creating screenshots and comparing them to reference images. Visual regression is crucial for all types of applications, as it impacts the first impression. I'll demonstrate using Puppeteer, Playwright, and Cypress.

Hello, and thank you very much for joining my talk. My name is Rainer Haneckamp. I am from Austria, and I am a trainer and consultant at Angular Architects. So this means I spend most of my time with Angular.

This talk is about visual regression. So this means that we will create automatically screenshots from our application, from our components, and we will match them against existing reference images. This is very similar to what we find with just snapshots, but whereas the snapshots are based on strings, visual regression uses real images.

And as I said before, we are doing a lot of workshops, and every time that I'm introducing this topic, there are two different types of developers. The first type says, well, that's great. I always wanted to see how this works. This is exactly what I needed. And it is very most likely that they are working on UI libraries or that they have applications. There a lot of time is invested into a very attractive design.

The majority though says, well, I'm writing business application or applications, and most of my time I'm using form elements. I'm using grids, and I don't create them by myself. I use already an existing framework like Material Bootstrap or whatever. Why do I have to care about visual regression? If you are sharing this opinion, then just think what happens when you're entering a website and you see that the buttons are overloaded, overflowing, or that the text is overflowing or something like that. Would you really trust this website? Would you really be ready to pay for some of their services? Very likely not. Why? Because the first impression was already a very bad one. And as everywhere in life, first, the first impression really counts. And as it is in applications, the visual part is always the first impression. So don't underestimate that, regardless what type of application you have.

And this is how we are going to do it. I will show you how to do visual regression by using three different libraries, frameworks. I will start with just Puppeteer, then Playwright, and then Cypress. Let's start. Okay, so here we are. This is our application, and what we want to test is this holiday card here. The first action is already that we don't create a screenshot of this URL, because it also contains the side menu here on the left side, the header. And if something changes there, then the test would throw an error, although the holiday card still looks nice.

2. Using Storybook and Puppeteer with Jest

Short description:

We use Storybook to extract components and show them in an isolated way. Puppeteer allows remote control of a Chromium-based browser for taking screenshots, combined with Jest. The Jest Puppeteer preset configures most things, and we have two important dependencies: Jest Puppeteer and ImageSnapshot.

And because of that, we are using Storybook. Storybook is a library that allows us to extract components out of our application and to present or show them in an isolated way. And this is something that I have already done. So this is the Storybook instance. I can click through the different variations, and that's what I'm going to use for visual regression.

Let's take a quick look into our code, how this can be done. So Storybook requires a default configuration. In my case, this is Angular. I need to define all the dependencies, and then I just create the so-called stories, which are the different variations. And for that I'm used here, a self-written factory method.

So we'll start with Puppeteer. Puppeteer is a library that allows you to remotely control a Chromium-based browser. This also allows you to make screenshots, and that's what we're going to combine here with Jest. You see here that I have already a Jest configuration that has a Jest Puppeteer preset. So everything is already, or most things are already configured by this preset. And in our package JSON we have these two important dependencies. The one is of course Jest Puppeteer along with Puppeteer itself. And the second is ImageSnapshot which is able to do the matching of the two images. And that's everything that we need to do except to write the test.

3. Visual Regression Testing with Jest and PlayWrite

Short description:

I have prepared a parameterized test using the global page object to open a URL and navigate to my storybook instance. I take screenshots and match them using the ImageSnapshot matcher. When running the test, Jest creates new screenshots if none exist. By changing the CSS, we can compare the reference and current screenshots to identify differences. PlayWrite is similar to Puppeteer and comes with its own testing library. It supports visual regression and allows configuration for multiple browsers. PlayWrite considers cases without a reference image as failures, while subsequent runs only fail if there are real differences.

I have already prepared the test here. This is how it would look like. This is a parameterized test because I am trying all the variations that I have. I am using the global page object to open a URL, navigate to my storybook instance. And once this is done I am creating a screenshot from what I see or what Jest sees and then I just call the matcher to match ImageSnapshot.

Let's run the test and we see that since no screenshots are there yet Jest is creating new ones and we find here this ImageSnapshot directory. If we open it we see the screenshots. Now let's try to change something and verify how it looks. So I am in my CSS of my holiday card and I'm changing it from font weight to normal so from bold to normal. We will see if our test will show us the difference now. We see that a lot of tests failed. Let's verify how we can see this. We find in the ImageSnapshots a further sub directory a div output and that shows us on the left side the reference, on the right side the current one, and in the middle we see the difference.

So the next thing is already PlayWrite. Very quick here. What do we require for PlayWrite or what is it? PlayWrite is very similar to Puppeteer. It can be seen as a rewrite done by Microsoft. It comes with its own testing library and it also supports visual regression out of the box. So we see here the dependency. There's just one package. We have a configuration here. We can define multiple browsers. We also see here WebKit. So this is the test for Safari and then we have the actual test. And as we see here, it looks more or less very similar to what we have seen before with Jest. Now let's run the test here as well. In contrast to Jest or Puppeteer, PlayWrite counts cases where we don't have a reference image yet as a failure. So that's why we see here everything red. When we run it the next time, it will only fail if there is a real difference. And of course, the images are also there.

4. Using Cypress for Visual Regression Testing

Short description:

To use Cypress for visual regression testing, add the Cypress plugin for snapshots, update the CypressJSON configuration, and run the test. Open the directory, import the snapshot in the support and plugins directories, open the Storybook instance, match the snapshot from the document, and check the image snapshots directory for the result. Thank you for watching and stay tuned for the Q&A.

Just need to open this directory and you see here that it's everything is there. And now to Cypress. Cypress is a very popular end-to-end framework. What we require? Cypress, of course, but also the Cypress plugin for snapshots. We need to add that snapshot in our support directory in the index, just import it. Also do the same in the plugins directory itself. You see it here. And you also want to update the CypressJSON itself. There you need to add the configuration.

What's left? Well, running the test, of course. How does it look like? We're opening the Storybook instance. We're then saying, well, I have the document and I want to match the snapshot from it. Let's try it out. The test is now running. We see the screenshot. We see everything's fine. Let's check for the snapshot itself. It's part here of the image snapshots directory. No surprise. Looks very familiar. And that's it.

So I hope that my talk was useful. Thank you very much, of course, for watching. My source code is available as well. It will be posted afterwards and I'm already looking forward to the Q&A. See you then.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.

Workshops on related topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.