Visual Regression Under the Hood

Rate this content

Visual regression is one of the hardest part in UI testing. And you will likely agree that it is extremely powerful. But how it works? What the problem it is solving under the hood? Why people choose visual regression services and how we build the fastest visual regression tool in the world :)

9 min
15 Jun, 2021

Video Summary and Transcription

Today's Talk discusses the value and challenges of visual regression in UI testing. It highlights the importance of predictability in loading pages and choosing the right screenshot resolution. It also mentions the use of Visual Regression Services, Docker, and the Odiff library as solutions to improve stability and efficiency in visual regression testing.

Available in Español

1. Introduction to Visual Regression

Short description:

Hello, everybody. Today, I'll talk about visual regression and its value in UI testing. Let's dive into a simple example and see how visual regression can automatically detect changes in UI screenshots, helping us build a stable and reliable system.

Hello, everybody. I'm excited to be here today and to talk about visual regression. My name is Dmitry. I'm from Ukraine. Working full-time at and doing some work at the open-source community.

Let's start. Today, we've been talking a lot about UI testing, but you probably will agree that the hardest part of UI testing is to test how UI looks for users, right? Because computers don't know anything about UI. And that's where visual regression gives us a lot of value.

Let's roll over a simple example of visual regression and then dive into the process. So here is an example. A simple screenshot of the homepage. And here is the next screenshot. You probably spot the difference, right, because they're changing too quickly. But visual regression can do this automatically. You can see that there are two changes in between of these screenshots. And this is extremely helpful when we want to build a stable and reliable system. Yeah, it makes us confident about how our UI changes for real users.

2. Challenges and Solutions in Visual Regression

Short description:

But in my experience, visual regression is also an extremely flaky test category. Today, I'd like to dive into visual regression and discuss the hidden problems in each step. Loading the page is not enough; it needs to be predictable. Different UI can occur on different operating systems or browsers. Visual Regression Services and Docker can help achieve predictability. Choosing the right screenshot resolution is crucial, as it should match the resolutions used by users. The comparison of screenshots can be slow, but the Odiff library offers a faster solution. In conclusion, ensure consistent environments, avoid unstable content, and test UI over user-used resolutions.

But in my experience, visual regression is also an extremely flaky test category. You probably know this reason when each literally each second screenshot, each second commit, has some visual regression noise and like we're all are humans and we are getting used to this starting ignoring, auto approving and so on and so forth. And this is a problem because once it become flaky, it lost the value.

So today, I'd like to discard this problem by diving into visual regression, the hood and try to get this knowledge and use this to build more reliable visual regression. So under the hood of visual regression always contain four simple steps. Firstly, you need to load a page, then you need to make a screenshot, compare it with previous approved version and see the difference. Looks pretty easy, but each of these steps has its own hidden problems and I'd like to discuss it.

So first of all, you need to load the page, but that's not enough to just load the page using your favorite browser-based test runner like Cypress, right? You need to make this page predictable, and this is a problem especially when you are not using visual regression services, because when your page is not in the stable state you can easily get a lot of noise, like for example, here. Most of the screenshots have sections that are changing from time to time, like the inline videos or changing carousel by timeout, and everything, all of this, can easily broke the visual regression process. Also, animation, times, random values can easily break this, so we need to be careful about this, but that's not everything we should care about.

Also, the different UI is possible even when you are running the same code, but in different operation system or in different browsers. Just because the different layout systems or different, like, operation system itself, can produce layout shifts or different default view, so this will break our code. And this is a real problem, which is perfectly solved by Visual Regression Services, but it gives a lot of problem for people that are trying to make the Visual Regression by themselves. Visual Regression Services solves this by loading your HTML, and not the screenshot, but HTML, running this HTML with all the styles in the specified browser, and only there to make a screenshot and compare it. But you can get the same level of predictability by running all of your tasks, and only run your Visual Regression tasks in Docker. It can be even reasonable to make a specific separate amount of tasks only for Visual Regression and run it only in Docker, even unproof it in Docker. And this will make you confident that your tasks are running in the same environment, and does not give a lot of noise and layout shifts in between local machines of developers. But there's also an interesting middle between these two approaches. There's a project called Visual Regression Tracker that gives you an ability to run these tasks inside the Docker in the self-hosted service, gives you an interface that allows to approve the screenshot, and is giving you the same level of predictability as Visual Regression Services, but self-hosted. I'm sure this project will make future of Visual Regression.

But then you need to make a screenshot, right? But which one? And here is a problem, because I'm constantly seeing, especially in the Cypress community, that people are giving default Cypress resolution, or some small resolution that are honestly not used by nobody in the world, yeah? We need to ensure that we are testing our UI over that resolutions that are used by our users. You can easily get this information from any analytics tool. For example, here is the stats of my personal website, and you can see that most of my users are using this weird resolution of some tablets, and honestly, I'm not testing my website over this resolution, and you probably know how it can be easy to lose some visual effect when this resolution is not widely popular, or too big, for example, and you need to ensure that you are testing over this resolution that's used by your users, and that's actually weird that by default, visual application tools and services are not using the most popular resolutions, like for example, Full HD. And the reason of this that we are doing screenshot testing over small images is that the comparison of screenshots is really slow. In order to compare two images with a Full HD resolution, you need to iterate over 2 million pixels, calculate the difference between each one using a specialized formula, and only then save the difference. It's a pretty hard and not performance-friendly task for computers, especially when you're trying to do this in JavaScript. And that's why I created and I'm right now working on the library called Odiff that allows you to do the image comparison not in JavaScript but in native, more performance language and save you a lot of time and allow you to test the screenshots that you want and make it fast.

So we probably are out of time. So let's discuss a conclusion and a key to painless visual regression. You need to ensure that your tasks are running in the same environment and you need to ensure that you don't have any unstable content on your page even if you're using visual regression services. And you also need to test your UI over that resolution that are used by your users and not just fast or performance-friendly for some service. And that's it. I'm happy to be here.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.

Workshops on related topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Remix Conf Europe 2022Remix Conf Europe 2022
195 min
How to Solve Real-World Problems with Remix
Featured Workshop
- Errors? How to render and log your server and client errorsa - When to return errors vs throwb - Setup logging service like Sentry, LogRocket, and Bugsnag- Forms? How to validate and handle multi-page formsa - Use zod to validate form data in your actionb - Step through multi-page forms without losing data- Stuck? How to patch bugs or missing features in Remix so you can move ona - Use patch-package to quickly fix your Remix installb - Show tool for managing multiple patches and cherry-pick open PRs- Users? How to handle multi-tenant apps with Prismaa - Determine tenant by host or by userb - Multiple database or single database/multiple schemasc - Ensures tenant data always separate from others
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.