Visual regression is one of the hardest part in UI testing. And you will likely agree that it is extremely powerful. But how it works? What the problem it is solving under the hood? Why people choose visual regression services and how we build the fastest visual regression tool in the world :)
Visual Regression Under the Hood
AI Generated Video Summary
Today's Talk discusses the value and challenges of visual regression in UI testing. It highlights the importance of predictability in loading pages and choosing the right screenshot resolution. It also mentions the use of Visual Regression Services, Docker, and the Odiff library as solutions to improve stability and efficiency in visual regression testing.
1. Introduction to Visual Regression
Hello, everybody. Today, I'll talk about visual regression and its value in UI testing. Let's dive into a simple example and see how visual regression can automatically detect changes in UI screenshots, helping us build a stable and reliable system.
Hello, everybody. I'm excited to be here today and to talk about visual regression. My name is Dmitry. I'm from Ukraine. Working full-time at Cypress.io and doing some work at the open-source community.
Let's start. Today, we've been talking a lot about UI testing, but you probably will agree that the hardest part of UI testing is to test how UI looks for users, right? Because computers don't know anything about UI. And that's where visual regression gives us a lot of value.
Let's roll over a simple example of visual regression and then dive into the process. So here is an example. A simple screenshot of the Cypress.io homepage. And here is the next screenshot. You probably spot the difference, right, because they're changing too quickly. But visual regression can do this automatically. You can see that there are two changes in between of these screenshots. And this is extremely helpful when we want to build a stable and reliable system. Yeah, it makes us confident about how our UI changes for real users.
2. Challenges and Solutions in Visual Regression
But in my experience, visual regression is also an extremely flaky test category. Today, I'd like to dive into visual regression and discuss the hidden problems in each step. Loading the page is not enough; it needs to be predictable. Different UI can occur on different operating systems or browsers. Visual Regression Services and Docker can help achieve predictability. Choosing the right screenshot resolution is crucial, as it should match the resolutions used by users. The comparison of screenshots can be slow, but the Odiff library offers a faster solution. In conclusion, ensure consistent environments, avoid unstable content, and test UI over user-used resolutions.
But in my experience, visual regression is also an extremely flaky test category. You probably know this reason when each literally each second screenshot, each second commit, has some visual regression noise and like we're all are humans and we are getting used to this starting ignoring, auto approving and so on and so forth. And this is a problem because once it become flaky, it lost the value.
So today, I'd like to discard this problem by diving into visual regression, the hood and try to get this knowledge and use this to build more reliable visual regression. So under the hood of visual regression always contain four simple steps. Firstly, you need to load a page, then you need to make a screenshot, compare it with previous approved version and see the difference. Looks pretty easy, but each of these steps has its own hidden problems and I'd like to discuss it.
So first of all, you need to load the page, but that's not enough to just load the page using your favorite browser-based test runner like Cypress, right? You need to make this page predictable, and this is a problem especially when you are not using visual regression services, because when your page is not in the stable state you can easily get a lot of noise, like for example, here. Most of the screenshots have sections that are changing from time to time, like the inline videos or changing carousel by timeout, and everything, all of this, can easily broke the visual regression process. Also, animation, times, random values can easily break this, so we need to be careful about this, but that's not everything we should care about.
Also, the different UI is possible even when you are running the same code, but in different operation system or in different browsers. Just because the different layout systems or different, like, operation system itself, can produce layout shifts or different default view, so this will break our code. And this is a real problem, which is perfectly solved by Visual Regression Services, but it gives a lot of problem for people that are trying to make the Visual Regression by themselves. Visual Regression Services solves this by loading your HTML, and not the screenshot, but HTML, running this HTML with all the styles in the specified browser, and only there to make a screenshot and compare it. But you can get the same level of predictability by running all of your tasks, and only run your Visual Regression tasks in Docker. It can be even reasonable to make a specific separate amount of tasks only for Visual Regression and run it only in Docker, even unproof it in Docker. And this will make you confident that your tasks are running in the same environment, and does not give a lot of noise and layout shifts in between local machines of developers. But there's also an interesting middle between these two approaches. There's a project called Visual Regression Tracker that gives you an ability to run these tasks inside the Docker in the self-hosted service, gives you an interface that allows to approve the screenshot, and is giving you the same level of predictability as Visual Regression Services, but self-hosted. I'm sure this project will make future of Visual Regression.
So we probably are out of time. So let's discuss a conclusion and a key to painless visual regression. You need to ensure that your tasks are running in the same environment and you need to ensure that you don't have any unstable content on your page even if you're using visual regression services. And you also need to test your UI over that resolution that are used by your users and not just fast or performance-friendly for some service. And that's it. I'm happy to be here.