Your Tests Lack Vision: Adding Eyes to your Automation Framework

Hello, everyone. Today's talk is about visual testing. But first let's start off with a little game of whodunnit.

Clearly somebody in this room murdered Lord Smythe, who at precisely 3.34 this afternoon was brutally bludgeoned to death with a blunt instrument. I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place.

I was polishing the brass in the master bedroom. I was buttering His Lordship's scones below stairs, sir. What, I was planting my petunias in the potting shed. Constable, arrest Lady Smythe. But how did you know? Madam, as any horticulturist will tell you, one does not plant petunias until May is out. Take her away. That's right, madam. It's just a matter of observation. The real question is how observant were you.

Clearly somebody in this room murdered Lord Smythe. Who, at precisely 3.34 this afternoon, was brutally bludgeoned to death with a blunt instrument. I want each of you to tell me your whereabouts at precisely the time that this dastardly deed took place. I was polishing the brass. In the master bedroom. I was buttering his lordship's scones below stairs and so. But I was planting my petunias in the potting shed. Constable, arrest Lady Smythe. Did you all happen to see any of the differences? Maybe the biggest one where the dead man got up and walked away? Probably not. And that's because we all suffer from what's called inintentional blindness. Inintentional blindness is where something new and unexpected enters your view, and you miss it because you're so focused on something else. And this is especially true when we talk about testing our software. We're focused on whatever feature it is that we've written and that's the one that we want to work, and we're missing everything that's going on around it. So how do we typically compensate for this? We automate the test. Because if we automate the test, we know that all of our steps will be recorded, and all of our verifications will be verified, and we can guarantee that those things will happen every single time. But what about the unexpected things, right? Sorry to tell you, but when you automate your test, you're actually scripting in the unintentional blindness, and you miss other things that may be happening in your application, such as the look and feel, right? Using the traditional automation tools, we automate those tests.

It's looking in the DOM, and that's where it's interrogating to give us the answer to our questions. For example, is this text present on this screen? It looks in the DOM. The text is present. It tells us, yes, the test passes. But what about if that text is the same color as the background and our users can't see it? What if it's bleeding off the edge of the page, right? What if it's covered by some other element, and therefore our users can't read it, or interact with it, if it's a button or something?

These are visual bugs. And when I talk about visual bugs and visual testing to solve for visual bugs, lots of times people will say, oh, that's interesting. You know, and think of it as this nice-to-have, but let me tell you, visual bugs live in everyone's applications, big or small. Let me give you a couple of examples. Here is Cineworld. Cineworld is the second largest cinema chain on the planet. And they had a visual bug on the page that asks you if you want to store your credit card information. This looks a little funny. Ha ha, we can laugh about it, right? These labels aren't aligned with the buttons, but let me ask you another question. Would you store your credit card information on this page? I wouldn't. Why not? Well, just as Kim has alluded in this tweet, it looks like they haven't tested their application. And if you look at this, you think, oh boy, if they miss this, who knows what else they miss, right, on the back end with my credit card. So no, of course not, I'm not gonna store my credit card information on this screen. But I have a theory. I think they did test this. In fact, I think they automated the test for this. If they automated the test for this, then with querying the DOM, these labels are there, the radio buttons are there. It's not a problem to interact with them. But from a user perspective, this is really bad, right? You lose trust in the application.

Here's another one, OpenTable. OpenTable is used to make restaurant reservations. I used to live in North Carolina a couple of years ago and I went back to visit and I wanted to catch up with some of my friends. I went to make a nice size reservation at one of my favorite barbecue places. North Carolina's known for barbecue. The barbecue place was The Pit and I was making a reservation for seven people. They gave me some times to choose from and I selected the 7 pm slot.

When I selected this slot, a modal appeared and it took me a minute to figure out, what am I supposed to do here because there's nothing in the center, right? As I got to looking around a bit, I noticed that over here in this corner, I can see a couple of select buttons, but this made me even more confused. What am I selecting? Why are there two of them that are not aligned and there's nothing here telling me what I'm selecting? So I did what any of you would do. I opened Chrome DevTools and I began to dig around the Dom to try to figure out what was going on. And when I did that, I realized that the labels actually were there. They were just as far away from the buttons as possible. So me being who I am, I did complete my reservation and I got to have barbecue, but I couldn't help but think about the many, many customers that are not engineers that have no idea what Chrome DevTools are. They're not looking in the dom to make a reservation on your application. No, they're closing this out. And if they're a millennial like me, they surely aren't gonna give you a phone call, they move on to another restaurant, right? So think about how much business was lost because of this visual bug.

These are what else. This happens to your favorite tech companies as well. Like Amazon, on their mobile view, someone not only wanted to buy something, they wanted to increase the quantity, meaning multiply the amount of money that they were wanting to spend in their face with this visual bug. This is Facebook marketplace specifically. So you go to the marketplace, you're trying to shop for something, look at the text over here and how it's just kind of going down the side of the page and overlapping on the images. This is very hard to read. This doesn't make me wanna spend money, right? But if I queried the Dom and asked if that text was present, it'd say, yep, it sure is. This is how this kind of stuff makes it to production.

Instagram, this is one of my favorite ones because this is sponsored content. Meaning someone paid Instagram for premium placement of their advertisement and this is what they got instead. All of the text is jumbled on top of each other, right? Again, querying the Dom, is this text present? Is this text present? Is this text present? Yes, yes, yes, ship it, right? Twitter, same thing. Overlapping text, automated test wouldn't be able to catch that. Google, even Google, my goodness. This one is not as bad as the others, but hey, it is a point of purchase. And so some highly paid Google engineer has to stop innovating new features to go and fix this. Fortunately, we don't have to live like this, people.

Visual validation can be automated. So what is visual validation exactly, or visual testing? Visual testing is where when you write your very first test for a feature, you know, the first time, and you verify that yes, this looks exactly how I want it to look. At that moment, we can take a screenshot of your application. And then for every regression run, we run another screenshot, take another one and compare the two. If there are any differences, your test fails. So this catches everything and it eliminates the unintentional blindness, right? But be careful. Visual testing is not something that's new. In fact, it's been around in the industry for a while, but the technique that many vendors were using was not ideal for testing, right? So the way that most of the visual testing solutions will compare the images is pixel to pixel. Well, that's extremely flaky, let me show you. So here is Google and let's say that very first screenshot is when our test is perfect, our application is perfect. And so that's the baseline screenshot. In the second screenshot, that's a regression run, we compare it in the regression test fields. Who can see why? Yep, that Google search button is bolded in the second one. Well, when your cursor is over that button, that's the hover effect.

So this is an automated test, maybe running as part of CI or something like that. You really didn't have control over where the cursor was happened to have landed and therefore now your deployment is blocked, not what you want.

Here's another one, who can see why this one failed? Yep, because of that cursor behind the word bad. Cursors blink. So if it happens to take the screenshot when the cursor is solid and then happens to take a regression screenshot when it's not, that can cause a failure as well.

You don't want your deployments blocked for this and having to investigate why that happened, nor do you wanna write automation code that tries to stop this sort of thing, right? Fortunately, Applitools has come out with a better approach to doing this using machine learning. So it mimics the human eye and brain and is able to highlight and detect only what matters in the test.

So I have another game here. You all don't get to play because you let the dead man walk in the first game. But in this one, I ran this through pixel to pixel and I ran it through Applitools machine learning algorithm to just see like, what's the difference and how does it detect. And this is the one with the pixel to pixel, notice everything is highlighted here, like every little white shift change, it's not met for testing. Versus when I ran it through the Applitools algorithm, it highlighted the things that I would have highlighted as a human being, right?

So okay, let's add this to a test, an existing test. So here's our application in the test, single page app, very simple, eight of my favorite books on test automation. You enter some text into that field and that will filter down the books, so the ones that have tests in the title, all right? So here's the test to do this. This is written in Cypress with JavaScript, but Applitools has SDKs for all of the different automation tools and all of the different programming languages. So if you use something different, good for you, but this is JSConf, right?

So, okay, so this is Cypress JavaScript. We are querying the word test. We are expecting these five books to come back. We enter that into the field, and then we have two validations here. One validates that the number of books that are visible is the same as what we expect, and each of those titles that's in the expected books array actually is visible, all right?

Great, so now I get a little wonky with the CSS and I want to add a bouncy effect, but I don't really know what I'm doing, but that's okay because I have test, which means I can do whatever I want. So I add line six here to this CSS and I check it in, right? My test will let me know if anything's wrong. Again, here's my test, look at it carefully. And this is what the application looks like with that new CSS. Will my test catch this? Let's see. I run this with Cypress again, notice all the green here. There's no red, even though I have literally flipped my application upside down. That's problem. That's why we need visual testing.

So how do we add it? Very simple. Three lines of code, almost like poetry, right? Starting on line three, we use the Applitude's eyes API and it integrates so nicely with Cypress.

So look here, becomes a side command. We say, open your eyes. We give it the name of our application. The next line, this is the magic, check the window, this takes the screenshot. It sends it out to the Applitude's cloud. Applitude stores all of your images on their cloud so you don't have to worry about disk space or anything like that. Final line, close your eyes. We're done here, folks.

So we run this test and now, notice the test does fail because it detects, girl, you flipped your books upside down, right? And it highlights all of the differences there for me, okay? So this is great for removing unintentional blindness, right, but we could even scope this down. What if we only wanted one book? For example, look at line two. I'm just gonna search for agile testing, knowing that there's only one book that comes back. And so I don't wanna test the entire page because there could be a lot going on in this page that I don't really care about and I wanna scope it down to the book. To scope it down, I use that same check window command, but I pass in a target, which saying, I wanna target a specific region and I give it the CSS selector for that region, okay? So that's the ID. And then notice the screenshot will only capture this one book, but when we scope it down like this, we have to be very careful because we could reintroduce the inintentional blindness. For example, let's say that this filter is broken. I enter the word test here and none of the books go away. Would our visual test pass or fail? If you said that it would pass, you are correct. Because remember, we only had a picture of that one book. We said, hey, look for that one book. Is it visually perfect? It sure is. It passes. So we have to still use our brain. Even though we're using machine learning in our testing, we are still in control. So I can mix and match my assertions. It doesn't have to be all visual. I can add that functional one back in, right? So look at line three. I'm saying, make sure that there's only one book visible. That's the functional assertion. And once that's the case, I can then say, now, do the visual test. So that makes it really, really powerful.

All right? I can also do dynamic content here. So you wouldn't think you could do dynamic, verify dynamic content with visual testing, but you can with Apple tools, right? So this is a match level. There's several different match levels. So this makes it very verbose and flexible. But this layout is saying, I want you to use machine learning in a different way to just detect the layout of my page.

So for example, this first screenshot, this is the books in one order. The second screenshot is the books in another order. Because I said to verify this using the layout algorithm, it's just going to detect the patterns on the page. Very popular for like news sites. When I used to work at Twitter, we used that match level for the tweets. We don't know what they're gonna say, but we wanna make sure that structurally, the page is okay, things aren't overlapping or bleeding off the edge, right? So even though those books are in different order, this still passes.

And then the final thing I wanna show you is the ability to run this across multiple platforms. So Applitudes has this ultra fast grid that works beautifully with eyes. And what this does is allow you to run your test across any browser or device that you want to simultaneously. And the way that it does this is it grabs the state of your application and then blasts it across all of those devices. So you don't need the device phone, you don't have to deal with that at all. You don't even have to change your test. You simply add line, starting on line four, we have an array that basically says, here's what I want. I want Firefox common IE in these dimensions. I want iPhone 10 in landscape and portrait mode. I want Galaxy. And this all runs lightning fast simultaneously. Pretty neat. And so that's what that would look like on the grid. Notice here, bunch of tests run here, all the different devices. Very, very, very cool. Also works for components. So if you have Storybook for Angular, Vue, React, this can verify your components and make sure your date pickers, your field, your buttons, everything remains visually perfect as the rest of your teams are gonna use these things. All right? So there you have it. That is visual testing in a nutshell.

What questions can I answer for you?

Hello, Alex. How are you?

I'm good. I'm trying to survive. I'm trying to survive. How are you?

Yeah, I'm good. I'm good. Hanging in there.

Cool. We've got quite a few questions. I'm gonna probably butcher the names of the people who asked the question. So Yuval wants to know, should I have visual regression on all my pages or just some critical ones? If so, which ones are critical? That's a good question. So you should have visual testing on your stable-type pages, right? So if you have an application that is undergoing a lot of maintenance, for example, this specific page, you're constantly adding new components to it and it's not quite stable just yet, that might not be a good candidate for visual testing. Now, you could use the API in some creative ways and say, kind of, ignore this active section of the page or verify it as like dynamic content that's going to change or whatever. So you could do it, but my recommendation is to use it on your pages that are most stable.

Okay, that's really good, that's really good. They have a follow up question that says, how do I minimize false positive? Is there a way to flag changes as intentional beforehand? Oh, that's good. So I would say it's not a beforehand kind of thing, but it could be before it's like, actually part of your CI or something like that, right? So I could run something locally. Let's say I develop, I change a feature, right? That changes the website. And so I know that the baseline is going to change. So I can run that locally. And then when it fails, I can just do a thumbs up. That's all it takes, a thumbs up or thumbs down to say intentional or not. And I can say, this is an intentional change, check that in. And then when it runs as part of the bigger suite, it'll be okay.

Okay, that makes a lot of sense. That makes a lot of sense. Manuel wants to know, how do you visually test the page that loads dynamic content, like a changing list of users, for example? Right, so you can do this on the dashboard. So Abilitudes has a dashboard that allow you to annotate the image. So you can mix and match the different match levels, right? So I talked about the dynamic match level. I talked about the default one, which was just kind of strict mode.

You can mix and match. You can say, for example, like that list, this part is dynamic, everything else, I want you to verify regularly. Some people use this a lot for like dates, timestamps, those sorts of things that are constantly changing on the site so you can annotate it. And you can do that on the dashboard or you can do it programmatically by giving it like the CSS selector of that element so I can say list of users, whatever that idea or whatever it is, I can specify that programmatically so I don't have to go back and annotate it on the dashboard.

A great way that I use that is with mobile testing. So on the top of your mobile is the status bar, right? With your notifications and your time, everything up there is dynamic. So programmatically I say always ignore that specific status bar element and so when I write new mobile test, it already ignores that part. Oh, that's actually quite interesting. That's quite interesting. Thank you.

How does Appy tools ensure that the test results are not flaky? Yeah, so it uses machine learning which is different than the pixel-to-pixel. So we have like a 99.5% accuracy and because it's using machine learning, it gets better and better, you know, over time as it learns more about the things that are kind of considered flaky or not.

Okay, okay. So we've got a lot of questions around Appy tools. I am gonna continue with that because it's something new for me at least. I haven't used it before and I wanna know more. Considering the capabilities and benefits of visual testing, is there still a benefit to snapshot testing or would incorporating Appy tools renders those obsolete?

Right, so if you're thinking of snapshot testing, like I know a couple of ways to do that. One is like capturing the DOM or whatever, the entire state of the application and then comparing that. So you have these textual type of changes, right? That in my opinion is a little bit more fragile than the picture one, right? Because I might change the underlying code without changing the visual aspects of the page, right? So it just leads to more maintenance to do this snapshot testing but that's still a viable option.

Okay, okay. That makes sense, that makes sense. I've got a question that I think is kind of related so I'm gonna jump to that. How about accessibility tests? Is there any possibility to automate them?

Yeah, so Apple IIs also has accessibility features and this is great. Like it's a great way to use visual testing outside of the box. So it has this contrast tester. So it's looking to make sure that all of your elements are adhering to like the AA or you know, AA guidelines or whatever, as far as the accessibility. So that's something you can turn on or turn off.

Okay, okay, that makes a lot of sense. That makes a lot of sense. One more question, it's a quick one and then we can, we can let you go to your speaker's room. So do you have an on-premise option for Applitools server?

Yes, so the on-premise option, of course that was more costly. There's a free plan for everyone. So everyone can use this for free. If you need like more features, more, you know, capabilities you can upgrade, but then the on-premise option is more, but yes, it is available in our customers that are like banks, healthcare. They use the on-premise option.

Okay, okay. Thank you. You've had so many questions, Angie. It's really, really hard for me to ask you all of those. So for the people which I've not asked the questions, I'm really sorry, but Angie has a speaker's room on spatial.chat. The link to join is on your timeline and you can join Angie in the room and ask her all the questions that I haven't asked. So thank you, Angie, for joining us today. It was really, really good to see your talk and get the chance to talk to you a little bit. Thank you so much. Sure thing. Bye, Alec.