Automated accessibility testing with jest-axe and Lighthouse CI

Rate this content
Bookmark

Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.

85 min
22 Nov, 2021

AI Generated Video Summary

This workshop covers automated accessibility testing, including linting, Just Axe, and Cypress Audit. Manual testing is essential for ensuring accessibility. The workshop explores the limitations of linting and introduces Just Axe to check for accessibility issues. Cypress Audit is used to detect contrast issues that code-based tests can't find. Lighthouse thresholds are set for accessibility, and a Lighthouse report is created to identify and fix issues. The workshop also addresses troubleshooting and setting background and foreground colors for maximum contrast.

1. Introduction to Automated Accessibility Testing

Short description:

This workshop will cover automated accessibility testing. Accessibility is about making your app or website usable for people with diverse abilities. Examples include providing alternate text for images and keyboard alternatives. The W3 Consortium and MDN have guidelines on accessibility. The abbreviation 'a11y' stands for accessibility, with 'A' representing accessibility, 11 letters in between, and 'Y' at the end. Automated testing is not a replacement for manual testing, as it only catches about a third of issues. Manual testing, especially with screen readers, is essential for ensuring accessibility.

So, this is going to be a workshop on automated accessibility testing, and I'll talk a little bit more about that in my introductory notes here. First, just a little bit about me. So, my name is Bonnie Shulkin, and I've been in the software industry since 2001. So, I've done a lot in the software industry. My current role is I am a full-time developer and trainer. I'm self-employed. You can find me at bonnie.dev, which also is on the lower right of all of the slides in case you forget that. Twitter, I'm at bonnie.dev, and GitHub, I am very proud to be at Bonnie. If you go to bonnie.dev and you go to the Talks tab, you can download these slides, which I would recommend doing. There's some links that it will be easier to click from the slides than it will be to actually type in.

So, Lars, thank you. I'm just looking at the chat. If I'm looking over this way, it means I'm looking at my other monitor, just seeing what's going on, and I saw that the chat came in. So, yeah, applications available to all citizens and organizations in Denmark and accessibility is a concern. So, yeah, automating quality checks is good. Let's talk a little bit more about what accessibility is. So, accessibility is making your app, your website usable for people with a diverse range of hearing, movement detection, sight, and cognitive ability. So, some examples of making a website more accessible are providing alternate text for images. So, if somebody is using a screen reader to view the screen, then they are going to be able to hear a description of what the image is. Also, keyboard alternatives, because some people are not able to use the mouse in precise ways. And using the keyboard is more – is more accessible. So, being able to have keyboard alternatives to any mouse clicks or going from section to section of your website. And I've added a couple of resources here. The W3 Consortium has a page on accessibility. And they have a lot of guidelines, as well. And then MDN, whose documentation I really like, they also have a page on accessibility. Thank you very much for posting the link to the slides in the chat. So you may have seen this a11y abbreviation. Does anybody know what it's an abbre... Well, I mean, it's an abbreviation for accessibility. Does anybody know where it comes from? Yeah, some people do. You can unmute yourself. Again, we're a pretty small group. So feel free to chime in. So it abbreviates accessibility. And, yeah. So it's an abbreviation for accessibility. And the A comes from the A in accessibility. Then there are 11 letters in between. And then there's a Y. So that's where the A, and then 11, and then Y comes in. And you can pronounce it A-11-Y. Or it's also sometimes pronounced ally. Now, as far as automated accessibility testing goes, and that's the subject of this workshop today, the goal is to make the manual testing less tedious. But it doesn't replace manual testing. So there was a study. The study found that only roughly a third of issues are actually found by automated testing. So the automated testing will catch things that will make it easier for your manual testers so they don't have to look for all of these things that can be caught by automated testing. But it's not meant to replace manual testing. So if you really want to make sure that your site is accessible to screen readers for example, you're going to want to have somebody test the site using a screen reader.

2. React App and Accessibility Issues

Short description:

In this workshop, we have a React app built with Vite. There are two versions: one with just HTML and CSS and another using the Chakra UI design system. The app has accessibility issues, such as missing alternate text for images and low contrast. Please clone or fork the repo and run npm install. Let's look at the app in Chrome, where the issues are more visible. We'll also check it in Safari to avoid screen sharing problems.

All right, so in this workshop, we have an app and it's a React app. But the tools that I'm going to introduce are they can work for other frameworks too. The React app is built with Vite, which is an alternative, it's sort of a lightweight alternative to create React app. And really, the app is just a very simple HTML and CSS about page for this fake healthcare company. So there are two versions, there's one that's just HTML and CSS and there's another which uses the design system, Chakra UI. And I used the design system or I have two different versions of this for a reason. There are some things that you can catch when it's just really bare bones, HTML, CSS, that's a lot harder to catch when you're using design systems, as we'll see. This is the GitHub repo. If somebody can copy and paste that into the chat as well, that would be lovely. If you haven't already, thank you very much. If you haven't already, please clone or fork this repo and run npm install, excuse me, and HTML, CSS, and Chakra UI. I'm going to just take a look at that. I'm going to get out of slide mode for a moment. I'm just going to go to a terminal. Here I am in the repo. You can see that I have Chakra UI directory, an HTML, CSS directory, and then a directory where you can copy and paste stuff from files, or we'll be moving files from this directory just to reduce the amount of typing that you have to do. If I go into HTML, CSS for example, it's a React app so the code is in source. Let me actually open up VS Code here. Here I am, and I want to look at the package.json. I have a bunch of scripts here. These came with byte, the top three came with byte and then the bottom three I added. But I'm going to be running this using npm run dev. Now it's running at local host 3000. And just go to Chrome. And if we look at localhost 3000, you can see that page and the background color is, I know why it's so terrible. It's because I have this dark mode on here. So it has some problems and some of the problems you can't really see. For example, this image doesn't have an alternate text tag and then this one, this contrast is not very high. So people who have difficulty seeing contrast might have trouble reading this. So that's another issue. There's also an issue around this label for region and I've sprinkled a couple of other issues in here as well. So that's going to be the backdrop app. Now, I'm going to control C out of this and I'll go into Shocker UI. This one is very similar, and if you look at the package JSON, you've got the same scripts. It's got some different installs because it's using the Shocker UI design system. But I can run this one as well. I can look here, and if I refresh, it's going to look pretty similar. There's a little bit of difference in the responsive this and you can see that we have some difference here. Actually, interestingly, in the dark mode that I'm using here, the contrast actually looks on this one, whereas it doesn't in light mode. Let me turn off the dark reader for a moment. Actually, you know what I think I'm going to do, is let's look at our sites in Safari instead. The Zoom screen share is getting in the way. This is what the site looks like. You can see that contrast is very poor. Any questions about the apps that we're going to be working with?

3. Automated Testing: Linting and Rule Details

Short description:

I'm going to talk about three different ways we can do automated testing here. The first way is linting. Linting is checking code while you're writing it using eslint. By running 'npm run lint', you can catch errors related to alt attributes, ARIA roles, and form labels. However, linting doesn't catch contrast issues. The linting tool used here is ESlint Plugin JSX Ally, which is included with the Airbnb ESlint plugin. You can find rule details on the ESLint plugin page for JSA 11Y. Fixing the errors will improve accessibility.

I'm going to talk about three different ways we can do automated testing here. The first way, automated accessibility testing, and the first way is linting. So, linting is checking while you're writing code, or I'm using eslint, so you can do this with an eslint command.

And you might have noticed that I had an npm script for linting, it was npm run lint. Now, if I... Let's actually go back to... Let's go back here. We're going to be working with the HTML CSS one for this linting to begin with. So if I run npm run lint, it's running this eslint command, and notice that we're getting some errors. We're getting some errors about this alt attribute, ARIA roles, and the form label. I was afraid this might happen. I've been working with both the solution and the initial ones.

This one, if we go back to code, I actually committed part of a solution. Let's go into source and app, and let's actually take out this alt tag on line 14. That's meant to be part of the problem here. Now, if I do an npm run lint, you can see I'm getting four errors and that's what I expected. The image doesn't have an alt prop on line 14. There is a redundant alt attribute. There's an abstract aria role that it doesn't like and there's a form label issue that it doesn't like. Notice it's not catching that contrast issue.

The contrast issue is not getting caught by linting. And the linting tool that I'm using here let's go back to is ESlint Plugin JSX ally. And so it's already installed and it's actually included with the Airbnb ESlint plugin if that's something that you use. And then you need to add the rules to your ESlint config. And then finally, here we have a place where you can look up the failures. So if you're not sure what a failure means, yeah, Vite is very fast. This isn't even my new fancy, I'm just responding to the chat. This isn't even my new fancy M1 MacBook, this is my old one. And yeah, it's a lot faster than create React app. It sounds like faster than Angular. So let's get out of here for a moment. And if you're not sure for example, what this ARIA role means or this label has associated control, you can go to the ESLint plugin page for the JSA 11 Y and there are rule details. Let's see. Oh, this one I'm actually in the Alt text rule details. So if we go back, here are all the rule details. And so if you want to know what one looks in particular, you can click on it. If you're actually in VS Code and you mouse over, you can click on the rule detail just from the VS Code pop-up and it'll show you it here as well. So I can put an Alt here and I'm gonna do something slightly not so great with the Alt so that it'll get caught later. I'm gonna say a health for life logo. And really logo is not a word that you want to put in your Alt text. You don't wanna put like photo or image or logo because it just is redundant for screen readers. But you'll notice that now that linting error is gone. I don't have the angry red squiggles anymore. And if I do the NPM run lint, now I only have three errors.

Okay, so I'm going to give everyone a moment to run NPM lint on your own machine and fix the remaining errors. Now let's go into this, let's get back to VS Code. All right, so for this one, it tells you don't use the word photo in the alt. So we'll just remove the photo of, and that lint error went away. This one, this role equals hidden, so this is the divider that you can see. That's this divider here, this long line that I decided screen readers didn't need to deal with.".

4. Automated Testing: Linting and Just Axe

Short description:

In this section, we discuss the different methods of automated testing for accessibility. We explore the limitations of linting when using a design system and introduce the use of Just Axe to check for accessibility issues in rendered code. The rendered code is checked against Axe standards, which include a list of rules. Installing Jest and Jest X is necessary for testing accessibility in Vite projects.

That's this divider here, this long line that I decided screen readers didn't need to deal with. And aria-hidden equals true is the correct way to do that. The role equals hidden, that's just not, that's not a legitimate role.

And then the final one, so some people were saying, well, can you just wrap the input in the label? And you can do that, and you'll notice that made it go away. You can also use the html4 and use the ID here. Oh, ah, because my label is actually not like surrounding any label I believe. Okay, there we go. Okay, so let's look at the chat. Oh, they need to have the same parent. Yes, thank you.

Okay, so great. And so I was also chatting with some people who came back a little early. And if we look at the Chakra project and we run the linter on the Chakra project, the Chakra project has the same four errors, but it only finds one of them. So if I look at Chakra in VS Code VS Code and I look at that app in that app file, you'll notice that we have that logo and it does not have an alt tag the way, for example, this splash image does. And I do have an ARIA rule hidden in here somewhere that... Yeah, so I have this role equals hidden. So that's the one that it found. And this one only has the divider once. Let's see, do I have... No, of course, I don't have that still in here. So I can update that. It has another issue with this label. This label is not within the form control for the region, but none of those are being found. I fixed that ARIA role, and now we could see it with no angry red squiggles, and we can see it here. The linter thinks this is totally fine.

So, linting doesn't work so well when you're using a design system, because it's looking specifically, here I am back in the HTML CSS, it's looking specifically for standard HTML elements, like image. And if I go back to Chakra, I'm back in Chakra here, you can see I'm not using the HTML image element. I'm using the Chakra image element that I've imported from Chakra UI React. And the linter is not familiar with that element. So we're gonna bring out a different tool here. We're going to bring out just Axe. Let me get back to my slides.

Okay, so this is just saying what I just said. So it's looking for standard HTML elements. And so any design system that you use that has custom React components is going to have this problem. Just Axe helps with this. It helps with this because it renders the code and it checks the rendered code, which has the standard HTML elements against, in this case, Axe standards. And I have a link here. Axe is, it's this DEC University or this company DEC here made these standards. So here's a list of all of their rules. And it's not exactly the same as the JSX Ally Linter, but it's very similar. Let me see. Okay, yeah. And so because the rendered code has those standard elements then it's going to be able to catch the errors. So I've already installed Jest to these Vite projects, but I have a link in case you want to install Jest to future Vite projects. That's a disadvantage of Vite with Create React app. Create React app is in some ways more bloated, but it's also more complete and includes Jest. And I've also installed testing library. Then we'll have to install Jest X, which has not been installed. So, let's go here to Jest X.

5. Automated Testing: Jest X and Issue Identification

Short description:

In this section, we install Jest X and discuss the usage of axe for automated testing. We create a test file in the Chakra UI project and copy the code from the workshop repository. The code comes from the just axe docs. After running the test, we find two issues, with one remaining. We address the issues in breakout rooms. Jestax doesn't provide line numbers, so we need to examine the code to locate the issues. We continue the discussion with Maria about the difficulty of understanding the HTML structure.

And this is where I got that only 30% of issues are found by automated testing. All right, so first, we'll install Jest X, and I'm in the Chakra UI project now. Let's paste there. And for TypeScript, we can also install Types. It's not necessary for this project because I'm not using TypeScript on the tests. And then you'll see that the usage, we actually have to import axe and to have no violations. And then we run axe on some HTML and we expect the output to have no violations. Let's go back to the slides for a moment.

With testing library, which is what we'll be using today, we use this container for testing library and that's where we get the HTML from. So we're going to make a test file in that Chakra UI project and we're just gonna copy the code from the workshop repo. So we're gonna copy it from the copy paste files. And the code actually comes from the just axe docs. So if we go here, you'll notice that it basically gives us everything we need for react testing library. So here I'm in the Chakra UI directory and I'm gonna copy from that copy paste directory which is one level up, and everything starts with A. So app test.jsx. I'm going to copy that to my source directory. And we can look at what we just copied.

So now that we have this testfile, and it doesn't like, my linter didn't like the order of the import so I just saved it to reorder the imports, but so I'm getting render from testing library. I'm getting axe and to have no violations from just axe, and then TSX likes to have react in there. And then I'm getting my app because that's what I'm going to render. I extend expect to be able to use this, to have no violations. And if you were running this in a project where you had more than one test file, you might wanna put this in your set up tests so that every file could use it. Then I'm rendering app, and I'm destructuring container from what comes off of it. So this is just the suggested way of just axe. This is the suggested way of getting HTML to pass to axe. So that I can take that results running axe on this HTML and I'm expecting that results to have no violations. So as you'll recall now, if I do an npm run lint, I'm not getting anything. And now if I do an npm test, this is running just in watch mode, running that test file we just made. And now you can see, it found some issues. It only found two issues, and we know that there were three remaining. We'll address that in a moment. But for now, I'm going to put you back in your breakout rooms to work on these issues. Now there's something that is less convenient about Jestax than the linter, in that it doesn't tell you what line on your code it is because it's working on rendered HTML. So you're going to have to look at this code and see if you can find it. And it's also the classes are not ones that you typed. You did not actually create, or you can't find those classes in the app.jsx file because these are from the rendered code from Chakra. So you have to do a little bit more work to find the issues here. I'm going to break you out into your breakout rooms again and so that you can find the issues and fix them. And then we'll come back and take a look at that third issue that wasn't caught. All right. So let me. Okay. All right. Goodbye. Sorry. Sorry that you have to leave. Bye-bye. Okay. But I was talking to Maria about, she wanted to be able to see what the actual entire HTML looked like just because it was so difficult to actually figure out what was going on. So we were talking about, I did a try and catch here.

6. Jest Error Recording and Custom Rules in Just AXE

Short description:

Jest records errors in tests by throwing an error. Running screen.debug provides a whole picture of the HTML. We fix errors in app.tsx by adding an alt for the logo and fixing the label. The tests are passing, but Just AXE doesn't catch all issues. We can add custom rules to Just AXE using an AXE config file. The new rule checks if an image has an alt attribute without 'photo', 'image', or 'logo'. The tests fail due to a redundant word. After fixing the issues, all tests pass.

And so you're probably aware the way Jest records errors or registers an error in a test is if there is an error that's thrown. And so if this line throws an error before we actually throw the error to fail the test, I'm running a screen.debug and screen comes from testing library React. And if I do that, then you actually get a whole picture of what the HTML is, which might make it easier to actually figure out what it's talking about here.

So we can fix the errors in app.tsx here. We can add an alt for the logo. And we can fix the label by putting the label within the form control. And now if we go back here, my test should be passing. So that's great, the tests are passing. It did not catch that photo of, if we go back to the Health for Life CEO, it did not catch that photo of, because that's not one of the AXE rules, that's a rule in the JSXA11Y linter, but it's not a rule in Just AXE. So we can actually add rules to Just AXE. AXE, that's a crawl there.

All right, so we can add a custom rule by creating our own AXE object and importing AXE from that, from the AXE object that we created with the custom rule, rather than Just AXE. So let's take a look at that, and I have in copy paste files, I have that AXE config. It actually took me a while to figure out how to do this, there's not great documentation for this. So if we look at copy paste files, we can see I have this AXE config. So I'm just going to copy that, I'm going to copy that just straight to, just straight to the top level of the shocker UI project. And then if I go back into the code, I can look and see what it is. So I just imported configure AXE from just AXE, and then I configured it with this rule that again, like I actually thinking of writing a blog post on this because it took me so long to figure out how to actually add this custom rule. But basically the test is, if this is an image, if this is an image element, cause it only looks at image elements, and it has an alt attribute. So it needs to both have an alt attribute and the alt attribute must not have photo image or logo in the attribute. So it's just a regular expression thing here. So now, and then I export, I export this axe here. So if we go into the test file, instead of importing axe from just axe, I'm gonna import it from that file that has the new configuration. So I'll import axe from axe config. This is where it's really useful for you to have your cameras on because I can see when people are typing and doing stuff. So now if we run that npm test again, it's going to be running with that new rule, and now it failed. Now it failed because it found something that has a redundant word. So, if we go back to app.tsx, and we remove that photo of, just make sure I'm in the Shockware UI, because it's easy to get these two projects mixed up. The watch mode will rerun the tests. Oh, that logo that I hid in my alt text for the logo, that's getting caught now too. So, we'll have to fix that as well. So, I'll give you a minute to get your tests passing as well. Go ahead and do that setup in the HTML CSS myself. So, I'm going to do the npm install. That's for the types. Then, I'll do the one for the actual package as well. Then, I'm going to copy that copy paste file, the app.test.jsx into the source directory. I'll do an npm test, which should run it. Yes, everything passed. No accessibility errors caught by just Axe. The reason it's not catching the logo is because I haven't done the custom rule yet. So, I'm going to copy that axe-config. Then, I will update that test file to import axe from that config file instead. So now, it should be using that new rule where it's testing to see whether it has photo, image, or logo in it. Oh, wrong up arrow. And now, it's finding that logo issue in the Health for Life logo. So I can go back and remove that. And now, all the tests have passed. Okay, so this is not going to take the entire three hours for the workshop. But let's take a ten minute break.

7. Cypress Audit and Installation

Short description:

We'll be back at 9:20 A.M. to continue with the final part, which includes Cypress for detecting contrast issues that code-based tests can't find. Cypress is a test framework that renders the page in a browser, and Cypress Audit provides accessibility testing tools using Lighthouse. To install Cypress Audit, use npm install with the save-dev flag. Set up the index.js file in the plugins directory and add Cypress commands. Then run Cypress with the npm script 'npm run cypress:open'.

And we'll be back at 20 minutes past the hour, 9 20 A.M. my time. And yeah, and then, we'll do the final part, which includes Cypress, which we'll actually tackle that contrast issue that the code-based tests just can't detect. Okay, so I'll see you in ten minutes. Are we on a break? Yep. For another like 10 minutes or so. Thank you.

All right. Let's take a look at Cypress Audit. All right. So errors beyond code. So for example, those contrast errors that we were seeing. If I go back and let's go into the Chakra UI and open. So here, this contrast error, where when I'm not mousing over, it's pretty hard to read that text. And if I were somebody who had trouble seeing contrast, I probably couldn't read it at all. So we're going to use Cypress, which is a way that actually renders the page in the browser. It's a test framework that renders the page in a browser, and Cypress Audit which is going to do some accessibility tests. So Cypress is an end-to-end tester that actually renders the page in a browser. And then Cypress Audit provides tools to use either Lighthouse, which is a Google-based tool for accessibility and a bunch of other webpage standards, or Pally. And we're going to be using Lighthouse. They do very similar things. So Cypress has been installed in both of the projects. And there is an npm script to run Cypress. So you can do npm run cypress colon open. So we do that, for example, in our shocker UI. It's opening Cypress. And then there's some integration tests that it just comes with by default that we're going to ignore. All right. So I'm going to Control-C that for now.

Cypress audit, we can install it just the way you'd install any other NPM dependency. So you install and do a save dev of Cypress audit. And then it needs some setup files. So we're going to keep clicking, and it thinks that I'm clicking on... There we go. So there's an index.js file in the plug-ins directory that we're going to use, and that is part of... We're going to get the text for that from the documentation. And then, we're also going to have to add some commands for Cypress. So let's do all of that together. We'll go here in Chakra UI, and we'll do an npm install, save dev of Cypress audit. All right, we'll let that do its thing. And in, let's see, I want to be in my Chakra UI directory. In Cypress at the top level, in plugins, I have an index.js and I'm actually going to just delete it. And I'm going to replace it with the index.js that I get from Cypress audit. So I'm just going to copy this whole thing here or I can click on the copied. This is the Cypress audit. And it's from the link from the slides. And I'll paste here. And then I'm going to comment out the Pali line because I'm not interested in that. And I'll paste this in the chat just in case people need that. Notice that we're using require and module.exports.

8. Cypress Audit and Lighthouse Thresholds

Short description:

With Cypress, you can't use the ES6 import. My linter has some problems with unused variables. In the support commands file, I import the commands from Cypress audit. I have a Cypress test file that visits my local host page and runs the Lighthouse command. I set up the Lighthouse thresholds for accessibility. I encounter errors on performance, accessibility, best practices, search engine optimization, and progressive web apps. I focus on accessibility and set the threshold to 100. The accessibility record is 98, which is under the threshold. I encounter the 'Cy Lighthouse is not a function' error. I add the function to the plugins index.js file as per the documentation.

With Cypress, you can't use the ES6 import, at least not out of the box. And then my linter has some problems with unused variables that I'm just going to ignore. All right. So we have this.

Now in the support commands file, I'm just going to import the commands from Cypress audit so I can use those in my Cypress test file. And then finally, I have a Cypress test file. And I have that in my copy paste files. And that's the A11y spec dot JS. So I'm going to copy that into Cypress and integration. So this has some lighthouse thresholds that we'll get to in a minute. But basically all it's doing is it's visiting my local host page and then it's running that lighthouse command, which I set up with that index in my plugins and the commands.js. Now in order for it to visit, let's go back to that spec file, in order for it to visit this page, I need to actually have my server running. So I'm going to open up a new tab and I'm going to have my server running in the background. And then I'm going to open Cypress again and I'm back to my original tab. Okay. And now I have this a 11 Y spec. Now I have Chrome here. If we try and run this on Electron, so Cypress actually loads the page. It would have a problem if I didn't have my server running on 3000 and notice it says Electron is not supported skipping. So you need to be using Chrome or I have actually haven't tried it on Firefox, but so now I'm going to try Chrome. Seems to be doing more here. Okay, so I got some errors and I got some errors on performance and accessibility, best practices, search engine optimization, and progressive web apps. Right now I'm not interested in anything but accessibility. So, let's go back to that spec file and I'm going to just use those Lighthouse thresholds. So basically my threshold for everything is zero except for accessibility where it's 100. When I save that, Cypress detected the change and now it's going back through its thing. Okay. So, now I've got that the accessibility record is 98 and it's under the 100 threshold. Oh, Cy Lighthouse is not a function. So, for that, you probably don't have this in commands.js. I'm sorry. Can you open the LI spec file? Yeah. Of course. Okay. I want to make sure I wrote it down correctly. Okay. Thank you. No worries. Lars, have you got it working? Yeah, now it's asking for those thresholds. Okay. Okay. So it sounds like you got past the lighthouse is not a function. So you may notice that this is not works as expected. Yes, and I'm surprised that I'm getting a 98 because I was getting an 83 and all of my and all of my practice runs, but it's yeah, it's failing so I'll take it. So this is this is not I mean it's a useful error in that it's telling me I'm failing but it doesn't give me any indication about where I'm failing. If we go back to the documentation it says okay well you can get a report by adding this function as an argument to Lighthouse in the plugins index.js file. Let's go back to the plugins index.js file and I'm going to add that function. You don't have to worry about following along with me here because this is not going to be our final step. Of course you can if you want. All right, so I did that and at this point sometimes Cypress will give an error.

9. Cypress Audit and Lighthouse Report

Short description:

See if we go into Chrome and we try and rerun this. There is a false error that happens when you change the configuration from underneath Cypress. The report gives more information than necessary but is not useful in finding the error. A Lighthouse report is created to parse the object and print a more useful report. The function takes the report, checks for violations, and makes them look pretty. The report is written to a file with an ISO timestamp. The issue is identified as a button without a background color and a label for signing up for the mailing list. The solution involves giving the button a background color.

See if we go into Chrome and we try and rerun this. Yeah, so this cannot destructure property errors of object null as it is null. There is a false error that happens when you change the configuration from underneath Cypress. So I'm just going to... I closed Chrome. I stopped this and I'm going to run this again.

Okay, so I got the error and that report is back in my console. And the report gives a lot more information than you necessarily need and honestly is not all that useful in finding the error. So back in the copy paste files, I'm going to control-C out of here for a moment. Back in the copy paste files, I've made a Lighthouse report that parses this enormous object and prints the report to a file in a way that I find more useful. So let's copy that Lighthouse report. We're going to copy that into Cypress Plugins. And again, basically all this is doing is it's taking the part of that object that I'm interested in. So you'll remember in the index.js file here, the function here takes that Lighthouse report and all I did was just like console logged it and just spit out that huge object. So this function takes that report. And if there are no violations, so I dig into where it actually prints the violations. If there's no violations, I'm just going to console log that there's no accessible errors and it's not going to make a report. Otherwise, I'm going to parse the violations and make them look pretty. And if I couldn't write the file, then I won't write it, otherwise I'll say I wrote the file. And then I also did some error management here because when I went from my main computer to my studio computer, it had some errors with the way I was formatting these multiline things. And so anyway, I didn't want it to fall over, so I wrapped the whole thing in a TriBlock. All right, so, and again, because this is Cypress, we need to use require everywhere and module exports. We can't use the ES6 import export. So let's go back here. And now I'm going to say const lighthouse. Actually, I think there's, I'm not destructuring. Lighthouse report equals require lighthouse report. And then I'm just going to pass that function along here. So let's open Cypress again. And we'll run our a11y spec. And now we can watch this, and there's a lot of information about what's going on with the server. So we're waiting for that info or error. So I got an info and it wrote it to this reports lighthouse a11y. And then I just put an ISO timestamp on it so that it'll make a new report every time. So if we look at reports, we look at this. Okay. So now I'm getting the actual issue, which is, it's just more useful than that entire object. So I have a, it's that, um, because this is rendered, it has some similar problems to just acts and that is not going to be able to tell me the actual line of code, but it does tell me that it's a button, type equals button. It has a label, sign up for mailing list. And it gave the foreground color and the background color. And it, it told me what the issue was. So if I go back to app.jsx, I can solve this in a pretty blunt force way, uh, by finding that button. Accessibility record is zero. That's interesting. So, for the button, I'm going to give it a background, oh, no such directory. Okay. So you might have to make a reports directory, um, in the no such, in the no such directory, you might have to make a reports directory in the Cypress file. I tried to, uh, make sure that that was there, that the reports directory was there with the, um, by making a README. Oh, no, actually your path is, oh, your path is using, you have, okay. I'll work with you on that in a moment, Lars.

10. Setting Background and Foreground Colors

Short description:

I'm going to make the background the darkest color and the foreground the lightest color. The accessibility record is now 100 with no errors. However, the report was still written, indicating an issue. Let's examine the report.

Thank you. I haven't tried this on a Windows machine. So, um, so I'm just going to make the background be, um, so here I have a theme for Chakra UI. I don't know if you've used Chakra UI before. But I have a theme that has brand colors. So I'm going to make the background the darkest color. And I'll make the foreground the lightest color. So the background is going to be brand 900. And the foreground or the text color is going to be brand 100. And so that should be rerunning the test. Maybe not. Let's rerun the test here. Yeah. So notice now my accessibility record is 100. And it found no accessibility errors. But it wrote the report anyway. So there's some issue there. Let's see the report that it wrote.

11. Troubleshooting FS Write File and Cypress Audit

Short description:

My length detector is having an issue. Lars, if you're on Windows, try creating the file first with FS. Use FS open for writing before opening it. If the error persists, let us know. You can also try running the Cypress audit on the HTML CSS folder. It works if you create the report file manually.

Oh. Okay. Well, my length detector is having an issue. I can work on that later. Right now I want to look at Lars. So you are getting... So it looks like Lars maybe on Windows, you need to create the file first with FS. It's trying to write the file and it can't find it. So you might want to... And I'm actually not an expert in this. But we're using FS write file. So FS open. If the file does not exist, an empty file is created. So Lars, you might try doing an FS open for writing before you actually, before it actually opens it. If we do fsopen and the file name for writing it. And then say open error here. And then, yeah, I don't love this, this nested callbacks here, but say if... So you might try that Lars and see if we get, I mean, my fear is that we're going to get the same error that it's just gonna be like, eh, can't open it. Let's see how we're doing. Yeah, so for everybody else, you can try running and running the Cypress audit on the HTML CSS folder and see if you can get that working there. And if not, let let us know what questions you have. That's supposed to be consistent. I probably should. The error is the same, but it works if you create the report file first manually.

12. Optimizing Web Accessibility with Cypress Audit

Short description:

We switch to the HTML CSS server for testing and address the poor contrast issue. We add the Cypress audit commands and run the test without thresholds. However, we only get an error without any indication of the cause. We copy the Lighthouse report function and process the raw report, writing a report to a file. The report shows insufficient contrast.

Okay. All right, so let me. I'm going to just. Make a note that. Is anybody else using windows I'm just curious. Okay, so we can get this working on on HTML CSS as well. I'm going to control C out of here. And I'm also going to control C out of my server because we want to make sure that the server that we're running is the HTML CSS server. Otherwise, we would just be testing against the Chakra UI server. If that's the one if that's the one that was serving on localhost 3000. So now the HTML CSS server is serving on localhost 3000.

So if we look here, we have that poor contrast here. I'm going to go back into HTML CSS. And we'll do the same things over again. So the first thing I'm just going to close out the Chakra UI. In Cypress and oh, so once you run Cypress it adds some folders so I can run that. So it added my integration and plugins folder. I'm going to remove everything from the plugins folder. And I'm going to add. The text from the Cypress audit documentation. Just going to comment out the Pali since we're not using that. Then in support and commands. I need to add the Cypress audit commands. So that it will recognize the lighthouse command in that spec file. So I'm going to copy from my copy paste files that spec file into Cypress integration. So now I have that spec file here. And that lighthouse command is now coming from the commands that I've added here. Oh, thank you, Linter. I forgot to actually install Cypress Audit. So let's do that. Oh, it wasn't done. Okay, so now the Linter is happy. And it's going to run this without the thresholds. So let's go, oh, we need to open Cypress again. So, we can see, well, if it gave us enough time, we can see that this is the HTML CSS one. And, again, we are getting some thresholds that we don't care about. So, we'll pass that lighthouse thresholds that I've written out, which is basically I don't care about anything. except for accessibility can have a threshold of zero. And so, now, I'm getting the error, but no indication of what's actually causing the error. And that's where I'm going to quit Cypress again. I'm going to copy that Lighthouse report from my copy and paste into Cypress plugins, because that's where I'm going to actually use that report function. And now, so now we have this report. I'm going to require it, since we can't use the ES6 import from Cypress out of the box. And then I'm going to pass that function to the Lighthouse function here. And this Lighthouse function what it does is, its argument is a function that takes that raw report and then does something with it. And we did a console log of the raw report. Found that the raw report had way too much information to be useful, so now instead we're going to be processing it and writing a report to file. So let's run this again. So, now it wrote this report and on windows you're going to have to do the same thing where you're going to have to create the report and have it overwrite, which again is not I'll look into that and think about it. And back here we can look at the reports and it's telling us it has insufficient contrast.

13. Setting Button Contrast

Short description:

To ensure maximum contrast, the background color of the button is set to black and the foreground color is set to white. The test is rerun and all tests pass. However, the mouseover contrast may still need to be checked by moving the mouse over the button.

The background, it has this foreground color and this background color. If we go into app.tsx and we'll take that button. And here I'm using a class of button. This one is the one that's really just HTML CSS. And so I'm going to go into this about.css. And this button class, oh, no, it's not. It's not in about it's in brand.css. And so for the button class, I'm going to, again, I'm just going to really brute force it here. I'm going to make the background color black and the foreground color white. Just for maximum contrast. So I'm going to run this test again. And now everything's passing. And if you look, yes, that's really bright contrast. Now it's possible that the mouseover contrast is not good enough. And for that you would have to... In Cyprus, you would have to move the mouse over this in order to check the mouseover contrast.

Watch more workshops on topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
WorkshopFree
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
WorkshopFree
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
Workshop
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!