Configuring Axe Accessibility Tests

Hello, my name is Steven Lambert. I go by he, him pronouns. I am a tech lead and people manager at Deque Systems. I'm also the primary developer on AXe core, the accessibility testing library.

In case everybody wants to get a hold of me later or in connect, you can find me on Twitter using the handle at StevenKLambert. You can email me using Steven at sklambert.com or you can visit my website, stevenklambert.com and contact me there.

So before we dive into how to configure AXe, I first wanted to give a quick overview of what AXe is and how to use it in case anyone was unfamiliar. So AXe is an accessibility engine for automated web UI testing. What that means is that AXe runs a set of rules on a page or your code to test for accessibility problems.

Here is an example of how you would use AXe on a webpage. You would first load a script whose source points to the AXe core library. In this case, it's from unpkg.com slash AXe dash core at latest slash AXe dot JS. That will load the main AXe core library onto your page. Once that's loaded, you can use a script of type module to await the AXe dot run results. Using those results, you can then log out the violations property, which will show you each rule that reporting a violation. Here is an example output of what that would look like.

So the violations property is an array where each index of the array is an object. Those objects will list the rule name that failed as well as things like the impact on the user and all nodes that failed the particular rule. So in this example, I'm showing that there is a landmark one main violation, a page has heading one and region rules are all failing.

So where can you use Axe? Well, Axe has many integrations into various languages and frameworks. The main library is axe-core and that can be used in the browser or node directly. We also provide a handful of integrations into the command line and popular testing frameworks. So on npm, you can look at the at axe-core namespace and there you can find the CLI integration, a Playwright, a Puppeteer, a React, a WebDriver.IO and the WebDriver.js integration. This year, we also released a new integration for VS Code, which is called Axelinter and it provides accessibility linting for your code that is consistent with Axe-Core's rule engine.

So what that means is that as you type, you will get linting suggestions for anything that we can detect. Lastly, we have other integrations such as the Chrome and Firefox extensions. We also have integrations into Java and various Ruby libraries, but we won't be covering any of those for this presentation. We'll just be talking about JavaScript and Node compliant integrations.

So now that we know what Axe is and how to use it, I wanna talk about how to configure which rules Axe runs. So as I mentioned, Axe runs a set of rules that determine the accessibility problems on a page.

By default, Axe will run all supported rules. Now, which rules are supported depends on which integration you are using. So for Axe core and its various node integrations, you can find the supported rules by going to the Axe core GitHub page. In there, we have a docs directory and the rule-descriptions.md file, which will list all supported rules. For Axe Linter, you can find the list of supported rules by going to the VS Code Axe Linter page. For Axe core, there's about 91 supported rules that you can look at. And for Axe Linter, as I mentioned, as only a subset of rules has about 33 supported rules.

So there are various ways that you can configure which rules Axe will run. For starters, you can disable a certain set of rules so that they won't run during a normal run. You could also specify a certain set of rules to only run, and you can also run rules that match a particular tag. Now, for this presentation, what I'm gonna do is I'm going to show an example of how to do this in only three integrations. I'll show you how to do it in Axe core, I'll show you how to do it in an example of a node integration like Puppeteer, and I will also show you how to do it in Axe linter.

So first, I wanna talk about disabling rules. So let's say as an example that you wanted to disable two particular rules, the button name and label rules. Now, the button name rule ensures that every HTML button element has an accessible name, and that can be used either through having text content in the button or that the button has an ARIA label, ARIA labeled by, or title attribute. The label rule does something similar where that it ensures every input element has an accessible name, either through an associated label element or using the ARIA label, ARIA labeled by, or title attribute. So for using this in ax-core, what you would do is you would pass an options object to ax.run. The object takes a rules property, which is whose value is also an object. Each key of that object is the name of the rule that you want to disable. And then the value of that is an object who takes an enabled property and can pass either true or false. Now true is the default behavior and that means that the rule will run, passing false will disable the rule so the rule won't run. So in this example, we pass the button name and label rules and we enabled false both of them. For a CLI and a test framework. So what you would do is you would initialize a new ax builder object that allows you to chain a couple functions off of it. One of those functions you can chain is called disabled rules. And the disabled rules function allows you to pass a rule name or an array of rule names that you wish to disable. So in this case, we can pass an array of button name and label to disable both of those rules. And then lastly, you would chain the analyze function and that would then run ax on that page. For ax linter, we don't have a an API you can use. So instead what you do is you configure it through a config file.

The config file you can use is called ax-linter.yaml. And that file must be at the root of your project for ax linter to find it. Inside of that YAML file, you would set the rules key, which is a dictionary. And each key of that dictionary is then the rule name that you wish to disable. And the value is a true or false again, where true means run the rule, which is the default behavior and false means disable the rule. So for this particular example, we pass button name and label and set them false in the YAML file. So that's how you would disable certain rules.

And now to do the inverse, we want to run only a certain set of rules. So now let's say that instead of disabling button name and the label rules, you wanted to enable them and make them the only rules that run. So for ax core, we once again will pass an object to ax dot run that options object takes now the run only property whose value is either a rule name or an array of rule names to run. So in this case, we would pass an array of button name and label and then that will make ax only run those two rules when it returns results.

For the CLI and test frameworks, what you would do is you would initialize ax builder once again and this time chain the function with rules and the with rules function will take a single rule name or an array of rule names and then only run those when you call analyze. Okay. For ax linter, unfortunately, we don't have a way for you to enable only a certain set of rules. So the only way that you can do this is you'll have to disable all other rules and then only enable the ones that you want. So ax linter has about 33 rules. So you would have to list all 33 rules, put false for 31 of them and then true for just button name and label rules. Again, to find the list of supported rules, you can go to the VS Code ax linter extension page.

So now we want to run only rules that match a particular tag. So what is a tag? First, tags are ways to categorize rules into groups. In most cases, they're used to group rules by the WCAG version and conformance level. So for example, the WCAG AA tag is used to group rules that pertain to WCAG 2.0 AA conformance. To see which rules are associated with which tags you can go to the AxeCores repo and look at the docs slash rule dash descriptions MD file once again. For a list of what tags are supported, you can go to AxeCores GitHub repo to the docs directory and look at the API.md file. In there we have linked a list of all supported tags. The quick rundown for those supported tags will usually be WCAG 2.0 AA and AAA as well as WCAG 2.1 AA and AAA.

Now for axe-linter again, they support only a subset of tags, which I believe are WCAG 2.0 AA and AA and WCAG 2.1 AA and AA. So axe-linter does not currently support any AAA rules.

For AxeCore to run only a certain set of rules, we once again will use the axe.run, passing an options object and using the run only property once again. But this time, instead of passing a set of rules, we will pass it a set of tag names. In this case, the WCAG 2 AA and the WCAG 2 AA tags.

Now, a thing to note about these tag names is that the axe will only run rules that match a specific tag. So for example, let's say you wanted to run all WCAG 2 AA conformance, which normally would mean that you've also fulfill WCAG 2 AA conformance. But axe will not run WCAG 2 AA rules if you only pass the WCAG 2 AA tag. It has to be a one to one match. So if you wanted to run all rules to conform to WCAG 2 A, you must pass the WCAG 2 A and WCAG 2 AA.

Now for the test frameworks, you would again initialize axe builder and then use the chainable function with tags and with tags takes a single or an array of tags. And then what's called analyze, only those rules would run. For axe linter, you would use the tags property, which is a list of tags to run, and then you would list whichever tags you wanted to run.

Lastly, I wanted to talk about some of the various options that axe allows you to pass when you run it. A list of all available options can be found on the axe course API doc file. A note about run options, run options are only supported in axe and its integrations, and they are not really supported in axe linter, with two exceptions.

So a short list of run options, we've already talked about the run only and the rules properties. They have similar things in axe linter, the run only being the rules or the tags properties. But we also support a few other things, so for example we can support passing the result types property, and the result types lets you limit which results are shown. So either pass results, incomplete or violations. You can also pass the iframes option and that will tell axe to run or not run within iframes. By default, an axe integration will run in all iframes on the page. So for an example, let's say that we only wanted to have the violations result shown, so we would pass the result types object to axe.run. And then since it's an array, and we would pass the violation string. And what this will do is when axe.runs, you will get back a list of all the rules still, but for those rules that didn't match the particular types, they will only show one node. So for any passes in this example, only one node will show a pass for that rule. What that allows you to do is it's useful for improving performance on very large pages or very complicated pages where you're only interested in a certain type of result.

For the test frameworks, you would use the options chainable function to pass the list of options that way. And that is how you configure axe. Thank you for coming to this presentation.

If you wanna get a hold of Deque, you can do so at any of these places. You can use their Twitter at DequeSystems. You can find them on GitHub at DequeLabs. You can also connect on LinkedIn at deque-systems-inc and also on YouTube at deque-systems, no dashes.

Thank you. Amazing talk, Steven. You will definitely win me over by preaching about web and accessibility and showing us how easy sometimes might be, or easier, not necessarily easy for everyone, but easier. Thanks so much for that. And I'm really curious to see the results from the pool. Let's welcome Steven and debate a bit about it.

What do you expect in this number, Steven? Yeah, thanks for having me. I didn't hear you well, sorry. I said that, and thanks for having me on. Welcome, of course, it's our pleasure. What do you think about the results? 65% replied with no. Not unexpected. Accessibility testing is hard to get started in a lot of enterprise-type systems that don't have them already there. Indeed, indeed. And I see a good chunk of people already using X, or XAI was literally wondering to us how we pronounce it better at the beginning. Or accessibility insights. And zero for the rest. How do you feel about the zero on the two other tools? I don't maintain them, so they're all great tools. As long as they're using something. Indeed, indeed. I agree that, I mean, it's not under you, but I am surprised that Wave has zero. It's one of the most how to some mediatized or promoted tools, especially when you research how to test accessibility. That's why I am a bit surprised. That's true. Yeah, indeed. Okay, then let's see how we do with questions.

And I do have really a personal one, maybe to tell you how was, what was the moment that triggered this accessibility passion maybe or exploration for your side? It's a good question. A lot, many years ago, when I was doing research on how to improve the user experience, I stumbled upon articles on accessibility and it just kind of struck me as interesting and I never really heard about it before. And so just digging into it over the years, I've just gotten more and more interested into it and wanted to keep pursuing it forward.

Yeah, that's really nice. I was asking you this because in my case, like you I was trying to, I was trying to learn it, but I wasn't necessarily driven by something specific. So I once would not choose a payment because of the color contrast. And then it was for me, I'm done. We need to bridge these more. We need to teach people about it. And I even interviewed developers on how much time they invest creating all these in a more accessible form. And to my surprise, the common answer that I got, it was like, Iona, it's just yet another library or just another tool that you use. Once you learn it to just apply next time is not the cost at all the projects. And that is my preaching moment for the developers or managers that don't believe that they have time or money to budget to invest. And after that for the users, I do mention all the benefits that even non-needing people will benefit from accessibility. Because at the end of the day, we benefit from better web, the same.

Thanks for joining in this intriguing and curiosity that I had. Milad was asking, in your presentation you configure Axe for text frameworks, should we use Axe-gest or Axe for right to left? So the ones that Axe and the Deque systems maintain are all under the axe-core NPM namespace. So I believe axe-gest is not maintained by Deque. But if you are using Gests, and I believe it's using Axe under the hood still, it's probably still good to use. I can't say what its API will be, though, for how to configure Axe under the hood. Okay, great. When using aislint-plugin-js-accessibility in React, do you think it's necessary to also write the end-to-end tests with Axe? I guess it depends on how you're doing your testing. A lot of times when you're doing Axe tests, I've seen them done as unit testing. So you're testing the individual pieces, making sure they're all accessible. That works for a good set of the rules Axe runs, but there are rules that are most focused on how the whole page works together. So, for example, we have a rule for heading order, which makes sure that you have your h1 through h6 tags in a correct order, and those can only really be run on a full page. So those are still probably good to use as an end-to-end testing of just kind of catch your page level type accessibility issues. Interesting. X and TypeScript.

X and TypeScript. Is there a TypeScript API for Axe or typings are definitely typed? Yes. All of our test framework integrations are written in TypeScript. Axe has a TypeScript definition file that you can use, but Axe itself is not written in TypeScript. And the X and suppressing warning. Is there a way to suppress warning because I don't want to fix it or because it's just not possible there are false positive reports? If you do encounter false positives, we would love it if you reported it on the Axe core GitHub issues page. We try to put very high priority on fixing false positives. In terms of ignoring warnings, there's currently not a way to do that unless you're using the Axe extension and you're paying for a license, in which case there is a way to ignore results that you don't care to see anymore.

Oh, indeed. And also one of the curiosities of how you introduced this accessibility testing in your company. Lot of curious people. So at prior companies, being a developer, it wasn't too hard to introduce it on my own code. So you can kind of just kind of sneak it in there and then your code as a unit test is running Axe. In terms of trying to get that integrated, you, let's see, what was it in the last Q&A session someone was talking about? Oh no, you were just mentioning it, yeah. Trying to convince them that, Hey, we can do this. I've already done it, it's not hard, it's now kind of just free to bring into a unit test. So anytime you wanna test your code, it's there, and then explaining all the benefits to users and really finding a user who struggles with the app or the webpage and showing them trying to use it is a great way to kind of get people to recognize, Hey, maybe we need to take this seriously. Indeed, indeed. And one of the things that I use to show people that they are users and probably so many other people is the example with the plane from the world war II when definitely you're looking at the wrong database, database of people. I miss how it's called now. I think something with bias it's... Yeah, survivorship bias. Yeah, exactly, survivors bias indeed. And people say, no, we don't have users. Well, of course, we never tried, or what helped or triggered them very fast was to find a solution for their daily stuff that applies to them, but both for accessibility. An example with real life sidewalks that need to have the ramp for disabled people, but are used for running too. Bicycles, scooters, mothers with kids or parents with kids in trolleys. Again, a solution that helps everybody is not necessarily dedicated to fix an accessibility issue. Yeah. That's cool.

Annie, I would like to ask you, how is your experience with the developers? So how you feel the checking that they do if you tested with them, how you feel the checks done, the scan are done automatically with developer tools scan? How do you think this brings awareness? How is done or is like you feel it's too little or enough to start? If you try them?

Yeah, so accessibility tests and at least automated accessibility tests can only catch about 50 to 60% of accessibility errors. Which introducing that on a slow bit is better than not catching any errors. So any accessibility testing is better than none, but you always then have to get a manual test, someone who understands the remaining 40% of errors to come and verify. If you really want your page to be truly accessible and ADA compliant and all that.

Yeah, well, I first started with these scans and at least on Firefox, because I work with Mozilla, they will give you even a link to MDN and will explain all the levels with that they have for different stuff. But they only cover as much as can be automating easily, like color contrast links, of course, like navigation tabs, or what else, something. Yeah, but like voiceovers were super hard to test on a mobile phone, how you test that accessibility stuff. And oh, another good example to show why accessibility is important is what I found for myself, is to have captions at videos on the phone. Because you're gonna listen to something and maybe you're not having all the time headphones and it's easier to see texts when you're like watching something not to disturb anyone, or you're with a kid and you want to... So those helped me to from the phone and I found them by mistake and after that, I use them. Again, great thing.

What else or which of these accessibilities issues would you like to be more implemented or more known and thought with solution? I guess the most common, there's a few common accessibility issues that if there was more awareness around them, you could solve. The most important thing a developer could do to help with accessibility is use semantic HTML. So there's the term called divitis or div soup where we just make everything divs and then add on the functionality we need. Doing that though makes it really easy to forget all the native benefits of accessibility that things like a button or a link already have on them. So just by using semantic HTML, you can cover a wide slew of accessibility problems and make sure those are fixed. Next to that, probably the next important issue would be making sure your webpage is just usable by a keyboard alone. Try testing your page, go to your keyboard, press tab, make sure everything that is clickable can be tabbed too that you can activate it with the enter or the space. Because most accessibility software or assistive technologies rely on a keyboard type of focus system. So you can cover a lot of problems as well by making sure that your page is keyboard accessible. So those two things are probably the most important starter points.

Hmm. Indeed. I agree. And one question more about X. X, the focus seems to be on the blind. But there are any checks for motoric disabilities. The button must be big enough to be hit with shaky hands, for example. Mm, there's... So in terms of how to automate test something like that, the WCAG, which is the definitive list of what makes things accessible for compliance, they have implemented a new version, WCAG 2.1, that introduced a new rule about what they call target spacing. So that says that any button or anything that's clickable needs to be a certain size and that it needs to have a certain amount of padding around it. And that is probably the closest you could get to something like shaky hands or tremors in terms of making sure your page is accessible towards that but there's not much that you could do. It's automated wise, that's probably more left to a manual tester to see if there's a enough space around the area to make sure that if you're having tremors that you can click on anything else accidentally. Indeed, oh, thanks so much for the answer. It really was interesting too especially because in testing we always consider from a UX point that maybe someone will have a stronger finger or just shaking one. Thanks so much for your amazing talk. I do invite everyone to join your chat room on spatial.chat and see around. Also tweet your solution for accessibility. I'm curious to read it more.