Configuring Axe Accessibility Tests

Hello, my name is Steven Lambert. I go by he him pronouns. I am a tech lead and people manager at Deque Systems. I'm also the primary developer on AXCore, the accessibility testing library. In case anyone wants to get a hold of me later or connect, you can find me on Twitter using the handle at Steven K Lambert. You can email me using Steven at SK Lambert.com or you can visit my website, StevenKLambert.com and contact me there. So before we dive into how to configure AX, I first wanted to give a quick overview of what AX is and how to use it in case anyone was unfamiliar. So AX is an accessibility engine for automated web UI testing. What that means is that AX runs a set of rules on a page or your code to test for accessibility problems. Here is an example of how you would use AX on a webpage. You would first load a script whose source points to the AXCore library, in this case it's from unpkg.com slash AX-core at latest slash AX.js. That will load the main AXCore library onto your page. Once that's loaded, you can use a script of type module to await the AX.run results. Using those results, you can then log out the violations property, which will show you each rule that reporting a violation. Here is an example output of what that would look like. So the violations property is an array where each index of the array is an object. Those objects will list the rule name that failed as well as things like the impact on the user and all nodes that failed the particular rule. So in this example, I'm showing that there is a landmark one main violation, a page has heading one and region rules are all failing. So where can you use AX? Well, AX has many integrations into various languages and frameworks. The main library is AX-core and that can be used in the browser or node directly. We also provide a handful of integrations into the command line and popular testing frameworks. So on npm, you can look at the at AX-core namespace and there you can find the CLI integration, a playwright, a puppeteer, a react, a WebDriver IO and a WebDriver JS integration. This year, we also released a new integration for VS code, which is called AX linter and it provides accessibility linting for your code that is consistent with AX-core's rule engine. So what that means is that as you type, you will get linting suggestions for anything that we can detect. Lastly, we have other integrations such as the Chrome and Firefox extensions. We also have integrations into Java and various Ruby libraries, but we won't be covering any of those for this presentation. We'll just be talking about javascript and node compliant integrations. So now that we know what AX is and how to use it, I want to talk about how to configure which rules AX runs. So as I mentioned, AX runs a set of rules that determine the accessibility problems on a page. By default, AX will run all supported rules. Now which rules are supported depends on which integration you are using. So for AX-core and its various node integrations, you can find the supported rules by going to the AX-core GitHub page. In there, we have a docs directory and the rule-descriptions.md file, which will list all supported rules. For AX linter, you can find the list of supported rules by going to the VS code AX linter page. For AX-core, there's about 91 supported rules that you can look at. And for AX linter, as I mentioned, as only a subset of rules, has about 33 supported rules. So there are various ways that you can configure which rules AX will run. For starters, you can disable a certain set of rules so that they won't run during a normal run. You could also specify a certain set of rules to only run. And you can also run rules that match a particular tag. Now for this presentation, what I'm going to do is I'm going to show an example of how to do this in only three integrations. I'll show you how to do it in AX-core. I'll show you how to do it in an example of a node integration like Puppeteer. And I will also show you how to do it in AX linter. So first I want to talk about disabling rules. So let's say as an example that you wanted to disable two particular rules, the button name and label rules. Now the button name rule ensures that every html button element has an accessible name. And that can be used either through having text content in the button, or that the button has an ARIA label, ARIA labeled by, or title attribute. The label rule does something similar where it ensures that every input element has an accessible name. Either through an associated label element, or using the ARIA label, ARIA labeled by, or title attribute. So for using this in AX-core, what you would do is you would pass an options object to ax.run, the object takes a rules property, which is, whose value is also an object. Each key of that object is the name of the rule that you want to disable. And then the value of that is an object who takes an enabled property and can pass either true or false. Now true is the default behavior and that means that the rule will run. Passing false will disable the rule so the rule won't run. So in this example, we pass the button name and label rules and we enabled false both of them. For a CLI and a test framework, so what you would do is you would initialize a new ax builder object. That allows you to chain a couple functions off of it. One of those functions you can chain is called disabled rules. And the disabled rules function allows you to pass a rule name or an array of rule names that you wish to disable. So in this case, we can pass an array of button name and label to disable both of those rules. And then lastly, you would chain the analyze function and that would then run ax for on that page. For ax linter, we don't have an api you can use. So instead what you do is you configure it through a config file. The config file you can use is called ax-linter.yaml and that file must be at the root of your project for ax linter to find it. Inside of that yaml file, you would set the rules key, which is a dictionary. And each key of that dictionary is then the rule name that you wish to disable. And the value is a true or false again, where true means run the rule, which is the default behavior and false means disable the rule. So for this particular example, we pass button name and label and set them false in the yaml file. So that's how you would disable certain rules. And now to do the inverse, we want to run only a certain set of rules. So now let's say that instead of disabling button name and the label rules, you wanted to enable them and make them the only rules that run. So for ax core, we once again will pass an object to ax.run. That options object takes now the run only property, whose value is either a rule name or an array of rule names to run. So in this case, we would pass an array of button name and label, and then that will make ax only run those two rules when it returns results. For the CLI and test frameworks, what you would do is you would initialize ax builder once again, and this time chain the function with rules. And the with rules function will take a single rule name or an array of rule names and then only run those when you call analyze. For ax linter, unfortunately, we don't have a way for you to enable only a certain set of rules. So the only way that you can do this is you'll have to disable all other rules and then only enable the ones that you want. So ax linter has about 33 rules, so you would have to list all 33 rules, put false for 31 of them, and then true for just button name and label rules. Then to find the list of supported rules, you can go to the VS Code ax linter extension page. So now we want to run only rules that match a particular tag. So what is a tag? First, tags are ways to categorize rules into groups. In most cases, they're used to group rules by their WCAG version and conformance level. So for example, the WCAG AA tag is used to group rules that pertain to WCAG 2.0 AA conformance. To see which rules are associated with which tags, you can go to ax core's repo and look at the docs slash rule dash description MD file once again. For a list of what tags are supported, you can go to ax core's GitHub repo, to the docs directory and look at the api dot MD file. In there, we have linked a list of all supported tags. The quick rundown for those supported tags will usually be WCAG 2.0 AA and AAA, as well as WCAG 2.1 AA and AAA. Now for ax linter, again, they support only a subset of tags, which I believe are WCAG 2.0 AA and AA and WCAG 2.1 AA and AA. So ax linter does not currently support any AAA rules. For ax core to run only a certain set of rules, we once again will use the ax dot run, passing an options object and using the run only property once again. But this time, instead of passing a set of rules, we will pass a set of tag names. In this case, the WCAG 2 AA and the WCAG 2 AA tags. Now a thing to note about these tag names is that ax will only run tags that match a specific tag. Ax will only run rules that match a specific tag. So for example, let's say you wanted to run all WCAG 2 AA conformance, which normally would mean that you've also fulfilled WCAG 2 AA conformance. But ax will not run WCAG 2 AA rules if you only pass the WCAG 2 AA tag. It has to be a one to one match. So if you wanted to run all rules to conform to WCAG 2 AA, you must pass the WCAG 2 AA and WCAG 2 AA. Now for the test frameworks, you would again initialize ax builder and then use the chainable function with tags and with tags takes a single or an array of tags. And then once called analyze, only those rules would run. For ax linter, you would use the tags property, which is a list of tags to run, and then you would list whichever tags you wanted to run. Lastly, I wanted to talk about some of the various options that ax allows you to pass when you run it. A list of all available options can be found on the ax cores api doc file. A note about run options. Run options are only supported in ax and its integrations, and they are not really supported in ax linter with two exceptions. So a short list of run options. We've already talked about the run only and the rules properties. They have similar things in ax linter, the run only being the rules or the tags properties. But we also support a few other things. So for example, we can support passing the result types property, and the result types lets you limit which results are shown. So either pass results, incomplete or violations. You can also pass the iframes option, and that will tell ax to run or not run within iframes. By default, an ax integration will run in all iframes on the page. So for an example, let's say that we only wanted to have the violations results shown. So we would pass the result types object to ax.run. And then it's an array, and we would pass the violation string. And what this will do is when ax.runs, you will get back a list of all the rules still. But for those rules that didn't match the particular types, they will only show one node. So for any passes in this example, only one node will show a pass for that rule. What that allows you to do is it's useful for improving performance on very large pages or very complicated pages where you're only interested in a certain type of result. For the test frameworks, you would use the options chainable function to pass the list of options that way. And that is how you configure ax. Thank you for coming to this presentation. If you want to get a hold of Deque, you can do so at any of these places. You can use their Twitter at Deque Systems. You can find them on GitHub at Deque Labs. You can also connect on LinkedIn at Deque-Systems-Inc. And also on YouTube at Deque-Systems-Inc. Thank you. Amazing talk, Steven. You definitely win me over by preaching about web and accessibility and showing us how easy sometimes might be. Or easier, not necessarily easy for everyone, but easier. Thanks so much for that. And I'm really curious to see the results from the pool. Let's welcome Steven and debate a bit about it. What are you expecting in these numbers, Steven? Yeah, thanks for having me. I didn't hear you well, sorry. I said that, and thanks for having me on. Welcome. Of course, it's our pleasure. What do you think about the results? 65% replied with no. Not unexpected. accessibility testing is hard to get started in a lot of enterprise type systems that don't have them already there. Indeed, indeed. And I see a good chunk of people already using X or X, I was literally wondering to ask you how we pronounce it better at the beginning. Or the accessibility insights. And zero for the rest. I will put the zero on the two other tools. I don't maintain them, so they're all great tools. As long as they're using something. Indeed, indeed. I agree that, I mean, it's not under you, but I am surprised that WAVE has like zero. It's one of the most how to some media ties or promoted tool, especially when you research how to test accessibility. That's why I am a bit surprised. That's true. Yeah, indeed. Okay, then let's see how we do with questions. And I do have really a personal one, maybe, to tell you how was what was the moment that triggered this accessibility passion, maybe, or exploration for your side? That's a good question. A lot many years ago, when I was doing research on how to improve the user experience, I stumbled upon articles on accessibility. And it just kind of struck me as interesting, how to, you know, never really heard about it before. And so just digging into it over the years, I've just gotten more and more interested into it and wanted to keep pursuing it forward. Yeah, that's really nice. I was asking you this because in my case, like you, I was trying to, I was trying to learn it, but I wasn't necessarily driven by something specific. So I once could not choose a payment because of the color contrast. And then it was for me, I'm done. We need to bridge this more, we need to teach people about it. And I even interviewed developers on how much time they invest creating all these in a more accessible form. And to my surprise, the common answer that I got, it was like, Ioana, it's just another library, just another tool that you use. Once you learn it, you just apply next time. It's not the cost at all the project. And that is my preaching moment for the developers and managers that don't believe that they have time or money to budget to invest. And after that, for the users, I do mention all the benefits that even non-needing people will benefit from accessibility because at the end of the day, we benefit from better web, the same. Yeah. Okay. Thanks for joining me in this intriguing and curiosity that I had. Milad was asking, in your presentation, you configure X for text frameworks. Should we use X-gest or X for right to left? So the ones that AX and the DQ systems maintain are all under the at AX-core npm namespace. So I believe AX-gest is not maintained by DQ. But if you are using gest, and I believe it's using AX under the hood still, it's probably still good to use. I can't say what its api will be, though, for how to configure AX under the hood. Okay. Great. When using islint-plugin-js-accessibility in react, do you think it's necessary to also write the end to end test with AX? I guess it depends on how you're doing your testing. A lot of times when you're doing AX tests, I've seen them done as testing/talks">unit testing. So you're testing the individual pieces, making sure they're all accessible. That works for a good set of the rules AX runs, but there are rules that are most focused on how the whole page works together. So for example, we have a rule for heading order, which makes sure that you have your H1 through H6 tags in a correct order. And those can only really be run on a full page. So those are still probably good to use as an end to end testing. They'll just kind of catch your page level type accessibility issues. Interesting. AX and typescript. Is there a typescript api for AX, or typings are definitely typed? Yes. All of our test framework integrations are written in typescript. AX has a typescript definition file that you can use. But AX itself is not written in typescript. And AX and suppressing warning. Is there a way to suppress warning? Because I don't want to fix it, or because it's just not possible. There are false positive reports. If you do encounter false positives, we would love it if you reported it on the AX core GitHub issues page. We try to put very high priority on fixing false positives. In terms of ignoring warnings, there's currently not a way to do that unless you're using the AX extension and you're paying for a license. In which case, there is a way to ignore results that you don't care to see anymore. Oh, indeed. And also, one of the curiosities is how you introduced this accessibility testing in your company? A lot of curious people. So at prior companies, being a developer, it wasn't too hard to introduce it on my own code. So you can kind of just sneak it in there, and then your code as a unit test is running AX. In terms of trying to get that integrated, you... Let's see, what was it? In the last Q&A session, someone was talking about... Oh, no, you were just mentioning it. Yeah. Trying to convince them that, hey, we can do this. I've already done it. It's not hard. It's now kind of just free to bring into a unit test. So anytime you want to test your code, it's there. And then explaining all the benefits to users and really finding a user who struggles with the app or the web page and showing them trying to use it is a great way to kind of get people to recognize, hey, maybe we need to take this seriously. Indeed, indeed. And one of the things that I use to show people that there are users, and probably so many other people, is the example with the plane from the World War II, when definitely you're looking at the wrong database of people. I miss how it's called now. I think something with bias. Yeah, survivorship bias. Yeah, exactly. Survivors bias. Indeed. And people say, no, we don't have users. Of course. We never tried. What helped or triggered them very fast was to find a solution for their daily stuff that applies to them, but both for accessibility. An example, you know, with real life sidewalks that need to have the ramp for disabled people, but are used for running too. Bicycles, scooters, mothers with kids or parents with kids in trolleys. Again, a solution that helps everybody, not necessarily dedicated to fix an accessibility issue. Yeah. That's cool. I would like to ask you, how is your experience with the developers? How do you feel the checking that they do, if you tested with them, how do you feel the checks done, the scans that are done automatically with developer tools can? How do you think this brings awareness? How is that? Or is like you feel it's too little or enough to start if you try them? Yeah. So accessibility tests, at least automated accessibility tests can only catch about 50 to 60% of accessibility errors, which introducing that on a slow bit is better than not catching any errors. So any accessibility testing is better than none. But you always then have to get a manual test, someone who understands the remaining 40% of errors to come and verify if you really want your page to be truly accessible and ADA compliant and all that. Yeah, indeed. Well, I first started with these scans and at least on Firefox, because I work with Mozilla, they will give you even a link to MDN and will explain all the levels that they have for different stuff. But they only cover as much as can be automating easily, like color contrast links. Of course, like navigation tabs, what else? Something. Yeah. But like voiceovers were super hard to test on a mobile phone. How you test that accessibility stuff. And oh, another good example to show why accessibility is important is what I found for myself is to have captions at videos on the phone, because you're going to listen to something. Maybe you're not having all the time headphones or just it's easier to see text when you're like watching something that to disturb anyone or you're with a kid and you want to have. So those helped me to from the phone and I found them by mistake and after that I used them. Again, great. Great. What what else or which of these accessibility issues would you like to be more implemented or more known and thought with solution? I guess the most common there's a few common accessibility issues that if there was more awareness around them, you could solve the most important thing a developer could do to help with accessibility is use semantic html. So there's the term called divitis or div soup where we just make everything divs and then add on the functionality we need. And then that doing that, though, makes it really easy to forget all the native benefits of accessibility that things like a button or a link already have on them. So just by using semantic html, you can cover a wide slew of accessibility problems and make sure those are fixed. Next to that, probably the next important issue would be making sure your web page is just usable by a keyboard alone. Try testing your page, go to your keyboard, press tab, make sure everything that is clickable can be tabbed to that you can activate it with the enter or the space because most accessibility software or assistive technologies rely on a keyboard type of focus system. So you can cover a lot of problems as well by making sure that your page is keyboard accessible. So those those two things are probably the most important starter points. Indeed, I agree. And one question more about X, the focus seems to be on the blind, but there are any checks for motoric disabilities. The button must be big enough to be hit with shaky hands, for example. So in terms of how to automate test something like that, it'd be the WCAG, which is kind of the definitive list of what makes things accessible for compliance. They have implemented a new version, WCAG 2.1, that introduced a new rule about what they call target spacing. So that says that any button or anything that's clickable needs to be a certain size and that it needs to have a certain amount of padding around it. And that is probably the closest you could get to something like shaky hands or tremors in terms of making sure your page is accessible towards that. But there's not much you could do automated wise. That's probably more left to a manual tester to see if there's enough space around the area to make sure that if you're having tremors that you won't click on anything else accidentally. Indeed. Oh, thanks so much for the answer. I really was interested too. Especially because in testing, we always consider from a UX point that maybe someone will have a stronger finger or just shaking. Thanks so much for your amazing talk. I do invite everyone to join your chat room on spatial.chat and see around. Also tweet your solution for accessibility. I'm curious to read it more. Don't forget to rate also the talk. Thanks so much again, Stephen.