Accessible CI/CD

Rate this content

CI/CD has been essential for any productive web product development and release. However, while accessibility is always an important aspect for any Web product included UI, it is often overlooked or considered as a time-consuming manual step, outside of the CI/CD flow. How can we automate accessibility testing within our CI/CD for better developer experience and team collaboration? What tools can we use to integrate and leverage accessibility compliance in our CI/CD? Join my talk and let's find out.

24 min
15 Feb, 2024

Video Summary and Transcription

Maya Min, a senior software engineer at Microsoft Industry AI, discusses the importance of accessibility testing and its relation to CIDI. WCAG provides guidelines for accessibility compliance, covering various aspects such as color contrast, navigation, and content layout. Maya explores automating accessibility testing through UI components and different testing levels. They recommend xCore and Playwright for end-to-end browser testing and integrating accessibility testing into CI/CD workflows using tools like GitLab and Azure Pipeline.

Available in Español

1. Introduction to Accessibility

Short description:

Hi everyone. Welcome to my talk on accessibility in a different context at CIDI. I'm Maya Min, a senior software engineer at Microsoft Industry AI. I've been in the industry for over 10 years, focusing on web development and frontend. I recently released a book called Learning View with Aurelie, which teaches frontend development using Vue and TypeScript. Let's discuss why accessibility testing is important and how it relates to CIDI. Accessibility ensures that users can perceive and interact with web applications. WCAG provides guidelines for accessibility compliance.

Hi everyone. Welcome to my talk and let's discuss accessibility in a different context here at CIDI, shall we?

Okay. But first and foremost, a bit about myself. My name is Maya Min. I'm a senior software engineer at Microsoft Industry AI. Our group is focusing on leveraging different LM technologies to develop AI-integrated solutions and applications for industry specific. I've been in the industry for more than 10 years, focusing on the web and the front end. And lately, I just released a book called Learning View with Aurelie, which this book will teach you and guide you how to develop web applications using view as a front end framework in the context of TypeScript. And if you're interested in learning a new awesome framework, check it out. I'm also a Cloudinary ambassador, Google developer expert, and also an organizer of the View.js Israel community. You can follow me on Maya Chavin in LinkedIn or Twitter or follow my blog posts at And that's enough for myself.

Let's go on to our first question. We know about accessibility. We know what accessibility is important, but why accessibility testing and why it has to put in the context of CIDI. Okay. First and foremost, try to click on this button. You can, right? Because this button, even though it looks like a real button, it has the text click here to continue. It's not a real button. It's just an image. And this is exactly what we talked about when we mentioned accessibility. The button is not accessible. No one can use it. It means we have an accessibility issue. So here comes the definition of web accessibility. So web accessibility or accessibility is how you see, how a user can hear, and how a user can perceive your application so that they can understand, or at least come up to some understanding how to use, navigate, or interact with your web applications. This is the full flow that's similar to user experience in a lot of sense. And when I say accessibility, I usually say it's user experience because it's including accessibility inside. And but based on this, it's very hard to decide what is the compliance level for accessibility. This includes a lot of things here, listening, viewing, or understanding, right? So for that, there's a group that WCAG come up with a very, it's a standard guideline of compliance for accessibility.

2. WCAG Compliance Standards

Short description:

WCAG provides guidelines for accessibility compliance. It includes color contrast, navigation, focus, landmark, content layout, zoom levels, understandable content, form handling, media, typography, and link behavior. These are essential, but there are other factors to consider depending on the complexity of your application.

They've been working very hard and it's become an industrial standard for enterprise and public domain to follow and to reflect on. So the normal level for WCAG, acceptable level for WCAG, accessibility compliance for enterprise and public domain is the version 2.1 level EE and above. So there's 3 level AA and AAA. But there's a lot of things mentioned in their website, and you may feel a bit overwhelmed when you go to the website because it's including a lot of explanation. It's very detailed explanations. So in this talk, I will just summarize the most common compliance standard and check that we need to follow. Color contrast, navigation, focus, landmark, content layout, 200% for 400%. You may think that 200% and 400% is not that common. In fact, it's very common because it included how the users zoom in and zoom out your website on a small device machine, like a small screen laptop. And 200% and 400% can define the maximum level the user can zoom in for your applications and make sure that your content stays consistently and usable even when users zoom in in that such case. Understandable content, form handling, media, typography, external internal link behavior. If you have an external link, which means the user will be redirected to a new page, open a new tab, you need to somehow inform the user that this is going to be navigating to a new website and the user won't be taken as surprised. Tab order and many more. Yes, many more. These are just an essential set of things to follow. And there's other things depending on how complex your application would be.

3. Automating Accessibility Testing

Short description:

Can we automate accessibility testing? Let's look at UI components, using atomic design. Testing is divided into different levels: atoms, molecules, organisms, and pages. For each level, we focus on specific accessibility aspects, such as color, semantics, navigation flow, layout, and content flow. Component testing includes labeling, attributes, semantics, focus, color contrast, typography, and form labels.

So that comes to the next question. Can we automate accessibility testing? Well, let's take a look at our components in UI applications. I like to use my favorite design pattern, atomic design, where we break components into smaller and smaller components to the lowest level. So we have atoms, which is the smallest isolated piece of core or component that is stand-alone and can be used for building a much more complex level of components like molecules. Molecules, built on top of atoms, can be used to build more complex organisms. So atoms, thinking like a button, an icon. Molecules, thinking like a form. And organisms, thinking like a gallery where you have a list of items and each item has another component. And then you also have field search, and pagination, and so on. And from this organism, you can build more and more complex, a full page. Page is less likely a component, but it's more like a stand-alone that you can do performance testing and end-to-end testing on this level.

So for accessibility compliance and accessibility testing for components, we also break it into according to different levels of a component. So for atoms, we don't really see the needs of testing everything. So for atoms, we only test color, focusable, semantics, HTML usage, informatic content, and typography. All the things that can be test isolated. It doesn't need to interact with other components. It can be interacted with by itself. So we check at this level only such things. When we move to molecules, which already have the interaction between one item and another item, we start to care about navigation flow, layout landmark, and focus flow. And if it's table and form, we also need to check if the table and form are built correctly when you do a navigation between one cell to another cell. How does it look, and how does it feel at the user? And for organism, you kind of repeat what molecules check for accessibility was, but you also add another level called content layout. Content layout is similar to landmark usage where the content, how the content flows, whether it flows correctly, whether it's understandable or not. And for pages, we just repeat what we test in our organization because they are very much the same, just a bit more complex. So for all of this, accessibility testing for component levels are divided. We come up with two sets of testing for accessibility. The first is component test, and the second is flow or user preference test.

The component test including what areas are labeled, whether, sorry, attributes, whether we're using the correct area label and attribute, and information. Semantic usage, are we using the right semantic or are we using deep for something that's not supposed to be deep? Focusability, color contrast, this is more about text versus background. Typography, accessible label, whether we have eternity accessible label for an image like an art text and so on. Form labels, each for each input have to have a unique form, and it cannot be missed.

4. Automated Accessibility Testing

Short description:

Component testing focuses on label attributes, semantic usage, content, and contrast. Automation testing is suitable for component testing, while flow and user experience testing require manual testing. xCore is an efficient open-source JavaScript package for automated accessibility testing. It allows configuration of rules and supports different frameworks and testing frameworks like React and Cypress. Chess is a convenient package that wraps around xCore for unit testing. It simplifies the process by providing a custom chess matcher. For end-to-end testing, there are broader considerations.

Do we have enough form to tell user what the input doing is? Do we have enough label to tell user what the input doing? Well, this is component test. The second one is flow and user preference test, where we do a bit more about user preference. Navigation flow using keyboard. Focus flow, when we do focus, you do tab navigation from one interactive element to another, what is the focus landing, where the focus flow that doesn't make sense to user. Content quality, screen reader support, where you need to hear what the screen reader is saying about your element on your page and see whether it also makes sense and include all the information that it needs to be included. Image versus text contrast on a banner, this is much harder to test compared to the text versus background, and error flow and landmark.

So based on this division, we now can understand more or less which one is more suitable for automation test and which one we cannot really automate, we have to do manual test. So for component test, we can see that most of them are just checking about label attributes, semantic usage, content, the very descriptive content, and check about the contrast of text and background, so on. For that, we can easily achieve with automated test, including unit test or integration end-to-end test. For flow and user experience, it's a bit more tricky. It's not that straightforward because it relies a lot on how the user and how human analyze your website and your flow. So for this, you have to do manual test, and manual tests incur in smoke test, post-production, and design evaluation before developer actually going on and develop a feature or component. This design evaluation is happening at the phase where designer design the whole flow and getting the feedback from developer before they hand over to developer. Okay, for automated testing, we have a very nice tool called xCore. It's an open-source JavaScript package that was developed by Deque, which is an organization that's dedicated for accessibility. And this xCore has been used in Google Chrome and Edge, adapted by Chrome and Edge in the analyzer for the browser and the web performance, such as Lighthouse. And it's proven to be very, very efficient. It allows you to config different types, set of rules for accessibility check. And it also allows you to config what level of WCG standard you want to test on your applications, depending on your needs. And it's always up to date so you shouldn't be afraid and will go out of date. Okay, and lastly, it has a very wide support for different frameworks and testing frameworks, such as React and ClearWire and Cypress. So you easily can create an end-to-end test integration with these extensions. So for xCore, it's an open-source JavaScript library for accessibility testing. You can use it also at the unit testing in your unit test suite. However, it's for convenience that you can use Chess, which is an open-source package, npm package, that already wraps around xCore and provides also the custom chess matcher. And you don't have to write your own matcher to fit xCore standard. And this project is pretty easy, pretty straightforward to use. So you include it in your chess unit testing system in your applications and install it with ChessDom to make ChessDom mimicking simulations for your test. And then you can use the render functions to render the component that you want to test, or the page you want to test, and pass the component to the x function and wait for it to assert whether it has any violations or not. For end-to-end tests, we have much more things that are wider than just support.

5. Accessibility Testing in CI/CD Workflow

Short description:

End-to-end accessibility testing with xCore and Playwright allows for real-world browser testing across different devices and browsers. It provides detailed information about accessibility violations and can be used for unit testing as well. Adding accessibility to the CI workflow can be done using GitLab's accessibility testing workflow or by integrating PA11Y framework for end-to-end testing in other CI/CD workflows.

Because end-to-end accessibility tests are mainly meant for end-to-end testing, where we have to see the actual layout and we can do the testing on actual layout, similar to how users perceive your application. So for end-to-end tests, for xCore, it has a package that goes with Playwright, Playwright.js, which is my favorite end-to-end testing framework because it's very fast and free and it's open source project. And it can run in a lot of different languages, not just JavaScript.

For Playwright, it also has a dedicated section for accessibility testing, how to integrate xCore in accessibility testing. So to use xCore in Playwright, we will install xCore xPlaywright package and we'll just import the xBuilder, which is the class function that gives us the instance, include only wrap around the page, and then we can check, we can analyze, we can trigger the function, analyze on it. So first, we will go to, we will navigate our test to the page that the target page in this test is a homepage and then we will wait for the xBuilder to scan and analyze the page instance that we will just go to, and then we finally assert if there's any evaluation or not. Okay, so using xCore and Playwright, we can cover the real-world browser testing, including different devices, different browsers, and see if it actually works and how it looks. We can also extend it, not just see if there's any violation, we can also extend it whether a component is visible, an element is visible, an element is focusable, because it's mimicking user actual experience.

So when you can also see if an event handler, when you click on it, what will happen, the dialogs open as expected on the page level accessibility rule check and you can also do the third-party widget app validations and color contrast, full color contrast, and add fully automated with predefined set of rule. So that is xCore with Playwright, and this is the example of how the violation would look. It will be an array of violations where it will give you some detail, the impact level, and all the information about the elements that have the violation. Okay, and that is xCore with Playwright as end-to-end test and also as unit test.

What about adding accessibility as part of the CI workflow? So if you're using GitLab, there's a very nice feature for accessibility testing which is accessibility testing workflow. It's included in your GitLab workflow. What you need to do is just download the template and include it in your applications, the YARN template as the workflow, it's on workflow for accessibility, then include it into your main workflow YARN plate. In here, you can define an accessibility stage and we can also provide some variables including all the URL you want to test for your application on accessibility and include the template. The template is working and in the back it will be on top of PA11Y framework for end-to-end testing, and this framework is a very convenient framework that has a lot of different packages that allow you to create your own custom dashboard for accessibility where you can see the violation and different level compliance and it's open source of course. So that is about GitLab CI workflow.

We're talking a lot about accessibility testing, automated testing, manual testing, what we can use to put an end-to-end and unit test. So where do we plug this in the CI CD workflow if we don't use GitLab workflow? What is CI CD? So CI CD stands for continuous integration and continuous deployment. So this is an example of how the flow goes between code development to continuous integration and continuous deployment. So literally as you can see here in the code development, we have a single source control where all the code will stay there and then developer can work on top of it with branches and merge back to the source. It can be GitHub, it can be GitLab, it can be Azure DevOps repo which they build on top of Git repository and the four levels of CI CD integration, the source, the build, the test, up the deploy.

What in the source we're talking about is the course version control. We also have some pipeline run on commit up here to make sure that nothing break or uses unit test whoops before we merge it back to source so we can control the quality of the source code. We also have coding standard operations such as LinkedIn, such as checklist, such as code review. All of this will make sure that our code is at least at a certain standard before we move to build. So build, we combine the actual source code into the actual consumable package such as NuGet package, such as ESC, such as JAR and a OS minify file so that we can deploy to the cloud for a web application. And in this build version, we're going to run a set of unit tests on top of the ready source code so we know that the source is stable enough to build. And then after we build, we will deploy these to a testing environment and then on that, we can perform more complex tests such as end-to-end test, integration test, performance and security test.

6. Accessibility Testing in CI/CD Deployment

Short description:

Accessibility testing is an integral part of the deployment process, including smoke tests, code standard evaluations, unit tests, end-to-end tests, performance tests, and smoke tests. It can be integrated into CI/CD workflows using various tools like Travis CI, Circle CI, Azure Pipeline, and others.

And finally, if things go well, we deploy to productions and the part of deployment have to be fully automated. And post deployment, we also have some tests called smoke test or where we actually have user feedback about product and we can do manual testing to see if in production it still look good and nothing break for user experience.

So where does accessibility testing fall into? So you can see here, it fall into all the highlight sections here. Code standard evaluations, unit test because we talk about unit tests for accessibility, end-to-end test, performance test and smoke test. Code standard evaluation is more about pre-code development and about co-review and collaboration between designer and developer. Unit test, end-to-end test, performance test is something we only talk about when we can plug in using Xcore or ChessX or Xcore with PlayLive for end-to-end test. And smoke test is manual testing on the user side or on the QA side for our product. That's how we integrated accessibility testing inside and we can build a CI-CD flow using Travis CI. Any Travis Circle CI checking Azure pipeline, any workflow, CI-CD workflow tool that can help you create a full flow for your application.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

DevOps.js Conf 2022DevOps.js Conf 2022
27 min
Why is CI so Damn Slow?
We've all asked ourselves this while waiting an eternity for our CI job to finish. Slow CI not only wrecks developer productivity breaking our focus, it costs money in cloud computing fees, and wastes enormous amounts of electricity. Let’s take a dive into why this is the case and how we can solve it with better, faster tools.
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
End the Pain: Rethinking CI for Large Monorepos
Scaling large codebases, especially monorepos, can be a nightmare on Continuous Integration (CI) systems. The current landscape of CI tools leans towards being machine-oriented, low-level, and demanding in terms of maintenance. What's worse, they're often disassociated from the developer's actual needs and workflow.Why is CI a stumbling block? Because current CI systems are jacks-of-all-trades, with no specific understanding of your codebase. They can't take advantage of the context they operate in to offer optimizations.In this talk, we'll explore the future of CI, designed specifically for large codebases and monorepos. Imagine a CI system that understands the structure of your workspace, dynamically parallelizes tasks across machines using historical data, and does all of this with a minimal, high-level configuration. Let's rethink CI, making it smarter, more efficient, and aligned with developer needs.
DevOps.js Conf 2024DevOps.js Conf 2024
25 min
Atomic Deployment for JS Hipsters
Deploying an app is all but an easy process. You will encounter a lot of glitches and pain points to solve to have it working properly. The worst is: that now that you can deploy your app in production, how can't you also deploy all branches in the project to get access to live previews? And be able to do a fast-revert on-demand?Fortunately, the classic DevOps toolkit has all you need to achieve it without compromising your mental health. By expertly mixing Git, Unix tools, and API calls, and orchestrating all of them with JavaScript, you'll master the secret of safe atomic deployments.No more need to rely on commercial services: become the perfect tool master and netlifize your app right at home!
DevOps.js Conf 2021DevOps.js Conf 2021
33 min
How to Build CI/CD Pipelines for a Microservices Application
Top Content
Microservices present many advantages for running modern software, but they also bring new challenges for both Deployment and Operational tasks. This session will discuss advantages and challenges of microservices and review the best practices of developing a microservice-based architecture.We will discuss how container orchestration using Kubernetes or Red Hat OpenShift can help us and bring it all together with an example of Continuous Integration and Continuous Delivery (CI/CD) pipelines on top of OpenShift.
TestJS Summit 2021TestJS Summit 2021
30 min
Configuring Axe Accessibility Tests
Top Content
Axe-core is a popular accessibility testing engine that is used Google, Microsoft, and hundreds of other companies to ensure that their websites are accessible. Axe-core can even integrate into many popular testing frameworks, tools, and IDEs. In this advanced session, we'll be learning how to configure axe and its integrations to fine tune how it runs and checks your pages and code for accessibility violations.

Workshops on related topic

React Summit 2023React Summit 2023
109 min
Web Accessibility for Ninjas: A Practical Approach for Creating Accessible Web Applications
In this hands-on workshop, we’ll equip you with the tools and techniques you need to create accessible web applications. We’ll explore the principles of inclusive design and learn how to test our websites using assistive technology to ensure that they work for everyone.
We’ll cover topics such as semantic markup, ARIA roles, accessible forms, and navigation, and then dive into coding exercises where you’ll get to apply what you’ve learned. We’ll use automated testing tools to validate our work and ensure that we meet accessibility standards.
By the end of this workshop, you’ll be equipped with the knowledge and skills to create accessible websites that work for everyone, and you’ll have hands-on experience using the latest techniques and tools for inclusive design and testing. Join us for this awesome coding workshop and become a ninja in web accessibility and inclusive design!
TestJS Summit 2021TestJS Summit 2021
85 min
Automated accessibility testing with jest-axe and Lighthouse CI
Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.
React Summit 2022React Summit 2022
161 min
Web Accessibility in JavaScript Apps
Often we see JavaScript damaging the accessibility of a website. In this workshop, you’ll learn how to avoid common mistakes and how to use JS in your favor to actually enhance the accessibility of your web apps!
In this workshop we’ll explore multiple real-world examples with accessibility no-nos, and you'll learn how to make them work for people using a mouse or a keyboard. You’ll also learn how screen readers are used, and I'll show you that there's no reason to be afraid of using one!
Join me and let me show you how accessibility doesn't limit your solutions or skills. On the contrary, it will make them more inclusive!
By the end, you will:- Understand WCAG principles and how they're organized- Know common cases where JavaScript is essential to accessibility- Create inclusive links, buttons and toggleble elements- Use live regions for errors and loading states- Integrate accessibility into your team workflow right away- Realize that creating accessible websites isn’t as hard as it sounds ;)
DevOps.js Conf 2022DevOps.js Conf 2022
76 min
Bring Code Quality and Security to your CI/CD pipeline
In this workshop we will go through all the aspects and stages when integrating your project into Code Quality and Security Ecosystem. We will take a simple web-application as a starting point and create a CI pipeline triggering code quality monitoring for it. We will do a full development cycle starting from coding in the IDE and opening a Pull Request and I will show you how you can control the quality at those stages. At the end of the workshop you will be ready to enable such integration for your own projects.