Who is Testing the Tests?


Have you ever wondered: "who's testing the tests"? Of course, tests are only valuable if they catch bugs, but how would one validate that? Well, let me tell you about mutation testing!

Mutation testing is the act of testing your test verifying that they catch bugs. Of course, you can do this manually inserting bugs and running the tests, but a mutation testing framework can do this for you!

Join me and learn the basics of mutation testing and how to use StrykerJS, the mutation testing framework for JavaScript or TypeScript.

At the end of this talk, you'll be the one that is testing your tests, and it won't even cost you much time!

8 min
19 Nov, 2021


Sign in or register to post your comment.

AI Generated Video Summary

Mutation testing is a method to improve test quality by inserting bugs into code to test if tests can detect them. Mutation-testing frameworks like Striker.js allow for various mutations to be performed. Mutation testing provides a mutation score that is a better tool than code coverage for measuring test quality. It can help identify missing tests or bugs in existing tests. Stryker is recommended for JavaScript and TypeScript mutation testing.

1. Introduction to Mutation Testing

Short description:

Hi, I'm Simon, a software engineer at InfoSupport. Let's talk about how mutation testing can improve test quality. Code coverage is a bad metric for measuring test quality. Mutation testing inserts bugs into code to test if tests can detect them. It generates mutants and runs tests to detect failing ones. Hundreds of mutations are performed to generate a comprehensive report. An example of mutating code is changing a greater than equals sign to a less than sign.

Hi, I'm Simon. I'm a software engineer working at InfoSupport, and today I want to talk to you about how mutation testing can help to improve the quality of your tests. But before we dive into mutation testing, I first want to talk a bit about code coverage.

Code coverage is often used by developers as a means to measure the quality of their tests. In my eyes, it's a bad metric to measure this. The only thing code coverage actually tests is whether or not codes get executed. The reason I write unit tests is to verify that my code works right now and to make sure it will keep working in the future. And if something happens and my code behaves differently, I will get a failing test. And again, code coverage does not measure this, it only measures if your code gets executed.

So, how do we actually measure the quality of your tests? Well, that brings me to the topic of this talk, mutation testing. Mutation testing is a way to insert bugs into your source code to actually test if your tests can pick them up. How does this work? Well, a mutation testing framework will start with your source code, which is all just happy and fine and nothing is wrong with it. Then it will make one small mutation for your source code, this will generate a mutant. For example, A plus B could be mutated into A minus B. This will result in a different outcome for your source code. And because your source code has changed, it should cause a failing test. So, the mutation testing framework will run your tests and it can have either outcomes. Either the mutant has been detected because of a failing test, in that case the mutant has been killed, or none of your tests failed and the mutant was able to survive. In mutation testing, we want to have killed mutants. So, we want tests to actually fail because we have inserted a bug. In mutation testing framework, we'll do this hundreds or thousands of times for your source code, and in the end, we'll combine it all together and generate a nice report.

So, how does this mutating work in actual code? Well, here I have a small JavaScript function on the top to check if a customer is allowed to buy alcohol. This customer is allowed to buy alcohol in this country if the age is at least 18. And below I have a test for Professor X, who is age 96 and is thus allowed to buy alcohol. In mutation testing look at the source code above and make a small change. In this case, the greater than equals sign will be changed into a less than sign, thus flipping the check around. If we make this change in our source code ourselves, the test will fail, thus the mutant has been killed. This proves that we have a test for this specific case and this bug cannot be inserted into our code without us knowing it. We can also change it in another way. For example, we can change it into a greater than.

2. Mutation Testing and Frameworks

Short description:

We started with greater than equals, now it's greater than. Changing the entire return statement to always return 2 also passes. Mutation-testing frameworks allow for various mutations like changing signs, emptying strings and arrays, and flipping operations. Striker.js is recommended for JavaScript and TypeScript, but there are options for other languages as well. A small demo of striker.js is available on our website. Running mutation tests can be slower for large applications due to the number of mutations being tested. The demo application generated 126 mutants from 12 source files.

We started with greater than equals, now it's greater than. Since we only have one test case for someone aged 96, all our tests pass and the mutant has survived. We can also change the entire return statement and just simply always return 2. In our case, it still passes and we have a surviving mutant.

This is very simple code. Most of our code does not look like this and most of our tests are also more complicated. While someone looking at this code might be able to say, hey, obviously you're missing some test cases, in the production code that we're writing, it's often a lot more difficult.

So, what kind of mutations can you expect for mutation-testing frameworks? Well, the definite list depends on the framework and the language that you're using. But in most cases, mutations like this are possible. For example, changing the plus sign into a minus sign, emptying strings, emptying arrays, flipping operations around, you can do a lot of things as a very small change in your code. And only one of these changes will be active at a time to ensure we know which mutation is causing a test to fail.

Well, mutation-testing is available for a lot of different frameworks. And since we're at a JavaScript conference, I would recommend striker.js as the framework to use for JavaScript and TypeScript. But it's available for pretty much every language that you know. If it's not on this list, simply Google it for your language and you will probably find something.

So, for striker.js, I have a small demo. The demo application that we have is also available on our website. The link will be at the end of the slides with information how you can set this up yourself. I can start striker from the command line. And it will start running our tests. Since the demo application is a small application, it's quite fast. If you have a large enterprise application, you will most likely notice that mutation testing is quite a bit slower than running the unit test. Because there are thousands of mutations being made and each and every one of them has to be tested. So, my laptop is starting up right now. It's testing. And in about a second or two, it will actually be done. For our application, it has 12 source files to mutate and it was able to generate 126 mutants. So, it's done. We get a nice table here as an output. But we also have a HTML report.

3. Importance of Mutation Testing

Short description:

Mutation testing provides a mutation score based on the number of mutants killed. It is a better tool than code coverage for measuring test quality. Looking at specific mutations can help identify missing tests or bugs in existing tests. I recommend trying mutation testing with Stryker for JavaScript and TypeScript.

And if we look at this report, we can see that we have a mutation score. That's the amount of mutants we were able to kill. The more mutants you killed, the higher your score. And just as with code coverage, you probably want a high score.

In this case, we have scored about 85% of our application. Well, if we look at code coverage for this application, we would have 100% code coverage. So, if we only use code coverage as a metric, we would be fooled in thinking we tested this properly.

Now if we look at specific files, you can also look at specific mutations, for example, the H-check here, and we notice that we have an equality operator that survives. To me this indicates that we are missing a test because we were able to change the greater than sign to greater than equals. So, in a report like this for your source code, you can actually figure out what kind of tests you are missing or if you have any bugs in your tests.

So, to summarize, mutation testing in my opinion is a far better tool to measure the quality of your tests. I would highly recommend you to try it out. For JavaScript and TypeScript, I recommend Stryker. There's an example available on our websites.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
34 min
Network Requests with Cypress
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.

TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!
But how do we do that?
Are the "unit" and "integration" terminology serves us right?
Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?
In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!
It’s time to go DEEPER!

TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2021TestJS Summit 2021
36 min
Effective Performance Testing to your Server with Autocannon
Performance testing expertise that is developed for a long time. In order to measure your server performance you need a tool that can efficiently simulate a lot of abilities and give you good measurements according your analysing criteria.
Autocannon NPM library gave me exactly that - that library is super easy to install and has a very simple API to work with. Within a really short amount of time you can start do performance testing to your application and get good measurements in development environment and in your performance labs, and generate complicated testing scenarios.
In this talk I will introduce Autocannon, explain how to efficiently analyse your server performance with it, and show how it helped me to understand complicated performance issues in my Node.js servers. At the end of this lecture, developers will be able to have the ability to integrate a fast and easy tool in order to measure your server performance.

Workshops on related topic

React Summit 2023React Summit 2023
152 min
Designing Effective Tests With React Testing Library
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents
- The different kinds of React application tests, and where component tests fit in
- A mental model for thinking about the inputs and outputs of the components you test
- Options for selecting DOM elements to verify and interact with them
- The value of mocks and why they shouldn’t be avoided
- The challenges with asynchrony in RTL tests and how to handle them
- Familiarity with building applications with React
- Basic experience writing automated tests with Jest or another unit testing framework
- You do not need any experience with React Testing Library
- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
147 min
How to Start With Cypress
Workshop Free
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
118 min
Detox 101: How to write stable end-to-end tests for your React Native application
Workshop Free
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and
used by React Native itself
as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
- iOS/Android: MacOS Catalina or newer
- Android only: Linux
Install before the workshop
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
Workshop Free
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.

TestJS Summit 2021TestJS Summit 2021
85 min
Automated accessibility testing with jest-axe and Lighthouse CI
Do your automated tests include a11y checks? This workshop will cover how to get started with jest-axe to detect code-based accessibility violations, and Lighthouse CI to validate the accessibility of fully rendered pages. No amount of automated tests can replace manual accessibility testing, but these checks will make sure that your manual testers aren't doing more work than they need to.