TestJS Summit

TestJS Summit 2021

<div>Have you ever wondered: "who's testing the tests"? Of course, tests are only valuable if they catch bugs, but how would one validate that? Well, let me tell you about mutation testing! Mutation testing is the act of testing your test verifying that they catch bugs. Of course, you can do this manually inserting bugs and running the tests, but a mutation testing framework can do this for you! Join me and learn the basics of mutation testing and how to use StrykerJS, the mutation testing framework for JavaScript or TypeScript.At the end of this talk, you'll be the one that is testing your tests, and it won't even cost you much time!</div> <a href="https://www.froala.com/wysiwyg-editor?pb=1"> </a>

Who is Testing the Tests?

Hi, I'm Simon. I'm a software engineer working at InfoSupport. And today I want to talk to you about how mutation testing can help to improve the quality of your tests. 
But before we dive into mutation testing, I first want to talk a bit about code coverage. Code coverage is often used by developers as a means to measure the quality of their tests. In my eyes, it's a bad metric to measure this. The only thing code coverage actually tests is whether or not codes get executed. The reason I write unit tests is to verify that my code works right now and to make sure it will keep working in the future. And if something happens and my code behaves differently, I will get a failing test. And again, code coverage does not measure this. It only measures if your code gets executed. So how do we actually measure the quality of your tests? 
Well, that brings me to the topic of this talk, mutation testing. Mutation testing is a way to insert bugs into your source code to actually test if your tests can pick them up. How does this work? Well, a mutation testing framework will start off with your source code, which is all just happy and fine and nothing is wrong with it. Then it will make one small mutation to your source code. This will generate a mutant. For example, A plus B could be mutated into A minus B. This will result in a different outcome for your source code. And because your source code has changed, it should cause a failing test. So the mutation testing framework will run your tests and it can have either one or two outcomes. Either the mutant has been detected because of a failing test. In that case, the mutant has been killed. Or none of your tests failed and the mutant was able to survive. In mutation testing, we want to have killed mutants. So we want tests to actually fail because we have inserted a bug.
A mutation testing framework will do this hundreds or thousands of times for your source code. And in the end, it will combine it all together and generate a nice report. So how does this mutating work in actual code? Well, here I have a small JavaScript function on the top to check if a customer is allowed to buy alcohol. This customer is allowed to buy alcohol in this country if the age is at least 18. And below I have a test for Professor X who is age 96 and is thus allowed to buy alcohol. In mutation testing framework, we'll look at the source code above and make a small change. In this case, the greater than equal sign will be changed into a less than sign, thus flipping the check around. If we make this change in our source code ourself, the test will fail. Thus the mutant has been killed. This proves that we have a test for this specific case and this bug cannot be inserted into our code without us knowing it. We can also change it in another way. For example, we can change it into a greater than. We started with greater than equals, now it's greater than. Since we only have one test case for someone age 96, all our tests pass and the mutant has survived. We can also change the entire return statement and just simply always return true. In our case, it still passes and we have a surviving mutant. This is very simple code. Most of our code does not look like this and most of our tests are also more complicated. So while someone looking at this code might be able to say, hey, obviously you're missing some test cases, in the production code that we're writing, it's often a lot more difficult. So what kind of mutations can you expect from mutation testing frameworks? Well, the definite list depends on the framework and the language that you're using, but in most cases, mutations like this are possible. For example, changing the plus sign into a minus sign, emptying strings, emptying arrays, flipping calculations around. You can do a lot of things as a very small change in your code. And only one of these changes will be active at a time to ensure we know which mutation is causing a test to fail. Well, mutation testing is available for a lot of different frameworks. Since we're at a JavaScript conference, I would recommend StrikerJS as the framework to use for JavaScript and TypeScript, but it's available for pretty much every language that you know.
If it's not on this list, simply Google it for your language and you will probably find something. So for StrikerJS, I have a small demo. The demo application that we have is also available on our website. The link will be at the end of the slides with information on how you can set this up yourself. I can start Striker from the command line and it will start running our tests. Since the demo application is a small application, it's quite fast. If you have a large enterprise application, you will most likely notice that mutation testing is quite a bit slower than running your unit tests because there are thousands of mutations being made and each and every one of them has to be tested. So my laptop is starting up right now. It's running the mutation testing and in about a second or two, it will actually be done. For our application, it has 12 source files to mutate and it was able to generate 126 mutants. So it's done. We get a nice table here as an output, but we also have a HTML report. And if we look at this report, we can see that we have a mutation score. That's the amount of mutants that we were able to kill. The more mutants you killed, the higher your score. And just as with code coverage, you probably want a high score. In this case, we have a score of about 85% of our application. Well, if we look at code coverage for this application, we would have 100% code coverage. So if we only use code coverage as a metric, we will be fooled in thinking we tested this properly. Now, if we look at specific files, we can also look at specific mutations, for example, the age check here. And we noticed that we have a quality operator that survives. To me, this indicates that we are missing a test because we were able to change the greater than sign into greater than equals. So with a report like this for your source code, you can actually figure out what kind of tests you are missing or if you have any bugs in your tests. So to summarize, mutation testing, in my opinion, is a far better tool to measure the quality of your tests. I would highly recommend you to try it out.
For JavaScript and TypeScript, I would recommend Striker. 
There's an example available on our website. 
If you have any questions, feel free to hit us up on Twitter at striker__mutator. 
And the slides are also on GitHub.