Exploring Node.js Test Runner

Rate this content
Bookmark

The talk "Exploring Node.js Test Runner" delves into the concept of a test runner, shedding light on its essential role within the Node.js ecosystem. It provides an overview of why the development of a test runner for Node.js took considerable time, and presents an exploration of its inner workings.

28 min
07 Dec, 2023

AI Generated Video Summary

Today's Talk introduces the new Node.js test runner and its features, including filtering, sub-testing, and reporting. It also discusses executing and writing tests in Node.js, as well as the features of the Node.js testing library. The advantages of the Node.js test runner include the ability to create custom test reporters and use TypeScript. However, there are limitations such as a small ecosystem and limited libraries. Upcoming features include test planning, faster test running, and continuous evolution. The Q&A session covers topics like test runner speed, reporters, sharding, and parallelization.

1. Introduction to Node.js Test Runner

Short description:

Today we're going to talk about the new feature of Node.js, the test runner. A test runner is a CLI tool that executes tests and integrates a reporter to export the outcome. Mocha, Cypress, and Chai are popular testing frameworks built on top of Mocha, providing different functionalities. These components enable testing in our applications.

Hello, everyone. Thank you for having me today. So today we're going to talk about the new feature of Node.js, the test runner. First let me introduce myself. I'm Marco Ippolito. I'm one of the maintainers of Node.js, mostly focusing on security. And I work a lot in the open source ecosystem of Node.js.

So let's start by definitions. So what's a test runner? So the test runner is actually the CLI tool that executes the test and integrates a reporter to export the outcome of the test. So it goes through your source code and picks the test files to run. So let's see a bit of classification. So Mocha, for example, is a very popular test runner and testing framework because the two things go together. And the test runner always includes the testing framework because otherwise, what you're going to pick the file to execute but what else are you going to execute? So testing framework provides the content that is inside the file while the test runner actually finds the file to execute. So Cypress is another popular testing framework that is built on top of Mocha. So Mocha takes care of going through the source code and finding which files you want to execute, and then Cypress is built on top of the BDD libraries of Mocha. While Chai is only an assertion library that's also built on top of Mocha. So we have three main components that allow us to perform testing in our application.

2. Node.js Test Runner Features

Short description:

In the Node.js ecosystem, there are many test runner testing frameworks due to the minimal core principle. Node.js prefers to keep its core minimal and rely on NPM packages for additional features. However, this has led to a large ecosystem, making it difficult for new developers to find resources and increasing the risk of NPM supply chain attacks. To address this, Node.js introduced its own native test runner and testing framework, inspired by Node.tap. The Node.js test runner provides features such as filtering, sub-testing, and reporting.

Okay. So now we talked a lot about libraries but what about Node.js itself? So in the Node.js ecosystems there are a lot of libraries and I want to, like, let's see the most popular ones. So we have Mocha that we talked about, Jest, TAP. We have like a huge amount of libraries. And while this is good to have a lot of variety, it's actually bad for new developers that want to learn how to do tests in Node.js because there is an incredible amount of information and it's hard to find a guide and something to begin testing with.

So what's the reason why there are so many test runner testing frameworks in the Node.js ecosystem? Well it's because Node.js always follows the minimal core principle and we can talk about this is also related to the whole JavaScript ecosystem. So Node.js doesn't want to add to its core nothing that can be a NPM package. So when someone requests a feature to Node.js what we say, just create a package. But with years this led to a huge ecosystem and it's deteriorated the developer experience. So people who wanted to learn how to use Node.js found it very hard because of the huge amount of packages and also having too many packages increased the surface for NPM supply chain attacks. And I'm one of the team members of the Node.js security team and we have been trying to decrease the risk for NPM supply chain attack with the Node.js permission model and many things. So having something built in the core is one package less, one potential vulnerability less.

Alright, so Node.js was actually shipping only an assertion library. Since Node 12 there was no test runner, no testing framework, just a very very small assertion library which was not very useful. So Node.js needed a native test runner and a native testing framework so you could execute tests in Node.js without installing any other dependency. So it took some time. It was marked as stable in Node 20. So from Node 20 which is the LTS and if you're not using Node 20 you should. You can use the Node.js test runner. So yeah, it took some time. It took like a couple of years to make it but we finally have it. And now we're going to explore some of the features. So the Node.js test runner was actually heavily inspired from Node.tap. Node.tap is one of the, in my opinion, best testing frameworks and test runners in Node.js. It's been created by Isaac, the creator of NPM, so it's quite popular and we took big inspiration for this library and it's actually one to one. You can just replace the import and it's the same. It uses the same functions. So let's see some of the features of the Node.js test runner. So we will see like filtering and how to execute certain tests and skip other ones. We will talk about sub-testing, reporting.

3. Executing and Writing Tests in Node.js

Short description:

We will discuss the importance of mocking, watch mode, and code coverage in Node.js. To execute tests, use the 'Node --test' command, which automatically selects the test folder and runs all JavaScript tests. You can specify the folder and extension, and use the '--watch' option to watch the test folder or a single file. Writing tests is easy with the standard syntax and assertion library. Positive and negative assertions can be made using the 'Node test' and 'Node assert' libraries.

This is very important and mocking. And we also have watch mode that if you don't know about watch mode, it's a very cool experimental feature that Node.js has. And code coverage, which is also very cool experimental features. And we will talk on why it's very important to have code coverage, an internal code coverage library.

All right. So we did a lot of theory and as someone much more important than me said, talk is cheap, show me the code. So yeah, we're going to talk about code. All right.

So how to execute test, you just run Node dash dash test. And from Node 20 you can do it. It will automatically select the test folder inside your application and run all the test in JavaScript files inside. So you can also specify the folder that you want to execute and specify also the extension if you're using a CGS file, MGS, depending on the JavaScript version that you're using. And you can also watch the test folder with the dash dash watch option. And it's very, it's very, like I always use it. All right. So yeah, you can also watch a single file instead of a folder. A watch is still experimental. So I suggest you to use it and provide feedback to the Node.js project. So you can, if you find some bugs, just open an issue and this will help the whole community.

All right. So how to write test. It's very easy. You just create your, you just have the standard syntax and the assertion library. You can import it from Node test and Node assert. And then you can assert something very standard. We will go through the assertion library in a while. It has some strict, a strict sub folder where there are like strict for deep equality checks and also not strict checks. So there are many assertions that you can do. There is the positive assertion. You can also do the negative assertion by adding a not at the beginning.

4. Node.js Testing Library Features

Short description:

The assertion library used to support snapshot testing, but it was unstable and removed. Now, efforts are being made to bring it back. Node.js testing library provides a range of features, including synchronous and asynchronous testing, BDD keywords, filtering, hooks, and mocking. It also offers various reporters, such as spec, tap, and dot.

Very handy. One thing, fun fact about the assertion library, we used to ship snapshot testing so you could test snapshot, but we decided to remove it because it was not stable enough. So we're trying to bring it back because I personally love snapshot testing.

And you can also obviously run a synchronous test with different kinds of level. So you have a top level and sub testing. This is quite powerful in the Node.js because most of the time they are going to be asynchronous.

And you can also use the BDD keywords like describe and it, they do the same thing. It's just another syntax for people who want to use behavior-driven testing. Node.js testing library also provides some keywords like to do, skip and only. So you can filter your test based on, you can write, you know that you will want to write the test later so you mark it as to do, you want to skip it because it's not ready yet, you can just skip it. So it provides a full, full utilities for testing.

Obviously all the hooks, before, before all, after, after all. This is our standard features that Node.js provides out of the box so you don't need to install any other library. You can filter by name pattern. So for example, if you want to write only the test that contain the word foo in the description, you can do it. So in this example, it will only execute foo but it will skip bar, this is very handy.

You can mock. It has the, out of the box, the mocking functionality so you can mock object functions, whatever, you don't need libraries like xenon and it comes everything out of the box.

Yeah, you can choose your own reporter. So most people use spec, which is the default one. This is what it looks like. But you can also use tap if you like YAML output. I personally don't like YAML, tap, because it's very verbose, but it can be parsed by machines. So it's actually pretty good for automations. And this is what tap output looks like. So pretty standard. We also have dot as a reporter. So if you are familiar with something like PHP units, yeah, the output is just a dot. And an X if it failed. I haven't used it yet.

5. Custom Test Reporter and TypeScript in Node.js

Short description:

You can create your own custom test reporter using a function. Node.js uses V8 internal coverage, making it powerful and eliminating the need for external libraries. TypeScript can be used in Node.js by requiring and using TSNode or TSX transpiler.

I don't know why should someone use it, but feel free to test it. And you can create your own custom test reporter. For example, if the test passed, you return a green check and if not, it returns a bug. And you can do this by creating a function. And it's just an async iterator. So I'm sure that all of you use async iterators every day, right? No? So yeah.

Okay. And now we talk about coverage. Node.js ships coverage. It's experimental. And it's very handy because unlike other libraries, for example, Istanbul, which they add checks during execution. So they take your code, add small counters, and check how many counters have been updated. And then it does some logic to actually check the coverage. This is not how Node.js does it. Node.js uses V8 internal coverage. So it's very powerful. And if you're using C8 as a coverage library, it uses the Node.js internals to do it. So you can have it out of the box. No need for external libraries. It's inside V8 and Node.js will offer it to you. And this is what coverage looks like with the default reporter, which is a spec. And you have for every file the lines of coverage, branch, and et cetera. So also here. Pretty standard if you're used to testing.

You can use TypeScript. So there is a lot of controversy in this because Node.js is the only runtime that doesn't ship TypeScript out of the box. And there is a reason behind it. So feel free to talk about this after the talk. But you can do it quite easily. You can just require the end use TSNode or TSX, whatever TypeScript transpiler you prefer.

6. Node.js Test Runner Advantages and Limitations

Short description:

And then you just use --"require." And you can run TypeScript tests very easily. The syntax actually changed pretty recently. Before it was a loader, --"loader." Now it's -"require." It's been a great update. Another important feature is shards. It allows you to split and run tests on multiple machines, making it ideal for large test suites in CI/CD. However, migrating an existing test suite may not be worth the effort. The test runner is still new, with a small ecosystem, limited libraries, and APIs. Additionally, there is no native TypeScript support, requiring the use of a transpiler like Node.js, TS Node, or TSX.

And then you just use --"require." And you can run TypeScript tests very easily. The syntax actually changed pretty recently. Before it was a loader, --"loader." Now it's -"require." It's been a great update.

And yeah, this is another very important feature. It's shards. So if you have a huge test suite, you can split the test into pieces. And you can run this test on multiple machines. This is something that has been released in Node 21. So it's not on the LTS yet. And this is huge because if you have a huge testing library in your CI CD, you can run just a small piece on one machine. Or you can run just like some half of it. You can basically split it and choose which part of your test to run. So yeah, this is one of the best features.

So let's move on. So should I migrate my testing suite tomorrow? No. You definitely should not. It's great to have something inside the runtime itself. But if you already have your test suite, it doesn't make sense to put a lot of effort into migrating it. But if you're starting a project from scratch, then it's something that you might be interested into. OK, so let's talk about the cons of the test runner, what work we are doing, and how we are improving it. So small ecosystem. It just has been released. The LTS was introduced last month. So it's very new. And therefore, small ecosystem, no libraries, no guides. So everything is new. Not many APIs, so it's not complete. It's not bloated like some other testing frameworks. And no native TypeScript support. You always need transpiler like Node.js, TS Node, or TSX, or whatever.

7. Upcoming Features and Code Coverage

Short description:

Upcoming features include the ability to plan and control the number of tests, black timers for faster test running, and continuous evolution with new APIs added daily. Other features like exit on failure, bail, and expect failure for negative testing are also worth exploring. Watch and code coverage are recommended, as they are experimental but highly useful. Stay updated on new features through Twitter and LinkedIn. Lastly, code coverage is a feature developers should be excited about.

So upcoming feature, planning. You can decide how many tests you want to run. And if they do not match, it throws you an error. Black timers. And it actually got implemented and released a few times ago. So test runners is running very fast. It's evolving very fast. New APIs are added every day. Community is very interested in it. So expect it to improve soon.

And exit on failure, bail. It's a cool feature. I actually tried to implement it myself, but failed because of complexities. And so I will try again in the future to implement the bailouts on failure. And the last one is actually expect failure. So negative testing. So if we actually expect a test to fail instead of a test to pass. So I suggest you to actually use these features. If not all of them, but at least watch and code coverage, which are very useful. They are experimental. So please use them and provide feedback so we can improve it for everyone.

And that was it. You can find all my links and work about Node.js. I will update about new features on Twitter, LinkedIn. That was it. Thank you. Really, really enjoyed your talk. So many new features. And one thing I always like asking people who build on things like this. What feature do you think developers don't know about yet, but they should be most excited for? So definitely the code coverage.

QnA

Node.js Test Runner Q&A

Short description:

It's very fast. It's faster than any other libraries you're using. And it's the most accurate because it actually uses V8 internally. Let's jump into the questions from the audience. The first one is about the reporters. They ask why there is no built-in JUnit reporter. We started from TAP, and no one in the Node.js maintainers team is interested in JUnit. Another question is about sharding. The test runner can shard based on the fraction of tests. You can play with the fraction depending on the number of machines. And finally, the Node.js test runner can parallelize tests.

It's very fast. It's faster than any other libraries you're using. And it's the most accurate because it actually uses V8 internally. So yeah, that one definitely. I'm really excited to try it out. I actually haven't tried it out just yet. So I'm going to try it out. And also I've actually got an idea for a talk that I want to give based off of this. I want to thank you in advance.

All right. Let's jump into the questions from the audience. We've got the first one, which is about the reporters. And they ask, why is there no built-in JUnit reporter? It's usually the most common use case since it's the de facto default format for machine-readable test results. So that's because we started from TAP, which was the one we took inspiration from. And yeah, probably because no one in the Node.js maintainers team is interested in JUnit. So we haven't built it yet. But if there is interest, we surely build it. So get on GitHub and let them know that you want JUnit test reporters as well.

I have a question, and this is something that I also am very interested in. So thank you, Norbert, for asking this. How does sharding work? Can it shard based on how long the test will run? Because I saw you had that kind of fraction. I was really interested. How is that fraction kind of made? Do you just make up on the fly, and it does one fifth of the tests? Yes, it actually does one fifth of the test. And you can play with the fraction, depending, for example, if you have five machines, then you split a fifth for every machine. And it's very handy. That is really cool. That is really cool. I'm really excited to try that out as well. And then another question, and I think I saw a piece of code that had this in it, can the Node.js test runner parallelize tests? And if so, how do you control it? Yes, it can. It can parallelize tests with the...

Node.js Test Runner Q&A (Cont.)

Short description:

The flag 'concurrency' allows running tests in parallel, sacrificing some control over their execution order. Test runner speed was a major focus during development, and it is significantly faster than alternatives. However, code coverage does not work well with the loader mechanism for TypeScript support, but this is being addressed. Feedback for the test runner is actively sought from maintainers and end users through various channels, including Twitter and GitHub.

There is a flag concurrency. And so how do you control it? You use it if you actually want to run tests in parallel, and you lose some kind of control because it will not execute them one by one. This is a choice that you have to make based on your use case. The classic it depends answer. There is always one it depends answer.

And then another one about test runner speed. Was it a concern when building it? Because this is a big part, especially when you're testing a lot, it becomes a big part of your time just waiting for tests to complete. Definitely, it was one of the main focuses, and it's fast. I did not add benchmark in my presentation, but you can find them on the Node.js project. It's very fast compared to the alternatives. That totally makes sense.

And another question as well, about code coverage, it works well when you use the loader... Does it work well when you use the loader mechanism to support TypeScript? No. That's one of the bugs that we are trying to fix right now. It doesn't work well, but we will make it work soon. And obviously, I just want to preface this by saying this is like the early days. And how are you kind of gathering feedback about features maybe developers want or features you maybe want to spend more time on? Exactly. Node.js became LTS last month. So it's only been in the market for one month for the public. We have been using it internally. We gather feedback by mostly maintainers that create new packages and use it in their new application. But we would really love end users to provide feedback on Twitter, on GitHub, on whatever. There is a discussion section on Node.js on GitHub where you can add ideas, improvements that you want for the test runner. Yeah. I love it. And I love it when there's like sort of an open feedback loop between the people working on projects and the people who use them. So please do contribute. One thing I loved about one of your slides is you had the slide and you had Bunn and Dino just like peeking in. It reminds me of this question where it's just thoughts on Bunn's test runner. Yeah, it's very fast.

Bunn: Collaboration and Standardization

Short description:

Bunn is great for the community, providing inspiration and feedback. It promotes collaboration and standardization through the WinterCG working group.

I love Bunn. It's great and it's helping the whole community because it allows us to see what the community wants and how other Node.js alternatives are doing. So it gives a lot of inspiration and feedback through also other runtimes. Yeah. Yeah. I love how it's an ecosystem and not just a competition between all these as well. They all help each other and build each other. Yes. This is very important. And right now we have created a working group called WinterCG where we all work together to standardize the ecosystem and not be competitors but like allies in the ecosystem. I love that. And thank you for all of that hard work that you're doing.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Do you know what’s really going on in your node_modules folder? Software supply chain attacks have exploded over the past 12 months and they’re only accelerating in 2022 and beyond. We’ll dive into examples of recent supply chain attacks and what concrete steps you can take to protect your team from this emerging threat.
You can check the slides for Feross' talk here.
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
In the early years of Node.js, diagnostics and debugging were considerable pain points. Modern versions of Node have improved considerably in these areas. Features like async stack traces, heap snapshots, and CPU profiling no longer require third party modules or modifications to application source code. This talk explores the various diagnostic features that have recently been built into Node.
You can check the slides for Colin's talk here. 
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring.
While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.
JSNation Live 2021JSNation Live 2021
19 min
Multithreaded Logging with Pino
Almost every developer thinks that adding one more log line would not decrease the performance of their server... until logging becomes the biggest bottleneck for their systems! We created one of the fastest JSON loggers for Node.js: pino. One of our key decisions was to remove all "transport" to another process (or infrastructure): it reduced both CPU and memory consumption, removing any bottleneck from logging. However, this created friction and lowered the developer experience of using Pino and in-process transports is the most asked feature our user.In the upcoming version 7, we will solve this problem and increase throughput at the same time: we are introducing pino.transport() to start a worker thread that you can use to transfer your logs safely to other destinations, without sacrificing neither performance nor the developer experience.

Workshops on related topic

Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Workshop
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.