Overtesting: why it happens and how to avoid it

Bookmark

Overtesting is a common problem with unit test suites. It's when you have too many tests that break often and take up a large amount of time to keep green. Here's a fresh perspective on why that happens and how you can avoid it.

7 min
03 Nov, 2022

AI Generated Video Summary

This talk is about overtesting, a technique to improve test suites and testing proficiency. It addresses the common problem of spending excessive time on maintaining test suites. The goal is to maintain 100% coverage while spending equal or less time on test suites. The talk suggests splitting tests into functional units, using expect object containing for manageable expectations, and prioritizing ease of testing and time spent.

1. Introduction to Overtesting

Short description:

Hello, this is a talk about overtesting, which is a simple technique to improve your test suites and testing proficiency. It's aimed at developers and QA teams who want to discuss testing strategies. Spending excessive time on maintaining test suites is a common problem. The goal is to spend equal or less time on test suites while maintaining 100% coverage. The key question is how many automated tests are needed for CI to fail. Over testing occurs when there are too many tests giving the same information.

Hello, this is a talk about overtesting, which is a very, you can probably understand what it is from the name, but it's a simple technique you can use to improve your test suites, improve your testing proficiency, so it's a model for thinking about your testing, really.

And this talk is aimed at developers, but it's also useful for QA engineers, QA teams who want to maybe discuss with their developers their testing strategies, their testing ideas.

So key question by overtesting, and this might be really obvious, but how much time do you spend maintaining your test suites versus your application code? A lot of my clients I've worked with, their problem isn't under testing, they're not testing too few of their use cases, they're actually testing a lot. And so what they end up doing is spending most of their time, I've observed this, up to 75% of their time or more in their test code, making the code changes to their application to add new features or fix bugs, but then having to go back and fix the tests, all these broken tests everywhere.

75% is too much. I love tests, I'm writing test left, right and center, but I'm not wanting to spend my life in the tests. I want to be at least in a situation where I'm spending an equal amount of time on both. And maybe even less, maybe I can be spending less time on my test suites, but still maintaining 100% coverage, for example. So, I'm not going to suggest what that number should be, but if you're kind of in this 75% territory, and you can think about whether you are personally and on your team, but you probably want to be moving down to improve your productivity.

So, here's a simple question. The top screenshot is showing GitHub workflow that's running a pull request. So, I'm sure a lot of you will be familiar with this setup. The red X is showing that CI has failed. So, continuous integration. Like I said, I don't see teams that don't have CI. I think most people are writing tests. Problem isn't under testing, it's over testing. So, to me, this question is key. How many automated tests do you need for CI to fail? The answer is just one. You need one test. So, ideally, whatever change you could make, if there was an issue with that piece of code you've written, just one test would fail because that's all you need to raise the flag and stop your pull request being merged. This is an ideal scenario. I don't think you'd ever get to one. But if you think about the times that this does happen, how many tests are failing for you? Are you in this scenario where multiple test suites are failing, hundreds of tests are failing because of a simple change you made? And this is the scenario where you then start spending that 75% of time fixing your test. So, over testing, very simply, you have too many tests, telling you the same thing. I'm gonna show a very quick example. There's plenty of examples of this. This is one I see a lot, where people are kind of doing scenario-based testing. And what they'll do is they'll set up the test and they'll print out the� they'll have an expectation for the entire response or payload that they've got here. So, this example is just calling fetch and I'm checking the method, credentials, headers, and the body.

2. Improving Test Suite Efficiency

Short description:

Split your tests into different functional units. Use expect object containing for more manageable expectations. Consider how much time you spend in your test code and find ways to improve productivity. Check out the second edition of Mastering React, test driven development, for effective unit testing practices. Have an open mind to new ideas and prioritize ease of testing and time spent.

But what I can do is split this test out into three different tests. The test on the left here is the key one, that's the body, that's probably the thing we're most interested in and the thing in our application code that will change the most. The two tests on the right, these are going to change less often. So, I shouldn't expect these to break. I'm not going to be changing the methods or headers often. So, these should hopefully just remain as they are. They're less brittle now.

The key here is using expect object containing. So, this is definitely your friend. You can make more use of this. Rather doing these huge expectations in your test, break them down into the functional units. So, that's it. That's really the idea. It's about thinking about how much time are you spending in your test code. That's an observation. You start thinking about that, and how does it make you feel? Are you frustrated by this? Rather than just being involved in that moment of fixing a test, getting the build working, stop and think how can you be more productive? And then figure out ways to improve it.

I don't want to suggest too many ways for you to improve things. This is the second edition of this that's just been released. Mastering React, test driven development. This is not just about TDD, it's about good unit testing practices that have helped me in my career. I recommend you check it out if you're interested in figuring out ways of writing tests that won't take up all your time. So to conclude, observation. Think about the time and feeling you've got with your test suite and what's serving you well. You should be happy with your test suites. They should be helping you out. Have an open mind to new ideas. Don't shut down ideas because you've read a blog post that this is a terrible thing to do, you shouldn't do that, this is how you should write tests. Don't write tests like this. Just keep an open mind on the ideas and always come back to the idea of how easy are your tests, how much time you spend in them, how do you feel when you're working in them. And that's it. Thank you very much for listening.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
23 min
Scaling Up with Remix and Micro Frontends
Do you have a large product built by many teams? Are you struggling to release often? Did your frontend turn into a massive unmaintainable monolith? If, like me, you’ve answered yes to any of those questions, this talk is for you! I’ll show you exactly how you can build a micro frontend architecture with Remix to solve those challenges.
TestJS Summit 2021TestJS Summit 2021
34 min
Network Requests with Cypress
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.


JSNation Live 2021JSNation Live 2021
30 min
Making JavaScript on WebAssembly Fast
JavaScript in the browser runs many times faster than it did two decades ago. And that happened because the browser vendors spent that time working on intensive performance optimizations in their JavaScript engines.
Because of this optimization work, JavaScript is now running in many places besides the browser. But there are still some environments where the JS engines can’t apply those optimizations in the right way to make things fast.
We’re working to solve this, beginning a whole new wave of JavaScript optimization work. We’re improving JavaScript performance for entirely different environments, where different rules apply. And this is possible because of WebAssembly. In this talk, I'll explain how this all works and what's coming next.
Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!
But how do we do that?
Are the "unit" and "integration" terminology serves us right?
Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?
In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!
It’s time to go DEEPER!


Workshops on related topic

React Summit 2023React Summit 2023
152 min
Designing Effective Tests With React Testing Library
Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents
- The different kinds of React application tests, and where component tests fit in
- A mental model for thinking about the inputs and outputs of the components you test
- Options for selecting DOM elements to verify and interact with them
- The value of mocks and why they shouldn’t be avoided
- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites
- Familiarity with building applications with React
- Basic experience writing automated tests with Jest or another unit testing framework
- You do not need any experience with React Testing Library
- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
147 min
How to Start With Cypress
Workshop Free
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
118 min
Detox 101: How to write stable end-to-end tests for your React Native application
Workshop Free
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and
used by React Native itself
as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites
- iOS/Android: MacOS Catalina or newer
- Android only: Linux
-
Install before the workshop
React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
Workshop Free
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
Workshop Free
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.


Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
Workshop Free
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:
- User authentication - Managing user interactions, returning session / refresh JWTs
- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents
- A quick intro to core authentication concepts
- Coding
- Why passwordless matters
Prerequisites
- IDE for your choice
- Node 18 or higher