Fighting Test Flakiness with Time Machines

Rate this content
Bookmark

What would you do differently if you could travel back in time? Modern testing frameworks have transformed this whimsical question into a practical one, by creating their own “time machines”.Cypress’ timeline, Playwright’s trace-viewer and Replay.io’s recordings have offered a retrospective look into the life of a test, ensuring that developers and testers are no longer limited to basic error messages on test failures.

However, these different time machines will bring different insights. So how do you decide? The decision on which one to use can make a significant difference in time spent on debugging a flaky test.In this presentation I will be focusing on comparing different time machine solutions and showing various flaky test examples to demonstrate how to navigate through debugging process and believe it or not - make it fun.Key Takeaways:

  • - learn about how different time machine solutions work
  • - discover how to effectively use time machines to debug a flaky test
  • - find out about sources of flakiness within the test and within the application under test
Filip Hric
Filip Hric
29 min
07 Dec, 2023

Comments

Sign in or register to post your comment.

Video Summary and Transcription

The Talk discusses the importance of learning from the past in software testing and troubleshooting test flakiness. It highlights the challenges of using memory and communication to gather information. The introduction of Replay.io, a time-traveling debugger, is proposed as a solution. The benefits of using Replay Chromium for recording and debugging, as well as the features of the Replay debugger, are emphasized. The Talk also addresses the relationship between test flakiness and app flakiness, and the significance of simulating real-world scenarios in testing.

Available in Español

1. Learning from the Past

Short description:

The answer to how I manage that with 4Kids is magic, and that magic is my wife. The topic is fighting test flakiness with time machines. We learn from the past. A lot of testing is actually the process of learning from the past. If you are doing SRE, you probably have conducted like a post-mortem meeting after an incident. How to learn from the past? We can either use our own memories, try to recall what has happened. We can talk to someone who witnessed something of interest. We can try again. We can try to recreate the past.

Thank you. Thank you very much. The answer to how I manage that with 4Kids is magic, and that magic is my wife.

All right. So, the topic is fighting test flakiness with time machines. So I have a question for you, for the very beginning. Would you like to travel back in time? Raise your hands. Who would like to travel back in time? All right. What would you like to do if you would travel back in time? Change the project that you're working on? That's a very software development answer. Who else? We got anyone else who raised their hand? You can shout out. Buy some Bitcoins. Buy some Bitcoins, all right. Those are great answers.

For me, actually, I would go to my own life and apply the knowledge that I have today. So yeah, maybe buy some Bitcoins or, you know, like when I'm arguing with my brother and I have a clever comeback, but like three months later, I would want to use that, you know, so like there, there you go. The point is, that I'm trying to make, is that we learn from the past. We are now at TestJS Summit and you might ask, like, what does this time traveling have to do with anything? It's that we learn from the past. If we can time travel and apply the knowledge we have today, we can do better. A lot of testing is actually the process of learning from the past. For example, if you write a bug report, you document the things that happened, you try to look into the past, and so you can try to fix them. With test automation, we use all kinds of different data and see traces of the test run. We, again, look into the past and apply that knowledge. If you are doing SRE, you probably have conducted like a post-mortem meeting after an incident. And all that is basically learning from the past.

All right, so now the question. How to learn from the past? If you think about it, we only have like a couple of options. We can either use our own memories, try to recall what has happened. We can talk to someone who witnessed something of interest. We can try again. We can try to recreate the past.

2. Using Memory and Communication

Short description:

Or we can use a time machine. And all of these ways of traveling into the past are good, but they have some flaws. The biggest problem is that even with all of this information, it might still not be enough. This often leads to an issue which many of us already had. You might lack information, clear communication, and technical knowledge. So, let's move on to the next point.

Or we can use a time machine. And we do actually have time machines and I'm going to be talking about them. And all of these ways of traveling into the past are good, but they have some flaws.

So let's go to the first one, using your memory. So if you're writing a bug report, I recently asked on LinkedIn, how do you write a bug report? And these are like the different answers that came in, what the bug report should do. And it's a lot. And I haven't even included everything. It's like descriptions, steps to reproduce, screenshots, videos, desired behavior, et cetera, et cetera. The biggest problem is that even with all of this information, it might still not be enough.

Which leads me to the other thing, talking to someone. Especially this happens in a team where the testing and fixing the bug or reproducing and fixing the bug is distributed among many different people or more people. And there's a communication bridge we need to create. And communication is hard. It's no easy task. And even in real life, if you want to talk to someone, let alone if you want to try to convey complex information.

This often leads to an issue which many of us already had. Could not replicate moving to backlog. Anyone has experienced this? Yeah? Okay. Then it's not just me. And it can be so annoying for all parties involved. Oftentimes, you can have amazingly smart people, and it would still happen. Like a couple of hands raised here. You are all smart people and it has happened to you. And by no fault of your own, you might lack information, you might lack clear communication, you might lack technical knowledge. And again, by no fault of your own, there's always something that can get in the way of finding the issue and then fixing the issue and learning from the past. Learning what happened.

So, again, talking to someone overall is great. But when you want to learn from the past. But also has some flaws. So, let's move on to the next point.

QnA

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Network Requests with Cypress
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
Full-Circle Testing With Cypress
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
Tiny Tests, Large Results
TestJS Summit 2022TestJS Summit 2022
21 min
Tiny Tests, Large Results
Yes, Big things do come in small packages. For example, isn’t a unit test’s speed, feedback, and reliability fantastic? Did you know we can also have fast, focused, and reliable feedback from our functional e2e tests? Atomic e2e tests are those that are targeted and focused. They’re tiny in size but large in their impact. This tutorial will teach you how to create atomic e2e tests with several code examples. First, we will use Cypress.io to authenticate by setting a cookie. Instead of using a UI. Second, we will use Cypress.io to set a JSON Web Token for authentication. Join me, and let’s write tiny tests for large results.
Visual Regression with Puppeteer, Playwright and Cypress
TestJS Summit 2021TestJS Summit 2021
9 min
Visual Regression with Puppeteer, Playwright and Cypress
Top Content
Visual Regression tests components via screenshot matching. I'll show how you do that in three different libraries/frameworks. Additionally, I will use Storybook to extract the components from your SPA choice.
Don’t Make These Testing Mistakes
TestJS Summit - January, 2021TestJS Summit - January, 2021
27 min
Don’t Make These Testing Mistakes
In this talk, I will discuss the common mistakes developers make when writing Cypress tests and how to avoid them. We will discuss tests that are too short, tests that hardcode data, tests that race against the application, and other mistakes. I believe this presentation will be useful to anyone writing E2E tests using JavaScript.
Gaining Confidence with Cypress Tests
React Advanced Conference 2021React Advanced Conference 2021
23 min
Gaining Confidence with Cypress Tests
Have you ever wanted to refactor mercilessly but didn't want to break the fragile tower? Or have you ever pushed to production only to spend the next few days cleaning up the regressions? You need end-to-end tests, and Cypress is a great, fast way to build them. With a simple JavaScript or TypeScript interface, you can automate browsers to hit those critical functions in your app to prove it works as expected -- this time and every time. Join us to dive into building Cypress tests and leave with confidence to refactor your way to greatness.

Workshops on related topic

How to Start With Cypress
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
Filip Hric
Filip Hric
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Testing Web Applications Using Cypress
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
Gleb Bahmutov
Gleb Bahmutov
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Best Practices for Writing and Debugging Cypress Tests
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
Workshop
Filip Hric
Filip Hric
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.
Flaky Test Management with Cypress
TestJS Summit 2021TestJS Summit 2021
114 min
Flaky Test Management with Cypress
Workshop
Cecelia Martinez
Cecelia Martinez
This workshop is for Cypress users who want to step up their game against flake in their test suites. Leveraging the Cypress Real World App, we’ll cover the most common causes of flake, code through some examples of how to make tests more flake resistant, and review best practices for detecting and mitigating flake to increase confidence and reliability.

Table of contents:
- Cypress Real World App Overview
- What is Flake?
- Causes of Flake
- Managing Network-related Flake (Activity)
- Managing Dom-relate Flake (Activity)
- Flake Detection and Mitigation Best Practices
- Q&A
Web Testing Architecture and Refactoring With Cypress
TestJS Summit 2022TestJS Summit 2022
158 min
Web Testing Architecture and Refactoring With Cypress
Workshop
Walmyr
Walmyr
In this workshop, I will introduce you to the migrator.cypress.io project in an unconventional way, where in addition to going over the Cypress commands equivalent to Protractor (and showing you how Cypress is simpler), I will also introduce you how I tested such a project evolving the automated test scripts on-demand, architecting the test suite in an evolutionary way.