Detox: The Unobtainable Test Stability (or is it?)

Rate this content
Bookmark

In this talk, we'll discuss how Wix is using Detox internally, how we manage configuration, how we fight flakiness, and some best practices we've developed over the ~3 years of building and using Detox in our CI process. We'll also discuss our endless striving for "0 manual QA", which always seems in reach, if we only overcome that one last technical hurdle.

Rotem Mizrachi-Meidan
Rotem Mizrachi-Meidan
36 min
15 Jun, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

Detox is a grey box testing solution for mobile applications that manages sync between test code and the app, eliminating the need for manual synchronization. It follows the grey box testing approach used by Espresso and Earl Grey. Wix's mobile app architecture consists of four types of parts, each with its own independent CI process. Test isolation and input consistency are important for stable end-to-end testing. Android emulators perform better on Mac VMs with nested virtualization, but our next-gen setup includes Bare Metal Mac minis for improved performance. Detox is primarily used for React Native applications and has limitations, but there are learning resources available. Detox supports camera functionality and provides solutions for debugging test fails and running tests on device farms. Currently, Detox supports simulators for iOS and devices connected to the computer for Android, with work underway to support real devices. The test APK for Detox connects to the production app and Node service, and Expo is responsible for providing a test APK for Android.

Available in Español

1. Introduction to Detox and Motivation

Short description:

Detox is a grey box testing solution for mobile applications that manages sync between test code and the app, eliminating the need for manual synchronization. In this talk, I'll explain how Wix uses Detox internally, including configuration management, fighting flakiness, and best practices developed over three years. We'll also discuss the drive towards zero manual QA.

Hello, everyone. I'll start a bit with the motivation for this talk. Detox is a grey box testing solution for mobile applications. It manages sync between test code and the app, so the users don't have to do it manually. Despite eliminating the users' need to do so and the abundant documentation and guides, developers and testers can still get tripped upon bad usage patterns, misconfiguration, and suffer from poor test stability. And we feel this pain every day internally at Wix.

So in this talk, I aim to explain how Wix is using Detox internally and how we manage configuration, how we fight flakiness, and some of the best practices that we've developed over almost three years building and using Detox in our CI process. We'll also discuss the endless drive towards zero manual QA, which always seems enriched if you only overcome that just one last technical hurdle.

So hi, I'm Rotem. I'm a software engineer working at Wix. In the past four years, I've been working on Detox since its inception. In the past two and a half years, I led the team behind Detox. And recently, I left this role in order to join Wix's server infra group. In the picture, you can see me with two of my favorite side projects, especially now during the third full lockdown in Israel.

2. Detox: Solving Flaky End-to-End Tests

Short description:

Detox was built at Wix to solve the problem of flaky end-to-end tests in React Native applications. It follows the grey box testing approach used by Espresso and Earl Grey, which involves running a synchronization mechanism inside the app to detect its busyness and idleness. This ensures that actions and expectations only occur when the app is idle and nothing will change until the next user action.

So in order for us to get everything into the right context, we'll start at the beginning. Detox was built at Wix in order to solve a growing problem with flaky end-to-end tests, especially on our then new React Native application. The main idea was to follow the approach successfully executed by Espresso and Earl Grey. These are two grey box testing projects created by Google for Android and iOS respectively. As opposed to black box testing, where the tester needs to decide how much time they wait for the app to finish what it's doing before sending the next action or expectation, where the grey box approach includes running a synchronization mechanism inside the app under a test to detect the busyness and the idleness of the process. Then it only interacts with the app when it is considered idle. Meaning it has no more events to handle, no more network requests, no more animations, no more transitions, it's actually doing nothing. This approach guarantees that any action or expectation that with the app will only happen when the app finishes processing everything and nothing will change anymore until the next user action.

3. Wix Mobile App Architecture and End-to-End Testing

Short description:

In Wix's mobile app architecture, there are four types of parts: the engine, the UI library (React Native UILib), modules, and other libraries. Each part has its own independent CI process, and when everything looks good, a new version can be published. End-to-end tests are split into different test suites and executed at different stages of the development cycle. The first type of end-to-end tests is production end-to-end, which ensures consistent features and tests on every test execution.

In order for us to talk of how we incorporate end-to-end testing inside Wix, let's also discuss how Wix's mobile app is architected. So, in a bird's-eye view, there are four types of parts which we build the app from. The first part is what we call the engine. It's the backbone of the application. It's an entity that includes all the native code and all the API registry for all the modules that are built on top of it, so they can communicate with one another and with the engine. The infrastructure libraries used with the engine, most of which are actually open source, and the engine itself are usually written in languages native to the platform, so it's Objective-C and Swift for iOS and Java and Kotlin for Android. And alongside, all this is wrapped with a unified JavaScript API.

The second type is our UI library, aka React Native UILib. This is an open source project, and this is the project which all the UI is built on top of, including, and it includes a lot of JavaScript and native components. Module is a single product. It's a blog or we have a lot of them in the app. These are, this can be a blog, store manager, CRM, chat, we have a few of those, and those expose an API of itself to consumers, and also it consumes the engines API. Also, it consumes, of course, UILIB to build its own UI, and other modules, of course. This is the actual product implementation. Here we'll have most of the screens, and the business logic, and all of those will be defined inside a module. And the fourth type is some other libraries, either if it's open or a closed source. These are somewhat disconnected from the release process, actually, because they are inherently just transitive dependencies of the above mentioned parts. New releases are being updated manually with those, mostly through PRs to the engine, to UI lib or a module. And regarding testing, they're being tested and treated separately with their own build, with their own test suites. Each of those parts has its own independent CI process, and when everything looks good to us, a new version can be published for everyone to use. The final stage is to take all of those with your configuration files and build one big binary that includes everything. And this is the app released to Google Play and the App Store.

Now that we have a slight overview of how everything is laid out, let's discuss testing. I will not discuss unit tests because those, though we have a lot of them in all the projects and they run all the time, but they mostly are very easy to run and they're very, very cheap to run. On the end-to-end side, we differentiate between a few types of tests and split those into a few test suites and execute them in different timings and different stages of the development cycle. Here are four types of end-to-end tests that we have. So the first one will be production end-to-end. Those are fully functional, those run on fully functional app with very minimal mocking. Wix use a lot of experiments and A-B tests all around, and with this type of tests, we want to make sure that we get the same features and tests on every test execution because they can differ on different runs of the app. So we developed an experiment override mechanism that enables configuration of predefined experiments blob and at the beginning of every test, and this launches the app with these experiments running, thus ensuring that we never get different behavior while the app is under a test.

4. End-to-End Testing and Test Selection

Short description:

Production end to end tests run when a module owner wants to publish their work into the app. Mocked end to end tests use mocked server endpoints and allow for control of inputs and consistent outputs. Screenshot testing is used for UI libraries, using Apple tools for comparisons. Component testing is done on top of Detox with Compot. Wix app modules are written in JavaScript or TypeScript, allowing for good coverage by running E2E tests on one platform. Infrastructure libraries are tested on both iOS and Android. Tools have been developed to improve stability and provide insight into test failures.

Production end to end run whenever a module owner wants to GA, publish their own work into the full app. Mocked end to end tests. These tests are actually just with mocked server endpoints, they can run on specific module or on the entire application. We usually just do it on a single module. They do not interact with production environments, this controls the mock server and we're able to control the mock servers outputs and tests. And test the actual module behavior with in predefined states. The big upside is the ability to control all the inputs to the test, and then we have consistent inputs with the local mock server, which guarantees consistent output of the tests, usually more stable and a bit faster than production end to end, and this run on CI on every push to the modules code base.

We also incorporate screenshot testing, those are mostly used in our shared UI library, the UI lib. Detox doesn't have any sort of screenshot comparison mechanism, and we take those, it just knows how to take screenshots on demand. We know how to do both device level screenshots and element level screenshots to be compared with external libraries, we use Apple tools in order to get smart comparisons to avoid false positives on slight pixel variations. It's worth mentioning here that slight pixel variations is not an uncommon issue. It can happen by comparing screenshots taken with two different graphics cards or drivers for instance. So if you take something with the local dev environment and you try to take a baseline on that machine and then run the test on the CI machine, you'll have different screenshots. This can easily fail and not be handled properly.

So component testing is another type of testing that is done. Some of the modules incorporate testing of actual React components states. These load components with their own states and then switching state and props programmatically throughout the test. This is being done on top of Detox with Compot. It's our React native component testing library. And these run on CI on every push to the module's code base, just like we do with mock end-to-end.

So how do we choose what to test? Wix app modules are written 100% with JavaScript or TypeScript. And this means that most of the module business logic runs on the same code, runs with the same code on the two platforms. A lot of times, product developers find the bug related to the module manifests on both platform similarly. So the bug which manifests only on one platform usually is an issue with an infrastructure library. This means that module devs are able to get pretty good coverage by only running E2E on one platform on top of their module. This is not however mean that the whole app can be tested on one platform. All of Wix's infrastructure libraries are running tests on both iOS and Android. So in order for us to trust end-to-end tests, they must be very stable, but also provide a good insight of what's happened and why the test fails. Over the years, we try to improve this in two ways. The first is by developing a series of tools to help us figure things out, view hierarchy dumps, trace logs, videos, test timeline, graphs, and also trying to make it easier for users to understand why tests hang when they do, and it's usually caused by some sort of something that keeps the app not idle either if it's an animation or a network request or something similar.

5. Stable End-to-End Testing and Configuration

Short description:

Two important aspects for stable end-to-end testing are test isolation and input consistency. Test isolation ensures that each test starts fresh and is not dependent on previous test execution. Input consistency handles the various inputs and configurations that may affect the app's behavior during different test runs. Detox's configuration files are crucial but fragile, and misconfiguration can lead to subpar results. To address this, we plan to provide a basic shareable configuration with Detox. CI for Mobile remains a challenge, but we have tested it on CircleCI, Travis, and Bitrise with good results.

The second is education, and the latter was mostly directed internally at Wix engineering team, but some of it also reached our documentation. So I want to go a bit over of the second one. So two things that are very important for us to get or to have when we want to have stable end-to-end, I want to have test isolation. This is probably one of the most important tips that I can give you. Make sure that every test is starting fresh, make sure it is not dependent on execution of a previous test.

The second one is input consistency. An app might have multiple inputs that might change the behavior when running various responses from servers. Some application might have experiments and A-B tests executing on different code paths in the app, causing it to behave differently during different runs. So you must make sure that these are handled and configured in your tests. And they're identical throughout all the iterations of those tests. Whenever it's locally or in CI. At least we developed an experiment override mechanism that helps us predefining those experiments and ensuring it works well while the app is under a test.

So regarding configuration, as Detox becomes more and more mature, we get more features and tighter integration with Jest, which we usually do that in order to leverage some features available in Jest or new cool features in Detox. Those configuration become harder to handle. Although the documentation itself is pretty good, it's very hard to keep track of the config. So as mentioned before, this is actually all the configuration files that we have in Detox, in one of our internal projects. So as mentioned before, Wix app is built from many independent developed modules. Each has its own git projects and build configuration in CI, and each of these was required to add the Detox configuration files into its own repo. Most of these were similar config files, and during the last six months, we got to a realization that it makes zero sense to keep those this way. And we went for and unified the configuration for everyone. We created a library that helps us share everything. These configuration files are an extremely important yet fragile part of Detox. It is easy to misconfigure. And whenever it's misconfigured, you get subpar results. For those reasons, we have decided to supply some sort of basic shareable configuration right out of the box with the open source Detox. And we hope to do this sometimes in the near future. This is something we already do internally. CI for Mobile is one of those things that are still pretty hard to nail right out of the box solution. So we've tried all the major SUS CI solutions running iOS E2E on either CircleCI, Travis, or Bitrise. It's pretty easy.

6. Optimizing Android Emulators and CI

Short description:

Android emulators perform better on Mac VMs with nested virtualization, but still lack hardware acceleration. Our current CI solution uses VMware ESXi on Mac Pros with Mac OS VMs, but performance is slow. Our next-gen setup includes Bare Metal Mac minis for improved performance. We recently released support for Genycloud SAS devices, which offer better performance, scalability, and the ability to offload device emulation. However, Genycloud is a paid service. Our main challenge is test execution speed in CI, especially for elaborate end-to-end tests. We aim to automate and speed up the release process.

And the big problem was with Android emulators. The real insight here is that Android emulators run better on Mac VMs than on the offered Linux ones. They are guaranteed to run with nested virtualization on Macs, which is required for running x86 emulators. We basically run a VM, the Android x86 emulator inside a VM which is the Mac OS so it requires nested virtualization. But those still miss hardware acceleration and the resulting with poor emulator performance although it works.

So the current solution for our in-house CI includes VMware ESXi on Mac Pros on InMix stadium with Mac OS VMs. These work for both iOS and Android as VMware supports nested virtualization. But performance for those machines is still pretty bad. Compilation time is slow and test parallelization is somewhat limited. We cannot use a lot of simulators or a lot of emulators on the same machine. And by that slowing down, or not slowing down decreasing the amount of time taken to run it.

So our next gen setup includes Bare Metal Mac minis such that run everything on the host OS. These have much better performance at least two times faster than the VMs on an Intel based Mac mini. Hopefully we can do it even faster with Mac minis with M1 chip. On the Android emulator front, we recently released support for a Genycloud SAS devices and we orchestrate all those with D-talks right out of the box. By using Genycloud emulators, we gain four key benefits. The first is we drop all prerequisites for the CI machines as these emulators run remotely on Genycloud infrastructure. The second is we get better performance per individual emulator. In our case, it decreased a 35 minute suite with two workers running Android emulators to 15 minutes with two workers running with Genycloud emulators. The third is maybe the biggest benefit is the ability to scale infinitely and run as many workers as we want in parallel. So all the SaaS CI options that we talked about now become valid as they can be used for builds and test orchestration and offload all the device emulation to Genycloud. The main caveat with Genycloud is the fact that this is a paid service, even for open source projects. So the holy grail for the release process is to have it all automated and fast. As we continue to solve our problems, we get just a bit closer to the goal on each step. The main issue we're facing today is the test execution speed in CI, especially for elaborate production end-to-end switch. This is absurd. The absurd is that it is faster for our QA engineers to go over the test suite manually than to wait for 150 minutes for everything to finish. So they actually do it. This will not stay this for long though. With just a bit more work, we'll get much faster test with execution with potentially infinite scale.

QnA

Detox: Limitations and Learning Resources

Short description:

Detox is a framework-specific tool primarily used for React Native applications. While it may not be the best choice for native applications or those written in different frameworks, it still has its advantages. When it comes to more complex flows, such as using device features or disabling internet, Detox has limitations, especially on iOS. However, there is documentation available to help users learn and navigate the tool.

Thank you. Thanks for having me, Alex. What did you think about the poll? I know Detox was in the top there and I kind of expected Detox to be the number one, what do you think about the other bit of it? Okay, so I guess two things. One is that I wonder what the other is, of course, and the second is that I know that Detox is probably the first choice when it comes to React native applications, but when it comes to native applications or applications written in different frameworks, then I see a point of using something that is not Detox. So yeah, it's not a total surprise, I guess, but the other is very interesting indeed.

That actually makes a lot of sense. The point about Detox being a really framework-specific tool, and from your talk you saw that. I've got a bunch of questions from a bunch of people. I'm gonna manual their names, so I'm not gonna focus on their names. I'm gonna just focus on the questions we've had for you during your talk. Don't forget, if you've got questions for Rotem, you can still go to the talk Q&A channel and Discord and ask a question. The first one was, I started working with Detox recently. It's pretty easy to start via getting started, but what about more difficult flows, for example, using device features like camera, disabling internet, blocking it, and unblocking the device? There are very few materials regarding Detox compared with other tools. Where and how can I learn the best Detox? That was a big question. So let's start small. How do I disable and enable things on the device, like blocking it and unblocking the device with Detox?

Yeah. Okay, so let's see. So regarding iOS, so Detox still supports simulators and it doesn't really run on devices yet. We have a plan and what we did with Detox17 and Detox18 is prepare ourselves to run on real devices on iOS. Now that doesn't solve the issue of disconnecting network and disabling things similar to that. And actually, I'm not really sure what has this kind of ability. It would only be, as far as I understand, it would only be to some kind of mock the network layer and in some way. On Android it's a bit more achievable, but yeah, we didn't really take this into account while developing Detox. So, we don't really use those, but if you think that this is like a super feature that you really want and think it's very important, then of course, let's have a discussion in our GitHub issues and let's see how we can make this happen.

That actually makes a really good point. I remember from my testing days, we didn't always try to test interaction with the device. The app was the most important thing. The app was the object of our test, right? What happened with the phone while we were using the app wasn't really the purpose of it. So how about materials? Where can people go to kind of find advanced detox materials? So I think that the documents are, I mean, there's a lot of documentation. And if you go, you'll find pretty good gems over there.

Test Stability and Camera Functionality

Short description:

There is a lot covered in the documentation, but there are also additional resources available on Wix's Medium. A recent blog post explains how to make tests super stable at Wix, including the collection of artifacts during testing. Regarding device and network functionalities, such as camera, Detox does not have any specific issues. If you have a camera module in your app, it should work seamlessly. Detox focuses on testing your own application, not system applications.

I'm pretty sure that there is a lot covered in the documentation. There are also things that I wrote previously on Wix's Medium that I think are a bit different from what's in the documentation. You can go and read those as well. Things regarding, I just recently published this week, the beginning of the week, a blog post about how we make our tests super stable internally at Wix. So what are the ground rules for making super stable tests? And I go over all kinds of features that help us achieve that.

Now, all kinds of artifact collection, artifacts that we collect during tests to make sure that we understand what's going on in during testing, especially when tests fail. And these are I think good materials as well. And so you said things about documentation of, what was that? About device and network like mocking network, bad network, which this is something that we don't really have. What were the other things, Alex?

So I think the examples mentioned specifically were device features like camera disabling internet and blocking and unblocking the device. So camera is something that... I'd be curious about the camera one, yeah. I don't see any issues specific to camera. If you have a camera module in your app, you can just open it. If you run emulator, then you can make sure that your emulator is connected to the camera or to like a mock camera that Android emulator provides. If it's a genie motion, it does the same. If it's the actual device, then it can open the camera as well. So if you have a camera module in your app and you open it, it should just work. There is nothing that we disable from like making it work, as far as I understand. I understand. So how do you deal with things like, for example, my phone has three cameras. It's a new iPhone. It has three cameras. I have to select which camera I want from every app that use the camera, right? If I use light, for example. Oh, yeah. There are two types of things that you wanna add, I guess. If you have, there is a way of going to the camera app and taking a picture. And this is something that, Detox usually just concentrates on the application that you let it work with. So it's your application, not other system applications. You don't wanna actually test those system applications. You wanna test your own application.

Camera Functionality and Debugging Test Fails

Short description:

Detox supports fetching pictures taken with other cameras or camera apps, as well as using specific camera modules like React Native camera kits. When debugging test fails, it is important to have consistent inputs and isolated tests. This ensures consistent outputs and makes it easier to identify and debug issues. When it comes to running Detox tests on device farms, there are solutions available for both iOS and Android.

So if you wanna take like pictures that were taken with other cameras, with other camera apps, then you can go and fetch those. Again, in your app if there is a way for you to go to the gallery and get those. And the other one would be if you have a specific camera module like React Native camera kits or something similar that opens the camera module as a view in your application. And with this, I don't think we have any limitation. I think it just works.

Okay, that's really good. That's really good. We've got a bunch of interesting questions. One of them says, what's the best practices of debugging detox test fails? There's a comment at the end that it says it fails very often in our pipeline whereas running locally is fine. So how do you deal with debugging test fails? What's your best practice for that? Okay. So this is exactly, I think. There is an exact answer of this with the latest blog post that I just published this week. Again, using all the tools that we provide, all those artifact collection tools and understanding the output of the tests themselves like the outputs of what detox outputs to the log. And there are a few ground rules of making sure that you have a base level of good testing setup and this is also written over there. But I would say in a nutshell, this is something that I ended the talk with is have consistent inputs. So you can have consistent outputs, that's for sure for any tests, not only end to end tests, but for every test. You don't wanna have different time in the clock for every test that you run. You don't wanna have like different inputs. That's a problem. And the second one would be have isolated tests. So it would be much, much easier for you to debug issues when they arise, because you know that your test begun on a specific screen and not on a screen that you don't really know how you got to. If it's like a residue of a previous test. That makes a lot of sense. That makes a lot of sense.

There's a question around device files that support DTOX. Someone says we have to run DTOX test on emulators in our pipeline. Are there any solutions for device farms that support DTOX? Running on simulators doesn't provide the same level of confidence as running on real devices. Right. So this one is a long answer as well. So I'll split it again for iOS and Android.

Detox Support for iOS and Android

Short description:

Detox currently supports simulators for iOS, but work is underway to support real devices. For Android, Detox supports devices connected to the computer, but there is no out-of-the-box support for device farms. However, Detox recently added native support for launching Geny Motion emulators, resulting in a significant improvement in test time. While running tests on actual devices has its advantages, running on Geny Motion Cloud has proven to be efficient and comparable to real devices. Detox can be used with X-Box React Native, and Expo is responsible for providing a test APK for Android.

So for iOS, we still support only simulators, but we're working on support for real devices. This would be with like wrapping or at least adding XEUI test to the mix that it would be what launches DTOX inside the iOS application. So for iOS, we don't really do that yet. Though we have sort of a plan with a big device farm provider, but it's still not something that we can make sure that, or at least not be sure that it will happen, but we really do want it to happen.

And for Androids, it does support devices and you can on every connected device on your computer, you can just run detox tests. I know that it's not the same as running on a device farm and this is not really supported. There is no device farm that supports detox out of the box, but we recently added support, like native support for launching Geny Motion emulators. These are different from Android emulators. They're also x86 but they run on designated machines on Geny Motion Cloud.

So for us, I know that you said that it's not the same as running on real devices and I agree to some extent, though we didn't really find a lot of issues specific to devices. Most of the issues we find are actually issues with the app, with the SDK, specific SDK levels. with Android. But the fact that we run on Geny Motion Cloud now, we started doing it internally at Wix and this dropped our end-to-end tests from, let's say I'm just taking an example. One of the suites took 35 minutes, we dropped to 15 minutes with the same worker count. So we're the same two emulators that run on that machine. With Geny Motion we were able to run with two workers in Geny Motion Cloud only on the 15 minutes for the same test suite. So it's like 60% improvement in test time. I know this is not exactly the answer you wanted to hear but I just had to talk about this. To be honest, it actually makes a lot of sense. For example, where we were running tests on a lot of Panda boards instead of rigging up actual devices in racks. We tried that for a while but it wasn't efficient and Panda boards gave us 99% the same deal, so it was good.

I've had so many complicated questions for you. I feel bad about picking the complicated ones. So I've got the simple one, your last question of the day and then you're done with me. I've got the really simple yes or no question. Can detox be used with X-Box React Native? Okay, yes. But we were contacted by Expo a few times in the past. For iOS, I think it should just work with the current setup. There isn't something that should be an issue as far as I understand. And with Android, it is actually up to Expo to also provide a test APK, a pre-built test APK like they do with the production APK.

Detox Test APK and Expo Integration

Short description:

The test APK contains Detox and connects to the production app and Node service. Expo was involved in the setup, which requires maintenance for new versions. Setting up in the Expo library should work. The talk was engaging with a Q&A session. Thank you Rothem for the insightful talk and challenging questions.

So the test APK is the APK that has detox actually inside it and it knows how to connect to the production app and ask questions and connect it to the Node service that we run and ask those questions. So we did speak with Expo in the past, I think about two years ago and it was done but it's something that needs to be maintained every time we have a new version, we don't really have breaking changes in Android, at least not in the past year I think.

So whenever someone will set it up in Expo in the library itself, yes, not like the project. Whenever someone from the Expo team will set it up, it should just work.

Thank you, thank you. That actually makes a lot of sense, the picture makes a lot of sense. Thank you Rothem, I've had a blast with your talk. The Q and A session was really engaging. Thank you so much, Rotherm. Thank you so much. Hard questions, thank you for those. Yeah, really hard questions, really hard questions. Thank you, thank you so much. Thanks. Thanks. Thanks. Thanks.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Network Requests with Cypress
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
Testing Pyramid Makes Little Sense, What We Can Use Instead
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
Gleb Bahmutov
Roman Sandler
2 authors
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
Full-Circle Testing With Cypress
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
Test Effective Development
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
Playwright Test Runner
TestJS Summit 2021TestJS Summit 2021
25 min
Playwright Test Runner
Top Content
Everyone Can Easily Write Tests
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.

Workshops on related topic

Designing Effective Tests With React Testing Library
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Top Content
Featured Workshop
Josh Justice
Josh Justice
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
How to Start With Cypress
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
Filip Hric
Filip Hric
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Detox 101: How to write stable end-to-end tests for your React Native application
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
WorkshopFree
Yevheniia Hlovatska
Yevheniia Hlovatska
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
API Testing with Postman Workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
Top Content
WorkshopFree
Pooja Mistry
Pooja Mistry
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
Testing Web Applications Using Cypress
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
Gleb Bahmutov
Gleb Bahmutov
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Best Practices for Writing and Debugging Cypress Tests
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
Workshop
Filip Hric
Filip Hric
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.