Testing Frameworks, Mobile Frameworks, and Browsers Love Developers and Testers

♪♪♪ Hi, everyone. My name's David. And today I'm going to be talking about testing frameworks, mobile frameworks, browsers, and how each of the people who work on those love developers and testers who use their products. The reason why I know this is, as Nathaniel said, I've been kind of in the industry for a while. I've spent nearly a decade working on Firefox. I've been doing browser automation for nearly two decades. And so every time I see these things and come to these conferences and see what people are doing and showing, all I see is kind of the love that they care and want to improve everything. And I want us to kind of care about it. I want everyone in this room to go, actually, it's nice to see that they care.

And so we're going to start with something I think is kind of a tree. Testing is hard. Whenever we look at how we, like, want to do our jobs, we start with testing and we go, actually, you know what? Testing is very, very hard. People build careers out of it, right? And they still go, I have no idea what I'm doing. And it's okay.

And the main reason why testing is so hard is that the systems we're working against are incredibly complex. We think about JavaScript. It's an asynchronous language. We've kind of been doing this. But whenever we think about testing, we think sequentially. We want to go from this step to this step to this step. So years and years ago when I started my career, there was callback hell. And you tried to do it and you hoped that things would run in the right time. The language has improved and it's got much, much better. But then we've got other things that kind of get in the way. We've got browsers. We've got web apps. We've got everything from multiple layers through the entire system that we've got to test. And we think about how we go about doing it and we go, actually, this is really hard. But, we've had like numerous people think about how we approach these problems. And they've tried to simplify them as they go along. And they've come up with kind of cool terms like the testing pyramid, unit testing, integration testing and so on. And I'm going to break it down. Because I think it's important that sometimes words matter. Sometimes I don't think they matter so much.

And so like with the unit testing, this is where I think people get a bit too fixated, but they've tried to break down the problem into something small. And it's a small test. Because sometimes people can't do it at a unit level. And we've got these integration tests where we test our contracts. And this is where we start to see some of the kind of real love that people have.

We get tools like Mocker and things like that have kind of grown, and newer tools, VTest, that have come out to allow people to do it. Testing in JavaScript when I started my career was a lot of just manually clicking, making sure that things worked as we anticipated. And it wasn't that easy.

And then we've got cool tools that are coming out that allow us to kind of think slightly bigger, but not as big as an end-to-end test. And we've got those component tests. We've seen the market exploding with people trying to test their front-end frameworks, which are really hard to test because they're complex. They've got their own complexity built in.

And then we've got these medium to large tests, as I like to call them. And the medium to large tests is, you're at least starting up a browser, but you're not doing a full end-to-end test. And then we've got our end-to-end tests. And so it's got all this complexity. And then you throw in different browsers, right? I've only just shown Chromium variations here, but you've got Firefox, you've got Safari, and then you've got mobile browsers on top of it. And this complexity is something that everyone starts thinking about, but when it comes to the case of actually solving it, improving it, doing that work, it's very hard. But there's bound to be someone who cares and wants to do it.

But we've got all this complexity. So we've got our frameworks, we've got browsers, and sometimes browsers are not always the browsers that we expect them to be, right? At the beginning of the year, the Chromium team said, oh, by the way, Headless is not what you think it is. Headless is a totally different browser in Chrome. So all your tests that you've been doing, thinking you've been doing the right thing, maybe it's not actually true, and we're sorry about that. But we fixed it, right? So, sorry about that. So now, not only have you got Chromium variations, Headless browser that you're thinking about, and you're trying to improve your tests, and it's not going the way you want. But this is a first sign that the Google team actually care about us. They've seen the problem. They've gone, actually, this is causing more problems than we care for. So let's go fix it. And this is another sign that our developers love us.

But then, we go straight from them loving us to them hating us again. Upgrading every four to eight weeks, right? When I started at Mozilla, the year before Firefox 3.6 had been released, I am that old, and we were six months out from Firefox 4 being released. Mozilla decided to have this wonderful all hands group where the whole company gets together. And at that stage, Mozilla was about 350 people. So we were small.

They invited a Googler over. The Googler gave a talk to the whole company, and he said, by the way, by the way, we're going to be shipping Chromium, because Chrome was new on the market. We're going to be shipping every 12 weeks. The web's not moving fast enough. We need it to move faster. Internet Explorer had kind of given up. And they were kind of just doing what they needed to at a bare minimum. Firefox was kind of growing, building things. And then the Chromium team were like, yeah, we need to do this faster. And it's only got faster since. The eight weeks is normally around Christmas time or kind of July 4th, when the U.S. teams need to have a break, and they don't want to be shipping then, because if something goes wrong, someone's going to have to work through the holiday periods.

So, I've talked about a bit of love from the developers, the Chromium developers, but where's the love for the testing community? Like, where are we seeing those things that kind of make our lives easier, better, faster, more productive? And where are we seeing those items? And we see them with tooling nowadays that is kind of allowing you to like, I need to set up a new project. How do I do that?

So, setting up devices, setting up browsers, right? I've been working on Selenium for about 16, 17 years, right? In one form or another. In that time, we never said, gave you the tooling to kind of download a browser, set that up, mostly because kind of making sure that was consistent everywhere was hard. And open source projects are mostly run by volunteers. There are a few exceptions, but setting those up was hard, but we tried to do everything else in the meanwhile. So, we worked with the W3C and got drivers and we standardized everything and got those into place. And then we got setting up the dependencies. I'm sure you've all seen like with component testing, we want to make sure that you have everything you need. But frontend frameworks have their own specialties. So, it's all about kind of download this, download this, not knowing when to do it.

Puppeteer famously started this with like, actually we'll download the Chromium for you. We'll make sure you can download, run on your first go. You couldn't do that with Selenium, not unless you already had a browser and you knew how to start up that browser. But there was a bit of complexity there. We've seen it grow with Cypress and Playwrights, Nightwatch, JS. I work on that now. WebDriver I-O, they've all gone actually, let's set up everything and improve it. So, they might not tell you that they kind of care for you, love for you, but their actions are starting to show it. And we've got to the stage where we can improve things.

So, in this case, like MPM and Nets, we see it with Reacts and other projects nowadays, where you can just go, I would set me up a project. You could do it. So, the picture on the right, if you can't see it, it's fine. But the last three lines is like, do you want to do end-to-end testing? Do you want to do component testing? Do you want to do mobile web or mobile app testing? And then you answer a couple of questions. You tell us where we're going to run it, and we go run it. And we make sure everything's in place. We give you example tests. This has now become a standard way for frameworks to set everything up, because getting started in a JavaScript space is hard, because we think sequentially, and we try to do everything sequentially. And we try to solve these problems in a language that's asynchronous and doesn't always work sequentially in the way we want it. And so it tries to improve these things. The other thing that's kind of trying to be improved along the space is the principle of least astonishment.

Or I preferred the principle of least surprise. So when you get something, you should intuitively know how to use it. If you work on it, like if you go to a mail client and you want to send an email, it should be obvious that there is a compose button or a new email button, right, things like that.

If you're working in an API, you should be able to kind of just go do this, dot, do that, dot, and it should just allow you to move it on if you wanted it that way, or you just kind of go, I think this is how I need it. I need to pull this information out. And it should be intuitive to do that.

Where some frameworks have started failing down, and this is mostly when it comes back to how Selenium never allowed you to download and always forced you, other frameworks were like, we'll download it for you, is that upgrading those browsers nowadays means you have to upgrade their framework, which then can cause other problems, but it's not necessarily because they don't love you, it's that their lovers could be shown in a different way. And so sometimes upgrading the framework can break your tests. I know I've seen this before with Puppeteer users. They were like, I'm stuck on a very old version of Puppeteer because every time I upgrade it, my tests break. I just can't handle that. I don't have time to fix my tests. And so you should be able to, through your tooling, whatever, be able to upgrade small parts, have the confidence and move on.

And this is where, kind of, it's a bit of a hobby horse of mine at the moment. Browser engines don't equal browsers. If you think back to the Chromium slide earlier, we had six different Chromium versions there, right? If I go speak to Brendan Eich, who's the CEO of Brave, he will tell you one or two things that's the same as Google Chrome or Chromium, but Brave is unique. Same with Opera, Vivaldi, Edge, right? They all take their different things. So how do we go about solving this? So in the Selenium space, which I spend a lot of time in, we've got tooling like Selenium Manager. This is how the Selenium project is trying to show you that they care, is that you want to update your drivers for Chrome because Chrome driver has to use the Chrome debug protocol, which changes with every version of Chrome. And you need to update that. What if we just took away that pain, solved it for you? The other thing that people have, that we were seeing, is that, like, the Puppeteer team were getting frustrated with the way people were raising issues. This bug is happening with Puppeteer in Chromium. And they would go, well, I can't reproduce this in Chrome. And so what they did, the Chromium team, was they created Chrome for Testing. And Chrome for Testing is to stop you using Chromium in your testing environment because it's not meant to be used as a test browser because there's no way to go from I've got this bug and track it all the way back to the commit that caused that bug. Because things are changing so fast in that code base that it's nice to have a standard way to do it. So Puppeteer, Selenium, things like that, they use it. And it's come through collaboration, improving what we want, that people start caring. And then it comes down to the other side of it, right? Shrinking desktops is not mobile. It's a bit controversial for some people, but it's true. And the main reason is how operating systems render web pages. They get complex, things like that. If you spend some time with a Chromium developer who's working on Android, or Firefox engineer working on Android, or Safari engineer working on iOS, it can be hard. But the way frameworks are starting to show people that they care and love for you is that we see this hard problem, we'll set it up for you. So one of the projects that I work on, Nightwatch.js, it sets up simulators and emulators for you if you want to test on mobile. And it's not relying on Appium to do things if you care about just the web. Because Chrome driver, Gecko driver, Safari driver, they work well with mobile browsers. So let's use mobile browsers. I will give you one thing, scaling on mobile is hard. But scaling up on desktop browsers is hard too, just in a slightly different way. Maybe a less expensive way, but it's still hard. And so people are giving you their love and giving you the frameworks and the tools. And all of most of this is for free nowadays, thanks to open source.

I think one of the things that would be awesome for people in this, like around the world, is to share their feature ideas. I think I would like this or that. This is the pain points I'm having regularly with your framework. There's a lot of noise that comes into a lot of frameworks and projects. But when there's someone going, hey, I love this idea, could you help me make it? I'm pretty sure every project that you use would be willing to help. So share your ideas, because sharing your ideas is you showing your love back.

And with that, I've come to the end of my talk. Thank you all. So yeah. One of the benefits of being an MC is I get to ask a question first. A question that I've kind of been wondering is, like you said, you've been in, especially browser automation for over 20 years, right? So you've been here for a while. You've seen a lot of ships come. You've seen a lot of ships go. Are there any ships that you see that just enter the station, leave the station, then come back around again, that maybe we should just start just putting them in place permanently?

I, not really. I think the interesting thing that is happening is that the JavaScript space seems very fickle sometimes, right? And, but when you look at it, some of the things that are always there, the kind of the old guard, are still there. They just might not get the publicity that they were getting, say, five, 10 years ago. And so kind of like if you just went on pure sentiment, people will tell you that like, oh, Cyprus and Playwrights are growing and they're doing this. They are growing and they're fantastic tools, right? But at the same time like the old guards, Lenium, things like that, they're still growing like 25% quarter on quarter, right? And so I don't see things like that. The things that I have seen are kind of how people approach the problem. So sometimes it was like, oh, we can just inject JavaScript into the page. That'll be good. And we started with that originally with Lenium and kind of went away from it. Cyprus then brought it back. And so it causes some problems, but it doesn't. But it's the uniqueness of the, kind of the web brings, there is always something moving forward, not necessarily going in a circle. Yeah, totally. Always moving forward, I like it.

Okay, so we've got a couple of questions coming in. The first one is asking about a different framework. So what do you think is different between the Nightwatch team's approach compared to other front end end-to-end frameworks like Cyprus or Playwright? This is a cheeky addition to the question.

Why is it better? So, the way kind of like Nightwatch and WebDriver are kind of core philosophies, be where your user is, right? So if you're going to test, do component tests, make sure it's in a browser, not using jsdom, because like that actually causes other problems. And it's like that Chromium versus Chrome problem that can kind of come up and you have these differences.

The thing that I think kind of like Nightwatch is, personally, I think Nightwatch is better than Cyprus because Cyprus uses like, the way it speaks into the browser is using older technology like the Selenium RC. And if you write any JavaScript in a page, you're bound by the sandbox in the browser because it's a security system, right? It's trying to keep everyone safe and it's not necessarily designed for automation where the way like Nightwatch and Playwright do it is that it speaks through the backend of the browser and go, hey, can you manipulate these things for us? And then it's more realistic. And so in that case, I think it's really good. I think there's a lot of similarities between like Playwright and Nightwatch and especially with the work that's going on in the W3C for like WebDriver Bidi to standardize some of those basic ideas so that Mindwatch can improve as it goes forward and be standard.

Can headless replace? No, I don't think it can. I've been in the camp that was trying to tell people when people moving to headless, same with Richard Bradshaw from Ministry of Test or ex-Ministry of Test was that like headless is not where your users are. Like it's great if you're locally developing and you don't want to see browsers popping up all the time, right? Like that can be annoying. And so it's cool. But like the minute it goes into your CI CD pipeline should be where your users are, same browsers, whatever. And you have a matrix and you just build and test through it. And you should do that as much as possible so that you can catch bugs, right? So like my favorite like browser engine versus browser bug was one where a couple of years ago, IndexedDB was broken in Safari, right? Like really, really broken. But if you were testing and like playwright with WebKit, you would never hit that bug because WebKit didn't have it, right? And then iOS Safari didn't have it either. But it was only on desktop, right? And so you had this weird bug that your CI CD would have been passing. And this goes against to me, the principle of least astonishment or least surprise, right? Like you shouldn't be getting surprises like that. And there's no way to track it back. So I'm very much use headed browsers as much as possible, but I do understand there's trade-offs, right?

And we're working on browsers and someone asks, especially with the way the web works, like, we're gonna have requests from the server. So should we have mock requests from a server or not? And why? I think the testing pyramid here comes in very quickly.

But if you were testing and like playwright with WebKit, you would never hit that bug because WebKit didn't have it, right? And then iOS Safari didn't have it either. But it was only on desktop, right? And so you had this weird bug that your CI CD would have been passing. And this goes against to me, the principle of least astonishment or least surprise, right? Like you shouldn't be getting surprises like that. And there's no way to track it back.

So I'm very much use headed browsers as much as possible, but I do understand there's trade-offs, right? So. No, totally, yeah. And I love the path of least astonishment. I will be using it again. In the future. And we're working on browsers and someone asks, especially with the way the web works, like, we're gonna have requests from the server. So should we have mock requests from a server or not? And why? I think the testing pyramid here comes in very quickly. Like when you've got your end-to-end tests, don't use mocks, but if you're doing component tests, do mocks, right? Like put everything in. Make those tests as fast as possible because you really wanna fail as fast as possible, right? Like ThoughtWorks decades ago, when they brought out their CI, the pipelines, they were like, yeah, builds should take no longer than 10 minutes because people like get distracted, right? Like mobile phones nowadays, people are like, oh yeah, it was cool building, let's do this. And it's like, oh wait, I've wasted an hour, right? Like having those quick feedback loops definitely have mocks and then kind of, but it's a balance. Don't do it everywhere. No, totally.

All right, we have time for one more sneak question, which is especially about the test of time. Do you think Chrome for testing will pass the test of time? Is it not better to be where your user is, which in my opinion is kind of the point of your talk, right? Yeah, so I do think it's gonna stand the test of time, right? Like when it comes to testing, we need, I'm trying to think of the right word here. We need to kind of make sure that our tests are deterministic. That's the way I was trying to think of, right? Chrome for testing allows for determinism, but you can still, it's going to be the same Chrome that ships, but they kind of have, they've taken the auto updater out and then they, so you're using what your users are and now you don't have the auto updates, but then tools like Puppeteer, Selenium Manager, things like that, they should be updating those as things go along. So you're kind of keeping where your user are, but you've also got a deterministic browser on your system. Now, totally. All right, we don't have time for any more questions. However, you will be able to find David throughout the day. And also he will also be at the speaker's discussion room later on. So definitely walk up and say hi and ask any other questions that you have. But let's give him one more round of applause. Thank you.