Testing for the Modern Web with Playwright

Hi, my name is Arjun and I'm excited to be here at TestJS and talk to you about testing. I'm a program manager at Microsoft where my team is building Playwright. Playwright is a new library for cross-browser end-to-end testing of web applications. In this talk, I'm going to talk to you about testing the modern web. My team and I have interacted with hundreds of testers and developers that are building modern web applications and we've shaped Playwright for their needs. I'm also going to show you the core ideas behind Playwright and hopefully convince you why Playwright is the right testing tool if you're building a modern web application. Let's jump right in. Let's start by talking about the modern web platform. The modern web platform is more capable than ever before. Web browsers have matured significantly over the last few years. They run on every device we have, including our desktop, our phones, and smart TVs. There's also been a continuous inflow of new features on the web platform. For example, the web can now detect your current location. It can connect to Bluetooth devices and also interact with your file system. The pace of innovation in the web platform has been incredible. These changes in the web platform have powered innovation in the application world. If you look at desktop applications, they're very similar to what they were 10 years ago, but web applications look very different. Web applications of today are full-blown applications that have rich and responsive interactions. We're building single-page applications that are built with newer reactive javascript frameworks. We're also building PWAs that are replacing native applications. The platform and our applications on the web have both evolved significantly. With these changes, we've also seen changes in how teams are operating. Teams of developers and testers are actually moving faster. Lots of teams are building applications today, and there's competitive pressures which are pushing teams to ship faster because that is how they achieve their business goals. This has led to teams adopting newer workflows like CI, CD, and continuous deployment. Teams are also looking for automated testing solutions so that they can release even faster. Unfortunately, end-to-end testing has not evolved as quickly as the web itself. Let's talk a little bit more about that. End-to-end testing needs to actually evolve for the changes that we talked about in the modern web. For example, tests need to be able to automate newer platform features that are showing up across browsers. These include capabilities that are available on desktop browsers, capabilities that are available on mobile browsers, and even capabilities that are being powered in smart TVs. Teams are actually struggling to be able to automate all parts of their application because their tests cannot actually cover all the capabilities that they're using. The applications of today are also harder to automate. Tests cannot keep up with how reactive frameworks work. Teams that are building rich single-page applications find that their end-to-end tests are flaky. They're not able to reliably automate user interactions on their applications. What happens as a result is that end-to-end tests that are actually supposed to help you move faster are actually slowing teams down. Teams end up having to add sleep timeouts or retries to be able to manage reliability issues. Teams are struggling with coverage issues because they're not able to automate all parts of their application. And therefore, end-to-end tests are actually slowing teams down when they actually need to be able to ship faster. These three problems are actually the reason why end-to-end testing has not been able to evolve for the modern web. And these three actually give us really the three pillars of modern web testing. Modern web testing needs to be capable so that it can actually automate all possible capabilities of the modern web platform. Modern web testing also needs to be reliable, which means it needs to be able to reliably automate user interactions with highly rich and responsive web applications. And finally, modern web testing needs to be fast, which means it needs to be able to speed up shipping and release processes instead of slowing them down. Any testing solution for the modern web needs to cover capable, reliable, and fast, the three pillars of modern web testing. We've actually focused on these three core ideas while building Playwright, which means Playwright is actually focused on capable automation without any trade-offs. It's also focused on reliable automation without any flakiness or time-outs. And finally, Playwright enables fast execution, just like unit tests. And so you can actually use end-to-end tests with Playwright to be able to actually speed up your desk workflows and not slow them down. For the rest of the talk, I'm going to zoom into each of these three core ideas and actually show you code on how they're actually implemented in Playwright and how you can actually make the most use of them in your tests while you're actually shipping your applications. Let's start with capable. Capable automation without trade-offs. Let's start and look at some code. So what I have here is a very simple snippet that uses Playwright in javascript. Now Playwright is at its core a tool for you to write end-to-end tests for your web application, which means with the Playwright api, you can actually automate browser interactions. You can launch a browser and automate interactions that are happening inside that browser window. In this particular code snippet, I'm actually using Playwright to launch an instance of Chromium. Now Chromium is the browser engine that's actually used inside Google Chrome and Microsoft Edge. Once I've launched that browser, I'm then opening a new page, which means I'm opening a new tab in that particular browser window with Playwright. And then that particular page is navigating to the test.js home page. Finally, I'm closing the browser and wrapping up my basic script. Playwright is enabling this, which means you can actually use Playwright to automate Chromium. Along with Chromium, you can also use Playwright to automate Firefox and WebKit. This is, you know, these three really make Playwright the right cross-browser testing tool for the modern web, as Chromium, Firefox, and WebKit actually cover the three main browser engines that are used in modern browsers today. We've talked about Chromium, which powers Microsoft Edge and Google Chrome. Firefox powers Mozilla Firefox. And WebKit is the browser engine used inside Apple Safari. Playwright provides you an api that can basically automate all the three browsers. And in this particular code snippet, we're actually launching an instance of WebKit and then doing the same interaction. One notable thing here is that Playwright WebKit is actually available across all platforms, which means you can actually use WebKit to test for Safari issues in your Linux CI environments, for example. And that's made possible with Playwright. Let's see how more of these capabilities work. With Playwright, you can actually have browsers running in both headless and headful modes, which means that I can actually run my tests on CI in headless modes so that the UI is not visible and my tests actually run faster. But I can also run these tests locally in headful mode, where I'm actually looking at the UI and seeing what has happened in the browser. This functionality, again, is available across all platforms. When we are testing for our modern web applications today, it's not just about testing them on our desktop environments. We are also building applications for mobile devices. With Playwright, we can actually emulate mobile environments with our cross-browser tests. So when I'm launching an instance of WebKit now, I'm actually specifying that the WebKit page should emulate the iPhone 11 device. This means that my page will have the right viewport just as an iPhone 11 and would enable mobile-specific features like touchscreen events. Talking more about capabilities, we talked about how the web today can detect your current location. When you're launching a browser with Playwright, you can actually use these newer web features. In this particular snippet, I'm actually launching the browser and specifying dark mode to be enabled. I'm also then specifying a particular geolocation for that browser window. I'm specifying a bunch of permissions and locale. This way, my browser is actually working as if I have a mock current location and gives you a quick preview of how Playwright enables newer web features so that you can actually test your applications for those capabilities. The other important capability that I want to highlight here is that Playwright can also intercept network activity. In this particular snippet, when I launched a browser and created a new page, I set up a mock api so that my api endpoint essentially returns an error code. This gives me the ability to actually test for an error scenario with Playwright's network capabilities. And then finally, another capability to highlight is that Playwright can actually automate interactions across multiple pages. You're not limited to one particular page. In this example, I'm launching an instance of the browser, creating a new page on it, and then opening a new pop-up. We can then automate that pop-up with Playwright. What we just saw was automation without trade-offs. You were able to automate all modern web browsers. You were able to get headless and headful execution across all platforms. You also saw device emulation for mobile testing and the best coverage for modern web platform APIs. If you're using any of these APIs, Playwright is really the best tool to be able to author and create your end-to-end tests so that you can actually ensure that all features of your application are covered with your end-to-end tests. That wraps up the capability section. Let's now talk about reliability. As we just talked about earlier, reliability is essentially the problem of being able to automate user interactions on your application without having your tests flake. In a lot of cases, users have to actually author timeouts, which means they need to make their tests sleep for some time so that they can actually reliably automate. This creates a whole bunch of problems because timeouts actually lead to flakiness. Playwright solves these problems in a very unique way. Let's see how. The key thing to note about Playwright is that Playwright can actually listen to browser events. It can actually get events from the browser and actually understand what's happening in your application when it's running inside the browser. In this particular snippet, I'm launching an instance of Firefox, I'm creating a new page, and then I'm setting up a new event listener for that particular page. I'm basically saying whenever my application throws a console log, it should show up in my test. Playwright is not limited to console logs. I can also listen for other events, such as the DOM content loaded event. The DOM content loaded event is an important lifecycle event when your application or your web page is actually loaded inside the browser. And I can actually listen for this event as a part of my test. I can also listen to network activity. For example, I can check for whether a particular web socket connection has been opened. This ability to listen to events gives Playwright the ability to wait for precise events instead of having to wait for blankets, sleep timeouts. This makes your tests highly reliable. Now Playwright uses this functionality internally on every interaction. Now in this snippet, I'm launching an instance of Firefox, and then I'm going to github.com and I'm clicking on a button. Notice that I did not have to use any sleep timeouts. This is because a page.click in Playwright is automatically waiting for the right conditions. It's waiting for the element to be visible. It's waiting for the element to stop animating. It's waiting for the element to receive click events. All of this is internally done for you so that your user interactions can reliably go to your applications and test your applications. This makes your tests significantly easier to write and maintain over time because you don't have to worry about custom wait conditions or sleep timeouts. In a lot of cases, these defaults just work out of the box. But in case you're looking for more control, Playwright also provides you APIs to actually author your own custom wait for condition. Now these wait for conditions again use the same entire event logic, which means that you don't have to worry about guessing the amount of time this would take. You can actually write that precise condition and make sure that your tests actually wait for it. Now this essentially shows how Playwright is more reliable. It essentially means that you don't need to actually guess on how long something would take. You can actually use events to verify that application state. This is wrapped up automatically for all interactions inside Playwright, which means everything that you do in Playwright has built-in auto waits and retries. And then finally, if your application requires custom logic, you have powerful wait for APIs that you can use and actually author your wait for conditions. This feature differentiates Playwright from other testing tools and actually improves reliability. Now let's talk about fast execution. Developers and testers are generally very happy with their unit tests because unit tests run reliably and are fast to execute. However, that's not the case with end-to-end tests because end-to-end tests depend on browsers and browsers can be slow, especially when it comes to launching and closing a browser. If you're running hundreds of tests, being able to sort of run hundreds of browsers is a very expensive operation and that actually drastically makes your tests slower. To solve this in Playwright, we've actually introduced a new abstraction there. Let's walk through this code snippet to show you how it works. So in this snippet, I'm using Playwright to launch an instance of WebKit, which is the browser engine used inside Apple Safari. Now in this browser instance, I'm actually creating a new context with the browser.newContext api. Inside the context, I'm then now opening my pages and doing interactions like I was doing earlier. What we've done here is that we've created a new abstraction layer on top of this browser instance called the browser context. This browser context essentially can host multiple web pages inside it. Now it's very cheap to create browser contexts, which means you can actually spin up multiple browser contexts without having to worry about the time it takes to launch and close a browser. This significantly speeds up your tests. Now there are a few properties of browser contexts that I want to highlight. The first is that browser contexts are actually completely isolated from each other, which means that you can actually create multiple browser contexts, run isolated tests without having to worry that they would affect each other. In this particular code snippet, I'm actually launching an instance of Firefox, and then I'm creating two contexts in it. The first context is an empty context, and the second context is context with authentication. In the second context, I'm adding particular cookies. Now this essentially means I can actually run my tests in two different scenarios. The first is no authentication, empty state, which is a great way to test your marketing websites. The second context is context with authentication, which means you can actually test for workflows that require authentication. These in the second context will not affect the first context. This basically ensures that if we have hundreds of tests, they can run in concurrent isolated environments through browser contexts without having to spend time to launch and close browsers. The other interesting thing that browser contexts enable is emulation. We looked at how you can use Playwright to automate device testing for iPhone or iPad devices. In this particular example, I'm actually launching an instance of WebKit, and then I'm actually specifying the two devices I want to test, the iPhone 11 and the iPad Pro 11. Now when I'm creating this browser instance, I can then create browser contexts on it. I'm basically looping over the devices to test and essentially creating a new context for every device that I want to test. Once I have that browser context, I can then open multiple pages inside it and navigate to the pages that I want to. This essentially means that we had one browser instance, but we were able to actually reuse that browser instance in form of browser contexts to be able to run multiple tests across multiple devices. Browser contexts therefore not only make your tests faster, they also give you a lot more capability to be able to run emulation scenarios, for example. This is essentially how Playwright is making your end-to-end tests a lot faster. Despite browsers being slow, we're able to create a new abstraction layer called browser contexts, which are much faster. Browser contexts parallelize execution with a single instance, which means you can just launch one browser, but then actually create parallelized execution environments. Each of them are isolated from the other, and you can have multiple pages being opened inside one browser context. This makes your tests significantly faster and gives you more capabilities like we just talked about. Capable, reliable, and fast. These are essentially the three core ideas behind Playwright. We've talked about how Playwright enables capable automation without trade-offs, how it enables reliable automation without having to worry about flakiness or timeouts, and how it enables fast execution, just like your unit tests. To use Playwright, just use npm. npm install Playwright can set you up with the latest stable version. When you install Playwright for the first time, it will download your browsers, which means you're ready to go for cross-browser testing. If you prefer to use something else other than javascript, you can also use Playwright in other languages. Playwright fits your preferred toolkit, which means you can use Playwright in javascript, Python, C Sharp, or even Java. You can also use Playwright with any test framework and any assertion library and deploy it to your preferred CICD environment. Finally, to learn more about Playwright, just wrapping it up with a few interesting links. The first is our documentation website at playwright.dev. You can also find us on GitHub in the Microsoft org under the Playwright repo name. Finally, you can join our Slack to actually interact with other users who are using Playwright for their end-to-end tests and also reach out to the Playwright team. We're all on Slack. Thank you so much for attending this talk. Hey, welcome. Are you happy with these numbers? 18% of the people saying that they're going to check it out? Yeah. I'm always excited to see folks trying out new tools. Playwright is something that we released last year and we've been seeing a good entrant, a good yearly somebody with our rising star somebody and the state of javascript survey. It's exciting to see the interest and excitement around Playwright. I'm sure it'll only grow and we'll be constantly tracking those numbers and making sure that we're actually satisfying that interest with the product. Yeah. You're talking about adaptation. Can you share some insights, maybe some secrets about companies that are using Playwright? We released Playwright as an open source repository almost a year ago. In fact, in January, in 21st January last year, and we released 1.0 in May. We've been ready for production really for about seven-ish months now. Since then, we've actually had thousands of developers and testers use Playwright across the different language bindings. You can use Playwright in javascript or Python and we recently also released Java support. There are a bunch of internal teams that are using Playwright. The VS Code team, for example, is testing everything on Playwright. The Bing team runs about 40,000 tests on Playwright. There are a bunch of external teams that are also using Playwright. Adobe is a big customer. They have a bunch of teams internally who are using Playwright. In terms of open source projects, the Material UI library is a big Playwright user. We actually just recently upgraded our homepage. If you go to playwright.dev, it actually shows logos of customers. It's a good place to track just the big names that are on the Playwright train. Well, probably after today, we're going to be able to add some logos on the website homepage. I want to go and ask you two questions from our audience. The first one is from Aaron McAdam. Which particular use cases is Playwright good for when you're comparing to cypress? Sure, yeah. cypress is an amazing tool. We are very impressed with their developer experience. In general, we've tried to focus on, to begin with, we've tried to focus on some of the scenarios that cypress cannot fully automate. For instance, if you want to actually run automation scenarios that are across domains, across multiple pages, across domain iframes, Playwright is super appropriate for that. If you want to have tests that involve native input events, or even like hovers or touch screen events, some of these capabilities are much easier to do in Playwright. In general, Playwright has very few trade-offs. Our pitch really is automation without trade-offs. If your tests actually require high capability automation tooling, Playwright is the right tool for you. In general, aside from the capability question, Playwright is also focused on a few gaps in the market that cypress is not essentially focused on. For example, if you prefer a different language, if your team actually prefers using something like Python or Java for automated testing, you can actually use that with Playwright itself and get the speed and reliability improvement. That prim Playwright makes available in javascript. Other languages are also possible. Even cross-browser support on Playwright is better in the sense that you can actually test for WebKit, which is the browser engine used inside Apple Safari. Notably, the Playwright support for WebKit is cross-platform, which means you can actually use WebKit testing on Mac OS or on Linux or Windows. If you have a mobile web application, which is used by users on Safari, and you have a Linux ci cd agent, you can actually use Playwright to run your automated tests for WebKit, essentially. Better browser support, better language bindings, and capability automation without any trade-offs is really our pitch for cypress. Super nice that you can also run this WebKit on a Windows machine. I think that really helps if you have a big team when people are using Mac OS and Linux and Windows, of course. Yeah, super powerful, I guess. Next question is from Danny. Can you share something about your roadmap for this in the upcoming year? Yeah, for sure. I think what's been amazing, actually, is we've seen lots of amazing traction over the last one year. With any new tool, there's tons of user feedback, tons of GitHub issues or Slack community messages or just engaged users getting on to conversations with us. That is essentially what's helping us shape our roadmap for this year. Our current focus is on making the getting started experience of authoring your tests even easier and debugging your tests easier. One of the things that we released in preview last year was an integrated test runner for Playwright, which means you don't actually necessarily have to use something like Mocha or Jest and figure that integration for yourself. With the integrated test runner, you'll actually get a full, complete end-to-end testing tool, which would simplify the experience of getting started significantly. Similarly, maintaining your tests over time is obviously the hardest part when it comes to end-to-end testing. Better debugging is something that a lot of our users have asked for and something that we always wanted to build that, but this is a priority for this quarter. We've been doing monthly releases. Playwright 1.8 was live last week. Playwright 1.9 goes live in February and actually has an amazing debugging UI, which will actually help you maintain your tests over time. Lots of authoring and debugging improvements, which will be live over the next quarter or so. That's really our priority. That's our roadmap for Playwright. People can just influence it by joining your Slack server. I understand. Danny, if you want to influence it, you can do it. Yeah, for sure. That's all the time we have. That's the beauty of... Oh, sorry. Go ahead. Yeah, I was just saying that's the beauty of open source software. GitHub issues or Slack, our product development and roadmap is fully transparent and we are always seeking feedback. All right. So, Danny, if you want to add something to the roadmap, you can just do it yourself. For the rest of the questions, Arjun is now going to go to the speaker room. And if you want to talk to him more about Playwright, now is the time. Arjun, I want to thank you a lot for joining us here and hope to see you again soon at the next big event. Thank you. Thank you.