Playwright Test Runner

Hi everybody, my name is Andrey and today I would like to introduce you to the new Playwright Test Runner. So let's start. So first thing first, what is Playwright Test? Well, it is a cross-browser web testing framework written in node.js, which basically means that you can author your tests in javascript or typescript. Now, it is free and open source and sponsored by Microsoft and is already extensively used in the industry. Now, this is all good, but it doesn't answer the big looming question. So why did we do yet another test runner? So honestly, we did not intend to. There is a lot of different test runners in javascript, but it turns out they are all unit test runners. Whereas in end-to-end testing, there's lots of very hard challenges. So for example, you want your test runner to support cross-browser tests out of the box. You want them to be parallelized out of the box as well. And you want them to be isolated. And Playwright, for example, has a set of unique APIs to make sure that test isolation is fast and reliable. And of course, we want these to be used. And last but not least, the history shows that end-to-end tests require a crazy level of flexibility in configuration, because there are many different configurations we want to use to run the same test. And we are inspired by PyTest fixtures, so fixtures are implemented in Playwright Test as well. So Playwright Test is our answer to end-to-end tests. And this is the test runner that was built specifically to address all of the end-to-end testing needs. So we have quite a lot to talk about in this talk, and we have just 20 minutes. So let's jump straight in to the first chapter, which is getting started. Okay, so the best way to show something is actually to show something. So let me hop to my terminal, and I will create a new folder, and I will start from scratch, and initialize a new npm project. Now, let's do npm init Playwright. And what this does is actually sets up this project to use Playwright for me. And while doing so, it will ask me all kinds of different questions. For example, what language do I want to use to author tests? And it is typescript, because I actually don't need to set up anything additional. I can just author my tests in typescript, and they will be transpiled to javascript and run seamlessly. No need to do Babel and webpack or anything. It will ask me where to put my tests. Of course, I want to have a github actions workflow. And yeah, examples. Examples are good. So now, once I answered all of the questions, it installs all the dependencies for the npm, downloads all the browsers. I have them downloaded already, so these have been reused. And now I am here with a bunch of sources pre-created for me. Let's look into these sources. First, we have this example end-to-end test. So, it is just seven lines of code, and let's pretend that we don't know what Playwright is. Let's just read it. So, first line, we import something, test and expect from the package we just installed. Now, test turned out to be a function, and we use it to create a test. We give it a basic name, and we pass in a callback, which is actually a test body. Interestingly, as an argument in the test body, we have this page that comes from the Playwright test framework. And this is a well-isolated test page that's different for every test, and no tests have pages or any of the state clashing. So, this is the isolation we talked about. Now, this is a browser page, so we can navigate it to a certain website, and then we can locate an element using a text on the page. And after that, we can click what we located. And everything is async, so we wait for it to complete. It looks like it does some kind of navigation, so after this, we can expect the page to have a certain title, like Getting Started Regex. Pretty transparent. Okay, what else do we have? We have this configuration file. This time, I will actually use Wim to go into it. Now, this is a generated file, and it has lots of comments so that you can easily go through it and understand what's going on. We don't need any of this stuff. I want to focus on this projects array. Now, projects are kind of environments. We have only one test, and we want to run this test in different browsers, like in Firefox or in Chrome. And this is what projects are for. So, here we have five projects pre-set up for us. We have Desktop Chrome, Desktop Firefox, Desktop Safari. All good. And we even have mobile emulation for Pixel 5, which is Android, and iPhone 12, which is a mobile Safari. So, we have one test and five projects, which means we'll run five times this test. Let's see how this actually happens. So, to run tests, I can do mpxplaywritetest. And as I run this command, it runs five tests using five workers. So, we've arrived at it actually runs five tests, which is nice. But the other important thing is that it actually uses five workers. So, all of these test runs are happening in parallel and actually only take eight seconds, which is pretty fast. Okay. So, out of the box, we didn't do anything. We already got cross-browser test runs. We got device emulation, and we got parallels, right? We can run these tests in parallel on our machine, which is very powerful. It saves lots of time. Now, I know people love reporters, and there are all kinds of different reporters that we can use to customize the output here. So, we can use a line reporter, for example, and this gives us this fancy interactive terminal prompt. Or I can say html reporter, and this will give me a visual result. So, it will run all the tests with a line reporter by default, and it will prompt me to open the result with this command. Now, I have my tests. They've been run into five different environments, five different projects, and I can click into my test to explore all the actions that happened inside of it. So, this is pretty cool and powerful. And I've also heard that folks love Allure, and you can actually do npm install dash d Allure playwright, and this will add a third-party Allure reporter that you can use with reporter Allure playwright, like this. And this will actually generate an Allure report for you. Okay, so this was a very fast getting started demo. Let's do a quick recap. So, we can create install playwright into projects with this npm init playwright helper script. And once we have playwright installed, we can actually run tests with npx playwright test, and it all runs in parallel. And this is what playwright actually looks like. It's very simple and intuitive api, and this is a single api to control all of the different browsers. And of course, we have reporters, we have Allure report, we have html reporter, we even have JUnit reporter, if you're into this kind of stuff. Okay, so we just played with how do we get started. Let's author some new tests. And now I want to show you a new thing, which is called CodeGen. So, for this, well, let me clear my screen. I'll do npx playwright CodeGen, and I'll save the results of what I will generate to github.spec.gs. So, as I do so, two windows are being opened. So, the first window, the browser window, is just a regular Chrome window. While the second one is this playwright inspector, and we'll talk about it a little bit later. And it has this glooming red record button highlighted. Now, everything that I do inside the browser is actually recorded by the recorder. And I just navigated to Microsoft, and I have this page code navigation. So, I can actually keep interacting with my web page. I can say playwright, wait for it to search, hit the playwright button. Actually, as you see, as I hover over different elements on the page, I have this black box with white text underneath my cursor. So, this shows me the selector that will be used as I will interact with the page. So, yeah, I'll hit issues, and I'll just keep going. Electron, type, and let's click on Pavel. And this is good enough. And I can turn off the recording and quit my browser. Now, let's see if we actually have things recorded. And we have our script, which is good, but does it work? So, to make sure that it actually works, I can run just this GitHub file. And this way, I actually filter by file name. But we have multiple projects. I don't need them for now. I just want to run in desktop Chrome, for example. And I actually want you to see how it's been run. So, I want to run it in a headed mode. And this is the browser window. It goes, navigates, and this is how the script progresses. And finally, it completes in a mere eight seconds. Pretty fast. Okay. So, it is really easy to author a new test with CodeGen. And if I want to change something, I can always go and hop into my editor of choice and change certain lines here and there. Now, let's pretend that this test actually does not work. How do I debug it? Well, it turns out it's pretty easy. I can just say test-debug for my test. And again, it will open two windows for me. This one is a browser window that will run this test. And I will resize it to uncover the second window, which is a Playwright Inspector window. So, I didn't have the opportunity to show you around this Playwright Inspector UI. So, here I am. Let me show you around. So, we have four parts in this UI. In the very center, we have this source code of the test that you are debugging. And we have a highlighted execution line, which is actually paused on the current Playwright position. I can control this execution line using the control panel here. For example, I can click step over to step over this command. And as I step over the navigation, well, the page in my browser navigates. Down below here, I have the log of all the actions that the Playwright is actually doing. I can see that page go to succeeded. There is a green check mark. It took just two seconds. And we are doing page click and we are paused. And you can expand these actions and see what's actually happening inside. And page click is already pre-expanded for me. And I can see all kinds of stuff that's happening here. And for example, Playwright was waiting for element to be visible and enabled and stable. And it scrolls in the view and does all kinds of stuff. And this is the auto-waiting that's built in. And this is how we actually make sure that all your clicks are actually happening for reals. We even make sure that the element receives pointer events in a certain location. And this is where we are about to click. And you can see this location in the browser window marked as a red dot. So this is where we are aiming right now. Okay. So last but not least, we have this fourth section, which is a selector playground. And it is currently highlighting this placeholder, find the repository. This is the selector that will be used for the action that is paused right now. I can change this and say, for example, text equals TS deep. And as I change and type different selectors here, elements are being selected in the browser page. And this is cool, but this explore button is even more cool. So as I click it, the Chrome window will activate. Bam. Now, as I hover over the different parts of the page, I can see different selectors here in this black box underneath my cursor. And as I click on this element, for example, this selector is being pasted into this input field. So I can further refine it, for example, or I can just copy it and paste it into my editor. And this is Playwright Inspector, and this is how we debug tests in Playwright. Let's go back to our presentation. So we just saw how we can generate tests with MBX Playwright CodeGen. And we saw Playwright Inspector, which is as easy to open as passing the debug flag to your command. And once you do so, you are presented with this beautiful, gorgeous UI that shows you everything you need to know about your test. So this all is very cool, but this all happened locally on my machine, which means the test was failing locally, and I had a very nice repro. I can reproduce the failure, and I can debug it using Playwright Inspector. But what would I do if this failure would actually happen somewhere in the cloud, and I would not have any way to reproduce it locally? Of course, I can take a screenshot or, for example, record a video of the whole run. But wouldn't it be cool if I can actually deep dive into the browser internals, into the browser console or network actions, or go and even maybe deep dive into the page DOM after each action of my test execution? So it turns out this is actually possible, and this is possible with something called Playwright Tracing. So let me tell you about Playwright Tracing. But before we actually talk about tracing, a little bit of terminology. So first things first, we call this term post-mortem debugging, and this is a debugging of the test failure that happens somewhere far away, and there is no way for me to reproduce it locally. So to debug this kind of failures, and these are very nasty failures, we usually use things such as test artifacts, which are any byproducts of test running. Things like logs or screenshots or videos, all of these actually count as test artifacts. Now in Playwright, we actually have this unique type of artifact, which is called tracing. And it turns out that tracing by itself is much more capable than both videos and screenshots combined. So let me explain you what it is, and I'll show you how it works. So Playwright Tracing is this technology that consists of two parts. First is this trace.zip files, and these are actually the artifacts. And this is a portable file format that has all the information about the Playwright test run, such as actions, events, screencast, network log, even console log, and DOM snapshots for each of the actions. And then we also have TraceViewer. And TraceViewer is this program that is actually bundled with Playwright that lets you look into these trace files and explore them. And since Playwright v1.17, we even have trace.playwright.dev, a special website where you can drop all the traces and it will be presented to you. So let me actually give you a demo of what it looks like. So here I conveniently have a repository created specifically for this presentation. But here in this repository, I have an action and it is failing. So let's explore. So for some reason, something is been timing out. And it even tells me go and open show trace. So let me actually do this. I'll go to summary and in github actions, we have all the artifacts uploaded. And if you actually use a GitHub action that we can pre-configure for you, it will upload all the artifacts automatically. So I can just download this trace file. I have it here. I unzip it. And this is a trace file. And I'll use this trace.playwright.dev file. And I will just get this trace and drag and drop it over the area. And this is the trace URUI. So a lot of stuff is actually going on here. So let me show you around. So up here, we have a timeline. And you can hover over timeline and scrub over it to see how things are happening inside your web page. This is pretty much like a screencast. But this screencast is actually overlaid with all the actions. These bars above the screencast, these are the actions. So for example, up here, I can see that navigation took something around 600 milliseconds. This is blue line is a page that go to call. And so on. I can see here's a click. Here is some expect. Here's another click. And so on. Now on the left-hand side, I have the list of all the actions in my script in a chronological order. And this is actually keyboard navigable. I can go up and down to select different actions. And for each actions, on the right-hand side, I have this panel with all the information about this action. And I have all the console messages that are happening during this action run. And even all the networks is happening here. So navigation is obviously very heavy on network. And in the middle here, we have a DOM snapshot. So this thing is not a screenshot, it's snapshot, which means it's a real html, real css. You can scroll it. You can even type into input fields. There is no javascript, so it will not react. But other than that, css, Hover, all of this stuff works. And you can even go and inspect it with devtools. So here I am. I can use devtools to look into the DOM of GitHub, which is very nice and gives me limitless capabilities. Now, for clicks and the way for navigations, like all of these actions we have here. And well, last but not least, we can see that this last one, page click, we tried to click on something called wolf and this action timed out. So, well, honestly, there is no wolf on this page, so it's very predictable. This was Traceviewer. So please use trace.playwright.dev. It's a very nice website. And once you do, you'll have this beautiful UI with all the goodies and bells and whistles that I showed you. And this is it for this presentation. Recap. So playwright test. Well, first of all, you can install it with npm new playwright. You can author new tests with CodeGen. You can debug your test with Inspector. And you can do postmortem debugging with tracing. So if you use playwright test, you are basically covered in all your life stages as a test author. If you like what you've seen, please head over to our documentation to learn more about playwright. We are also present on social channels. Please join our Slack. We are very active there. Follow us on Twitter, subscribe to our YouTube, and give us a star on GitHub. Thank you for listening.