Playwright Test Runner

Rate this content
25 min
18 Nov, 2021

AI Generated Video Summary

The Playwright Test Runner is a cross-browser web testing framework that allows you to write tests using just a few lines of code. It supports features like parallel test execution, device emulation, and different reporters for customized output. Code-Gen is a new feature that generates code to interact with web pages. Playwright Tracing provides a powerful tool for debugging and analyzing test actions, with the ability to explore trace files using TraceViewer. Overall, Playwright Test offers installation, test authoring, debugging, and post-mortem debugging capabilities.

1. Introduction to Playwright Test Runner

Short description:

Hi everybody, my name is Andrej, and today I would like to introduce you to the new Playwright Test Runner. Playwright Test is a cross-browser web testing framework written in Node.js. It is free and open source, sponsored by Microsoft, and extensively used in the industry. Playwright Test was built specifically to address all end-to-end testing needs. Let's jump straight into the first chapter, which is getting started. I will create a new folder, initialize a new npm project using Playwright, and answer some questions to set up the project.

Hi everybody, my name is Andrej, and today I would like to introduce you to the new Playwright Test Runner. So let's start.

So first thing first, what is Playwright Test? Well, it is a cross-browser web testing framework written in Node.js, which basically means that you can author your tests in JavaScript or TypeScript. Now it is free and open source and sponsored by Microsoft and is already extensively used in the industry. Now this is all good, but it doesn't answer the big looming question. So why did we do yet another Test Runner? So honestly, we did not intend to. There is a lot of different test runners in JavaScript, turns out they are all unit test runners. Whereas in end-to-end testing, there's lots of very hard challenges. So for example, you want your test runner to support cross-browser tests out of the box. You want them to be parallelized out of the box as well. And you want them to be isolated, and Playwright, for example, has a set of unique APIs to make sure that test isolation is fast and reliable. And of course, we want these to be used. And last but not least, history shows that end-to-end tests require a crazy level of flexibility and configuration, because there are many different configurations we want to use to run the same test. And Vivio is inspired by PyTest fixtures, so fixtures are implemented in PlayWriteTest as well. So PlayWriteTest is our answer to end-to-end tests. And this is the test runner that was built specifically to address all of the end-to-end testing needs.

So we have quite a lot to talk about in this talk, and we have just 20 minutes. So let's jump straight into the first chapter, which is getting started. Okay, so the best way to show something is actually to show something. So let me hop to my terminal and I will create a new folder. And I will start from scratch and initialize a new npm project. Now let's do npm init PlayWrite. And what this does is actually sets up this project to use PlayWrite for me. While doing so, it will ask me all kinds of different questions. For example, what language do I want to use to author tests? And it is TypeScript because I actually don't need to set up anything additional, I can just author my tests in TypeScript and they will be transpiled to JavaScript and run seamlessly, no need to do Babel and Webpack or anything. It will ask me where to put my tests, of course I want to have a GitHub Actions workflow, and yeah, examples are good. So now once I answered all the questions, it installs all the dependencies for the NPM, downloads all the browsers, I have them downloaded already so these have been reused. And now I am here with a bunch of sources pre-created for me. Let's look into these sources. First we have this example end-to-end test.

2. Playwright Test Runner: Features and Configuration

Short description:

The Playwright Test Runner allows you to write tests using just a few lines of code. It provides a well-isolated test page for each test, ensuring no clashes between pages or state. You can navigate the test page to a specific website, locate elements, and interact with them. The configuration file allows you to set up different environments, such as running tests in different browsers. The Playwright Test Runner supports cross-browser testing, device emulation, and parallel test execution out of the box.

It is just seven lines of code and let's pretend that we don't know what Playwright is, let's just read it. So first line we import something, test and expect from the package we just installed. Now test turned out to be a function and we use it to create a test. We give it a basic name and we pass in a callback which is actually a test body. Interestingly as an argument in the test body, we have this page that comes from the Playwright test framework. This is a well isolated test page that is different for every test and no tests have pages or state clashing. So this is the isolation we talked about. This is a browser page so we can navigate it to a certain website and then we can locate an element using a text on the page and after that we can click what you located. And everything is async so we wait for it to complete. It looks like it does some kind of navigation so after this we can expect the page to have a certain title like getting started regex. Pretty transparent.

OK what else do we have? We have this configuration file. This time I will actually use Wim to go into it. Now this is a generated file and it has lots of comments so you can easily go through it and understand what's going on. We don't need any of this stuff. I want to focus on this projects array. Now projects are kind of environments. We have only one test and we want to run this test in different browsers like Firefox or in Chrome and this is what projects are for. So here we have five projects pre-setup for us. We have Desktop Chrome, Desktop Firefox, Desktop Safari. All good. And we even have mobile emulation for Pixel 5 which is Android and iPhone 12 which is a Mobile Safari. So we have one test and five projects which means we'll run five times for this test. Let's see how this actually happens. So to run tests I can do mbxplaywrite test and as I run this command it runs five tests using five workers. So we will write that it actually runs five tests which is nice. But the other important thing is that it actually uses five workers. So all of these test runs are happening in parallel and actually only take eight seconds which is pretty fast. OK so out of the box we didn't do anything. We already got cross browser test runs, we got device emulation and we got parallels.

3. Running Tests in Parallel and Code-Gen

Short description:

We can run tests in parallel, customize the output with different reporters like Allure, HTML, and JUnit. Playwright has a simple and intuitive API to control different browsers. We can also use Code-Gen to generate code and interact with web pages.

We can run these tests in parallel on our powerful machines, which saves a lot of time. There are various reporters available for customizing the output, such as the line reporter for an interactive terminal prompt or the HTML reporter for visual results. By default, the tests run with a line reporter, but we can also generate an AllureReport using a third-party AllureReporter. Playwright provides a simple and intuitive API to control different browsers, and we can use different reporters like Allure, HTML, and JUnit.

Now, let's do a quick recap. We can install Playwright into projects using the npm init playwright helper script. Once Playwright is installed, we can run tests with npx playwright test, and it will run in parallel. The Playwright API is straightforward and allows us to control all different browsers. We also have options for generating reports like Allure, HTML, and JUnit.

Next, let's talk about Code-Gen. We can use the command 'npx playwright code-gen' to generate code and save it to a file. This opens two windows, a regular browser window, and the Playwright Inspector. The Inspector has a record button that allows us to record interactions with the web page. We can navigate to different pages and keep interacting with them.

4. Using Playwright Test Runner and Code-Gen

Short description:

We can install Playwright into projects using npm init playwright helper script. Once installed, we can run tests with npx playwright test. Playwright provides a simple and intuitive API to control different browsers. It supports various reporters, including LUR, HTML, and JUnit. Now, let's explore a new feature called Code-Gen. By using mpxplaywrite code-gen, we can generate code and interact with web pages. The generated code can be used to navigate web pages, interact with elements, and perform actions. We can also run the generated script to verify its functionality.

So we can create install playwrights into projects with this npm init playwright helper script. And once we have playwright installed, we can actually run tests with npx playwright test. And it all runs in parallel. And this is what playwright actually looks like. It's very simple and intuitive API. And this is a single API to control all of the different browsers. And of course we have reporters, we have a LUR report, we have HTML report, we even have the JUnit report if you're into this kind of stuff.

Okay, so we just played with how do we get started. Let's author some new tests. And now I want to show you a new thing which is called Code-Gen. For this, let me clear my screen. I'll do mpxplaywrite code-gen and I'll save the results of what I will generate to As I do so, two windows are being opened. The first window, the browser window, is just a regular Chrome window, while the second one is this Playwright Inspector, and we'll talk about it a little bit later, and it has this gloomy red record button highlighted. Now, everything that I do inside the browser is actually recorded by the recorder, and I just navigated to Microsoft, and I have this page code navigation. So I can actually keep interacting with my web page. I can say Playwright, wait for it to search, hit the Playwright button. Actually, on the web page I have this black box with white text underneath my cursor. So this shows me the selector that will be used as I will interact with the page. So I'll hit issues, and I'll just keep going. Let's click on Pawel. This is good enough. I can turn off the recording and quit my browser. Now, let's see if we actually have things recorded. We have our script, which is good. But does it work? So, to make sure that it actually works, I can run just this github file, and this way I actually filter by filename. But we have multiple projects. I don't need them for now. I just want to run in desktop chrome, for example. And I actually want you to see how it's been run.

5. Debugging Tests with Playwright Inspector

Short description:

In a headed mode, the script progresses in the browser window and completes in 8 seconds. Authoring a new test with Codegen is easy, and debugging is simple with the dash dash debug command. The playwright inspector UI provides source code, execution control, and action logs.

So, I want to run it in a headed mode. And this is the browser window. It goes, navigates, and this is how the script progresses. And finally, it completes in mere 8 seconds. Pretty fast.

OK. So it is really easy to author a new test with Codegen, and if I want to change something, I can always go and hop into my editor of choice and change certain lines here and there.

Now, let's pretend that this test actually does not work. How do I debug it? Well, it turns out it's pretty easy. I can just say, dash dash debug for my test. And again, it will open two windows for me. This one is a browser window that will run this test. And I will resize it to uncover the second window, which is a playwright inspector window. So I didn't have the opportunity to show you around this playwright inspector UI, so here I am. Let me show you around. We have four parts of this UI. In the very center we have this source code of the test that you're debugging. And we have a highlighted execution line, which is actually paused on the current playwright position. I can control this execution line using the control panel here. For example, I can click step over to step over this command. And as I step over the navigation, the playwright page in my browser navigates. Down below here I have the log of all the actions that the playwright is actually doing. I can see the page go to succeeded. There is a green check mark. It took just two seconds. And we're doing page click, and we're paused. And you can expand these actions and see what's actually happening inside. And page click is already pre-expanded for me. And I can see all kinds of stuff that's happening here. And, for example, playwright was waiting for element to be visible, and enabled, and stable.

6. PlayWrite Test Runner: Debugging and Tracing

Short description:

The auto-weighting feature ensures that all clicks are happening accurately. The selector playground allows for easy selection and exploration of elements on the browser page. PlayWrite Inspector provides a comprehensive UI for test debugging. PlayWrite tracing enables deep diving into browser internals, console, network actions, and page DOM after each test action.

And it scrolls into the view, and it does all kinds of stuff. And this is the auto-weighting that's built-in, and this is how we actually make sure that all your clicks are actually happening for reals. We even make sure that the element receives pointer events in a certain location. And this is where we are about to click, and you can see this location in the browser window marked as a red dot. So this is where we are aiming right now.

Okay. So last but not least, we have this fourth section which is a selector playground. And it is currently highlighting this placeholder, find the repository. This is the selector that will be used for the action that is paused right now. I can change this and say, for example, text equals TS lib. And as I change and type different selectors here, elements are being selected in the browser page. And this is cool, but this Explore button is even more cool. So as I click it, the Chrome window will activate. Bam! Now, as I hover over the different parts of the page, I can see different selectors here in this black box underneath my cursor. And as I click on this element, for example, the selector is being pasted into this input field. So I can further refine it, for example. Or I can just copy it and paste it into my editor. And this is PlayWrite Inspector, and this is how we debug tests in PlayWrite. Let's go back to our presentation.

So we just saw how we can generate tests with MBX PlayWrite CodeGen. And we saw PlayWrite Inspector, which is as easy to open as passing the "-debug flag to your command. And once you do so, you are presented with this beautiful, gorgeous UI that shows you everything you need to know about your test. Okay. So this all is very cool, but this all happened locally on my machine. Which means the test was failing locally and I had a very nice repro. I can reproduce the failure and I can debug it using PlayWrite Inspector. But what would I do if this failure would actually happen somewhere in the cloud and I would not have any way to reproduce it locally? Of course, I can take a screenshot, or for example, record a video of the whole run. But wouldn't it be cool if I can actually deep dive into the browser internals, into the browser console or network actions? Or, you know, go and even maybe deep dive into the page DOM? After each action of my test execution. So it turns out this is actually possible. And this is possible with something called playwrite tracing.

7. Playwright Tracing: Exploring TraceViewer

Short description:

Playwright tracing is a unique type of artifact that is more capable than videos and screenshots combined. It consists of files, which are portable file formats containing all the information about the test run. TraceViewer is a program bundled with playwright that allows you to explore trace files. Additionally, is a website where you can drop traces and view them. In a demo, the speaker shows how to open a failing trace file and explores the features of TraceViewer.

So let me tell you about playwrite tracing. But before we actually talk about tracing, a little bit of terminology. So first things first, we call this term post-mortem debugging. Because this is a debugging of a test failure that happens somewhere far away. And there is no basic way for me to reproduce it locally. So to debug this kind of failures, and these are very nasty failures. We usually use things such as test artifacts, which are any byproducts of test running. Things like logs or screenshots of videos, all of these actually count as test artifacts.

Now in Playwright, we actually have this unique type of artifact, which is called tracing. And it turns out that tracing by itself is much more capable than both videos and screenshots combined. So let me explain you what it is and I'll show you how it works. So playwright tracing is this technology that consists of two parts. First is this files and these are actually the artifacts. And this is a portable file format that has all the information about the playwright test run, such as actions, events, screencast, network log, console log and DOM snapshots for each of the actions. And then we also have TraceViewer and TraceViewer is this program that is actually bundled with playwright that lets you look into this trace files and explore them. And since playwright v.1.17, we even have, a special website where you can drop all the traces and it will be presented to you.

So let me actually give you a demo of what it looks like. So here I conveniently have a repository created specifically for this presentation. But here in this repository I have an action and it is failing. So let's explore. So for some reason something is, is been timing out and it even tells me go and open show trace. So let me actually do this. I'll go to summary and in GitHub actions we have all the artifacts uploaded, and if you actually use a GitHub action that we can pre-configure for you it will upload all the artifacts automatically. So I can just download this this trace file. I have it here, unzip it, and this is the trace file and I'll use this trace.playwright.def file and I will just get this trace and drag and drop it over the area. This is the trace URUI. So a lot of stuff is actually going on here, so let me show you around. Up here we have a timeline and you can hover over the timeline and scrub over it to see how things are happening inside your web page. This is pretty much like a screencast but the screencast is actually overlaid with all the actions like these bars above the screencast. These are the actions.

8. Using Traceviewer and Playwright Test

Short description:

In Traceviewer, I can see the navigation, clicks, expects, and other actions in my script. The panel on the right provides information about each action, including console messages and network activity. The Dom Snapshot allows me to interact with the real HTML and CSS of a page. I can scroll, type in input fields, and inspect the DOM using DevTools. Traceviewer is a powerful tool for debugging and analyzing test actions. If you're interested, visit for a beautiful UI and more features. Playwright Test offers installation with npm, test authoring with code.gen, debugging with inspector, and post-mortem debugging with tracing. Check out our documentation and join our social channels for more information.

So for example up here, I can see that navigation took something about 600 milliseconds. This blue line is a page that goes to call and so on I can see here's a click, here's some expect, here's another click, and so on.

Now on the left hand side I have the list of all the actions in my script in a chronological order and This is actually keyboard navigable. I can go up and down to select different actions and For each actions on the right hand side. Oops, sorry I have this panel with all the information about this action and I have all the console messages that are happening during this action run and And even all the networks that's happening here Right.

So navigation is obviously very heavy on network And in the middle here We have a Dom Snapshot. So this thing is not a screenshot It's snapshot, which means it's a real HTML real CSS. You can scroll it. You can even you know type Into input fields. There is no JavaScript, so it will not react but other than that CSS hovers all of the stuff works and You can even go and inspect it with DevTools So here I am I can use DevTools to look into the DOM of GitHub Which is very nice and it gives me limitless capabilities now for clicks and With Navigations, we like all of these actions we have here and Well last but not least we can see that this last one page click we tried to click on something called wolf and This action timed out so well, honestly, there is no wolf on this page. So it's very predictable This was Traceviewer

So, please use it's a very nice website and once you do you'll have this beautiful UI with all the goodies and bells and whistles that I showed you and This is it for this presentation recap

So playwright test Well, first of all, you can install it with npm playwright You can author new tests with code.gen You can debug your tests with inspector and you can do post-mortem debugging with tracing so If you use playwright test, you are basically covered in all your live stages as a test author If you like what you've seen Please head over to our documentation to learn more about playwright We are also present on social channels. Please Join our slack.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.
TestJS Summit 2021TestJS Summit 2021
36 min
Effective Performance Testing to your Server with Autocannon
Performance testing expertise that is developed for a long time. In order to measure your server performance you need a tool that can efficiently simulate a lot of abilities and give you good measurements according your analysing criteria.Autocannon NPM library gave me exactly that - that library is super easy to install and has a very simple API to work with. Within a really short amount of time you can start do performance testing to your application and get good measurements in development environment and in your performance labs, and generate complicated testing scenarios.In this talk I will introduce Autocannon, explain how to efficiently analyse your server performance with it, and show how it helped me to understand complicated performance issues in my Node.js servers. At the end of this lecture, developers will be able to have the ability to integrate a fast and easy tool in order to measure your server performance.

Workshops on related topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.