Testing Mail Service With Playwright

Rate this content

We send emails to our users - account verification and newsletters. We allow the user to contact us by sending an email via inbuild form. Do we? Does the user receive an account verification email or exactly what notification they signed up for? We can cover this functionality as part of E2E tests: get an email and open it to check what is in it. We will need Playwright and a fake SMTP server to capture emails sent by the app.

17 min
03 Nov, 2022

AI Generated Video Summary

This Talk discusses how to test mail service with Playwright, covering e-mail verification, reset password user journey, and more. It explores the use of third-party providers for reliable e-mail delivery and demonstrates how Playwright can help perform checks on e-mail content. The Talk also introduces the concept of a fake SMTP server and showcases how fixtures can be used to access the SMTP server and perform assertions on the HTML body of emails. Playwright's HTML rendering feature allows for interaction with email content as if it were a regular web page. It highlights the ability to render HTML from API calls, perform assertions on the rendered page, and exclude dynamically generated data from visual regression tests.

1. Testing Mail Service with Playwright

Short description:

In this talk, I will show you how to test mail service with Playwright. We will cover e-mail verification, reset password user journey, and more. By using third-party providers like Amazon SES, MailChimp, Postmark, or SendGrid, we can ensure reliable e-mail delivery. However, the problem lies in what gets delivered to the user. Today, I will demonstrate how Playwright can help perform checks on e-mail content and ensure a seamless user experience. Additionally, I will introduce the concept of a fake SMTP server, which captures and stores e-mails in a test environment. Tools like Mailhug provide a user-friendly interface for manual verification and comprehensive API documentation.

Hi, everyone, my name is Kat Kniotek, I work as a quality engineer at Zoopla, and thank you for joining my talk today. I will be talking about testing mail service with Playwright. I will show you how easy it is to add e-mail verification or reset password user journey to your end-to-end tests. And honestly, I really like how clever you can go here with Playwright. So let's start.

Depending on the application you are working on, you may or you may not communicate with the user by the e-mail. But if you do, you would send them newsletters, order confirmations. If you work on the e-commerce website, reset password link if the action is requested by the user, e-mail verification request and any sort of receipts or notification they may ask for. And you usually use the third party providers to handle this, may transportation for you. And those would be Amazon SES, MailChimp, Postmark and SendGrid, just to name a few. And it's a really good idea because we can rely on them to make sure that e-mail is actually delivered. But the problem statement of this talk is what gets delivered to the user. Can they click the link and reset their password? Does verification e-mail actually take them to the correct domain? If body content of the e-mail actually matches our design system, and of course, if by mistake, we don't send any sensitive data or any random data like this test as a subject of the e-mail. So today, I will show you how all those checks can be performed with help of playwright. So let's start.

So to describe the e-mail communication in a production application, you will have application communicating with third party service, and third party service will be responsible for delivering e-mails to the user or user groups. In a test environment for this demo, instead of sending requests to the third party service, we'll be sending e-mail to fake SMTP service and then from there, we will be able to access data from playwrite tests. OK, I just introduced a new thing, a fake SMTP server, what it is. You can think about it as a container that will be capturing and storing all e-mails sent by application in a test environment. And for this purpose, you can choose one of many tools available. I listed just a few here. Some of them are paid and some of them are open source. It's up to you which one you will use. For this demo, I'm using Mailhug. All of them usually come with really nice graphical user interface that you can access in your browser that looks exactly like the mailbox. You can browse e-mails, you can delete e-mails, store attachments or forward e-mails. It's great for manual verification of what gets sent to the user. That sort of fake SMTP server will also come with great documentation about their API. So with Swagger. Swagger would list all available end points on the service with methods that are available and how the payload should look like and what you expect to see as a response of the API call to the server.

2. Integrating Mailhook API with Playwright Testing

Short description:

Here's the example call to the Mailhook API, querying for emails sent to the test user. The response includes the status, number of emails sent, and the email objects with ID, from, to, content, and body. We can perform assertions on the email content, including checking for specific strings, elements, and styling. To translate this API testing to Playwright tests, we need to access the SMTP server, retrieve the most recent email, and perform assertions on the HTML body. Playwright's HTML rendering feature allows us to interact with the email content as if it were a regular web page. We can click links, check for elements, and verify URLs. To start, we'll access the SMTP server from our end-to-end tests using fixtures, which provide isolated contexts with separate configurations.

Here's the screenshot of the example call to the Mailhook API. So as a search point I'm querying here. Based on their documentation, I provided query params to get all emails that has been sent to the test user at example.com and on the right-hand side, you can see how the response looks like. So good, because status is 200. And then you can see how many emails have been sent to this user, so the service endpoint returns three.

Brilliant, because I was just testing a little. And all of them are listed as email objects in the items array, so on line five, and each individual email object would have ID, from, to, content, and body. And if you look closer, below line 34, well, from line 34, you can see that body is actually HTML body, HTML code. So again, here we can do some assertions as well. We can check if the certain strings are present, if the certain elements are present, let's say button or link, but also verify if styling, right, styling has been applied, right? Just to mention, I'm using Thunder Client here. So, tools similar to the Postman or Insomnia, if you're familiar with them.

A fake SMTP server comes with some nice features allowing manual testing, but we'll use the same SMTP server in our play write tests, so let's have a look how to translate what I just showed with, let's say, API testing to play write tests. Let's start with pseudo coding, what do we want to do? So, first, our test will need to have precondition and that depends on your application. That could be the action that triggers email verification to be sent. Let's say, user creates the account or using completes the order and order confirmation gets sent and so on. Because I'm not connecting to any real application here, it was just a script that triggers sending email.

Okay, so what do we want to focus in the test? We want to do exactly what I did manually with connecting to the SMTP server. So, I want to access SMTP server, I want to get most recent email that has been sent to the user. So, in a response there was items array, so I want to get the first element of it, so with index zero. And I want to get an HTML body part, because that's where I want to do assertions on. And the great magic feature of Playwright, you can actually use this HTML body to render a new page and do the same assertions as you would do on a regular page in a browser. So, you can click verify button link, you can check if certain elements are present and you can also verify if by clicking the verify button, you will open a new tab and you can do assertion if the new tab which is also page, will have expected URL. Okay, so, that's a lot to do. So, where to start? I will start with accessing SMTP server from my end-to-end tests. And for this purpose, I will be using fixtures. What fixtures are? So, probably in your application, you have Playwright Config. Well, in your end-to-end tests, you have Playwright Config and you specify base URL value there. And that would be fronted of your application client code, the base URL of your old tests. But you can specify also fixtures, so like isolated contexts that will be used in a certain test. So, here I'm creating the fixture called email API and that one will have separate base URL value, extra HTTP headers and so on.

3. Accessing SMTP Server with Fixtures

Short description:

Fixtures allow you to add different endpoint tests from the same repository. The API context is defined on line 11 and used for all calls made to the email API. On line 14, the get method is used to query the search endpoint and return the response. The HTML body of the most recent email is obtained on line 26, and any encoding is cleaned up on line 28. The HTML body is then used to render the email page and interact with it. The test consists of preconditions, using the email API to get the email body, rendering the email page, clicking a button, and asserting the URL of the newly opened tab.

So, that fixtures allows you to add maybe different endpoint tests from the same repository test also backend or other services like here example, Mailhook. So, as you can see, I provide the base URL value which is running on localhost Mailhook server. I pass authorization as an extra HTTP headers and I also specify to ignore HTTP errors as true because I'm trying to connect to localhost. And this API context is defined on line 11, will be used on all calls made to email API.

Okay, and how it looks like actually when you use it. So, on line 14, here I see I can see that yeah on the left-hand side you probably don't see this 14, it's 14, line 14, I do exactly the same what I did in the Tandr client whether it was manually testing endpoint. So, on the API context that I just defined, I do use the get method to query the search endpoint for the query params the same as in the Tandr client example. So, I query all the emails that has been sent to the test user and then I return the response.

And here, this may look a little bit uh complex, what is happening here? Here on line 26, I get the first email, so items index one, content and body to get the HTML body of the most recent email that has been sent to the user. What is happening on line 26 might look a little bit confusing, so when I was showing in a tender client how the body looks like, the HTML body of the email, maybe you noticed that they were appended equals 3D strings in some places. This is because the type of encoding used for email transportation called quoted printable. So here on line 28, I just tidy up the email body from this encoding. Looks complex, but I rely on helpers libraries to do it for me. And then at the end, I just return the HTML body.

So now this HTML body can be used to render the page. So here in a email page object, I define my custom go to method. That takes as a parameter, actually the HTML code from the email. And on line 14, I instruct playwright. Every time I would be going to forward slash email URL, I want this route request to be fulfilled with the email body. So then when on line 16 actually I go to this URL, what will be rendered in the browser will be actually email body. And then I will be able to interact with it as I would do with any page. So I will be clicking verify button on line 22 and wait until new tab will be open and will return the new tab.

Putting it all together on line 15, basically that's how the test will look like, so obviously first precondition. But then as I said, I specify the separate fixture so that's how I imported it to the particular tests, email API on line 15. So, then I use the email API to get email body on line 18 for the test user, the test user I use for manual verification as well. Once I have this HTML body online, I think it's line 12, I can use this HTML body to render the email page, click the button on this email page and then assert that actually URL of this newly opened tab matches test.js summit. Okay, that may work, may not, but if I will run the test actually in a debug mode, I can see browser opening and Playwriting Inspector opening beside it, so I can step from the test and see that it actually works. So what is happening here? That's the actual first page that opens, because the API call happens in seconds, just before the browser opens. So here we are in the step that I go to forward slash email, and the email is actually rendered. And here, player attempts to click element button with ID Button 1.

4. Advanced Email Testing Techniques with Playwright

Short description:

You can use Playwright to render HTML from API calls and perform assertions on the rendered page. Playwright allows you to exclude dynamically generated data from visual regression tests. With Playwright, you can check if emails are delivered, verify button functionality, and perform assertions on DOM elements. The links provided include Mailhook Swagger, a Mailhook Docker image with OAuth, and information about equal 3D encoding and the Tandr client for manual API tests.

And once that one is successful, the new tab is open. And as you can see, the URL is exactly the one that we expected to appear here. So when I've seen it for the first time, I was like, whoa, that's amazing. You can actually get HTML from the API call and use this HTML to render the page. Yeah, it's amazing. And going from here, you can think how many other assertions you could do on the actually rendered page.

So you can do visual regression tests. So do the screenshot comparison to make sure that nothing has changed in the version of the email that you are sending now comparing to the baseline image. Here you may say, okay, I sent to the user as a maybe title or, I don't know, a header of the email. I rendered their username or their first name or initials. And that will be different with every test run, because you dynamically generate test data. So here with Playwright API, you can exclude this area from the comparison. So let's say mask the H1 locator. And that will help you with the testing the email if the data is dynamically generated.

You can also, as I mentioned, once we have this email rendered as a page, you can do all sorts of assertions like as you would do on a regular page. So let's say that check the verify button actually has the correct attribute or if the state is changing when you hover over and so on, so on. So I hope that I showed how the playwright can be used for testing email body and what actually gets sent to the user. So I checked that email got delivered to the user. I checked functionality of the button so you could see that action click has been successful, that the navigation happened to the correct URL. With Playwright you are also able to do visual check on the email, on the content body and obviously all other assertion on DOM elements as you would do.

OK, that's end of the talk. I hope that you found it useful and the links that I just posted here will help you to maybe try and build a similar solution and apply it to your tests. As I mentioned, I was using Mailhook so I provided a link to Mailhook Swagger. Mailhook docker image with OAuth so you can just run it on locally. More explanation about equal 3D, so the quoted printable encoding. A few words about Tandr client, so my go-to tool for manual API tests and link to contact with me if you would have any questions. So, yeah, I just look forward to hear your questions now. If you want to access the presentation, you can just scan the QR code and that will take you to the slides.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
23 min
Scaling Up with Remix and Micro Frontends
Do you have a large product built by many teams? Are you struggling to release often? Did your frontend turn into a massive unmaintainable monolith? If, like me, you’ve answered yes to any of those questions, this talk is for you! I’ll show you exactly how you can build a micro frontend architecture with Remix to solve those challenges.
Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
JSNation Live 2021JSNation Live 2021
29 min
Making JavaScript on WebAssembly Fast
JavaScript in the browser runs many times faster than it did two decades ago. And that happened because the browser vendors spent that time working on intensive performance optimizations in their JavaScript engines.Because of this optimization work, JavaScript is now running in many places besides the browser. But there are still some environments where the JS engines can’t apply those optimizations in the right way to make things fast.We’re working to solve this, beginning a whole new wave of JavaScript optimization work. We’re improving JavaScript performance for entirely different environments, where different rules apply. And this is possible because of WebAssembly. In this talk, I'll explain how this all works and what's coming next.
React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2022TestJS Summit 2022
20 min
Testing Web Applications with Playwright
Testing is hard, testing takes time to learn and to write, and time is money. As developers we want to test. We know we should but we don't have time. So how can we get more developers to do testing? We can create better tools.Let me introduce you to Playwright - Reliable end-to-end cross browser testing for modern web apps, by Microsoft and fully open source. Playwright's codegen generates tests for you in JavaScript, TypeScript, Dot Net, Java or Python. Now you really have no excuses. It's time to play your tests wright.

Workshops on related topic

React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
React Summit US 2023React Summit US 2023
96 min
Build a powerful DataGrid in few hours with Ag Grid
Does your React app need to efficiently display lots (and lots) of data in a grid? Do your users want to be able to search, sort, filter, and edit data? AG Grid is the best JavaScript grid in the world and is packed with features, highly performant, and extensible. In this workshop, you’ll learn how to get started with AG Grid, how we can enable sorting and filtering of data in the grid, cell rendering, and more. You will walk away from this free 3-hour workshop equipped with the knowledge for implementing AG Grid into your React application.
We all know that rolling our own grid solution is not easy, and let's be honest, is not something that we should be working on. We are focused on building a product and driving forward innovation. In this workshop, you'll see just how easy it is to get started with AG Grid.
Prerequisites: Basic React and JavaScript
Workshop level: Beginner
TestJS Summit 2023TestJS Summit 2023
89 min
Building out a meaningful test suite that's not all E2E
We're all taught to follow the Testing Pyramid but the reality is that we build out the Testing Christmas Tree. In this workshop, David will talk you through how to break down projects and put the tests where they need to be. By the end of the workshop you will be able to update your projects so that anyone and everyone can start contributing and truly living up to "Quality is everyone job".
He will walk you through:- Component Testing- API Testing- Visual Regression Testing- A11Y testing
He will also talk you through how to get these all setup in your CI/CD pipeline so that you can get shorter and faster feedback loops.