Synthetic Monitoring and E2E Testing - 2 Sides of the Same Coin

Rate this content

Over the last few years, there has been a great focus on shifting tests left to earlier in the software development cycle. It's great that the emergence of trends such as e2e testing has enabled us to validate the user workflow earlier. However, they do not eliminate the need to monitor this workflow against a live production system. I have seen that the segregation of development and support teams leads to both teams using different tools to build similar automation that simply targets different environments of the same software.
By joining forces, we can build automated scripts that can be used both for production monitoring and testing within CI. In this talk, I will discuss the causes of this cultural divide, and why this needs to change for us to shift user testing both left and right. Furthermore, I shall show how this can be achieved using Playwright and Elastic Synthetics and also chat through the challenges you may face shifting left and right.

9 min
06 Jun, 2023

Video Summary and Transcription

Carly Richards discusses the challenges of using different tools for synthetic monitoring and end-to-end testing, emphasizing the need for a unified approach with playwright and Elastic Synthetics. The Playwright API and Elastic Synthetics can be used together to create tests and monitoring tools, ensuring consistent user experience and documenting application features. By bringing development and SRE teams together and using a common tool, collaboration and defect identification can be improved.

Available in Español

1. Challenges of Tooling and the Need for Unification

Short description:

In this part, Carly Richards discusses the challenges of using different tools for synthetic monitoring and end-to-end testing. She explains how the lack of collaboration between development and operations teams, differing priorities, and the use of different tooling can hinder automation efforts. Carly emphasizes the need for a unified approach and introduces playwright and Elastic Synthetics as a solution. By utilizing these tools, teams can create scripted journeys that serve both as tests and monitoring tools, ensuring consistent user experience and documenting application features. Carly also provides an overview of the implementation process and highlights the importance of maintaining the monitors within a project structure.

Hi, everyone. It's great to see you at React Summit. My name is Carly Richards, I'm a developer advocate at Elastic. And I want to talk about synthetic monitoring and end-to-end testing. You might wonder why these things might seem totally different constructs. But despite the emergence of DevOps and SRE practices, and about trying to unite among custom workflows, I'm finding that, for various reasons, we're still using different tools to try and achieve rather similar levels of automation, albeit at different points in the development cycle.

I'm going to talk about my own experiences of why I think that happens. And then I'm going to show how we can unite alongside a common tooling by using embedded end-to-end testing within a synthetic monitoring tooling, to allow us to basically do both of these things using the exact same scripted journeys.

So I used to work in a bank before I joined Elastic, and for the majority of that time, I found that dev and ops were not as joined up as we'd like to say. They were basically warring factions with competing interests, and the emerging best practices from SRE and DevOps coming through the observability space often were picked up by production management, but it never really permeated back very easily into the developer side of the fence. And it took a lot of convincing for that to happen. And I honestly think that stemmed from three key reasons that sadly I think still exist and some way today. The first is that we often have Dev and SRE as separate departments or separate teams, rather than being part as one multidisciplinary team. And with the separate departments with competing agendas, lack of communication and empathy tends to build up, which can be a big problem because it causes each group to feel like when someone comes with an idea of how to collaborate together, that we're throwing things over the fence for the other to pick up and break the flow and basically call into question the direction in which we knew we were meant to build as a developer or as an SRE. The second is often that there isn't a common prioritisation between these two groups. SRE can often find minor defects or have ideas for automation or enhancements to the application ecosystem but quite often those are deprioritised and instead you'll have a product owner pushing for new features for users that developers will happily be going away to build and we don't ever address this balance. And the final challenge that I've seen the first hand as well is that often development production teams are using different tooling to achieve similar objectives. So in my last role at the bank we were actively using cypress for developing end-to-end tests meanwhile colleagues in SRE were writing similar automations in Selenium because it integrated into their observability platform and they could use it for periodic monitoring and checks against the production application. And it meant that we couldn't really use the end-to-end tests in production without either of us switching to a new tooling. And the reality is the end-to-end testing using automation frameworks to validate the user journey through automated tests and synthetic monitoring where we run periodic scripts to either check the accessibility or endpoints or the achievability of user behavior to make sure it's still operating correctly are effectively two sides of the same coin. And if we unite amongst a common tooling we can end up using these same scripts not only as testing and monitoring tools in the same application suites across different environments, but we can also use them as a common way of documenting the user journey and how we expect users to use their application features.

So using playwright and Elastic Synthetics we can do that. The way that this works as you'll see in grey going through each of the boxes is that we'll have JavaScript and TypeScript journey files that will basically utilize playwright to automate those interactions and run locally as end-to-end tests against a locally running web app. Then through peer review, we can run the same monitors as end-to-end tests within the GitHub actions CI pipeline or any other CI provider that you're using. Once we get to deployment stage of our app, we can then push these same monitors using our own API key into the appropriate location, either the Elastic-specific location or our own running private location, to do the monitoring instead against the production web application, which will then store the results within Elasticsearch and make available the dashboards in Kibana so that we can actually see what's going on.

So how do we get started with this? Well, we need to create a project to house these monitors so that they can be maintained within source control using from both parties. The easiest way to do that is using the init wizard after installing the Elastic Synthetics Playwright wrapper via NPM. It'll give you a sample project structure looking like this, which will include synthetics configs for the specific configuration for the project and some example monitors including the TypeScript journeys that you can see there under the journeys folder. We then are able to write our own behavioral tests building on the examples using Elastic Synthetics. You'll see here on the top that the page object which is the playwright page object is exposed within our journey and also within each stick.

2. Using Playwright API and Elastic Synthetics

Short description:

We can use the Playwright API to locate elements, perform actions, and make assertions to ensure the expected results. These tests can be run locally and in CI pipelines, and the same definitions can be pushed to production using Elastic Synthetics. SRE audiences can monitor the state and track performance over time. By bringing development and SRE teams together and using a common tool, we can collaborate, identify defects, and build better applications. Thank you for attending React Summit!

And we can make use of the Playwright API to do things like locate elements via CSS selector, which you can see on line 16, by the various helper attributes introduced in versions 2017 and later, such as get by test ID. We can perform clicks and actions that a user would against these appropriate HTML elements and then we can perform various assertions to see that the appropriate result is showing there.

And then we can run these locally and make sure that all of the changes associated with our new feature are passed. We can then run the same things within our CI pipelines as well and then we can push those same definitions to production to the Elastic Synthetics location which will allow you to run those same tests as monitors against your production application.

Then SRE audiences can monitor the state. They can see the basic pings, they can see as highlighted in the background of these cards what the duration has been over time using the chart in the background. Then when things go wrong we can actually see the individual steps, we can see what particular expectations or errors happened and we can then do smart things like have alerts or anomaly detections to try and pick up if there's perhaps a degradation in performance if these particular tests are taking longer as per our trends.

So for me DevOps is always meant to be about bringing the dev and the SRE factions together. We need to leave that cultural baggage behind that's been sitting for so long and work together and if we use a common tool there's some great benefits to be afforded in terms of documenting the user journey, collaborating together, to pick up defects in testing and in production and just generally best building better applications for users.

So thank you so much it's been great speaking here at React Summit. Hopefully I get to catch you around the conference and if not feel free to check out the code example on the QR code on the screen or you can also contact me with any questions at currentlyellrichmond.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2022TestJS Summit 2022
20 min
Testing Web Applications with Playwright
Top Content
Testing is hard, testing takes time to learn and to write, and time is money. As developers we want to test. We know we should but we don't have time. So how can we get more developers to do testing? We can create better tools.Let me introduce you to Playwright - Reliable end-to-end cross browser testing for modern web apps, by Microsoft and fully open source. Playwright's codegen generates tests for you in JavaScript, TypeScript, Dot Net, Java or Python. Now you really have no excuses. It's time to play your tests wright.
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.
TestJS Summit 2022TestJS Summit 2022
21 min
Tiny Tests, Large Results
Yes, Big things do come in small packages. For example, isn’t a unit test’s speed, feedback, and reliability fantastic? Did you know we can also have fast, focused, and reliable feedback from our functional e2e tests? Atomic e2e tests are those that are targeted and focused. They’re tiny in size but large in their impact. This tutorial will teach you how to create atomic e2e tests with several code examples. First, we will use to authenticate by setting a cookie. Instead of using a UI. Second, we will use to set a JSON Web Token for authentication. Join me, and let’s write tiny tests for large results.
TestJS Summit 2022TestJS Summit 2022
17 min
Testing Mail Service With Playwright
Top Content
We send emails to our users - account verification and newsletters. We allow the user to contact us by sending an email via inbuild form. Do we? Does the user receive an account verification email or exactly what notification they signed up for? We can cover this functionality as part of E2E tests: get an email and open it to check what is in it. We will need Playwright and a fake SMTP server to capture emails sent by the app.
TestJS Summit 2022TestJS Summit 2022
21 min
E2E Tests for Web3 Applications
Top Content
We will go through a brief explanation of what is Web3 and the architecture of a web3 application. Then we will talk about how to end-to-end test, its challenges, some test tools that are available, and a demo using cypress and metamask.Agenda: What is Web3; The Architecture of a Web3 Application; Web3 E2E Tests Introduction; Web3 E2E Tests Challenges; E2E Test Tools; Demo.

Workshops on related topic

React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
TestJS Summit 2023TestJS Summit 2023
89 min
Building out a meaningful test suite that's not all E2E
We're all taught to follow the Testing Pyramid but the reality is that we build out the Testing Christmas Tree. In this workshop, David will talk you through how to break down projects and put the tests where they need to be. By the end of the workshop you will be able to update your projects so that anyone and everyone can start contributing and truly living up to "Quality is everyone job".
He will walk you through:- Component Testing- API Testing- Visual Regression Testing- A11Y testing
He will also talk you through how to get these all setup in your CI/CD pipeline so that you can get shorter and faster feedback loops.
TestJS Summit 2021TestJS Summit 2021
146 min
Live e2e test debugging for a distributed serverless application
In this workshop, we will be building a testing environment for a pre-built application, then we will write and automate end-to-end tests for our serverless application. And in the final step, we will demonstrate how easy it is to understand the root cause of an erroneous test using distributed testing and how to debug it in our CI/CD pipeline with Thundra Foresight.

Table of contents:
- How to set up and test your cloud infrastructure
- How to write and automate end-to-end tests for your serverless workloads
- How to debug, trace, and troubleshot test failures with Thundra Foresight in your CI/CD pipelines
TestJS Summit - January, 2021TestJS Summit - January, 2021
127 min
Uniform Browser Automation Infrastructure
In this workshop, I will show you how to quickly deploy and use browser automation infrastructure with Moon solution. We will start deploying everything on your workstation and will soon be able to run Selenium, Playwright and Puppeteer tests in parallel in the same cluster. Then I will demonstrate how to easily deliver the same experience for your team using a remote cluster in the cloud platform.