Synthetic Monitoring and e2e Testing: 2 Sides of the Same Coin

Hi, DevOpsJS! It's great to see you all today. My name is Carly Richmond. I'm a Senior Developer Advocate at Elastic. I'm a front-end engineer. And also, I am a bit of a testing nerd. And today I'm here to talk to you about monitoring, which from a DevOps conference, you probably expected, but I'm also here to talk to you about testing, which might have been a bit of a surprise. I'm here to make the case that synthetic monitoring and end-to-end testing are two sides of the same coin. Very similar constructs. And in fact, using different tools for these is effectively duplicating effort on the dev and op side. We're going to walk through an example of how this can be done for JavaScript web applications, making use of Playwright for JavaScript and TypeScript, GitHub Actions, and also an observability provider, which in this case is Elastic Synthetics, so that we can try and shift our monitors left as code and try and collaborate more together.

So first, I need to set the scene of my experience because this is something that at the start of my career, I really wish I had. It would have solved several issues that I kind of battled through. So my first developer job, which was 2011, I was a bit of a rarity because I worked on a tiny team. We had to do everything. So we were developers, we were support analysts, we were dealing with outage situations, user acceptance testing, requirements gathering. I pretty much did everything and wore so many different hats.

And then later, it was only really when we ended up offloading things like some testing and some support where I got to work with individuals and actually learn some best practices rather than muddling along as I went. But then when I moved roles, I found that actually that prior experience is very, very rare indeed. In fact, the normal situation, which I still hear from speaking to many DevOps engineers and SREs is that it's a bit more like a game of Rock'em Sock'em Robots when it comes to developers and support kind of almost at loggerheads sometimes with the conflicting ideas. And quite often there's testers maybe in the background, unless we've ended up with the situation where developers are responsible for their own testing and validation.

Now, shifting left is not a new thing. And when I first heard about this, I thought, this is a great idea. This will lead to perhaps a little bit more harmony in that situation. Perhaps, you know, with the emergence of common practices, we can work together. Myself and other devs can learn more about how to build more maintainable applications. You know, maybe everything will be all great. And instead, I've seen something a little bit more different happen where actually in some regards, we've got all these great additional disciplines. If you think about SREs, DevOps engineers, there we've got prompt engineers, all these people that are able to dive deep into topics and show off expertise. But we still don't necessarily collaborate together very well. And this can still lead to a very fractured experience. And there's several reasons for that. The first is that particularly in large organizations, those existing silos still persist. And in fact, DevOps is often adopted out of an existing production management function, meaning that it doesn't really align with developers because they're still having those kind of cultural mishmashes and barriers in place. Our empathy isn't great. We're not really great at trying to understand the other side of it, because SRE activities, ML engineer activities, developer activities, these things are all surprisingly different, to the extent where even the priorities that we have as teams is different and can be struggled to align together too. Sometimes quality is a bit of an afterthought. Developers and the notion of shifting left means that we've had to learn all sorts of wacky things around security, best practices, error handling, monitoring diagnostics. And actually, we need more help because otherwise we end up just covering over the cracks. But the biggest problem, to be honest, is that we're using very different toolkits, which sometimes makes sense for the role that we're doing. But when we're using the same different tools to achieve the same ends, it doesn't really make sense anymore.

And in my last engineering job, the one that really got me was the fact that we're using different tools for end-to-end testing and synthetic monitoring. And I couldn't understand why. So before I go into the idea about how similar they are, let's first set the scene for anyone who's new to these terms.

End-to-end testing refers to the ability to use an automation framework, such as Playwright, like we're using today, or perhaps Cypress, or in a prior life when I was building Angular applications, Protractor, which thankfully isn't here anymore, to try and automate the user workflow, those operations they do to interact with their web apps, such as clicks and text entry, in order to try and validate that what we've built corresponds to that specification and will work as they should intend it to. Synthetic monitoring is where we basically, on a periodic schedule, run some form of script to test the availability of our application every few minutes or a few hours potentially. We might think that's something simple, like pinging an HTTP endpoint just to see that it's still alive, but with the emergence of scripted monitors in recent years, that can also mean performing user journeys, automating those same steps that they do to make sure that a user can indeed perform that workflow at that point in time in the system.

So we're automating the same thing. We're automating the user workflow, the clicks, and because we're building them in different tools, so while I was building in Cypress in my final developer role, I had colleagues who were SREs that were writing these monitors in Selenium using Apica as a wrapper. And I just don't understand because that leads to different understandings about how the system is intended to work because the workflows aren't always necessarily functioning exactly the same way. And it's also leading to duplicate effort fundamentally too. So I think we need to bend that notion. I think we need to come together and try and use a common tool set and basically shift your monitors left so that we use monitors as code, which can then double as an end to end test specification. And we're going to show an example of how to do that today. I'm going to use Playwright for JavaScript, which is a Microsoft maintained end to end testing library. And I have a TypeScript examples here too. You have GitHub actions so that we can run these tests in CI as part of any kind of pre-deployment or pre-integration checks. And then we need an observability provider because we're monitoring our app. And we are going to use in this case, Elastic Synthetics to wrap our Playwright test and allow that to capture the monitor activity and run it on a schedule against production. All the codes and the slides I'm going to walk through today are available in that QR code that you can see off in the corner and don't panic. It will come back at the end. But before we dive into the example, let's just get right in our head exactly how these elements fit in terms of a traditional flow.

So let's take the example of developers working on a new feature. So what they'll do is build out the feature alongside writing a journey file, a monitor that they can use as an end to end test making use of Playwright. And they will run that as part of their local development practice. Hopefully engaging in test driven development. They'll then push their changes, the monitor and the code changes into source control and undergo any reviews and checks running these monitors as an end to end test within the CI pipeline. Then when we come to deploy our application to keep our monitors in sync with the production app, we need to be able to push them out to where they need to run using API key authentication, which will then run either in your own location or within the location provided by the observability provider. And then in this case, persisting the results down to Elasticsearch and then being able to see the results in a nice shiny dashboard. So the example I'm going to cover today is the login flow that you're seeing flashing in front of you at the moment. So a simple manual text entry of entering username and password to go into an order screen and then start adding items to cart.

So first, we need that Playwright test that we can then convert later into a monitor. You'll see here I've got a very vanilla test up in front of me here that is basically going through starting from line four. We are going to the login page. We're pulling out the login form using the data test ID attribute as part of this selector. This is what I would recommend as a practice because the last thing you want to do, which I've done in the past, is couple this to CSS classes that you're also using for styling or to complex CSS structures with nested parents, meaning that as soon as you move something or change something in the view, you break all your tests. So try and avoid doing that. Checking the login form is defined, then going off and getting the button, checking it's disabled, adding in our credentials. So this is just a single test I have. So if you want to reuse your credentials, Playwright does have the option to do that by persisting their credentials in a JSON file. Check out the authentication documentation for that. But we are adding it to the text input boxes for the username, checking the value of the text box is actually what we added. Same thing for the password. We're then checking that we can now submit the buttons enabled, and then go off and make sure that we're navigating to the page as we expect.

So what we need to do next is wrap this as a monitor. And the way we do that is we create a new Synthetics project. So we'll have a global install, which you can see up here, then use the init wizard as part of the third command to generate that sample project, which will generate the skeleton code. So you see, you have the journeys folder. So these are example monitors that we're going to run as tests to show you how to get started.

You've then got lightweight heartbeat monitors. So if you're wanting to do things like pings, you can keep them all in the same repository as code, not going to cover those today. And then you have the project configuration with all your settings. So if we move on to that, you'll see here, this is the config file. I am specifying params. So I've got the URL, you can see that this is local host. So this is what I would ping under a default scenario. But if we scooch all the way bottom to line 30, you'll see that actually when the environment's production, I changed the URL to match. And that means that we don't have the situation where the monitor is going to fail when we push it because it's trying to access a locally running app that doesn't exist.

I'm also passing in the username and password so that I can then use the same values picking up from the environment in the monitor as well as NCI and under local development. So you would use an environment file locally to manage that. Then you have your playwright options, as per the playwright documentation, any specific settings you need. And then the monitor settings for all the monitors in the project, unless you say otherwise. So we're setting the schedule to 10 minutes. We have it running as a UK location, and then I haven't set up any locations myself. This is an elastic location in the UK. So that's all fine. If you set up a private location, that will be listed here. And then I've got the project settings. So I need to know where my deployment is. So this is the cloud deployment details. This is a point to a local running version, if that's what you had, the cabana space and then the ID.

So we need to now transform that test that we had before into a journey, which means we want to try and break up into smaller steps, which allows us to basically get more fine-grained information as to what's failing. So the first thing you'll notice, if you look at the top, is that our imports are now different. So we have a journey and a step construct here.

You'll also see for the journey that actually I'm passing in the page object. This is the playwright object, and I'm also passing in the parameters and the configuration. I can override the monitor with default settings, so if I want it to run on a different schedule, less often or more often, I can do that. I can set things up and tear things down with before and after, similar to unit tests. And then I have two steps. I have the login page steps, which are just exactly the same as what we covered in the playwright example before. And then I've also got the manual login steps, which have been split out as well.

And then when I run this locally as a developer, everything's green. I then know I can push my changes. So that means when it comes to the CI pipeline, there's two things when I'm trying to merge things into the main branch on any kind of reviews being complete, there's two things that I want it to do. I want to be able to run this suite, and I want to be able to push the monitors out to production alongside the deployment. So here, specifying the node environment as Dev, basically to pick up the correct configuration, and adding in the username and password so that those parameters can be picked up, I am going to run the suite exactly the same way as I did before with the same command. So what I will also do is specify a JUnit file for the output for the report, which obviously I didn't need to do for local development.

So scooting down to 18, I am, you know, sorting out my installs, I am starting the app running locally. And then what I'm doing here is I am running Elastic Synthetics within the journeys folder itself, so it can pick up the definitions, which is why I've specified the working directory. And then the JUnit reporter output option means that my publish unit test results task underneath is able to pick up that definition and show me my test results. But I also need to push the monitor, so when I deploy my app, which is happening in the background with Netlify whenever this flow passes, I want to be able to push those definitions to run as a production monitor. And the way I do that is specify the node environment's production, specify the API key. This has to be a secret. You don't want to be publishing your keys in your publicly visible GitHub workflows, so make sure you're using an appropriate vault. Making sure it's dependent on tests and doesn't push in the event of a failed test run. Again, running this time within the main test folder rather than the journeys folder. And then the tasks are that we again do run the installs and then run push, and this is generated when you run the generate wizard. And when we do that, we then go into Elastic. We see we have monitors, but we go, oh, you're failing. What's happening there? It needs to be able to pick up the parameters. Unless you specify them in the global parameter settings, it's not going to be able to pick up the environment variables. So you need to be careful of that. This is also a good time to mention that if you're adding credentials in for these tests, you need to make sure that this is a test account running a production that's limited to performing particular actions. Ideally, you don't want it to be changing state in a way that is irreversible.

You don't want it to be able to do things like making payments. So make sure that the roles are configured correctly in your application to make sure that someone is not going to be able to make tons of transactions against a legitimate card rather than a test card or be able to do anything they shouldn't. Because at the end of the day, if these credentials get out, you need to assume a worst-case scenario.

You also are able to see how long these are taking, how many failures have had over the period of time, and then we can start to do smart things. We can see when it's failing. We can see what step it's failing at. We can see what the trace is. We can see things like how long the steps are taking, which for me is useful because I used to have a situation where my intent test would get longer and longer and longer. So the idea for a monitor being able to see if it's starting to run a bit longer and you can then figure out if perhaps they're starting to degrade, maybe there's optimizations you can make to make it be more performant experience for the user. That's all super useful.

Many potentially do smart things like alerts. Now the hope is that this will catch something before your user does, but that's not always necessarily going to happen. Sometimes a user is going to find an issue and what we can do is we can use these same definitions as communication for what a user's encountering. So if we write specification or we record a specification, we can then use that to recreate the problem, work on the fix and then push all the way back through from that starting flow that we covered before.

Because in this particular situation, DevOps for me, given my experience, given those challenges I talked about at the start, it's all about coming together, making sure we collaborate irrespective of whether us as SREs, as DevOps engineers, as developers, irrespective of the challenges and the priorities that we have, we all have one goal and it's to basically ensure that we're building awesome applications that users are happy using. So it's time to come together and if common tooling can help us do that, in this case with end-to-end testing and synthetic monitors being two sides of the same coin, let's do it.

So the next time you are looking at your end-to-end suite or you're looking at your scripted monitors, just pop your head over that wall and ask the question about are they writing the same thing? Are the workflows in sync? And is actually there a way where we can write the same definitions using common tools? So thank you very much DevOpsJS, it has been an absolute pleasure. I will be floating around so feel free to ask me any questions about playwright, about synthetic monitoring, I'll be around for that. Also dive into the code which you can check out at the QR code which is listed there and there's a copy of the slides there as well. If you don't catch me around and you think of something after, I'm on X, I'm on LinkedIn, I'm on Masterdod, just reach out and say hi and I'll see you next time. Bye!