TestJS Summit

TestJS Summit 2022

Testing Web Applications with Playwright

Debbie O'Brien

Tiny Tests, Large Results

In today's tutorial, we're going to talk about automated atomic tests. Automated atomic tests are a wonderful way to update your UI tests and make them less brittle and drastically faster. We're going to see a few examples of those automated atomic tests, and we're going to even implement those automated atomic tests on two web applications. One application will have an HTML web form, and another will use a JSON web token for authentication. My name is Nikolay Advolatkin, Senior Solutions Architect at Sauce Labs and founder at UltimateQA.com. So, what are you waiting for? Let's go and check out how to implement automated atomic tests. An automated atomic test is a test that tests only a single feature or a component. They typically have very few UI interactions and typically touch a maximum of two screens, especially with front-end UI automation. The reason why I say they typically touch a maximum of two screens is because these automated atomic tests will typically have a setup state where they need to set up the state of an application, and then a post-setup state where they need to perform a validation. Automated atomic tests have several good environments in that they are much faster and much more stable than your typical automated tests due to the fact that they decrease the amount of latency that we have to deal with, and they decrease the amount of UI interactions that we have to perform. Now, I've got a question for you. This test over here, if you look at it and analyze it, do you believe this test is atomic? At first glance, this test might actually look atomic, right, in that it's signing in, stating that we were able to sign in, and then it adds an item to a cart and asserts that an item was added to a cart. It's close to being atomic. However, it's not actually atomic. The reason why it's not is because it has to log in and validate that we have successfully logged in. There is no reason why we can't test the UI login in a separate test and ensure that that works, and then afterwards, what this test really cares about, what it actually wants to test, is this portion right here. And so, we can save time and stability by performing these interactions without using the user interface. Atomic tests can come in many forms. Some of them can be extremely simple. For example, when we used to write automated atomic tests that click links, we actually have a wasted action and a wasted validation in that we need to click a link and assert that the link goes to the right location. What we're actually interested in is the href attribute of the link, which we could easily validate in this manner, making our test faster and less brittle. Let's go ahead and take a look at our first web application, which is going to have an HTML web form. We're going to be able to log in to this HTML web form, but instead of using the UI, which is not efficient and not atomic, we're going to be able to make a web request that's going to end dropping a cookie in our browser, and then we're going to be able to bypass the login without interacting with the UI. If we come to this application, there are many tests that are here, but I'm going to focus on one at a time just to give you a better understanding. The very easiest test to understand, which is the positive test case, is going to be this test case here called Redirect to Dashboard on Success. This is the very positive test case where we visit the login page. By the way, you can see our application on the right-hand side if you're not familiar with Cypress. It displays the test execution on the left with the actual application on the right. If you did want to explore the application in another browser, you definitely can. Here it is, and it performs all the same exact operations. But with Cypress, for example, it's very easy in that you can see what the test is doing and then see the actual application on the right-hand side and even interact with it if you wanted. You can see that the very first step that this test does is it visits the forward slash login of our application that is on local host 7077. After that, it performs the standard operations that you would expect in order to actually be able to log into the application. So it's going to get the username and type in Jane Lane. It's going to get the password field and type in password. And then it's going to submit the form. Once the form is submitted, you can see we do a couple of assertions. First, that the dashboard page is there. So our URL includes dashboard, forward slash dashboard. So that means that we were able to successfully log in. Second, we get the header and assert that it contains Jane Lane. So almost checking not only that we're logged in, but that contains the right user. And we get the cookie that corresponds to our login and make sure that there is a cookie that exists. So take a look at the application tab here. In our cookies currently, there is nothing available here. I've cleared it. But if we rerun the test, and of course, as I mentioned, it does interact with the application and actually logs in. You can see that now there's a Cypher session cookie that's dropped. Hence, why we are able to log in through the application. The challenge with this test is that, again, it's not atomic, right? In that it does interact with this login form, and then it logs in, and then it validates that we're logged in. So that's great. But if you wanted to do any kind of actions beyond this, for the login, we'd be doing the same thing over and over every single time. And so it's repetitive and unnecessary, and maybe we can make that action more atomic. But let me actually show you the rest of the tests so that you better get an understanding of what the application can do. So I uncommented the only part of the test. And so now we have all of the tests that are available here. And these tests, of course, as good end-to-end tests or any kind of test should do, they kind of help us to understand the functionality of the application without actually manually stepping through it. So the very first test here is called unauthorized, right? And so the expectation here is that you're redirected on the visit to forward slash dashboard, right? So if you're not logged in, you should be redirected. And so this is what this test does. It tries to visit dashboard, but then it gets redirected to you're not logged in and cannot access this page. Because there's an assertion that gets the H3 and checks for that message, and it checks the URL includes unauthorized in it, right? So if you're not logged in, can't access the dashboard page. If we try to log in by doing a get on the dashboard page, we will also be unable to do that. So we're expecting a 302 to come back to us as a result when we try to log in without a cookie. And we'll also expect the response to come back with unauthorized in the body. If you try to interact with the actual form and you try to log into it with, for example, invalid user and the valid password, we expect there an error to be visible and the error to say that username and our password is incorrect. And on success, you've already seen this. If we visit the dashboard, type in the valid user and the valid password, we get a success in that you're able to visit the dashboard. So in and of itself, this test is atomic and great and very useful to test a single time. Once you're able to log in with the web form through the UI, you know that it's going to work and it no longer needs to repeat it in the rest of the tests. And so how do we make it atomic? On this case, for this web application with an HTML web form, the way that it works is you submit a request. How would an atomic login actually look under the hood? Well, as you can see from the documentation of this test here, we can simply perform a web request that will allow us to log in. The web request is simply going to be a post to the login URL, passing in the valid username and password, and then that will automatically set the session cookie. And so now after this point, we can perform any operations that we want that are behind the private application interface. So let me show you what exactly that looks like. And so here you can see doing the post to the login, right, and then showing you that the cookie does exist. And in fact, we can even simply do – just take a look at this test so that it's focused. Go back here to the application, get rid of this. And then if we rerun it, we're going to be able to see that there is a cookie. And nothing happened in our application, right? But if, for example, now we come and try to access the dashboard because we are logged in and we have a cookie in our browser, we're able to access it without actually logging in. This time, instead of our application being an HTML web form, it's actually going to use JSON web tokens. JSON web tokens are a very common industry standard method of basically authenticating into applications and allowing access to certain parts of the application based on this web token. If you want to learn more about it, you can go to this website here, JWT.io. And basically the way it works is you submit a request with your username and access key or username and password. And then based on that, the application will verify whether those exist in the database. You get back a token that looks like this. And now this token is what you can use for authentication. Very common in many applications in today's day and age. So extremely useful to be able to do this in our automated test. Here is the repo that we're going to be working out of. As before, it's very similar. This folder that I'm in is part of a much larger folder. I can show you exactly here. So you can see the folder I'm in is logging in JWT. If I navigate up, you can see I'm in the larger Cypress examples repo. And then as I said, we're working out of logging in JSON web token folder. The application we're going to be working on is a Vue application. And you see the instructions on how to start the app here, which I did over here. Actually, I simply ran npm run start to start the application. And so that's running in here. And then in another terminal window, I ran npm run cypress open. And that opened up my Cypress browser. And so now let's go take a look at the tests to see exactly what they're doing and how to do it through the UI and then how to do it without a UI through an atomic manner. The way this app is going to work is it's going to send the login credentials. And then those login credentials are going to be stored in local storage in an item named user. You can see from this screenshot here, there's our token. And then any requests that we make afterwards that are behind this authorized information, that's going to require the authorization header with a bearer token in order to be able to access the resources. And so if we, for example, take a look at the network tab here under the users, you'll see that there is an authorization and a bearer token with the JSON web token. And that's how users can get access to different parts of the application. Once Cypress is up and running, we'll be able to see these two specs here. One is using UI and then the other one is not. And so if we pull up this using UI spec, you'll see all of the tests that are performed. Right? Using UI, it'll be something very standard you've seen before, which is basically getting the username, typing in that username, typing in the password into the password field, and then clicking the Login button. And you can see what happens before and after. Afterwards, we are basically checking. Here you can see there's a web request happening of a Pulse 200. And then we're doing a number of operations, such as we're making sure that we are at the correct URL, expecting this header to be visible. And then we're even expecting that there is an item with a username, first name, last name. And then it has this token that is a JSON web token, and we're expecting it to be a string. And then there's a bunch more assertions on different parts of the web request. There's even more assertions like this, checking that there's a logout link, and then clicking the logout link, and logging out, and asserting that we're on the login page. So if you try to access the localhost 4000 slash users, this URL, you will get unauthorized. So you get this error message. So if we come here and try to access this application, invalid token, because we can't get there. And so we're supposed to get back A401. So if we come here, inspect, and come here to the network, and then we do yelp xhr, and to users, we will get here a 401. You can see the user's document return was 401, because you're unauthorized. And so this basically obviously shows a positive test case and a negative test case, both through the UI. So here is actually one with the UI that will try to log in with an invalid username. So typing in username, typing in a wrong password, and then clicking to log in. And then of course, it's going to get that username or password is incorrect. And hence, it's doing that assertion to make sure that it exists. So that's our UI test. Wonderful, again, fast on localhost will be much more cumbersome when we actually deploy this to an environment and have to travel over the network to interact with this, and more flaky as well as a result. Hence, we can bypass all this using web requests as we did with the previous web application. So if we wanted to make the same test atomic, we have these specs here, and I'm going to run them. Unfortunately, they're not as useful to see through the Cypress UI, because they do make a bunch of web requests. And so the information behind them is not super useful. But you can see how we can make an authenticated request, make sure that we're logged in, and show loaded user. So instead, we can actually look at the code to understand how to make this possible with JWT. So if we look at this spec.cy.js, we can see that the very first step that we do is actually hit an endpoint to authenticate a user with a username and password, and then store the returning body inside of the user. Then, in order to be able to add the item to our local storage, we need to set that item called user, and we stringify that user, hence enabling that bearer token. And so once that's set, we can try to hit this endpoint with this bearer token, and we are able to access this user, without which right now we would not be able to access it. But with it, we are. And then, of course, as a result, we can check for different kinds of conditions to be or not to be visible. And that's it. That's how you create an automated atomic test using a JSON Web Token login mechanism. Now there are actually lots of different types of login mechanisms, and certain ones may be relevant to your application. Others may not. There are many more examples here. As you'll see in this repo, you'll see all of these login examples right here, from basic auth all the way to single sign-on. So depending on your application, you're going to use the relevant login mechanism. And also, as a bonus, I've linked a bunch of bonus resources to how to do automated atomic testing in the readme. It's at the bottom of the readme. So definitely be sure to check that out. Thank you so much for joining me in today's tutorial about automated atomic tests. And if you want to learn more about me or catch up with me, you can find me at ultimateqa.com or my YouTube channel or any of these other social medias down below. Thanks again so much for your time, and I'll see you next time. Take care.


Nikolay Advolodkin

Full-Circle Testing With Cypress

Hey everyone, I'm glad to be here. My name is Filip and I'm a tester with maybe some developer skills or developer with testing skills. I don't know, depends how you look at it. I'm also a Cypress ambassador which means I like to talk about Cypress and I love to teach people how to use this amazing tool. So that's what I'll be doing today. The topic of today's talk is full circle testing with Cypress. So what that means, well, it's a completely made up term. I made it up to make this talk a little interesting and hope that you will take a look at it. But I do have an idea that's behind it that I would like to share with you. So let me drive a circle or a ring, but don't worry, you're not going to die when you look at it, hopefully. But hopefully you'll learn something new and yeah, let's take a look. So as you know, there's something that's called testing pyramid which I'm not going to draw here because everyone knows what it is. But it also contains something that might be at the top of the pyramid and which is something that I'm going to start today's talk with and that's end to end testing. So when talking about Cypress, this is where Cypress shines the most and this is what Cypress is mostly known for. So if you have never used Cypress, I recommend you to check out the workshop. I have a link at the bottom of this slide. But also there is going to be a workshop on TestJS Summit in just a couple of days. So make sure you check it out if you still can. Hopefully you can. And we can meet there and you can learn to work with Cypress with me. So if you can make it to that one, make sure to check out my website and you'll find one at the end of November. So let's take a look into Cypress and how you can actually write a test with Cypress. So on the right side I have my Cypress running in graphic user interface mode. So that would be the one you run with npx cypress open in your terminal. And on the left I have a test which I'm going to show you later. I want to show you this one right now, the end to end test. So inside this test I'm doing a simple flow. Visiting an application. Visiting an application and then selecting an element, typing into it, hitting enter and then doing some actions again. Getting an element, clicking, typing, interacting with my page. Basically what I want to do is to go to my home page, create a new board, then create a new list and inside it create a new card. And this is what the flow automates. So whenever I make an error, for example I change my selector to not be found and when I save my test it will automatically rerun. And if I make an error during my test I get this really nice assertion error explaining to me what actually happened. So in this case what actually happened is that it couldn't find the selector because I made a typo on it and it was nowhere to be found. Cypress is actually going to retry searching for that selector for 4000 milliseconds, so for 4 seconds. Now this means that Cypress works really great for the applications that are all the time re-rendering and doing some sort of fetching from API, etc. as modern applications do. So when you are testing you get this instant feedback into what your test has done. And also the important thing here is that we are running inside the browser. So while we are doing that we get all of the advantages that browser will provide us. So for example we can open up a console and take a look into what's happening inside the console. So I'm getting a couple of warnings here but these are not that important. Cypress actually takes a great advantage of the console and of the fact that we are running inside the browser. So when I click on any of my commands, for example this visit command, Cypress will print out information into the console with the details of that command. It will also print out information for every HTTP request that is being done with the application. So let's maybe fix the test, save it to make it pass and also what Cypress enables us as you can see is to sort of travel back in time and examine the application as it was looking while the test was being executed. So this is pretty neat. This gives you an insight and especially if you are looking for a selector that is kind of hard to find because the element appears and then it disappears. This is a really, really helpful tool. So you can see different API calls and Cypress sort of invites you to get into the application and examine the application and get to know its nuts and bolts. So a great area to start examining your application a little bit more is to do API testing. And this is actually a second part of our circle of tests. So when we are doing API testing in Cypress, we can use a command called request. So let me show you that one. And let's change it because I'm going to show you the API command in just a second. So let me save my test and let me open that in my Cypress runner. And oops, sorry, that was not the correct test. We want to run the API testing. All right. So now we have it. So inside here, what I'm doing is using the request command to send a post request to the URL of API boards. So as you can see, the syntax of my test is pretty simple. I'm using the cy-global object, which is available throughout the whole test suite. And I'm using the request command that is there for sending API requests. So of course, what I'm doing is sending the post API. And here is the body that I'm sending towards the server. Now when I get a response, I take with the its command, I'll take the status and make sure that it is 201. Now how do I know to look for the status and for the contents of the response? Well, the first way of how I can look into that is to click on my command. And just as I've shown you a couple of seconds ago, open the console and examine what the command has yielded. So with my its command, I have actually taken out the status and then checked whether the number of that status is 201. So it is. And my test is passing. So this is really cool, right? But I see a little problem here. Maybe you see that too. We have that big white space over here, which is not used for anything. So that's kind of not so good, right? So what I did with that is I got inspired by the open source community and namely Cleb Bafmuto who has built a plugin for showing the API in the browser. And I have made sort of my own version of that. So I have this installed in my repository. And what this plugin will do is that will add a new command to the Cypress library. So it's called API and it works exactly the same as CYRequest. So when I save my test, it will print out the information about the request into the browser. So I can see the method. I can see the URL, also the status. And I also see the request body and response body and all that information. I can even copy the contents of the response body to the clipboard. And maybe I can just try to write a test. So let's go with its body. And let's do deep equal. And I'll paste my object over here. I think that should work. Let me save that. And yeah, we're good. So we are testing the API with our shoot command and with a visual feedback that we got here. So as I said, Cypress is known for end-to-end testing. And as you can see, it's a great tool even for API testing. So what we can do is we can combine these two worlds together. We can do end-to-end testing and API testing. And since all of the requests that Cypress sends are actually sent from Node, so they are not sent from out the browser, what we can do is to call any API we want. So actually, I have a test over here. Let me show you that one. Here I'm testing an email. So let me open that one. And let's explain what's happening in here. So I'll go through the timeline so I can travel back in time and guide you through what's happening. So I'm visiting a signup page. And then I'm filling in the email and the password. And I'm checking this nice little checkbox. So let me show that to you. And this says, send me a welcome email. So what this application will do is that it will actually send you an email if you check that checkbox. And what I can do with my request is to access the mailbox that I'm using with an API request. So there are different kind of tools that enable you to access your APIs with HTTP requests. So one of that is TestMail, but we got MailSAR, MailSlurp, and other services that provide this feature. So that's exactly what I'm doing in this test. Although it may look like a little bit complicated, it's actually not. I'm just generating a random number over here. I'm assigning it to a variable or an alias. And then I'm using this alias as a part of my email address so that it can be identified on the server. And then also using that tag in the request itself as a query string. So when I send that request, I get the contents of the inbox as a response. And I'm checking the body emails. And I'll check the subject of that email to be Welcome to Trello App, which is exactly the subject of that email. So yeah, that's pretty powerful. But you probably noticed that when I'm going over to timeline, that Cypress actually takes notice of all the API requests that are being done by the application itself, which brings me to another area of API testing, which is network layer testing. So not only we can send requests, we can actually observe the request that our application is doing. And let me show you on the example that I have prepared for you. So I'll open another test that's going to be network. And I'll filter out the first test using data only. Maybe you know this trick, maybe you don't. But if you do that only, you can filter out the only tests you want to run. So let me save that, open the test. And you can now see it in action. So what's happening here is my test is actually failing. So let me walk you through that. Inside this test, I'm using the intercept command, which when I explain this on my workshops, is something that tells Cypress to watch for a certain request to happen and to be called by our application. So this request is actually being called, the post API boards is actually being called when we create a new board on our page. So when I go to my homepage, fill in the details and hit enter key, that's where the API is being called. That's exactly the place where that happens. And we have that highlighted in Cypress. I have aliased it with a name of create board. And here is the alias. So what I do after I do the action, after I type in new board and hit enter, is that I wait for this API request to happen. And then I'm testing the request body name to have a certain text. And as you can see, this test is actually failing because my input is adding a white space after my input. So whenever I hit the enter key, it will register that as a new line and it will add that slash and sign into that, which is not good because I'm polluting my database. This demonstrates that I can not only test the response like to go one, two, or two or something like that, I can actually test whether the application is doing what it should be and whether it is sending the proper data to the server, which is great because now for API testing, I'm not using the direct requests that I know how to prepare, but I'm actually testing whether the application is sending the proper data to the server and checking if it's not sending some corrupted data or doesn't handle the inputs properly as it does here. So what Cypress and the intercept command actually enables me as well is to check for different edge cases. Not only I can observe an API request happen, I can also sort of cut off the server response and provide my own. So what I can do, I can change the body, I can change the response status, I can even change the response time, which is exactly what I'm doing over here. So I have a callback and I'm handling the reply to sort of be delayed by 10 seconds. So when I now save my test, you can see that the API boards request is actually taking a longer time and I have this reload button being shown in the application, which normally would be a really hard place to sort of reach and to test, but with intercept, we can actually show that and see if it behaves properly, if it appears after three seconds as it should and if it works properly. So we can actually click on it and make sure that our application will reload. So with the testing of network layer, we can sort of get to different hard to reach edge cases and one of the great examples of how we can reach different edge cases is testing the application layer. So that's what Cypress enables you to do as well. That's what I want to show you. So in this example, I'm using something that's called app actions. So let me change this test a little bit, save it now and run it. All right, I got it. So Cypress takes a great advantage of the fact that it is running in the browser. I've already shown you a couple of those, like printing out things into the console, taking notice of what the application is doing on a network, but also you can take a look directly into the application state. So let me show you in the application, I have this store.ts, which this application uses Spina, it's written in view and for store management, it used this Spina package and it uses this use store. So inside here, I have all of the state or all of the default state defined and I also have defined some of the actions that mutate this state. Now what I have done with my application to make it more testable is that whenever we are testing, whenever there is Cypress on the window context, we are exposing the use store, so the whole state of the application to the store variable on the window object. So what this enables me to do is to access the properties of the store and even the functions of that that mutate the store right from the window context. So in Cypress, I have this cywindow command which access the window object and then I can invoke different functions. So here I'm invoking the store and then I'm invoking the getBordList function, which actually triggers the API call for fetching all of the boards from the database. Inside here, I also have a pause function, so maybe even if you are using Cypress, you haven't heard of that one, but it's super useful when I'm demonstrating something. So inside here, I can just go step by step and move on to the next command, which is API call. So I'm creating a new board, but as you can see, it is not appearing in our application and this is because we don't have some sort of WebSocket connection to the database or with the server that would update the contents of our application. We would basically need to reload the application to make that getAPIBoards call to be triggered again. But with AppActions, we actually don't have to do that because we will invoke the store and then call the getBordList action, which is responsible for fetching all of that data from the server and which is the function that gets called whenever we open the application. So let me go to the next step. We get the window, we get the store and now we call the getBordList, which will trigger this API call and our board is now loaded. So with AppActions, we can access all kinds of stuff. We can even access the things that are hidden from the application. So let me comment this out and show you this because I have some hidden API tools that I made for this application and now I'm basically calling toggleTools to appear and I have this nice neat API tools available for me here and I access this using the invoke and using the internal store of the application. So this is a huge advantage compared to maybe different test automation tools because since we are running in the browser, we have direct access to the application. As opposed to tools that sort of drive the browser and access the browser sort of from outside to automate actions on the browser. Now of course this has some advantages and it has some disadvantages, but I view this as a really big win for Cypress when it can basically access any part of your application that you give the authority to and manipulate it in a way that helps you reach maybe a hard to reach edge case or some part of the application which might be hard to set up because of the data or because of some other factors. Now Cypress is of course a great tool for getting into the app and testing different states and making the app do stuff so we can test it in a better way, but we can dive even deeper and break down our application into smaller chunks. So Cypress has released this killer feature called component testing. So it recently got out of beta, it has been here for quite some time, but with versions of Cypress 10 it reached the whole crowd. So I want to show you this because our application is sort of built from different small components that make our application. I like to think about it as sort of Lego blocks that are used to build the application. So in here I'm basically using CyMount function that is going to render my component and I'm passing different properties into here. So if I type hello world, my button over here is going to render hello world. If I type hello test.js, it's going to render out this. So I can test different kinds of states of my component and it doesn't have to be button, we can go to something more complicated. So for example, we can test different states of our notification in here, test an info message or an error message. Or we can go to our navigation bar, click this home icon and see if it behaves differently when we are at home or when we are at some other sub page. We can even render out sub components. So this drop down items actually contains some slots with components in them and yeah, you can go basically anywhere with this. The biggest advantage of this one is that we get to render our components inside a real browser. So this is exactly how our components would be rendered inside the browser and we can interact with them, we can test them as you can see and we get all of the API of Cypress enabled for us. So we can test different states, if the component calls an API, we can use intercept to sort of change the state. We can test, we can send requests, we can add, we can change the state of the app using the app action that I have just shown you. Or we can just render multiple components and see how they interact with one another. But if you are doing that, you kind of reached the full circle and go back to end to end testing. So you can go all around as many times as you want and hopefully you get a good coverage of your application. This has been really fun, so thank you very much for listening this far. I will be really eager to meet you on one of my workshops, whether on TestJS Summit in here or in November on my other workshop. You can find the link at the bottom of this presentation. For now, I will be looking very much to hear your questions. See you in a bit.


Filip Hric

E2E Tests for Web3 Applications

Hello, everybody. I'm Rafaela and I'm going to share a bit of my experience with end-to-end tests for Web3 applications. So, just a bit about me, I have 15 years experience in QA, test automation, test strategy, architecture, and leadership. And I got really passionate about the QA area, so I created this blog in 2011, where you can find everything that I have learned in the past couple of years. So code snippets, programming, DevOps, leadership, and test automation, of course. I was born and raised in Brazil, to be more specific, I'm from a coastal city called Santos in Sao Paulo. And you can see the picture on the screen. I'm also a really big fan of Harry Potter, as you can see, and this is one of the reasons why I moved to London eight years ago. And the weather as well, but nobody believes when I say that. But yeah, the agenda is going to be this one. We are going to talk about what is Web3, then the architecture of the Web3 application, some introduction about the end-to-end tests, and the challenge, then the tools that you can use. And we are going to have two demos, one just using a mock and another one using a framework called SyncPress. And then in the end, we are going to talk about the trade-offs of each approach. So what is Web3? Web3 is just a new version of the web. We are on Web2 right now. And this Web3, it comes with the ideas of decentralization, token-based economics, and also blockchain technology. So it offers a read-write-own model. So people, they have financial stake and more power over the online communities they belong to. So they own the data, which is not the case right now. The online experience is expected to change as cell phones and smartphones and the pieces they did in the past. So it's going to be a big change for us that we are living in this moment. Some business, they try to join this new market, but they encounter some pushbacks with blockchain as well, like the negative impact on the environment and the financial speculation as well. This is the evolution of the web. So from the first version to now, so the first version we had the static read-only pages where you couldn't interact too much with them. And then now what we have is a bit more dynamic and interactive. So we can go there and post things. So Twitter, we can just send our data away basically to these big corporations. And the Web3, which is the one coming now, is more private, secure, and decentralized. So the idea is our data is not going to be owned by these big corporations, but by yourself. And the architecture of the Web3 is also quite different because it's based on smart contracts like blockchain. So if we see now what we have is basically the frontend backend database inside of this web server, and this is controlled by the corporation or the company, somebody. And the Web3, we have just the frontend in the web server. So this is going to be controlled by the company or the corporation. But then the smart contracts, the EVM and the Ethereum blockchain, this is something that is going to be decentralized. And the smart contracts, they have the business logic or the protocol of these transactions and this business. So basically now what we have is just we are going to interact with the frontend and then the frontend is going to interact with the smart contracts and then EVM, the Ethereum virtual machine. And this is going to add the transaction to the blockchain. So it's going to add a new block to this Ethereum blockchain. And this is the part where it's decentralized. So this is an example of a Web3 stack. So we have some examples of like tools and frameworks, but you can see the layers. So you have the first layer with decentralized applications. So it can be something like MetaMask, which is a plugin to authorize identity, have wallets. It's a cryptocurrency wallet. Then you have Uniswap, then you have another layer, presentation layer, which is called and you have the developer environments where you can test your product. And then you have the other layers like blockchain interaction layer and the network layer with the EVM blockchains. So this is not all the tools that you can use. It's just like an example with the most popular ones, I believe. But at least you have an idea of how it is, the Web3 stack. And then the tasks, the end-to-end tasks are more like the problem with Web3. But if you want to do the unit tasks, it's fine. You can use something that you already use for Web2 applications. You can use Jest, you can use Mocha, it's still the same kind of framework. And then you can just verify if the components are operating as intended. So UI tasks are also not a problem because you can still use the same frameworks that you use right now to do the tasks. And yeah, the integration tests, they are not a problem as well because you can just use the smart contract libraries to mimic their own changes by just mocking the underlying EVM. So the problem is when you go and you try to do the full end-to-end tests without the mocking because you have this third-party provider that you need to use to communicate with the blockchain and submit the transactions. And most of the frameworks right now that do end-to-end tests, they cannot access this plugin. So one of the most used plugins is Metamask, as I said, it's a cryptocurrency wallet. And you need these to interact with the Ethereum blockchain and sign the private keys and do the full transaction cycle flow. So yeah, faking these UI interactions is really complex and the test frameworks that we have right now, they don't support these plugins, even when running headless browser. So one of the approaches that I'm going to show you after is just mocking the web tree, but this is basically what is the flow. So the user goes and interacts with the web tree that interacts with the Metamask or the third-party provider. And then finally, Metamask is able to send this transaction to the Ethereum blockchain and add the block there. And these transactions, they are signed by a private scheme. So UI frameworks, they cannot do that at the moment as well. And one of the approaches is using the web tree mock. So now, while it is a good practice anyway, not using the real third-party integration because you shouldn't be dependent on this because you don't have control of this third-party, you don't have control of the plugins. So if it crashes or if there is a bug, you might need to pause your entire development just because you cannot test your own application full end-to-end cycle. So one idea is to mock the web tree and then just do the same sending transactions and then simulating this part with the Ethereum blockchain. And then yeah, with the web tree mock, you can solve the problem by signing the transactions with a private key automatically and then sending to the EVM and the Ethereum blockchain. Another approach would be using this framework called Synpress, which is a wrapper from Cypress.io and has MetaMask support. But yeah, the problem with this is you are tied to use only MetaMask with this. It's the only third-party provider plugin that supports at the moment. And you need to use Cypress.io as well because it's a wrapper from Cypress.io. But at least you can choose between locking the version of the MetaMask so you don't have unexpected failures or you can just make sure that Synpress can just run the tests but before that update the MetaMask plugin just to have the latest one. So you can decide what you want to do. Basically if you want to lock the version or if you want to update with the latest MetaMask before running your tests, which would be good to figure out if your application has any problems with the new version of MetaMask. And you can still use any of other instrument test tools that are in the market already. The only problem is they don't have something like Synpress. They don't have a framework that is built on top of them, apart from Cypress.io, just to do the integration with this third-party provider. But you can use the mock strategy. So you can still use, for example, Taskafa that I really like as well. You can use that and then mock the Web3. But of course they have downsides as well. So if you want to use something different from MetaMask you can use it with the mock but then if you go with Synpress the only thing that you can use is the MetaMask plugin. So now we are going to just see the first demo which is with the mock. So it's going to be Cypress and the mock of Web3. Let me just go to the other screen. Oops. Sorry. This one. Yeah, this is on my GitHub and it's just a fork from another project but you can see already how that works. So it's a simple... You just need to use ER and start to start a local host webpage, simple page, and then you can use ER and Cypress run to run the tasks. But basically it's just this task is sending the transaction. It's really fast so you cannot see properly. But the first one is just connecting to MetaMask. You can see... Oops. You can see the connect with the login and this is the sample they created. So you can see here it's expecting to have the address of the wallet that you connected. Then you are signing the messages as well and then you need to check here the data response if it's exactly what you expect and if it's doing the right transactions. And then in the end, yeah, try to send a transaction which you sent to this wallet, this address, this amount, and then you should have... Yeah, in this case, it's okay to have this service. It cannot send transactions which was expected in this case. And then signing out as well which you should be... It's really, really simple this one but you can at least see how the mock is done. And whoops. And the other... This is the address so if you want to have a look. And the other demo is going to be using the SyncPress tool. So you can see that we are using the SyncPress and we are really integrating with MetaMask. So this one is just running Cypress and it's loading Cypress and then it's going to load the specs and just run straight away. So this is the MetaMask plugin. You can see that it's starting and it's not mocking. So if you go and install MetaMask plugin, you can see that it's exactly like that. And also on the top, you can see Chrome extension so it is the real plugin. Then now, it was just setting up the MetaMask with importing the account. And now it is using the plugin to send... Yeah, to accept the connection request and just connect to this account. And now we're just creating the network as well. And yeah, basically this is another demo with the SyncPress. It's really straightforward as well, but it's well-maintained. You can also check on the official SyncPress repo on GitHub. They have other examples there, if I'm not wrong. But this one is really simple and is on my GitHub repo as well. GitHub account, sorry. And yeah, of course, the trade-offs. So we have the test speed that we need to take into account. So when you use the mock, of course, it's faster than using the third-party plugin and waiting to load and also the network. All the problems that you have when you are not mocking this third-party provider. Then you have the dependency on third-party app when you are doing the full real end-to-end tests. So if something crashes, if there is a bug, you might need to just stop or revert the version that you were using before the bug. Then you might need to pause, or if you are really testing and you want to make sure that it's going to work with the latest version. If there is a bug, you might need to wait until this bug is fixed just to make sure that it's okay. And also end-to-end with the end-to-end test with SyncPress is the real-world test. So you know that this is really what the user is going to do. They are not going to mock, of course. And then the other one is mock maintenance. So you need to worry about that if you are mocking, just to keep up to date if you want to make sure the latest is working with your app. And then you have the test app in isolation as well. So when you do with the mock, you don't care about this third-party if it's working or not, then you are making sure that your app is always working fine. You don't need to pause. You don't need to stop your development. You don't need to worry about the third-party should be fixed or anything like that. It is a good practice anyway to do that, to mock. So one thing that you can do as a strategy is just to have a good balance between the unit tests, the integration tests, so you can mock most of the tests, and then having, I don't know, maybe 10% or even less of end-to-end tests. So you can use something, for example, the framework in automating these 10%, or you can have some exploratory tests doing the full end-to-end, or you can just have manual tests for that. It's really up to you, your project, the strategy that you use, what are your needs, but a good balance always make sure that, okay, we cannot test end-to-end with the latest version of the third-party, but we know that with integration is working fine, we have everything covered, so it's more robust, it's more trustworthy in the end. And yeah, that's it. So thank you. If you want to connect to me in any of these social medias, just feel free to do so, or send any questions or feedbacks. I'm open for these as well. It's always good to improve. So yeah, that's it. Thank you.


Rafaela Azevedo

Playwright Can Do This?

 Stefan Judis

Unit Testing Angular Applications

Hi, everyone. Welcome to my talk about unit testing Angular applications. First, just a quick intro about me. So my name is Filip Voska. I am from Croatia and I'm working at Infinium as JavaScript team lead. Throughout my years, one thing that I've learned is that you really want to be writing tests for your code. Nobody writes perfect code and you want some confidence that it's running correctly, that it's doing what it's supposed to do, and that's what you use testing for. So quickly, to go over the topics, we'll take a look at what are some of the ways to test frontend applications, what are some specifics to testing Angular applications. Then in third part, we will take a look at some best practices which are not really necessarily related exactly to Angular, but they're also concepts that you can apply in other frameworks. And in the end, we will have an overview of some of the things which you should test, which you shouldn't test, and what's the next steps, what are some educational resources. Okay, so let's first take a look at what to test. If you go online, you will probably, and you Google testing, you will probably see something related to the testing pyramid, and it's what describes the ratio between unit integration and end-to-end tests. So in, let's say, the most common version of it, people say that you should have the most amount of unit tests, then a bit less of integration tests, and then even less end-to-end tests. Now there are variations of this where some say that you should have something like this, which is like a testing hourglass, where you should have, let's say, equal amounts of unit and end-to-end tests, but a little less integration tests, and then you will also find this sort of shape, which is some kind of vase where you can put flowers in. And yeah, those are all kind of different philosophies, and that's all a topic on its own. We won't go into details about that. We will just take a look at what matters for Angular. And for front-end applications, you have usually a choice between Jasmine and Jest. Those are two most popular testing frameworks. With Jasmine, you would also use Karma as a test runner. Jest is kind of all-in-one solution. And Angular ships with Jasmine and Karma out of the box, but it's also quite easy to use Jest with it. Then for end-to-end, you have things like Cypress, Selenium, Playwright, and these are all ways to run an automated browser, where it's an actual browser running your code and you're simulating user behavior. So I will be focusing on Jasmine today for unit testing. I will not be covering end-to-end or integration tests. Now integration tests, you can really do them with any of these tools, because the line between end-to-end and integration and unit tests, it's kind of blurry. It depends on what you define as a unit and how many units are involved in a test. So what is a unit in context of Angular? Well, Angular is mostly composed of different classes, and those classes can be like components, directive, pipes, modules, services, et cetera. And then you also have functions like regular helper functions that you might have. So mostly we are talking about testing classes in Angular and creating instances of those classes. So let's take a look at some of the tooling which Angular gives us that makes testing and creating those instances a bit easier. So this is a component which we will be working on. It's quite simple. For those who are maybe not familiar with Angular, Angular consists of a class component that has inputs and outputs, and these are the ways that the component can communicate with parent components. So parent can pass something via input down to the child, and the child can basically emit an event back to the parent using an output. And in our example, we have a really simple component where you can see in the template on line four, we render the amount of time that the button has been clicked. And on line six, we have a button with attached click handler that calls some method, and that method is defined in the class, and it just increments the counter and emits the event to the parent. So this is one of the most basic components that you could have in Angular. So let's take a look at an anatomy of a unit test. And this is really the same in both Jasmine and Jest. There are some other differences between them, but this is really the same, and this is the same in many other languages, I would say, as well. So first you define a test suite. So we are defining a test suite for counter component. We use describe function for that. Then we have some, we usually have before each. Before each is a piece of code which will run after, which will run before each test, each individual test, and here you would set up some state and some initial state. Then between lines four and six, we finally have one individual unit test, and it's usually, it's using function it. It's a bit weird naming, but it's called like that because the way you're supposed to read this is counter component, it should do something. So that's why the function is called it. Okay. So let's expand a bit on this basic scaffolding for the test by adding in some Angular specific things. So here we have component and fixture. Component and fixture are related. So fixture is something that has some extra methods and helpers that make interaction with the component a bit easier in the test, and it allows us to query some stuff as we will see a bit later. And the component is the instance of the component class itself. So that's, we will have references to two of those and we will be using them quite a lot throughout the tests. And then on lines 11 through 13, we set up a testing module. So Angular components are run inside of a module, both in runtime and in tests. So here we are setting up a testing module inside which the application, inside which the component will be running. Here we define any dependencies that the component might have. Here we're dealing with a really simple component which doesn't have any dependencies. So we just import that component. Then on lines 15 and 16, we create the fixture using create component and we get the component instance. Line 17 is something quite specific to Angular where we call detect changes. This is what triggers the re-rendering. So in runtime, when the application is running, change detection which is the process which is in charge of re-rendering when necessary. It is, let's say, automatic, but in tests you have to do it a bit more manually. And now finally between lines 20 and 22, we have our first unit test for our counter component and we simply assert that the component is initialized, that the class instance exists. So it's a truthy value, the component. Now let's test some initial state. So we will use our before each method to query the pre element and to query the button element. And we will store them in some variables in the test suite because we'll be using them in multiple places so it's more reusable. So here we can see that we use fixture debug element query by CSS and we get the pre element. We do the similar thing for the button and for the pre element we also read the native element of it. Then finally on lines, now we have three unit tests, so we check that the rendered pre element exists, we check that the rendered button element exists. Those are the first two tests and in the final test we check that the initial state of the counter is set to one. And that's kind of just the baseline test to see that everything works correctly initially. Now we want to check what gets rendered. It's quite important that you check what actually gets rendered as final HTML because someone could like delete the whole template of the component and if you were only testing the component state like checking the values of class properties, all your tests would still pass even if someone deletes the whole template. So that's why it's important to check what gets rendered and that's why it's important to actually click on elements etc. We'll see more about that. But yeah, here you remember we queried our pre element, it's an HTML element and we just checked that the text content is times clicked one and that's also like the initial state. This is how it relates to the template that we have. Now let's just like we want to check what gets rendered, we also want to check what happens when we do user interaction. And now it's important when doing user interaction testing, you shouldn't be calling handler methods on the class directly. What you have to do is you have to trigger the event handlers for similar reasons as why you want to check what gets rendered. Someone could delete your template, if you're just calling the click handler directly it will still pass. But if you're actually trying to click on the button and the button doesn't exist, the test will fail. Okay, so that's why it's important. So here on line 7 and 8 we assert that the counter is initially one and the times clicked one is the rendered text. Then we click and then we check, so the way we click is we trigger event handler programmatically. This relates to the click handler that we attach. And this test pass. Now we check that the counter is two, but you will see that it seems like I have maybe a typo here. So I checked that the component counter is two, but the test passes and what is actually rendered is that the text is still times clicked one. It's not times clicked two. And this is because as I've mentioned change detection, which is the process for re-rendering, it doesn't run automatically in tests like it does in runtime when the application is actually running. This is why you have to call it manually after some user interaction simulation. So if you trigger some user interaction programmatically you have to call detect changes to re-render. Now the test will fail because times clicked one is not what actually gets rendered. What actually gets rendered is times clicked two. And you can see in the error log you get like a nice message where it says what was the expected value and what was the actual value. Now we just fix it by updating one to two. So now this test is valid. So, two most important things here is you have to kind of know when you have to re-render in tests. It's a bit tedious, but you have to do it. And the second thing is you have to check what gets rendered in order to be fully sure that everything is working correctly. Now we're writing unit tests for our counter component. But let's say we are now looking at the template of a parent component. So we want to test how is our parent component using this child component. And this is now a unit test of a parent component where we can get the component instance of the child component. So here you can see we will use a query by CSS test JS counter. And test JS counter is the selector of our child component. You can see in the template it's being used like that as an element selector. Then we get the component instance and we check that the counter input is set to 100. So we are basically checking that our template is doing what we are expecting it to do, setting counter to be 100 initially. We can test the other way around. So if our parent component wants to react on counter change event, we can write tests for that as well. So let's say that on counter change we just console.log the new value in the parent component. This dollar event, it's a special piece of syntax, but this will be the value with which the event is emitted. So in this case it will be a number that gets emitted on counter change and that's the value that gets passed to the handler method. So here the test is expanded a bit more because we are now using spy on. So spy on is something from Jasmine. It's very similar in Jest as well. So basically you take console.log and you spy that it has been called. So on line three we set up the spy and on line five we expect that it hasn't been called. On line seven we trigger the event handler to let's say emit 101 as the new counter value and then we expect the console.log has been called once with value 101. So it's important to check how many times something was called beforehand, how many times it was called after, and with which value it was called. And it's all kind of nicely, you usually want to check that something was called once and with some value and that's why you have this nice method to have been called once with something. And this is how the event relates and how the event handler name relates between template and the test. Okay, so let's take a look at some of the best practices. Test doubles, so generally if your application depends on some let's say service and services in Angular are just classes that offer methods and they are most commonly used for fetching data. And here our user service is fetching, like making an API call and fetching the list of users. However, when you're unit testing you don't really want to make API calls and if your component depends on user service you want to avoid making API calls. So for that purpose you would create a test double of user service. We usually call it here user testing service, but you could call it something like user service mock or something like that. And here you define all the same methods that the real service has, but it doesn't make a real API call, but it returns some mock data. And this is how it's used in some user table component for example. So Angular uses dependency injection and this component would inject the user service. But when we are writing tests, we provide user testing service under user service. So the component still expects that it will get like user service instance, but when the tests are run, then you get the mock version, the test double, the testing version of the service, whatever you want to call it. And you can do pretty much the same thing for a component. So component also has some methods, it has some dependencies. You can create a testing version of the component for which it's important that it has the same selector, but it doesn't have to have the complex template and it doesn't have to have any implementation logic. It just needs to have all the inputs and outputs. And you do that very similarly. So if your parent component depends on the counter component, in the unit test for the parent component, you would import counter testing component. And that way you don't use the real component, you use a mock. And this is what makes it a unit test, not an integration test. So you're not testing both components at the same time. You're not testing the parent component and the counter component at the same time. You're just testing parent component and using a test double version of the counter. Next topic is harnesses. Now I won't go into harnesses in too many details, but they are something that's provided by Angular CDK. And you can define a harness for some component. And that harness will have some extra methods which are useful for testing. So in example of material checkbox, there you have a harness which makes it easier to like toggle the checkbox in test, etc. But yeah, I won't go into too many details. You can also write harnesses for your own components. Now last topic about best practices is coverage. So coverage is a metric which tells you when you gather all of your tests, like how much of the code has been covered. And you can get a nice looking report like this. And if you'll take a look into one specific file, you can see which parts of the code have been executed, which have not. So this is a good thing, but it's not something that you should really blindly chase because coverage in Angular doesn't include HTML templates, so you won't get that in the coverage. 100% coverage of course doesn't mean you don't have bugs. And depending on the setup, some files might not be even included in the coverage, so you will get actually wrong numbers. And we have some documentation about that. So yes, code coverage, it's often a lie, but that's a number that project managers like to hear about. Okay, so let me conclude some of these topics. So what you should unit test. You should unit test business logic, services, core UI components, sensitive data handling, helpers, errors and loading states, test if something gets re-rendered correctly, and anything really related to date time handling and time zones, because that's really painful to debug. So having good unit tests for those things, that's really nice. But there are also some things which you shouldn't really unit test, and that's like third party libraries, your implementation details. And what I mean by this, you shouldn't test private methods. You shouldn't be calling them directly. You should always test something in a way that it's used. So if you're testing a component, write your unit test as if you were a parent component. Don't go calling private methods. You also shouldn't test complex interaction between units. That's like integration tests. That's like not for unit tests. It's a different kind of test. Complex UI flows are also quite hard to do with unit testing, especially if it's things like swiping, etc. So that's best left to end to end testing. If you have some really trivial code or some code that is here only temporarily, you won't really get too much benefit from unit tests. That code probably won't break. You want to test things which are more complex, which could break, and you want to ensure that those don't break. And again, setting up unit tests can sometimes be quite complex. So if the time it takes to set up unit test is much greater than the feature is worth, then maybe consider skipping it and do some other type of testing. So here, just like in the end, I will leave you with this short list of do's and don'ts. You can run through this. But it's really important to write. I think what I want to stress most here is that you should write tests based on your requirements. So tests in this way will also be a documentation of what your code should be doing. And it's also documentation that can be executed to verify that the code is doing what it's supposed to be doing. So I think that's the most important note to take here. Make sure that your requirements are covered by tests. Here are some additional resources that I have. Official Angular documentation for testing is actually quite good. It covers a lot of ground. We also have Infinium handbook regarding Angular testing. And there we have a lot of different test cases, let's say recipes, how you should test something. You can also find some useful courses online for testing. I'll here link Angular Academy testing course. I'm not in any way related to them. I just think they have good content. And yeah, that's pretty much it. I thank you for your attention. I know this has maybe been a lot, but please, you can go back and look at some certain sections and see if there's something that you maybe want to take a second look at. I'll be in the chat to answer any questions. And yeah, thank you. Bye.


Filip Voska

Testing Mail Service With Playwright

♪♪♪ ♪♪♪ ♪♪♪ Hello, everyone. My name is Kat Kmiotek. I work as a quality engineer at Zoopla. And thank you for joining my talk today. I will be talking about testing mail service with Playwright. I will show you how easy it is to add email verification or reset password user journey to your end-to-end tests. And honestly, I really like how clever you can go here with Playwright. So let's start. Depending on the application you are working on, you may or you may not communicate with the user by the email. But if you do, you would send them newsletters, order confirmations if you work on the e-commerce website, reset password link if the action is requested by the user, email verification request, and any sorts of receipts or notification they may ask for. And you usually use the third-party providers to handle this mail transportation for you. And those would be Amazon SES, MailChimp, Postmark, and SendGrid, just to name a few. And it's a really good idea because we can rely on them to make sure that email is actually delivered. But the problem statement of this talk is what gets delivered to the user. Can they click the link and reset their password? Does verification email actually take them to the correct domain? If body content of the email actually matches our design system? And of course, if by mistake we don't send any sensitive data or any random data like this test as a subject of the email. So today, I will show you how all those checks can be performed with help of Playwright. So let's start. So to describe the email communication in a production application, you will have application communicating with third-party service. And third-party service will be responsible for delivering emails to the user or user groups. In a test environment for this demo, application, instead of sending requests to the third-party service, will be sending email to fake SMTP service. And then from there, we will be able to access data from Playwright tests. Okay, I just introduced new thing, fake SMTP server. What it is? You can think about it as a container that will be capturing and storing all emails sent by application in a test environment. And for this purpose, you can choose one of many tools available. I listed just a few here. Some of them are paid and some of them are open source. It's up to you which one you will use. For this demo, I'm using Mailhook. All of them usually come with really nice graphical user interface that you can access in your browser. That looks exactly like the mailbox. You can browse emails, you can delete emails, store attachments or forward emails. It's great for manual verification what gets sent to the user. And that sort of fake SMTP server, it will also come with great documentation about their API, so with Swagger. Swagger would list all available endpoints on the service with methods that are available and how the payload should look like and what you expect to see as a response of the API call to the server. Here's the screenshot of the example call to the Mailhook API. So as a search point I'm querying here. Based on their documentation, I provided query params to get all emails that have been sent to the test user at example.com. And on the right-hand side, you can see how the response looks like. So good, because status is 200. And then you can see how many emails have been sent to this user. So the service endpoint returns three. Brilliant, because I was just testing a little. And all of them are listed as email objects in the items array, so on line five. And each individual email object would have ID, from, to, content and body. And if you will look closer below line 34, well, from line 34, you can see that body is actually HTML body, HTML code. So again, here we can do some assertions as well. We can check if the certain strings are present, if the certain elements are present, let's say button or link, but also verify if styling, right styling has been applied. Just to mention, I'm using Tandr client here. So tools similar to the Postman or Insomnia, if you are familiar with them. Okay, so fake SMTP server comes with some nice features, allowing manual testing, but we'll use the same SMTP server in our playwright test. So let's have a look how to translate what I just showed with, let's say, API testing to playwright test. Let's start with pseudocoding, what we want to do. So first our test will need to have precondition. And that depends on your application. That could be the action that triggers email verification to be sent. Let's say user creates the account or using completes the order and order confirmation gets sent and so on. Because I'm not connecting to any real application here, it was just a script that triggers sending email. Okay, so what do we want to focus in the test? We want to do exactly what I did manual with connecting to the SMTP server. So I want to access SMTP server. I want to get most recent email that has been sent to the user. So in a response, there was items array, so I want to get the first element of it. So with index zero. And I want to get HTML body part because that's where I want to do assertions on. And the great magic feature of Playwright, you can actually use this HTML body to render new page and do the same assertions as you would do on a regular page in a browser. So you can click verify button link, you can check if certain elements are present, and you can also verify if by clicking the verify button, you will open new tab and you can do assertion if the new tab, which is also a page, will have expected URL. Okay, so that's a lot to do. So where to start? I will start with accessing SMTP server from my end-to-end tests. And for this purpose, I will be using fixtures. What fixtures are? So probably in your application, you have Playwright config. In your end-to-end tests, you have Playwright config. And you specify base URL value there. And that would be fronted of your application client code, the base URL of your old tests. But you can specify also fixtures, so like isolated contexts that will be used in a certain test. So here I'm creating the fixture called email API. And that one will have separate base URL value, extra HTTP headers, and so on. So that fixtures allows you to add maybe different end point tests from the same repository test, also backend or other services like here, example, Mailhook. So as you can see, I provide the base URL value, which is running on localhost Mailhook server. I pass authorization as extra HTTP headers. And I also specify to ignore HTTP errors as true, because I'm trying to connect to localhost. And this API context that is defined on line 11 will be used on all calls made to email API. Okay, and how it looks like actually when you use it. So on line 14, here I can see that, yeah, on the left-hand side, you probably don't see this 14. It's 14, line 14. I do exactly the same what I did in the Tandr client, where there was manual testing end point. So on the API context that I just defined, I use the get method to query the search end point. For the query params, the same as in the Tandr client example. So I query all the emails that has been sent to the test user. And then I return the response. And here, this may look a little bit complex. What is happening here? Here on line 26, I get the first email, so items index one, content and body, to get the HTML body of the most recent email that has been sent to the user. What is happening on line 26 might look a little bit confusing. So when I was showing in the Tandr client how the body looks like, HTML body of the email, maybe you noticed that there were appendant equals 3D strings in some places. This is because the type of encoding used for email transportation called quoted printable. So here on line 28, I just tidy up the email body from this encoding. It looks complex, but I rely on helpers libraries to do it for me. And then at the end, I just return the HTML body. So now, this HTML body can be used to render the page. So here in the email page object, I define my custom go to method that takes as a parameter actually the HTML code from the email. And on line 14, I instruct playwright. So this time I would be going to forward slash email URL. I want this route request to be fulfilled with the email body. So then when on line 16 actually I go to this URL, what will be rendered in the browser will be actually email body. And then I will be able to interact with it as I would do with any page. So I will be clicking verify button on line 22 and wait until new tab will be open and will return the new tab. Putting it all together on line 15. But basically that's how the test will look like. So obviously first precondition. But then, as I said, I specify the separate fixture. So that's how I imported it to the particular test email API on line 15. So then I use the email API to get email body on line 18 for the test user. The test user that I use for manual verification as well. And then once I had this HTML body on line, I think it's line 12, I can use this HTML body to render the email page. So I click the button on this email page and then assert that actually URL of this newly opened tab matches test.js summit. Okay. That may work, may not. But if I will run the test actually in a debug mode, I can see browser opening and play around inspector opening beside it so I can step through the test and see that actually works. So what is happening here? That's the actually first page that opens because the API call happens in seconds just before the browser opens. And here we are at the step that I go to forward slash email and the email is actually rendered. And here, player attempts to click element button with ID button one. And once that one is successful, the new tab is open. And as you can see the URL is exactly the one that we expected to appear here. So when I seen it for the first time, I was like, whoa, that's amazing. You can actually get HTML from the API call and use this HTML to render the page. Yeah, it's amazing. And going from here, you can think how many other assertions you could do on the actually rendered page. So you can do visual regression tests. So do the screenshot comparison to make sure that nothing has changed in the version of the email that you are sending now comparing to the baseline image. Here you may say, OK, I sent to the user as a maybe title or I don't know, header of the email I rendered their username or their first name or initials. And that will be different with every test run because you dynamically generate test data. So here with Playwright API, you can actually exclude this area from the comparison. So let's say mask the h1 locator and that will help you with the testing the email if the data is dynamically generated. You can also, as I mentioned, once we have this email rendered as a page, you can do all sorts of assertions like as you would do on a regular page. So let's say that check that verify button actually has the correct attribute or if the state is changing when you hover over and so on and so on. So I hope that I showed how the Playwright can be used for testing mail, email body and what actually gets sent to the user. So I checked that email got delivered to the user. I checked functionality of the button. So you could see that action click has been successful, that the navigation happened to the correct URL. So with Playwright you are also able to do a visual check on the email, on the content body and obviously all other assertion on DOM elements as you would do. Okay, that's the end of the talk. I hope that you found it useful. All the links that I just posted here will help you to maybe try and build a similar solution and apply it to your tests. As I mentioned, I was using Mailhawk. So I provided a link to Mailhawk Swagger, Mailhawk Docker image with OLF, so you can just run it locally. More explanation about equal 3D. So the Quotent printable encoding. A few words about Tundr client, so my go-to tool for manual API tests. And link to contact with me if you would have any questions. So yeah, I just look forward to hear your questions now. If you want to access the presentation, you can just scan the QR code and that will take you to the slides. Thank you. ♪♪♪


Kat Kmiotek

React Advanced

React Advanced Conference 2022

How I Test a Million UI States with Every Merge — Visual Testing with Storybook

Michael Chan

TestJS Summit 2021

How to Catch a11y Defects During Unit and E2E Testing

Hello everyone, I'm going to show you some ways to catch accessibility defects during unit and end-to-end testing. First, a bit about myself. I'm Emily. I'm a developer in Toronto and I currently work for a company named Narwhal. So this is how fixing accessibility bugs feels like to me. Basically doing a lot of patch fixes. As a developer, accessibility bug is a type of bug I do not wish to be assigned to. In my past projects, accessibility testing is usually done at the very end of a project. By the time I need to fix those bugs, the page is already built. It is very hard to make any structural change and move components around. This talk will show you some ways for developers to automate some accessibility testing while we are doing the development. I'm going to show you how I add accessibility testing to this simple website I've created. It is a search engine for Studio Ghibli. So Studio Ghibli is a Japanese animation company. This simple app would allow you to search any films and characters under this studio. So this is what this website looks like. You can search anything. For example, princess. If I enter princess here and search, it will show me any princess related films. If I search for example Kiki here, it will show me films and characters related to Kiki. I can click the film here and it shows me the film detail. Quickly go through the tech stack. Frontend is React. Monorepo tool is NX, which is built by my company Narwhal. NX is a powerful tool that will set up unit and end-to-end testing for your outdoor box. So I highly recommend you to check it out. For unit testing, I'm going to use jazz and jazzX. For end-to-end testing, I'm going to use cypress and cypressX. For unit testing, I use jazz and jazzX. You can install jazzX and type in, use the command here. Now with jazzX installed, I need to set up. I add a setup file named jazz.setup.js here. And inside the jazz config file, I need to add a key setupFileAfterEnvironment and point to jazz.setup.js here. Inside jazz.setup.js, this is what it looks like. It's very simple. You just have an import from jazz.so.x here, and it also extends jazz's expect function to include, to have no violations from jazzX. Now the setup is done. I want to unit test the search form component on the top here. It's a really simple search form. It has a text field and a search submit button. Currently, you have a default test to check whether it renders successfully or not. Now I want to add unit test for the component to check whether it has any accessibility violation. Note, I've imported from jazz.so.x on the top here. Then I add a unit test to check for accessibility violations. It renders the component here. And then you pass the container to the x from jazz.so.x library. You expect the output here to have no violations. The unit test will be very simple. It has two lines here, and it's pretty easy to add. Now when I run a unit testing again, I would notice that the test failed, which is actually great. It means the tests are actually working. It will show the accessibility errors in the console here. Developer could add similar tests to each component for the app. If all the components are accessible, the entire app are more likely to be accessible. It is good that we allow developers to add accessible unit tests while just building the components. The components could be standalone and do not even need to be shown up on the browser page. This will help to avoid accessibility bug at the component level. How about page level? I'm going to talk about end-to-end testing. It's going to use cypress and cypressX. Enter the command to install cypressX. I have a cypress test for the search page, which is basically entering text in the search field, submit the search form, and make sure it redirects to the results page with the right query parameter. So this is what the cypress test looks like. You enter the text field and redirect to the results page. To set up cypressX testing, I need to add typing first. In the tsconfig file, under compile option types, I need to add cypressX here. Also in the support file, I add the import from cypressX. Now I'm done with the cypressX testing. I'm ready to add cypressX test. In the before-each statement, I need a line to inject X here, then I simply add a test. I just have one line. Check accessibility. Now I run the cypress test again. Notice it failed. I have added a console log to output accessibility violations on the search page here. This will help test app on the page level and prevent accessibility-related regression bug. With all these tools, it will be easier for developers to avoid accessibility bug during development rather than going through them when the entire app is built. This is the end of my presentation. Thanks for listening.


Emily Xiong

E2E Tests for API – Saving Nerves and Hours

Hello everyone, my name is Valentin Kononov and today my lightning talk is about end-to-end tests for API, how I did it, how it saved my nerves and hours and so on and so forth. But still just a little bit about myself. So I am a developer, mostly working with TypeScript and JavaScript these days, so with Node.js and especially with the backend. And on one of my last projects we had a backend written in NestJS and we struggled a lot with testing it, especially when we need to change some functionality. So that's how we tried to eventually move to end-to-end tests and here is what my talk is about. At the moment I am working at a Unity company and it's a lot about fun and games and stuff but I am working in an advertisement department so we have a lot of services and both frontend and backend and of course we need to test them carefully. So straight to the point, we don't have too much time. So when we talk about the API testing, what do we usually mean? First of all we need to test the data flow, so how the data flows through our services, what are DB queries, what input, output and data transformation. And in most cases we just need to test that crude operations work fine and the syndication works fine and all the endpoints are correct. And what we usually do to achieve that? Of course, in most cases we have integration tests. And how does it work? Integration tests usually test the whole module, the whole service. But service is not something which is present by its own, it's something which communicates with other modules like database or data repository or some syndication or something else. So that's why to test it we need to mock some dependencies. And when we did that and the test is working fine, what do we do next? We update the code, we make some new features and what we need to do next? We need to update mock dependencies so the test keeps working. And what else? Yes, we need to update mock dependencies once again and again and again and again. So I really faced it. We had a really big, important, super mega critical test about some export process and every time we changed the minor thing in the export code, we needed to completely rewrite the whole test. That was bad. So how usually the integration test could look like with all the mocks? So it's actually a huge pile of different kinds of mocking functions like you see in the slide. So sometimes we mock the whole module, sometimes we mock just one function. And of course when something is changed in the function, especially in the last example here, like the update function was changed and its output was changed. In the format of how it was an object, it became just one simple number. So we need to update the mock. And that actually sucks because we spend a lot of time just for supporting the tests. But it should be vice versa. Tests should save our time for the development. So is there any alternative? So how we can proceed so that the whole process would be easier for us as developers? And basically my recipe for that was end-to-end tests. We discussed with the team, okay, we have a NestJS framework, it has a lot of testing capabilities, so why don't we use it for end-to-end tests? And what is the end-to-end test for when we're talking about the API? It's when you start up the whole API, like the whole Node service in your testing environment, and you test it at point by point. So in this slide you can see how initialization of the Nest API could look like. So you see, you create a testing module, you initialize it, you create a Nest application, and then you should close it and of course you can override anything you want. For example, Nest uses so-called guards for authentication. Here we can override it. Initialize and close in the very end when all the tests pass. And authentication guard mock is a pretty simple thing, which you just write once and use everywhere. It's really such simple stuff. And here is how the whole test would look like. So you literally request, you do a POST or GET request to some certain endpoint for some certain URL, and in this sample, because it's POST, you can send somebody details, like what kind of project data we should add. And you can expect, like here in the last line, you can expect some kind of code, like it should return 201, which means successfully created. And you can also do a GET request and also form some URL, send request, expect some code, and also expect some response and check that the response properties are correct. So the pros and cons. The pros, easy to fix, easy to modify, and you test the whole workflow. And as you see right here, the whole test is very readable, and it's also an additional source of documentation for your API. But of course there are some cons and bad sides. So it can take time to write the initial version of this test, because in some cases you need to prepare the data. Of course when you have just a crude API, you can use just only GET or POST requests to create data and to make sure it's created. But in some cases you need to prepare the data in a database, and this minimal database should be run somewhere like in a continuous integration service. And just for you, we have a really special trick, really small, sometimes, in my experience, only once. But we needed to fake the date. So I prepare some data and I should make sure that this date is created with this date now. And usually we did it like that. But in some code we actually use not just the NOW function, but also CONSTRUCT. And there is a little bit more comprehensive solution how to mock the date for both places, for NOW and for the constructor. So just make a screenshot. It's a very useful trick. See you later. And here's the link for my slides, my website and my doc, which inspired me for all of that stuff. So, as a summary, make tests. It's really a great thing when you do tests. And entering tests makes the changes faster and you can cover the whole workflow with the last mocks and be much more happier. See you guys!


Valentin Kononov

It's a (Testing) Trap! - Common Testing Pitfalls and How to Solve Them

Ramona Schwering

Test Time Execution! Why it Can't be Ignored? 

Hello everyone, I'm Jay, one of the founders at LambdaTest. Thank you for tuning in and also glad to be a part of TestJS event here. So just to give you a quick background, I started LambdaTest four and a half years ago. I've been building our complete test execution platform on cloud. That's our vision that we're working on. And today I just want to take a few moments to share about the common problems that I get to hear and often hear from the modern QA and development teams. And before that, I would like to give you a quick gist of where we come from. We started in 2017 and we recently got a series refunding from Sequoia and Telstra ventures. And from the time we released our first product, we have close to half a million users who have signed up. And out of our 7,000 active customers, we have 500 enterprises who have been a part of the journey. And before we talk about what we do, I would like to highlight on the problems. And I often have been speaking to a lot of leadership, engineering leadership, and the modern QA and agile teams. And one of the things that I often hear is that they're aiming at shortening their release cycles. Now, that could be because of a number of reasons. That could be because of innovation where they want to drive, roll out new features. And or maybe they just want to fix the bugs so that they can make the experience better for the customers. Or maybe just overall, just refactor the code for better stability of their application as such. Now, we all know that test automation tools have been existing. And although the users and the customers that I've been talking to have implemented end-to-end testing suit, but there still exists a bigger problem. And that's the test infrastructure on which they run the tests on. And unfortunately, this infrastructure on which they run the tests on, if it is not really up to the mark, it really increases the test execution time significantly. Now, let's talk about the infrastructure as such. Now, talking about the in-house infrastructure, firstly, it takes a lot of time to build and maintain this infrastructure. And if it's not really up to the mark and customized as per the needs of the QA team, it becomes very tough to scale it. And also, this adds a lot of cost to maintain this entire effort. And the second aspect is high test execution time. Now, it's very, very important, especially for the modern QA and dev teams who are really focusing on increasing their release velocities and ship code faster. High test execution is one of the biggest roadblocks that we often hear. And again, like I said, it's not the problem about the end-to-end automation suit, but it's about the release velocity they are aiming at. And one of the roadblocks, again, is because of the hours and hours that it takes to run your test suit. Now, if you look into this graph in the right-hand side, as you start, your test suit is typically it just takes seconds to run. And as you scale your test suit, as you know that your test suit, the test execution time is proportionately related to the code you ship. So the more code you ship, the more test scripts are written and your test cases grow. Now, as you go towards the enterprise suit where you have 10,000s of tests, your execution time takes from minutes to hours to complete. And that is a bigger problem to solve because your developers are actually waiting to get the feedback of the code they've written. But here we're waiting for the test to complete. And again, with the poor infrastructure that we are running the test on, we are impacting the productivity of both the QA and development teams. Now, third problem is flaky test at scale. Now, flaky test could be because of a number of reasons. Now, especially when you're running a test on infrastructure, now there could be a lot of problems related to the CPU consumption. The disk errors or there could be high resource consumption in the machine that you're running a test on, which leads to the flaky results. Now, this is a bigger problem altogether to solve. And one of the key aspects that if you see, if you correlate your high test execution time with your release cycle, so you can directly see that the high test execution time is leading to delayed release cycles because the feedback time that you're taking to get back to the developer is also getting lengthier. Now, we all know this problem exists. Now, this is exactly where LambdaTest comes into picture. So we are building a continuous quality platform on cloud where you can test with confidence on our infrastructure and scale your test as well. So we started with our product, the first product that we launched is live testing where you can fire up the virtual machines of 2000 plus browsers and operating system combination and test your applications. You can also test your native applications on simulators and emulators. And then we have the automation cloud where you can run your automation test suits and you can run hundreds of parallel sessions to make sure you're able to kind of reduce the test execution cycle. And most importantly, the day we started the company, we focused on three parameters here. We focused on performance, we focused on reliability and scalability. These were the three important aspects because we understand that this is one of the most crucial aspects for the QA and development teams. Now we built this fastest Selenium grid on the cloud and where you can also perform headless browser testing and also get advanced reporting and analytics. And we just recently launched a real devices cloud where you can perform your manual native app testing. You don't need to maintain those infrastructure in-house. You can also perform automation testing and also look into the device logs as such. So along with the infrastructure that we built, we also give other functionality like you will be able to run your pre-pod, you'll be able to run your test on pre-pod environments by using our enterprise ready tunnel. You can run your tests in parallel across different browsers, devices and OS so that you can fasten your test execution time. You can reduce your test execution time. And we also give you support for geolocation and this is easily integrable with your CICD tools. If you want to fire up your testing from CICD into our infrastructure, it just takes just five minutes for you to integrate and get started. So apart from that, I also want to highlight on the integrations that we have. Our integration landscape spans around 120 integrations that we've built with frameworks, with codeless automation, with CICD and bug tracking tools because one of the aspects when we started the Lambda test, building Lambda test, so we wanted Lambda test to be a part of the ecosystem, not work as a standalone system. So we try to build an integration as much as possible to make it more easier for the QA and development teams. In a short span of time, we have garnered good attention in the JITO Crowd space and this is just because of our, I would say, the reliability and scalability and performance that we provide on our infrastructure. And we've been leading and training on JITO Crowd as momentum leaders and you'll be able to see the reviews that we get and the ease of use. It just takes around 30 minutes maximum for a user to onboard by integrating this test suite and CICD tools and start firing up the test. And yeah, so this is a quick overview about us and I really thank you for hearing me out. I really appreciate your time and I hope you enjoy the TestJS conference. Take care. Bye bye.


Jay Singh

Why Tracing is a Lifesaver for E2E Tests of Distributed Systems?

Hey everyone, I'm Burak from Tundra. I'm super excited to be with you all today and I hope you're having a great time watching these all excellent speakers here at TestJS Summit. Today I'm going to talk about how tracing might help you finding the root causes in your test failures, end-to-end test failures in your distributed systems. Before we get into it, let me introduce myself and also Tundra to you. I'm Burak and working as a product manager here at Tundra. Apart from my professional career, you can see me playing tennis on the course and Tundra builds modern tools for developers and QA engineers who develop and deliver apps, modern applications on the cloud. Our flagship product, Foresight, helps you to monitor and debug your CI-CD workflows and tests. And today, we are going to talk about why do end-to-end tests fail, how to debug them, and how tracing can help you to find the root causes easier. Why are they failing? Why do end-to-end tests fail? So they are trying to check every part of your system works together correctly. In short, they are checking the whole application from back to the front. And there may be a couple of reasons of failures and one of them is time waiting reasons. If you have multiple services communicating with each other, the responses won't be instantaneous. So there might be time-related issues. So another thing is a test may be dependent to another one. So if one fails, there may be 10 other failures occur since that one failure. And another and most important one is inter-service communications. Since end-to-end tests may have different services waiting the responses for each other, so think about like if you have a checkout system waiting for a shopping cart return, so there may be a possible failure from a different service. Even though you get a 500 service error, it doesn't tell you what actually happened and where it happened. You can, when there is an error happens, you can debug them by checking the logs obviously. They provide you a summary of the whole execution. And although modern tools and IDs highlight the exact error, you may end up getting lost in thousands of lines to find the exact reasons for your failure. And also you can check artifacts such as screenshots and video recordings. Since end-to-end test provides end-to-end test scenarios contain user's experience, so you may get help from those kinds of artifacts. However, it may not be enough for you to see what happened behind the curtains as I mentioned, like if you have especially working on microservices. So how tracing can help you detect those kinds of failures easier. And tracing actually is a common developer practice for analyzing and understanding the whole application data and the flow. And suppose that something goes wrong with a monolithic application, a developer needs to just go over the stack trace since the whole application runs on a single execution, runs on a single process to figure out what actually happened. Now actually imagine that scenario with a distributed app potentially consisting multiple lambda functions and there are some external services, external databases, authentication services that your app needs to work with. Yet alone testing those kinds of applications is not easy, but also you are trying to find the root cause of the failures. It is really a challenging issue and those kinds of scenarios need some special tools. And fortunately, Tundra Foresight is that special tool for QA and build engineers. You are seeing a result of a single test case visualized on Tundra Foresight dashboard, not only providing you the logs, messages, error messages, and performance history, if your test case includes some screenshots, you can also see them on Tundra Foresight dashboard. And also by the help of our novel distributed tracing technology, it's easy to visualize the whole app in a single trace map. You can also, with the help of distributed tracing, you can track message exchange between different kinds of services. And if I click on the trace map, I'll get a map, actually a flow that visualizes the whole JST execution. You are seeing a JST execution and the whole end-to-end test requires some transaction between AWS services such as API gateway and lambda. And if you go over and see the red arrow between the lambda and DynamoDB here, you're seeing a red arrow. So if I click on that, I will get a summary of those transactions. And after navigating through the trace chart, I'll find the error newspan, I'll get, with the help of time travel debugging, I can see that there is some error, saving an item to R in my add map request, so I can quickly detect the failed reason. For summary, everything fails all the time, but we have limited time to find those kinds of errors. So distributed tracing and foresight capabilities helps you to find them easily, saves your time, saves, you don't need to reproduce the whole application flow again and again. And you just need to only navigate through the application flow and find the root causes of your tests and fix bugs and detect bottlenecks on your test. So thanks for listening to me. This is all for me today. If you have any questions, feel free to ask me and I will be very glad to hear your feedbacks and answer your questions. If you want to give it a shot, go check Tanra.io.


Burak Kantarci

JSNation Live

JSNation Live 2020

Adaptation of the E2E Tests for a Big Project

Hi, I'm Vadim. I'm a front-end engineer from Delivery Hero, and today with my colleague Devesh, I want to present you the adaptation of the end-to-end tests which we have done for our project in Delivery Hero. The agenda for today would be the case study for the end-to-end tests, or why we start doing that, the end-to-end test setup, and the best practices we use, how to avoid repeating the cycle while developing tests, and how we are testing different variations of the feature. So the case study for the end-to-end tests were pretty simple and common, like for everyone, because Delivery Hero is a huge company with a huge front-end project. It takes a lot of time and effort to maintain and develop it. So we thought that it's a big improvement to the situation to introduce the end-to-end tests, so we don't need to do manual testing features every time, or debug fixes, so we can improve our resilience and the calmness while deploying a big change. So just to have an overview of the scale, we have more than 20 front-end developers. We do the 10 pull requests daily, our micro front-end repo has more than 50k total lines of code without node modules, and the old monolith has 200k total lines of code. So it's quite big. The end-to-end test setup we have is basically a separate repo, because we added the tests while we were migrating from the old project to the mono repo with the micro front-end, and we have the simplicity of running the tests in the separate repo, so we can easily onboard developers. For now, we run them as a cron job. We know that it's not a standard, let's say, but we are working on the integration to the CI-CD pipelines, and we created our own custom Slack integration with Cypress. We know that there is an official plugin developed last week, but we still have our own for now. You can see the example of what the Slack bot is messaging with the report, and you can click on the report button and check this kind of a report, which is very helpful, because it shows the stack trace, the error, exactly error, and the screenshot. We also, of course, are using some best practices from the Cypress team and some of our own, which is trying to use accessibility principles in the test created by Ken Dodds, which is the testing library creator. So instead of using data CI, which is a best practice from Cypress team, we can use the area labels and other accessibility attributes, which is very helpful and handy. We also, of course, use the Cypress commands for general helpers, like if something is used in separate specs, or more than two times, we just wrap it into the command and use it as a one-liner, which is simpler. As well, we are using a given annotation, but not only as a structure for the test, but also as a command, because as you see here in an example, it's very helpful, because even if you remove the whole code, you still have an idea of what the test is going to test, because sometimes the commands can be complicated. And now I want to give over for Dabesh. Please. Thank you, Vadim. Hi, everyone. So, Vadim, you told quite a good scenario, like what are the best practices we are following, and so I'll be guiding you through about two particular things that we are doing at our end, which really helps us achieve E2E test very quickly. This one particular thing is about how to avoid repeating the cycle. Most of the E2Es that you see nowadays, it's like a single flow. First, the E2E goes to the one page, then the second page, then the third page. It's a sequential manner. But considering we are the 20 plus dev team, as Vadim mentioned, and different teams are working on different pages or different modules, we don't want to actually repeat the whole cycle during the development. We want to test our features, we want to test only our page. So this will be something like this, going from one page to the other page. So how are we actually achieving it? So we actually created a utility service in our utilities, I mean, not the utility service, but you can say a command that Vadim mentioned, we have a custom command, which is about go directly to the page. So what happens in this particular command? So this could be any page, like this could be a go directly to my page, and then I can actually mock the previous data about all the pages which came before it, like all the data that they added to the browser, to the storage, everything will be mocked here. And what I'll be doing after this, so I'll be using the like, C++ window, I'll be setting the local storage, all the data, then I'll perform the test related to my particular page. And if something is wrong, like something is wrong during the mocking of the data, I can just avoid doing the test. And I can just redirect to the default page. Now, so what comes next is like, how are we actually testing the different versions of a feature? So suppose you are on a page, so you have reached your page, but now you want to test different variations of your feature on that particular page. So what would you do? So for this particular thing, we went to a simple solution, like use the leverage, the cookies from the browser. So for that, we actually created a utility service in our app, which uses the cookies from the browser, and it actually changes the values which are coming from the feature flag service. And it actually maps to it, it merges with the new values. So in this case, this utility service can be used to change the variation of the feature flag. And it looks something like this. So in this case, you can see like we have the feature flags coming from the service, we have the flags from the cookies, we merge it and we store it in the in-memory cache. And what happens after that? Okay, our app is ready to handle it, handle the cookies. Now we just need to actually set this particular key inside a cookie with the variations mentioned over here. So as you can see in the test, so test for feature flag variation, and I'll just set the cookie for those variations. And then after that, I can just keep on writing my test. And so as you see, like we have a lot of good practices, and we have good things in place right now, but nobody's perfect, and there's always room for improvement. So a few things that we noticed, and we have a plan for it. So one thing is like moving away from the Cron job and adding the E2E test to a CI-CD test. And the next thing is like, as Vadim mentioned, we have the E2E test in a separate repo. And we had the use case because we have two different projects we are supporting. We are migrating from the monolith to a micro frontend. So once everything will be migrated to micro frontend, we will be adding the E2E to our monorepo for the micro frontend. And definitely always keep improving. So that's what we are doing. We'll keep improving the quality of our test. And last but not least, please, please join us. We are looking for some developers like you. Please join us at Delivery Hero. Thank you. Thank you.


Vadym Kukhtin

Devesh Jadon

TestJS Summit - January, 2021

Testing for the Modern Web with Playwright

Arjun Attam

Don’t Make These Testing Mistakes

Gleb Bahmutov

Play it Right with CodeceptJS: An Introduction to Supercharged End-to-End Testing

Hello, I hope you enjoyed the conference and I hope you learned a lot today. Lots of new tools, new approaches, you met wonderful people and I'd like to jump on this train and introduce you to a new very cool tool, Concept.js. So I would call it supercharged end-to-end testing, well, because I can and because I really do enjoy testing with Concept.js. So my name is Michael and I will introduce you into this tool and show you the problem it actually solves. So I'm from Kiev, Ukraine, I'm actually a web developer, but I also work as a test automation engineer to practice, to see how people, what problems people do have when going with automated testing. And that's how I take this experience to make it an open source testing framework like, surprise, surprise, Concept.js. So yes, I am lead developer of Concept.js as well, but also I like do testing with Concept.js. Well, I think it's fair enough to say this. I started my journey into open source testing frameworks with my previous project, Conception, which is very popular in PHP community. And Concept.js takes this experience and brings this to a new level and gets into JavaScript ecosystem. Nowadays, I'm working for my own company and we are building a test management system for automated test, Testomato. Testomato is focused on JavaScript tests. So if you have a total mess with your tests, Testomato can organize it for you. That's the idea. So I would start my talk with the question about, do you remember that old good days when everything was pretty predictable, everything was stable? And I'm not talking about the things like that you could travel to all the places you want, you could talk to people you want to. I'm talking about just engineering stuff that previously we had only one tool. It was called Selenium and we used it to test everything that is related to browser. If you need Selenium, you would probably choose Java. If you need, maybe you will need Python or Ruby, but you will choose Selenium. It's certain decision. There was absolutely no alternatives on those days. But today we live in the era of uncertainty. The JavaScript spawned and it created a wonderful ecosystem of different tools. And today choosing a right tool for the job is a tricky question because we have Cypress, we have WebDriver, Playwright, Cucumber, Jest, Mocha, and wow, how much of things you need to learn today to choose the right tool for the job. We definitely live in the era of uncertainty. And much more stress brings the questions I'm trying to list when I'm asking a client for a new test automation project. Because we can start writing tests, but then client came out and said, hey, hey, hey, I learned about the BDD yesterday. It's so awesome. I want to write this test myself in Gherkin language. Can our tool write tests in Gherkin? Good question, especially if we already have our tests not in Gherkin but in JavaScript. Can we easily introduce BDD level on top of it? Oh, the client came. I need Internet Explorer. You will hate me for this, but we need Internet Explorer because I have lots of enterprise clients. Can your tool easily switch to Enterprise Explorer on the fly when all the tests, especially with BDD and all the stuff were already written? That's a good question. A client came again. I need Safari support. Well, Safari is fair enough. Safari is a modern browser. Everyone uses it. And if our tool can do Safari or not, this also can be tricky for some kind of tools. But we need to be prepared for all those changes all this time. Can we adjust our tests to be executed on all those different environments? And especially for mobile browsers because the world goes mobile and testing in mobile is also a hard requirement that the client may not be fully aware of at the beginning of the journey, but will be asking when you have some tests and they will ask you, could you execute this test on mobile and this test? So mobile browser can be a new requirement as well. And again, it's an era of uncertainty. We don't know what a new tool will come up this year because every year some good test automation tool emerges. The previous year we received Playwright from Microsoft. Previously, we got Puppeteer. There was Test Cafe Cypress. So lots of new tools are coming out here and there, and maybe 2024 brings us something new. Will we be able to switch to it if it's so new, so-called Cypress killer, Playwright killer, absolutely anything killer? Maybe we should try it as well and without rewriting tests. Is it ever possible? So this question we can address to concept.js. Concept.js is BDD style testing framework. BDD style means that we write our tests in human-like language and we try to avoid some complex JavaScript interaction and keeping the code as readable as possible. And it's solely focused on browser testing that makes it different from Mocha or Jest, but it has API for all browser engines. So unlike Cypress, concept.js doesn't try tests on its own. It delegates the running to Playwright, to WebDriver, to Protractor, to Puppeteer, to Test Cafe. So concept.js can easily switch execution between all those engines. And yes, it comes with Cucumber BDD support in case your client really needs it. So here is the test in concept.js. As you can see, you can read it. It's very straightforward. You don't need to look into documentation. You don't need to think of what is happening on the screen. You can already understand everything that is happening here. So concept.js supports clicking by button names, by button text, by CSS, everything. So you can make your test as readable as possible when writing tests with concept.js. So concept.js is pretty easy to install. It's just one command. And by default, it comes with Playwright because I think Playwright is really awesome nowadays. And I would choose Playwright for my next test because it's multi-browser, multi-platform, and I would recommend it to everyone. But concept.js can be installed without Playwright. It's just an opinionated decision which you can easily switch. So concept.js comes with a demo project, but let's start with a new empty project. You will need to initialize your new test and select browser engine, select browser, answer this boring question, and create a first test. So when we did this, we get into the first test created. The test should contain only these two steps, open a page and launch the pause. Now I'd like you to introduce another cool feature of concept.js is pause. Pause replaces all your tests that need to go after they open the page. So pause is awesome. Let me show you. So I am executing this test. It's running in my browser. It opens Airbnb website with Playwright. And after the page is opened, the execution is paused. So now I can open my dev tools and think of what should I do next. You see this is my terminal. You see this pause message here. So when I'm not sure what should I test, what should I start with, I'm just starting with a pause. The page is opened, the dev tools are opened. I can navigate through the page and pick different elements, pick different comments, try them. And this is how it goes. For instance, let me show you how we deal with calendars here. I have my dev tools opened on the bottom and terminal on the right. I write my comments into the terminal and I check if I can find elements by the locators I just picked up. For instance, I found that I can select date with test ID locator and I try it. Yeah, it worked. And this is the date was selected. Then I press key enter and I see that browser acts correspondingly. So I continue to do my actions by writing comments, selecting elements. And this is like I am writing my tests in the real time. So when we finish picking up comments and we finished our tests, we go back to the editor and our comments were saved into CLI history file. What I do, I copy my comments from this file. Only successful comments were saved. I remove the pause and voila, here is my test. It's included all the comments I had. So I was executing this browser once and I didn't do any blind guesses, blind searches. I executed the test and the test is here. By the way, I could write my comments and I have very nice auto-completion support. So everyone can start with Cutsape.js because the comments are very easy to catch up. For instance, this is the test we finished with, but as you see, it's very easy to read. So junior engineers can read it and they don't really care of how it is executed. They don't need to learn APIs of different frameworks, only these comments, which are very well documented and auto-completed. How hard is it to switch browser engine? Basically like this, we go to config and we change from player to web driver and we can expect that the test will be executed similarly. Not the same, because you see the web driver will work a bit differently than playwright. Maybe at some points you will see crashes, but the idea is that the comments are the same. So all the difference can be easily fixed with maybe add some weighting, change some locator. So for most of the tests, the transition will work smoothly. For 95% of test code, you will require no change. So this solves the problem I mentioned. So if you need to transform your test to playwright, you can easily transform. If you need web driver, you can just change config. Maybe you will need to change to TestCafe for Internet Explorer support. This also works. And yeah, this is really powerful. Concept.js also comes with built-in UI, a graphical interactive test runner that can show step-by-step report of all executed tests. So you can use it as a reporting tool. You can see how the execution goes. And it has also some features that terminal has, but some more advanced as well. For instance, you can keep one browser without relaunching a test and switch easily between headless and window mode. Also in Concept.js, we have a built-in reporting support. So one of the best supported tool here is Concept.UI for sure, because Concept.UIs are done to Concept.js, but the screenshot on the right is not from Concept.UI, maybe you already guessed what it is. It's allure reporting. So Concept.js has built-in support for allure reports. In case you need better reports to show to your clients or just to see what was happened on CI in production, so you could track the changes on your tests. We also do support the more enterprise technologies like Report Portal. We have official Concept.js agent in Report Portal organization, so you can have Report Portal reports with Concept.js. And Testumatio, the tool I'm working on, it also tries to bring better reporting to your end-to-end tests. So reports are pretty important in nowadays testing because everything can be shown from console and for sure your managers, your clients won't look to console to see what tests are correct, what tests are not, for what reasons, they want a clean, nice graphical UI to understand this. This is why Concept.js provides so many options for reports because depending on your budget, depending on your conditions, you can choose whatever tool you want. By the way, all those three tools are absolutely open source. You can start working with them and with Concept.js today. So let me sum up some important features of Concept.js. Not mention features of Concept.js because you already heard of switching between engines, interactive polls, easy to learn syntax. What else do we have? And yes, I was already showing this a bit, maybe you have already seen it. So we support parallel execution by workers. So very modern, very elegant API. You can write your own custom runner so you could parallelize your tests however you want. For instance, group run tests in two groups, in two browsers from Firefox or make four groups at Safari. So you can split tests however you want into groups, bring as many browsers you need and have parallel executions and have single report after this. That's really cool. Also built in and simply working solution for failed steps. So automatically tries four steps. So if some element is not found on page on first try, you won't even notice it. Concept.js will re-try this and wait for it to appear without even notifying you. So it works for any command, click, see, so whatever it is. So it just works. Just don't touch it and you are fine. Concept.js takes the traditional concept of page objects and it's built in into the framework so you can easily refactor your code to make it easy to be used in other tests. So page objects, step objects, data objects are all built in. And as I mentioned previously, we have Cucumber BDD support built on top of Concept.js. So this is really awesome because you can combine true BDD with specifications in JavaScript like Concept.js tests. This is important because not all tests should be written in the truly BDD manner because not all tests are features. So because some you need to have regression tests for bugs. So my point is that you should combine BDD approach with classical approach and only Concept.js testing framework allows you to do this. So another cool feature I really like is if you have API and you could communicate with your application via that API, for instance, you could create features through this API, delete pictures. And it is much easier to do it through the REST API interface instead of web interface. So you can easily combine them together. And that's my final points. Concept.js is a testing framework and what differs it from others, it focuses on end-to-end testing without executing your end-to-end test by itself. So it tries to take the best user experience into itself to make testing as easy as possible. So Concept.js is absolutely free, open source. You can contribute on it. There are already big and small companies using Concept.js and enjoying it. So join our team, join our community, try Concept.js on your own to check it. And it works really great with Playwright. Thanks for watching. My name is Michael and see you soon. Yeah, I thought that Cypress will take even greater numbers here. I'm pretty impressed that someone is using Concept.js. What I wanted to show with this poll is that there are different engines and for different purposes you can choose the tools that fit right for you. So I think Cypress is very popular because it's very developer friendly and it took on board lots of developers who didn't like testing, who didn't want to do this and browser testing seemed to them complicated because it was always a product of Selenium to deal with. And the part with WebDriver is more for QA engineers who work on end-to-end testing. They need more to browser control. So I think it's like two audiences here. And yeah, they're quite different. But we need to engage all of them to collaborate and make better tests. That makes a lot of sense. We've also started getting questions for your talk. So I've got a few questions for you, if that's okay. I'm going to start with them and I'm not going to mention the names of the people who have asked the questions because I'm going to butcher the names. So there's no point. But one of the first questions was, does CodeSav.js work in the multi-browser mode? For example, a pair of browsers are interacting with each other? Sure, sure. We had this feature implemented like a year ago. No, two years ago. So yeah, it's quite some time. So it was working, it was better tested. And yes, you can use it. Two different browsers if you want it. Okay, okay. That's really good. Somebody made a comment about CodeSav having different trade-offs like Cypress does. Cypress, for example, says it doesn't have multi-tab support and you can't use it to drive two browsers at the same time. How's CodeSav faring compared to those? Cypress doesn't solve this problem. It delegates it to the engines that already solved this problem. So this is why I prefer tools like Playwright or Puppeteer, which have full browser control. This is why I dislike Cypress for most of my testing work. But yeah, this is like a user preference and whatever is better fits to your job. So if you need to test single page application with a single component, Cypress is really fine. But if you need full browser control and more and more features, ConceptJS is better because you can choose the engine while you write your tests. So the code base is growing, but you're still able to choose a better engine if you need it. That makes a lot of sense. And I like that, especially since people were really, really skewing towards one of the engines out there. How does CodeSav deal with things that take a while to appear in the page? For example, a single page app? Sorry, how does it... What? How does CodeSav deal with things that take a while to appear? So they're slow to load on the page because it's a big page. Like a single page application where everything kind of has to load before you can interact with it. Yeah. So the problem with single page application, we don't know is it loaded or not. So it never says us anything about it. So there is no state of this application. So we have different strategies on this. The default strategy is outer a triangle each step. So each step in the test will be automatically retried on failure. We have explicit waits. So you can wait for anything you want there. You can add manual retries. And if you have very, very bad applications that are not very flaky, you can retry the full scenario. This is what also works. I think that's all strategies, but maybe there are a few more of them. But this is what I use regularly and that works. People want to know more about the restrictions of using code, code.js. Are there any restrictions like fire log loading or working with iframe? What are some of the limitations of the framework? That's quite incorrect questions because I really feel it's like Cypress auditory and you can't do it because concept.js is not driving the browser. It's delegated to playwright. And playwright has no restrictions. So the only restriction you need to understand is that concept.js can't do everything that playwright can. So you need to write some playwright code or puppeteer code or web driver code if you need some very special APIs to be used. But yeah, you can do everything that you can do with playwright, puppeteer, web driver, et cetera, with concept.js. Okay. That makes a lot of sense. That makes a lot of sense. Someone wants to know if you can generate a concept.js test by recording interactions in a browser? I'm very skeptical about this idea. So I tried to use it. Then I switched to this interactive pause mode and I found this much more effective because I'm not just recording my actions. I'm testing it in the real time, which I can't do with recorder. So this is much more flexible and effective for me as a tester. That's really good. There's a lot of questions coming in from Michael. We won't get to all of them. We'll take one more. So when CodeSep does the auto retry feature, is that recording only on success on the first try or, for example, on the third try? The person asking the question has a concern about retries that can mask intermittent bugs. So how does the auto retry record happen? I think it's quite a technical question. I'm not sure I will fit in time to answer it. But I would like to talk about it after my talking in the room. Oh, perfect. In that case, that's a perfect plug for me to remind everybody that they can join you in your speaker's room on spatial.chat. The link to join is on the livestream timeline. And please join Michael for more questions. We won't get time to go into any more. Thank you, Michael, for your talk and for joining me for the Q&A. It was really good. Thank you for inviting me and thank you for the opportunity to be here with you. Thanks for watching, everyone.


Michael Bodnarchuk

The Evolution of Browser Automation

Christian Bromann

#e2e testing