How to Choose an Automation Tool

Rate this content

What are the aspects you should think when choosing an automation framework. Some main differences on available frameworks.

Explain the main differences between the most known frameworks for automation: Selenium, Cypress, Playwright and Robot Framework. Positives and negatives about the frameworks. Depending on the needs of the project your work what is the best framework to use. Small talk about how frameworks integrate with some test management tools.

21 min
03 Nov, 2022

AI Generated Video Summary

I am Lia, a software engineer at Nocare, and I'm going to show you a process to choose an automation tool. I'll compare three frameworks and discuss other integration options. Selenium is a well-established framework with a large community, but it requires third-party integrations for certain types of tests. Cypress is an all-in-one tool that supports unit testing, but it only works with JavaScript and lacks parallel testing capabilities. Playwright is a newer framework that is more complete and supports parallel testing, but it has a smaller community and less mature documentation. If you want a framework that does it all, choose Playwright. Documentation is essential to remember your initial thoughts, show others what is being tested, and establish good practice. Once you have your test list, chosen framework, and prioritizations, schedule a meeting with management. Present your work confidently, as it is the best approach for the team and product.

1. Choosing an Automation Tool

Short description:

I am Lia, a software engineer at Nocare, and I'm going to show you a process to choose an automation tool. I'll compare three frameworks and discuss other integration options. Analyzing your application is the first step, understanding its weak points and creating a test list. Talk to the development team to avoid duplication of work. If they don't have unit testing, consider implementing automation tests throughout the process. Decide if you want to code and choose a comfortable language. If not, there are inclusive options like testing and perfecto. If you prefer coding, I'll compare Selenium, Cypress, and Playwrights.

I hope everyone is feeling good and before anything, I want to thank you all so much for being here today. I am Lia, and I'm going to show you a process that you can use to choose an automation tool. I am going to do an overall comparison of three frameworks that I think are the best at this moment, and I'm going to talk a little bit about other tools that you can integrate in your automation process.

So a quick insight into who I am. I am a software engineer at Nocare. My main focus is on test automation. And let's dive into this. So this is a three-step process, like all the good ones, of course. But each step will need attention and work. So let's move on and dive into the first phase, that is, analyze.

So analyzing means answering some questions. And the first one is not really a question. It's understand your application. So look at it and be really critical about it. Try to understand what are the weak points, the features that you can break more easily, and try to think of edge cases in the application. So just have an overall picture of the application that you are going to test and make a list. Make a list of the tests that you want or need to do. And at this moment, it will be a good idea to talk with the development team and try to understand what are the coverage of unit testing that they have, if they have, and knowing their coverage you can avoid duplication of work by your side.

And if they don't have unit testing, it's a really good opportunity to meet with them and with the management team and think of a way to do a coverage since the develop until the production. And this means that have automation tests in all parts of the process. So the development team will have unit testing and you can have end-to-end security, accessibility, whatever types you think are more suitable for you. So having this, you will need to ask yourself if you want to call or not, and ask your team if they feel comfortable doing this. And in which language do they feel they will be more comfortable working. So having this, you will end up with a list more or less like this in what you have, the development team has unit testing or not, what type of tests that you are going to do, and you will have a language that you feel more comfortable working with, if you choose to code.

So if you do not want to code, that is not a problem because here we are inclusive and we have options for everyone. And if you don't want to code, I will give you two options that I know, and be aware that I believe there are a lot more options for you, so if this option that I am going to give you does not suit your or accommodates your needs, just search for it. And we are kind of entering here in the next part that is search, but I am going to previously tell you about testing and perfecto, and this two are similar in the things that they can do, but just in a quick explanation, this two record your manual testing, and for each action that you do in the browser they make a block. And when you finish you will have your test case and your automation that is a group of blocks and just like in coding you can create a group of blocks that you can reuse between tests, you can create folders to run in each deployment, you have a full group of options that you can have just like in the coding. So if you don't want to code, just take a look at this tool and dive into this because I'm sure that you will have options for you as well. So if you do want to code, I'm going to do some comparison between Selenium, Cypress and Playwrights.

2. Comparing Selenium, Cypress, and Playwright

Short description:

Selenium is a well-established framework with a large community, but it requires third-party integrations for certain types of tests. It does not support parallel testing or iframes. Cypress is an all-in-one tool that supports unit testing, but it only works with JavaScript and lacks parallel testing capabilities. Playwright is a newer framework that is more complete and supports parallel testing, but it has a smaller community and less mature documentation.

And this is more or less what you are going to do in the search part with the frameworks that you choose to compare. And for Selenium, starting with Selenium. Selenium is a framework that is here for a long time, and this means that Selenium has a large community supporting it. And you will not be alone working because you will have a Stack Overflow to help you with a large group of people that can help you.

And overall, it is a really good framework, but it will need third parties to work with certain types of tests. So Selenium automates the browser. So it simulates a real browser interaction, meaning that if you need to do API testing, you will need to integrate with a third party to have the API testing. And since Selenium does not support by itself parallel testing, you will need to integrate with Selenium Grid and a cross-browser tool to make it work. And for me, the most downsides for Selenium is it does not have an in-built report tool, so you will need to have a third party like X-Ray or Test Rail, that I'm going to talk a little bit in the end, to support this. And it does not support iframes or popups. So, if your application depends on third party iframes, Selenium is going to be a really not go-to framework. But, if this is something that you can live without, Selenium is really really good to work with a large set of tests, but, like I told you, it will need third parties integration for some type of test, and this will make it more difficult to work in the beginning. I'm not saying that it's impossible because nothing is impossible, but, you will need to have some experience and some expertise to understand what are the workarounds that you need to do here, okay.

So, it is a good tool. If you want something easier to start with, let's move on to the other tool and understand what are the differences here. So, with Cypress, Cypress is a all-in-one tool, meaning that you can have the development team doing unit testing and you doing the other types of tests in your site, and have the whole team working with the same framework, and this is cool because you can take it out with each other and try to understand and review code from each other, and it is very complete, it has an in-built report, you don't need to integrate with any third parties here, but the downside is that you only work with JavaScript here and it does not support parallel testing and multitap testing, meaning that if you need to have certain types of configurations that need to be run at the same time, for example, if you want to have a browser with a device and various configurations about this, you will need to run them separately. So, it will be a very huge time running the test case. If you need to have various set of configurations here, various environments running at the same time, you need to consider if Cypress is going to be a good tool for you. But for starters, and for a small medium group of tests, Cypress is really good. It's a very simple tool to work with if you work with javascript, of course. It's a very simple tool to work with and it has a lot of documentation, the community is growing so you will have help for with Stack Overflow for any type of data that you need. But it has its downsides, just like all frameworks that you will encounter here.

Moving on to Playwright, Playwright is a fair new framework. The first time that I heard about it it was here in 2020, if I'm not mistaken, and it is, I think, the most complete of the three. You can have all types of tests running in Playwright, it supports parallel testing so you can have various sets of configurations and environments running at the same time. It has context isolation so you can have tests for new tabs and clicks open in a new tab, it can capture videos and screenshots so you can have visual testing in comparison. But it is a new framework so be aware there are some types of features that are not completely developed so if you look for a framework that is settled, maybe playwright is not the go-to. And since it is a new framework, the community is not very large compared to the other two, so you will need to have some experience with coding and expertise, programming and searching for things because you are going to be a little bit alone in this. You will have the community of playwrights taking doubts, but Stack Overflow is not really filled with questions and answers for how to move with playwrights. So, be aware that you will need to take a lot of time trying to move with playwrights and documentation is a bit confused because since it does a lot of things it's kind of messy to work around with.

3. Choosing Frameworks, Reporting, and Planning

Short description:

If you want a framework that does it all, choose Playwright. Choose a framework that fulfills your needs. Each framework has options for accessibility testing. Xray and TestRail are options for reporting, with Xray being more integrated and visible for the team. Prioritize your tests and document your work.

But if you want to take the time to work with the framework that does it all for itself, I think it will be very good to work with playwrights. So just in an overall of this tree I think that you need to choose what you feel comfortable with. So have a look at the tree, just play around with each one of them, if you have that time, and choose one that you feel comfortable working with. Because this is something that you're going to work with, like, six hours of your day.

So don't choose one because your friend told you that this is the best or I tell you this is the best. Just choose one that you feel comfortable and that fulfill most of the needs that you have for testing an application.

So, moving on to this, you will end up with a board, or you should end up with a board, more or less like this, where you have the type of test that you need at the right, at the left, I'm sorry, and check and cross or ups and downs of the framework that you are comparing. And just in a side note, each of them has options to work with accessibility tests. The one that I know is AX, that you can integrate with the frameworks, and you can have ApliTools to each one of these three to run visual testing. Have in mind that Cypress and Playwright do this by itself with screenshots and videos, but if you want to have a third party working with this, you can work with ApliTools.

So after this, you will choose, or not, a reporter, and this is going to be a quick comparison because I don't want to go very deep into this, because I think that choosing a reporter is not a decision of the quality team only, it should be a decision of all the product team because everyone should be integrated in the process. But just in a quick comparison, Xray has integration with Jita, TestRail also has integration with Jita, but it's not so obvious that it's there. And Xray, you can create test plans, test execution to integrate with the issues that you have in Jita, this having in mind that you work with Jita. If you don't work with Jita, this is not a very good option for you. But in my opinion, Xray is a more integration tool with Jita and to be visible for everyone in the team what is been testing and what is being doing for the quality team. And TestRail is more on the downside. So if you want a tool that's the quality team will work with separately from every other team I will go with TestRail. And if you want a tool that is more integrated with the process and the product team and the development team I will go with X-Ray.

But moving on to the plan part. That is the important... I will not say the important, but the part that you have to take a little more time to understand how you are going to do this. So the key word here is prioritize. And what I mean by prioritize is choose what tests are you going to do first. And this could be the most painful end to end test that you do manually. And it can be the simpler test that you do just to start with and try to understand if you can work with this framework. So just unite a team and if you are by yourself don't feel sad, just do this. And think about it, really sit down and take a look to what you are going to do and what is going to be the first thing that you are going to do. And parallel to this, do documentation. This is very important because the knowledge cannot be only in your mind, you need to share it with the world, with the team, with the company so that people know what you are testing to be reassured that the application is stable to go to production. And just for you to have some place where you can rely to understand what you did in the past.

4. Importance of Documentation and Confidence

Short description:

Documentation is essential to remember your initial thoughts, show others what is being tested, and establish good practice. Despite some reluctance, taking the time to document will be appreciated in the end. Once you have your test list, chosen framework, and prioritizations, schedule a meeting with management. Present your work confidently, as it is the best approach for the team and product. Share your confidence with management, and they will support your efforts. Enjoy the process and don't hesitate to ask questions.

Because right now if you are starting, it is very easy to understand what you are doing, but in 6 months, or 1 year from now, I really doubt that you remember what you are thinking when you start the project. So, if you have documentation, it will help you to have place to go, to remember what you are thinking in the beginning, to show to other people what are being tested in the application, and it is good practice.

I know that some development person people really don't wanna take the time to do documentation, but just do it. It is good practice and you are going to thank me in the end, believe me.

And after you have this, after you have the list of test types that you wanna choose and that you choose to do, I'm sorry, the framework that you choose to work with, the author that you think is best for the team to use and the prioritizations of the tests that you are going to implement, now it's time to schedule a meeting with the management team or with the manage. So have this really structure in a way that when you present this, there is no option for them to say no.

So you did all the work, you analyze the product use search for the frameworks, you plan the work that you are going to do in the next month, two months, in the next time. So present this with the certainty that the team and you want to do this and are very sure that this is the best thing for the team and for the product and be confident about it. This is important to tell you because I think that sometimes people just think that this is my opinion but be really confident that this is the best thing to do and you are going to do this. And share that confidence with the management team and be reassured that they will not cut your legs, they will be really really happy to let you do this.

So, have lots of fun working with this, programming or not programming and if you have any doubts just go for it. Thank you so much and see you next time.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
23 min
Scaling Up with Remix and Micro Frontends
Do you have a large product built by many teams? Are you struggling to release often? Did your frontend turn into a massive unmaintainable monolith? If, like me, you’ve answered yes to any of those questions, this talk is for you! I’ll show you exactly how you can build a micro frontend architecture with Remix to solve those challenges.
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
JSNation Live 2021JSNation Live 2021
29 min
Making JavaScript on WebAssembly Fast
JavaScript in the browser runs many times faster than it did two decades ago. And that happened because the browser vendors spent that time working on intensive performance optimizations in their JavaScript engines.Because of this optimization work, JavaScript is now running in many places besides the browser. But there are still some environments where the JS engines can’t apply those optimizations in the right way to make things fast.We’re working to solve this, beginning a whole new wave of JavaScript optimization work. We’re improving JavaScript performance for entirely different environments, where different rules apply. And this is possible because of WebAssembly. In this talk, I'll explain how this all works and what's coming next.
React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.

Workshops on related topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.