Test Time Execution! Why it Can't be Ignored?

Rate this content

For modern agile teams, release velocity is very critical. Unfortunately, one of the biggest bottlenecks for faster release velocity is testing times. Even after teams implement end-to-end test automation, long testing times significantly delay the pipelines. This delay means developers get feedback of their efforts very late, which can snowball, affecting the shipping velocity. Hence test execution time becomes one of the most critical KPIs for any business that wants to ship features faster than the approaching deadlines. LambdaTest helps modern dev teams release faster executing tests faster on a scalable, reliable test cloud.

10 min
18 Nov, 2021

Video Summary and Transcription

Lambda Test is a cloud-based continuous quality platform that addresses common challenges faced by modern QA and development teams. These challenges include the need to shorten release cycles, the increase in test execution time due to test infrastructure, and the impact of flaky tests on developer feedback and productivity. Lambda Test offers a wide range of features, including live testing for virtual machines and native app testing, automation cloud for running test suites, and integration with various tools. The platform is known for its reliability, scalability, and performance, and can be quickly onboarded and integrated with existing test suites and CI-CD tools.

Available in Español

1. Introduction and Background

Short description:

Hello, I am Jay, one of the founders at Lambda Test. Today, I want to share the common problems faced by modern QA and development teams. We started in 2017 and have a large user base and enterprise customers. Before discussing what we do, let's highlight the problems I've heard from engineering leadership and agile teams.

Hello, everyone. I am Jay, one of the founders at Lambda Test. Thank you for tuning in and also glad to be a part of TestJS event here.

So, just to give you a quick background. So, I started Lambda Test four and a half years ago. I've been building our complete test execution platform on cloud. That's our vision that we're working on. And today, I just want to take a few moments to share about the common problems that I get to hear and often hear from the modern QA and development teams.

And before that, I would like to give you a quick gist of where we come from. We started in 2017, and we recently got a series refunding from Sequoia and Telstra Ventures. And from the time we released our first product, we have close to half a million users who have signed up. And out of our 7,000 active customers, we have 500 enterprises who have been a part of the journey. And before we talk about what we do, I would like to highlight a number of problems. I have been speaking to a lot of engineering leadership and the modern QA and agile teams.

2. Challenges in Test Infrastructure and Execution

Short description:

One common problem faced by teams is the need to shorten release cycles, driven by factors such as innovation, bug fixing, and code refactoring. While end-to-end testing is implemented, the test infrastructure can significantly increase test execution time. In-house infrastructure requires time and effort to build and maintain, and high test execution time becomes a roadblock for teams aiming to increase release velocity. As test suites scale, execution time increases, impacting developer feedback and productivity. Flaky tests at scale further complicate the problem. Lambda Test offers a cloud-based continuous quality platform to address these challenges.

And one of the things that I often hear is that they're aiming at shortening the release cycles. Now, that could be because of N number of reasons. That could be because of, you know, innovation where they want to drive roll out new features and or maybe they just want to fix the bugs so that they can make the experience better for the customers or maybe just or you know, the the kind of just just refactor the code for better stability of their application as such.

Now, we all know that this thing, this automation tools have been existing. And although these these the users and the customers that I've been talking to have implemented end to end testing suit, but there still exists a bigger problem, and that's the test infrastructure on which they run the tests on. And unfortunately, this this this infrastructure on which they run the tests on, if it is not really up to the mark, it really increases the test execution time significantly.

Now, let's talk about the infrastructure as such now talking about the in-house infrastructure, if you if you're in it, firstly, it takes a lot of time to build and maintain this infrastructure. And if it's not really up to the mark and customized as for the needs of the Q8 team, it becomes very tough to scale it. And also this this adds a lot of cost to to to maintain this entire effort. And the second aspect is high test execution time. Now, it's very, very important, especially for the modern QA and dev teams who are who are really focusing on increasing their release velocities and ship code faster. High test execution is one of the biggest roadblocks that we often hear. And again, like I said, it's not the problem about the end-to-end automation suit, but it's about the release velocity they are aiming at. And one of the roadblocks again, is because of the hours and hours that it takes to run your test suit.

Now, if you look into this graph on the right-hand side, as you start your test suit, it typically just takes seconds to run. And as you scale your test suit, as you know that your test suit, the test execution time is proportionally related to the code you ship. So the more code you ship, the more test scripts are written, and your test cases grow. Now, as you go towards the enterprise suit, where you have ten thousands of tests, your execution time takes from minutes to hours to complete. And that is a bigger problem to solve because your developers are actually waiting to get the feedback of the code they've written. Now, you may have noticed that here we're waiting for the test to complete, and again, with the poor infrastructure that we are running the test on, we are impacting the productivity of both the QA and development teams.

Now, third problem is flaky test at scale. Now flaky test could be because of a number of reasons. Now, especially when you're running your test on infrastructure, there could be a lot of problems related to the CPU consumption. There could be high resource consumption in the machine that you're running a test on, which leads to the flaky results. Now, this is a bigger problem altogether to solve. And one of the key aspects that if you see, if you co-relate your high test execution time with your release cycle so you can directly see that the high test execution time is leading to delayed release cycles. So the feedback time that you're taking to get back to the developer is also getting lengthier. Now, we all know this problem exists. Now, this is exactly where Lambda test comes into picture. So we are building a continuous quality platform on cloud where you can test with confidence on your infrastructure and scale your test as well.

3. Overview of Lambda Test Features

Short description:

We offer live testing for virtual machines of 2000 plus browsers and OS combinations, as well as native app testing on simulators and emulators. Our automation cloud allows running test suites and ballot sessions to reduce execution time. With a focus on performance, reliability, and scalability, we provide the fastest selenium grid, headless browser testing, and advanced reporting. Our Real Devices Cloud enables manual native app testing, and we support automation testing and device log analysis. Integration with CICD tools is quick and easy, with support for running tests in parallel across browsers, devices, and OS. We offer 120 integrations with frameworks, codeless automation, CICD, and bug tracking tools, making Lambda Test an ecosystem-friendly solution. Our reliability, scalability, and performance have garnered attention in the JITU crowd space. Onboarding and integrating our test suite and CI-CD tools takes just 30 minutes. Thank you for your time and enjoy the TestJS conference.

So we started with our product, the first product that we launched is live testing, where you can fire up the virtual machines of 2000 plus browsers and operating system combination and test your applications. You can also test your native applications on simulators and emulators.

And then we have the automation cloud where you can run your automation test suits and you can run hundreds of ballot sessions to make sure you're able to reduce your test execution cycle. And then most importantly, the day we started the company, we focused on three parameters here. We focused on performance, we focused on reliability, and scalability. These were the three important aspects because we understand that this is one of the most crucial aspects for the QA and development teams.

Now, we built the fastest selenium grid on the cloud and where you can also perform headless browser testing and also get advanced reporting and analytics. And we just recently launched our Real Devices Cloud where you can perform your manual native app testing. And to maintain those infrastructure in-house, you can also perform automation testing and also look into the device logs as such.

So along with the infrastructure that we built, we also give other functionality like you will be able to run your prepod, you'll be able to run your test on prepod environments by using our enterprise-ready tunnel. You can run your test in parallel and across different browsers, devices, and OS so that you can fasten your test execution time, you can reduce your test execution time. And we also give you support for geolocation. And this is easily integrable with your CICD tools. If you want to fire up your test from CICD into our infrastructure, it just takes 5 minutes for you to integrate and get started.

So apart from that, I also want to highlight on the integrations that we have. Our integration landscape spans around 120 integrations that we have built with frameworks, with codeless automation, with CICD, and bug tracking tools. One of the aspects when we started building the Lambda test, we wanted Lambda test to be a part of the ecosystem, not work as a standalone system. So we tried to build in integration as much as possible to make it more easier for the QA and development teams. In a short span of time, we have garnered good attention in the JITU crowd space. And this is just because of our, I would say, the reliability and scalability and performance that we provide on our infrastructure.

And we've been leading and training on JITU Crowd as momentum leaders, and then you will be able to see the reviews that we get and the ease of use. It just takes around 30 minutes maximum for a user to onboard by integrating this test suite and CI-CD tools and start firing up the tests. And yeah, so this is a quick overview about us. And I really thank you for hearing me out. I really appreciate the time, and I hope you enjoy the TestJS conference. Take care. Bye-bye.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
TestJS Summit 2022TestJS Summit 2022
20 min
Testing Web Applications with Playwright
Top Content
Testing is hard, testing takes time to learn and to write, and time is money. As developers we want to test. We know we should but we don't have time. So how can we get more developers to do testing? We can create better tools.Let me introduce you to Playwright - Reliable end-to-end cross browser testing for modern web apps, by Microsoft and fully open source. Playwright's codegen generates tests for you in JavaScript, TypeScript, Dot Net, Java or Python. Now you really have no excuses. It's time to play your tests wright.
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!

Workshops on related topic

React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Featured Workshop
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.