Fantastic Bugs and Where to Find Them

Rate this content
Bookmark

Every bug is different: Some are lurking around for months, others appear suddenly after the upgrade of a dependency. Some are introduced us, others other teams or systems. Some are painfully obvious and affect all users, others only occur in edge (cases). And the ways of finding, and eventually, preventing them, are just as diverse: be it snapshot, unit, integration, end to end tests or automated visual tests, every kind comes with its challenges and opportunities. Testing UIs is hard, but in the end, only test automation can give us the confidence we need to move fast and refactor our code relentlessly. In this talk we are going to look at what kinds of bugs there are, which tests are most effective for catching which, and how we can implement them using modern front end technologies.

Iris Schaffer
Iris Schaffer
34 min
17 Jun, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

TypeScript is a JavaScript-based language used for mobile apps. Testing rules include ensuring tests fail when something goes wrong and only fail when necessary. Integration testing can be done using tools like Cypress and Netflix PolyJS. Visual regression testing compares screenshots of changes and helps prevent visual bugs. Starting with end-to-end tests and writing tests for bugs encountered are recommended for beginners.

Available in Español

1. Introduction to TypeScript and Front-end Testing

Short description:

TypeScript is a JavaScript-based language used for mobile apps. I will share my experience and discuss front-end testing, emphasizing the importance of investing in lower levels of the bug-finding pyramid.

What is TypeScript? It's a JavaScript-based JavaScript language that's designed for a mobile app. That means, it's used by both Android and iOS, for both physical and real-world applications. You can create templates for your own applications, and provide your own code base for your own applications.

What have you learned so far? Most of my career, I've spent working on short-lived campaigns at creative agencies. And although I learned a lot there, there were certain things that I didn't really get to learn there. For example, maintainability of software, scalability. So when I joined a product development team about two and a half years ago, I changed my attitude towards software quite a lot. With hundreds of thousands of monthly active users, all of a sudden, the quality of your delivery becomes a lot more important than the speed at which you deliver. So what I've learned in the last couple of years is what I want to talk to you about today.

My name is Iris. I'm originally from Austria, but I've moved all over Europe over the last couple of years, and currently I'm based in Stockholm, Sweden. This is my Twitter handle, in case you want to follow me or watch me rant about things every now and then. Although I have arguably the worst taste in music ever, I somehow managed to land a job at a music company. In case you're not familiar with Spotify, we're one of the world's biggest audio streaming services with about 60 million tracks, including 1.5 million podcast titles. And with almost 300 million users, you can imagine that any edge case that you can or can't think of will happen for quite a lot of people. So we've had our share of extraordinary bugs and bug fixes, like, for example, in 2009, when John fixed Eric's matrix factorization job by installing curtains in the server room so they wouldn't overheat. I personally don't write matrix factorization jobs, but instead, my team owns the Premium Family, Premium for Students, and Premium Duo products on the main Spotify.com website. And those are the floors that I will show you today. Or the bugs that I will show you come from these floors. But speaking of bugs, how do we actually find them? This is what I call the bug-finding pyramid. And at the lowest levels, you have automated testing and type systems, as well as manual and exploratory testing. All of that happens before you deploy to production. After that, we have monitoring systems, and at the end, we also have our users complaining, for example, on Twitter or to our customer service. In an ideal world, all of our bugs would be found in the first two layers, and as few bugs as possible will be actually exposed to the user. And how to we ensure that we avoid exposing bugs to user? By investing in those lower levels of the pyramid. And that is what I want to talk about today. Specifically, I want to talk about front-end testing, because for some reason, 36% of front-enders still didn't test their code in 2019. And to make sure that we're off for a good start, let me first introduce a couple of ground rules that I like to think about or keep in mind when I write my tests. A good test has a couple of characteristics. It should be readable, writable, isolated, fast.

2. Testing Rules and Bug: Parental Control Lock

Short description:

The most important rules for testing are: 1) Will the test fail when something goes wrong? 2) Will the test only fail when something goes wrong? Treat tests like normal code and test the public interface. The first bug discussed is the parental control lock not updating. A unit test was written for the toggle filter function, which is a pure function that always produces the same output given the same inputs. Testing the function involves setting up the data, defining the expected outcome, calling the function, and comparing the output to the expected result. The test should fail when something goes wrong and only fail when necessary.

But the most important rules for me are the following. Number one, will the test fail when something goes wrong? If it doesn't fail if something goes wrong, what's the point of even having the test? And there's another important question that we can derive from that, which is, which parts can go wrong in which ways? And we can use that to decide which test cases to cover, what scenarios to actually test.

The second question is, will the test only fail when something goes wrong? And this is almost equally as important, because tests are codes too, and if we have to constantly change our tests so they don't fail, first of all, we're wasting a lot of time, but also we lose a lot of trust and confidence in our tests. So treat your tests like you do your normal code, make it as maintainable as possible. And the way we do that is by testing the public interface. So if we only test the public interface, we can ensure that we only have to update the tests if something actually has to be updated in other parts of the application as well.

And with that, let's look at the first bug. Parental control lock not updating. Within our app, we have this explicit music filter where parents can allow or disallow explicit music for other people on the plan. Unfortunately, while developing this feature, this is what I was seeing. It loads and it jumps back. So what was wrong? In our application somewhere we have this data structure and we have this allow explicit music, true flag or false flag. Unfortunately, after communicating with the back end, this flag was not updated accordingly. Luckily, this never made it into production because I wrote a unit test for it. And unit tests are tests that cover or test the functionality of one unit. In this case, the toggle filter function.

The toggle filter function just takes an old state, an ID, and returns a new state. And this toggle filter function is a pure function because it only has two inputs and one output. No matter how often you call it, given the same inputs, it will always produce the same output. And what I love so much about these pure functions is that, first of all, they're extremely easy to reason about. They're very simple to understand. But also, they're very simple to test, and how do you test them? Well, given the same input, you have the same output, so you can give it some input and assert that the output is something like here. We first set up the data that we want to pass in, so a members array with one user to remain the same and one to be updated. We then set up what our expected outcome should be. And mind you, the allowExplicitMusic flag is updated from false to true. We then actually call the function, and lastly we compare the output of the state after data.member to the expected members that we set up before. So, time to ask our two questions. Will the test fail when something goes wrong? Yep! Will the test only fail when something goes wrong? Unless your business logic changes, yes, and in that case we obviously want to update the test as well. Some of you might now be thinking, I thought this was about testing UIs. Let's look at another bug forever, a loader.

QnA

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Network Requests with Cypress
TestJS Summit 2021TestJS Summit 2021
33 min
Network Requests with Cypress
Top Content
Whether you're testing your UI or API, Cypress gives you all the tools needed to work with and manage network requests. This intermediate-level task demonstrates how to use the cy.request and cy.intercept commands to execute, spy on, and stub network requests while testing your application in the browser. Learn how the commands work as well as use cases for each, including best practices for testing and mocking your network requests.
Testing Pyramid Makes Little Sense, What We Can Use Instead
TestJS Summit 2021TestJS Summit 2021
38 min
Testing Pyramid Makes Little Sense, What We Can Use Instead
Top Content
Featured Video
Gleb Bahmutov
Roman Sandler
2 authors
The testing pyramid - the canonical shape of tests that defined what types of tests we need to write to make sure the app works - is ... obsolete. In this presentation, Roman Sandler and Gleb Bahmutov argue what the testing shape works better for today's web applications.
Full-Circle Testing With Cypress
TestJS Summit 2022TestJS Summit 2022
27 min
Full-Circle Testing With Cypress
Top Content
Cypress has taken the world by storm by brining an easy to use tool for end to end testing. It’s capabilities have proven to be be useful for creating stable tests for frontend applications. But end to end testing is just a small part of testing efforts. What about your API? What about your components? Well, in my talk I would like to show you how we can start with end-to-end tests, go deeper with component testing and then move up to testing our API, circ
Test Effective Development
TestJS Summit 2021TestJS Summit 2021
31 min
Test Effective Development
Top Content
Developers want to sleep tight knowing they didn't break production. Companies want to be efficient in order to meet their customer needs faster and to gain competitive advantage sooner. We ALL want to be cost effective... or shall I say... TEST EFFECTIVE!But how do we do that?Are the "unit" and "integration" terminology serves us right?Or is it time for a change? When should we use either strategy to maximize our "test effectiveness"?In this talk I'll show you a brand new way to think about cost effective testing with new strategies and new testing terms!It’s time to go DEEPER!
Playwright Test Runner
TestJS Summit 2021TestJS Summit 2021
25 min
Playwright Test Runner
Top Content
Everyone Can Easily Write Tests
TestJS Summit 2023TestJS Summit 2023
21 min
Everyone Can Easily Write Tests
Let’s take a look at how Playwright can help you get your end to end tests written with tools like Codegen that generate tests on user interaction. Let’s explore UI mode for a better developer experience and then go over some tips to make sure you don’t have flakey tests. Then let’s talk about how to get your tests up and running on CI, debugging on CI and scaling using shards.

Workshops on related topic

Designing Effective Tests With React Testing Library
React Summit 2023React Summit 2023
151 min
Designing Effective Tests With React Testing Library
Top Content
Featured Workshop
Josh Justice
Josh Justice
React Testing Library is a great framework for React component tests because there are a lot of questions it answers for you, so you don’t need to worry about those questions. But that doesn’t mean testing is easy. There are still a lot of questions you have to figure out for yourself: How many component tests should you write vs end-to-end tests or lower-level unit tests? How can you test a certain line of code that is tricky to test? And what in the world are you supposed to do about that persistent act() warning?
In this three-hour workshop we’ll introduce React Testing Library along with a mental model for how to think about designing your component tests. This mental model will help you see how to test each bit of logic, whether or not to mock dependencies, and will help improve the design of your components. You’ll walk away with the tools, techniques, and principles you need to implement low-cost, high-value component tests.
Table of contents- The different kinds of React application tests, and where component tests fit in- A mental model for thinking about the inputs and outputs of the components you test- Options for selecting DOM elements to verify and interact with them- The value of mocks and why they shouldn’t be avoided- The challenges with asynchrony in RTL tests and how to handle them
Prerequisites- Familiarity with building applications with React- Basic experience writing automated tests with Jest or another unit testing framework- You do not need any experience with React Testing Library- Machine setup: Node LTS, Yarn
How to Start With Cypress
TestJS Summit 2022TestJS Summit 2022
146 min
How to Start With Cypress
Featured WorkshopFree
Filip Hric
Filip Hric
The web has evolved. Finally, testing has also. Cypress is a modern testing tool that answers the testing needs of modern web applications. It has been gaining a lot of traction in the last couple of years, gaining worldwide popularity. If you have been waiting to learn Cypress, wait no more! Filip Hric will guide you through the first steps on how to start using Cypress and set up a project on your own. The good news is, learning Cypress is incredibly easy. You'll write your first test in no time, and then you'll discover how to write a full end-to-end test for a modern web application. You'll learn the core concepts like retry-ability. Discover how to work and interact with your application and learn how to combine API and UI tests. Throughout this whole workshop, we will write code and do practical exercises. You will leave with a hands-on experience that you can translate to your own project.
Detox 101: How to write stable end-to-end tests for your React Native application
React Summit 2022React Summit 2022
117 min
Detox 101: How to write stable end-to-end tests for your React Native application
Top Content
WorkshopFree
Yevheniia Hlovatska
Yevheniia Hlovatska
Compared to unit testing, end-to-end testing aims to interact with your application just like a real user. And as we all know it can be pretty challenging. Especially when we talk about Mobile applications.
Tests rely on many conditions and are considered to be slow and flaky. On the other hand - end-to-end tests can give the greatest confidence that your app is working. And if done right - can become an amazing tool for boosting developer velocity.
Detox is a gray-box end-to-end testing framework for mobile apps. Developed by Wix to solve the problem of slowness and flakiness and used by React Native itself as its E2E testing tool.
Join me on this workshop to learn how to make your mobile end-to-end tests with Detox rock.
Prerequisites- iOS/Android: MacOS Catalina or newer- Android only: Linux- Install before the workshop
API Testing with Postman Workshop
TestJS Summit 2023TestJS Summit 2023
48 min
API Testing with Postman Workshop
Top Content
WorkshopFree
Pooja Mistry
Pooja Mistry
In the ever-evolving landscape of software development, ensuring the reliability and functionality of APIs has become paramount. "API Testing with Postman" is a comprehensive workshop designed to equip participants with the knowledge and skills needed to excel in API testing using Postman, a powerful tool widely adopted by professionals in the field. This workshop delves into the fundamentals of API testing, progresses to advanced testing techniques, and explores automation, performance testing, and multi-protocol support, providing attendees with a holistic understanding of API testing with Postman.
1. Welcome to Postman- Explaining the Postman User Interface (UI)2. Workspace and Collections Collaboration- Understanding Workspaces and their role in collaboration- Exploring the concept of Collections for organizing and executing API requests3. Introduction to API Testing- Covering the basics of API testing and its significance4. Variable Management- Managing environment, global, and collection variables- Utilizing scripting snippets for dynamic data5. Building Testing Workflows- Creating effective testing workflows for comprehensive testing- Utilizing the Collection Runner for test execution- Introduction to Postbot for automated testing6. Advanced Testing- Contract Testing for ensuring API contracts- Using Mock Servers for effective testing- Maximizing productivity with Collection/Workspace templates- Integration Testing and Regression Testing strategies7. Automation with Postman- Leveraging the Postman CLI for automation- Scheduled Runs for regular testing- Integrating Postman into CI/CD pipelines8. Performance Testing- Demonstrating performance testing capabilities (showing the desktop client)- Synchronizing tests with VS Code for streamlined development9. Exploring Advanced Features - Working with Multiple Protocols: GraphQL, gRPC, and more
Join us for this workshop to unlock the full potential of Postman for API testing, streamline your testing processes, and enhance the quality and reliability of your software. Whether you're a beginner or an experienced tester, this workshop will equip you with the skills needed to excel in API testing with Postman.
Testing Web Applications Using Cypress
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
Gleb Bahmutov
Gleb Bahmutov
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Best Practices for Writing and Debugging Cypress Tests
TestJS Summit 2023TestJS Summit 2023
148 min
Best Practices for Writing and Debugging Cypress Tests
Workshop
Filip Hric
Filip Hric
You probably know the story. You’ve created a couple of tests, and since you are using Cypress, you’ve done this pretty quickly. Seems like nothing is stopping you, but then – failed test. It wasn’t the app, wasn’t an error, the test was… flaky? Well yes. Test design is important no matter what tool you will use, Cypress included. The good news is that Cypress has a couple of tools behind its belt that can help you out. Join me on my workshop, where I’ll guide you away from the valley of anti-patterns into the fields of evergreen, stable tests. We’ll talk about common mistakes when writing your test as well as debug and unveil underlying problems. All with the goal of avoiding flakiness, and designing stable test.