Debugging a Non Reproducible Crash

Rate this content
Bookmark

POV: Your app has a crash affecting thousands of users, but for the life of you, you can't reproduce it and have no idea what's causing it. Hear the story of an epic struggle to vanquish a non reproducible bug and learn what to do (and what not to do) when facing such a foe.

Alexandre Moureaux
Alexandre Moureaux
20 min
25 Oct, 2021

Comments

Sign in or register to post your comment.

Video Summary and Transcription

The Talk discusses a vicious bug that caused 20,000 crashes in a JS application. The bug was an array index out of bounds exception in the SimplePool class. The team used a debugger to analyze the bug and discovered a race condition caused by an upgrade to React Native SVG. They collaborated with React Native contributors to fix the issue and deployed a patched version. The Talk emphasizes the importance of using a crash reporting tool, monitoring release health, and learning from bugs and source code analysis.

Available in Español

1. The Story of the Vicious Bug

Short description:

Today, I'm going to tell you a story, the story of a bug and our fight against this bug. A bug so vicious and cruel that it actually caused us no less than 20,000 crashes. Our crash rate goes up significantly and our crash reporting tool is reporting an exception every minute. It's a JS application illegal argument exception error while updating a style property in a shadownode of type a React Native component. It happens for every user, every Android device, and all Android devices are affected.

Today, I'm going to tell you a story, the story of a bug and our fight against this bug. A bug so vicious and cruel that it actually caused us no less than 20,000 crashes. But introductions first.

Hi, everyone. I'm Alex. I'm very excited to be here at React Advanced London. I'm a tech lead at BAM. We're based in Paris and we develop mobile apps in Flutter, Native, and of course React Native. And our story begins in October. And we're a team of nine people, and we're very happy and proud to release version 4.3 of our app. Why are we so happy and proud? Well, because actually we were getting ready for our October 11th live event that the app was covering, and we were adding a lot of essential features to the app. Super. We're super happy.

But then, the unexpected hope occurs. Suddenly, our crash rate actually goes up significantly. And actually, our crash reporting tool that we're using, Sentry, is under heavy fire. It's reporting an exception every minute, then, a lot of exceptions every minute. Then, it's basically an exception every second, and it's getting overwhelming. And all of those exceptions are a bit different, but they're all kind of have the same shape. They're like this. Basically, it's a JS application illegal argument exception error while updating a style property in a shadownode of type a react Native component.

And so, well, first thought, is like, well, you know, we did QA this release, we did test it out a lot. Why did we not see this happening? And also, if you search a bit more about this error, this tends to happen if you set a wrong value to a style. For example, if I set padding top to NAN, not a number, this is what would occur. So it kind of sounds like something quite easy to detect. So, like, well, maybe it happens only in certain extreme cases that we have not tested properly before. But it turns out that Sentry is basically reporting that it happens for every user, every Android device so this is an Android issue only, but all Android devices are affected. And also in our app you can actually favor the team, for example, to change a bit the experience of the app. But it doesn't matter whichever team you're actually favoriting doesn't impact this. You're getting the crash.

2. Analyzing the Crash and Reproduction Attempts

Short description:

We have a big crash on startup that affects any device and user. We couldn't reproduce it, so we analyzed the stack trace and found an array index out of bound exception in the SimplePool class. Rolling back the release was not an option, as it had high user value. With a 10% crash rate, we attempted to reproduce the issue on multiple devices but didn't get any crashes.

All right. Well, we have a big crash, we have a big fire to put out, so let's start by trying to reproduce the crash, right? So fortunately we configured for Sentry or crash reporting tool to tell us what the user was doing before triggering the crash. So here we see that the user is actually opening the app, starting the first screen of the app, which is called Home. And boom, actually it crashes instantly.

All right, so basically you're telling me that it affects any device, it crashes on startup, it affects any user and we can't reproduce it? We've never seen it before, how is that even possible?

All right, well I guess step two, if you can't really reproduce, is analyzing the stack trace. So let's take a look. Okay, I did say that we have several different errors. I guess, let's take a look at the first one. So this one is an array index out of bound exception. It's a Java error. And it's happening in the class called SimplePool and it's a class from Android v4 support library. And it's happening in SimplePool.release, like 116 of pools.java. And well, to be honest, at this point I'm like I don't even know what SimplePool is. And I don't even know why I'm even in the Android source code. Like there's a big fire to put out and it feels like it's going to take a lot of time to actually figure out what's going on because I don't really understand this. So I guess let's find an easier solution to put out the fire.

So one idea would be, well, could we just roll back our release? Well, if you're a mobile app developer you know that we can't actually really roll back the release? We actually have to deploy a new release with the old code. It's kind of annoying and it means that certain users, you know, the users will get an update of the app just reverting everything. And at this point in time, we actually know that our crash rate is about 10%. So it seems that basically a user opening the app has one out of 10 chances to crash the app. But it seems that whenever they try to restore it, it works. And also, this release has actually great value for users. It turned out to be one of the highest-rated releases despite this outstanding crash. So we thought, well, no, let's not roll back. It's not the end of the world. It's outrageously big to have 10% crash rate, but let's try to fix it in another way.

All right, we know that the crash rate is 10%, so I'm like, okay, I can devise a battle plan. I'm just going to take six Android devices, I'm going to trigger with a script 10 app launch per device, so statistically I should get like five to 10 crashes, right? And at least that would be some kind of reproduction. I would be able to finally see the issue, and if I get a fix, then I would be able to test it out. The result was that I didn't get any crashes.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Don't Solve Problems, Eliminate Them
React Advanced Conference 2021React Advanced Conference 2021
39 min
Don't Solve Problems, Eliminate Them
Top Content
Humans are natural problem solvers and we're good enough at it that we've survived over the centuries and become the dominant species of the planet. Because we're so good at it, we sometimes become problem seekers too–looking for problems we can solve. Those who most successfully accomplish their goals are the problem eliminators. Let's talk about the distinction between solving and eliminating problems with examples from inside and outside the coding world.
Modern Web Debugging
JSNation 2023JSNation 2023
29 min
Modern Web Debugging
Top Content
Few developers enjoy debugging, and debugging can be complex for modern web apps because of the multiple frameworks, languages, and libraries used. But, developer tools have come a long way in making the process easier. In this talk, Jecelyn will dig into the modern state of debugging, improvements in DevTools, and how you can use them to reliably debug your apps.
Jotai Atoms Are Just Functions
React Day Berlin 2022React Day Berlin 2022
22 min
Jotai Atoms Are Just Functions
Top Content
Jotai is a state management library. We have been developing it primarily for React, but it's conceptually not tied to React. It this talk, we will see how Jotai atoms work and learn about the mental model we should have. Atoms are framework-agnostic abstraction to represent states, and they are basically just functions. Understanding the atom abstraction will help designing and implementing states in your applications with Jotai
A Framework for Managing Technical Debt
TechLead Conference 2023TechLead Conference 2023
35 min
A Framework for Managing Technical Debt
Top Content
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt.In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.

Debugging JS
React Summit 2023React Summit 2023
24 min
Debugging JS
Top Content
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.
Fighting Technical Debt With Continuous Refactoring
React Day Berlin 2022React Day Berlin 2022
29 min
Fighting Technical Debt With Continuous Refactoring
Top Content
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt. In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.

Workshops on related topic

React Performance Debugging Masterclass
React Summit 2023React Summit 2023
170 min
React Performance Debugging Masterclass
Top Content
Featured WorkshopFree
Ivan Akulov
Ivan Akulov
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
React, TypeScript, and TDD
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
Paul Everitt
Paul Everitt
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
Web3 Workshop - Building Your First Dapp
React Advanced Conference 2021React Advanced Conference 2021
145 min
Web3 Workshop - Building Your First Dapp
Top Content
Featured WorkshopFree
Nader Dabit
Nader Dabit
In this workshop, you'll learn how to build your first full stack dapp on the Ethereum blockchain, reading and writing data to the network, and connecting a front end application to the contract you've deployed. By the end of the workshop, you'll understand how to set up a full stack development environment, run a local node, and interact with any smart contract using React, HardHat, and Ethers.js.
Remix Fundamentals
React Summit 2022React Summit 2022
136 min
Remix Fundamentals
Top Content
Featured WorkshopFree
Kent C. Dodds
Kent C. Dodds
Building modern web applications is riddled with complexity And that's only if you bother to deal with the problems
Tired of wiring up onSubmit to backend APIs and making sure your client-side cache stays up-to-date? Wouldn't it be cool to be able to use the global nature of CSS to your benefit, rather than find tools or conventions to avoid or work around it? And how would you like nested layouts with intelligent and performance optimized data management that just works™?
Remix solves some of these problems, and completely eliminates the rest. You don't even have to think about server cache management or global CSS namespace clashes. It's not that Remix has APIs to avoid these problems, they simply don't exist when you're using Remix. Oh, and you don't need that huge complex graphql client when you're using Remix. They've got you covered. Ready to build faster apps faster?
At the end of this workshop, you'll know how to:- Create Remix Routes- Style Remix applications- Load data in Remix loaders- Mutate data with forms and actions
Vue3: Modern Frontend App Development
Vue.js London Live 2021Vue.js London Live 2021
169 min
Vue3: Modern Frontend App Development
Top Content
Featured WorkshopFree
Mikhail Kuznetcov
Mikhail Kuznetcov
The Vue3 has been released in mid-2020. Besides many improvements and optimizations, the main feature of Vue3 brings is the Composition API – a new way to write and reuse reactive code. Let's learn more about how to use Composition API efficiently.

Besides core Vue3 features we'll explain examples of how to use popular libraries with Vue3.

Table of contents:
- Introduction to Vue3
- Composition API
- Core libraries
- Vue3 ecosystem

Prerequisites:
IDE of choice (Inellij or VSC) installed
Nodejs + NPM
Developing Dynamic Blogs with SvelteKit & Storyblok: A Hands-on Workshop
JSNation 2023JSNation 2023
174 min
Developing Dynamic Blogs with SvelteKit & Storyblok: A Hands-on Workshop
Top Content
Featured WorkshopFree
Alba Silvente Fuentes
Roberto Butti
2 authors
This SvelteKit workshop explores the integration of 3rd party services, such as Storyblok, in a SvelteKit project. Participants will learn how to create a SvelteKit project, leverage Svelte components, and connect to external APIs. The workshop covers important concepts including SSR, CSR, static site generation, and deploying the application using adapters. By the end of the workshop, attendees will have a solid understanding of building SvelteKit applications with API integrations and be prepared for deployment.