Deconstructing Distributed Tracing

Rate this content
Bookmark
Slides

Distributed tracing is a powerful technique that allows you to track the flow and timing of requests as they navigate through a system. By linking operations and requests between multiple services, distributed tracing provides valuable insights into app performance and helps identify bottlenecks. In this talk Lazar will explain the concept of Distributed Tracing by walking you through how monitoring tools build tracing solutions.

8 min
12 Dec, 2023

AI Generated Video Summary

Distributed tracing is a powerful technique for tracking requests and operations in a system, especially in full stack and microservice applications. The reinvention of distributed tracing introduces the concept of a trace and spans to capture debugging data. Enhancements include tags and a status field for better analysis, and the distribution of traces using a trace context for continued tracing.

1. Introduction to Distributed Tracing

Short description:

Distributed tracing is a powerful technique that helps track the flow and timing of requests and operations in a system. It is especially useful for full stack and microservice applications, allowing for better understanding of system performance and identification of bottlenecks. The technique has been around since the early 2000s but gained popularity in the 2010s. As libraries and frameworks evolved, so did debugging tools, from logs in Apache Server to handling multiple requests in a single process with separate threads. With advanced concurrency, frameworks like Node.js allow requests to start and finish in different threads.

♪ ♪ Reconstructing distributed tracing. Hello, everyone. My name is Laza Nikolov, and I am a developer advocate at Sentry. Today on my talk, we're going to talk about distributed tracing. First explain what it is. Then we're going to get into a little history on the debugging tools to find out why distributed tracing existed in the first place. And then in order to understand it better, we're going to rebuild distributed tracing from scratch or at least just the concept of it.

All right, so let's dive in. Distributed tracing is a powerful technique that allows you to track the flow and timing of requests and operations as they flow through your system. This is especially useful for full stack and for microservice applications. Distributed tracing helps you understand the performance of the system and also identify any bottlenecks. It's especially useful for debugging complex and weird bugs like race condition bugs that require a lot more than just a console lock and a stack trace. It's not new by any means. There are white papers mentioning tracing since the early 2000s, but it got popularized during the 2010s. So to understand why it exists, we need to go back in time.

As our libraries and frameworks evolved, so did our debugging tools. For example, back in the early days of Apache Server, logs were one of the few methods for debugging. As requests arrived, Apache forked a child process and handled the requests. If you wanted to debug what happened during that specific request, you could just pull the process's logs and you'll see the whole operation flow. And that worked. We were happy. Then we got basic concurrency. Think of IIS in ASP.NET. Instead of forking a process for every request, we started handling multiple requests in a single process, but in a separate thread. Logs are still a good debugging method, but to isolate the request's logs, we need to prefix them with the thread name and then filter the log messages based on it. Not a big deal, but we made it work. Then we got advanced concurrency. Our frameworks evolved into async, multithreaded, futures and promises, event loop-based frameworks. This is Node.js. So now our request can start at one thread, but finish at a different one, going through many other threads along the way.

2. Reinventing Distributed Tracing

Short description:

Prefixing logs with a unique ID for each request no longer solves the problem in a distributed system. With the rise of containerized services, backends are spread across multiple machines, making it difficult to trace operations. To address this, we reinvented distributed tracing from scratch. We introduced the concept of a trace, which follows a request and captures debugging data. Within the trace, we have spans that represent the smallest unit of work, such as an HTTP request or a function call. Spans can create child spans, allowing us to mirror the structure of our software. Each span has a unique ID and holds data like its parent ID.

Prefixing them with the thread name doesn't really solve our problem now. We need to prefix them with something unique to the request itself, and that's what we did. We generated a unique ID for each request and prefixed it, our logs.

But our frameworks didn't stop evolving. About 10 years ago, Docker and AWS made way for containerized services. And now our backends don't even live on one single machine. Each container and microservice handled multiple requests and produced its own logs. Our logs are all over the place now. It was very hard to make sense of the operation flow, so we needed a better debugging tool that can trace the operations as they jump between containers and services. That's when distributed tracing became a necessary tool for debugging.

In order to understand how it works, we're going to reinvent it from scratch. Since our backends now have a very distributed nature, we needed to define a vehicle for each request that will follow it around and capture debugging data along the way. Let's call that a trace. The trace will start when the operation flow starts, and it's going to have a unique ID. That can be the frontend, for example.

If we think about logs, they usually tell us what happened at a particular time. They try to mimic the structure of our code. So let's invent that now. Let's invent something that's going to describe the smallest unit of work, like an HTTP request or a function call or anything specific that our software does at a specific time. We're going to call that a span, and we're going to create one immediately when the trace starts. That's going to be our root span.

So just like the log, the spans are going to mimic the structure of our software. But since we're reinventing it, let's make it much smarter than simple messages. So since spans are the smallest unit of work, like a single function, and we know that one function can invoke another function, which in turn can also invoke a third function, we're going to design our spans so they can create child spans, which can to create their own child spans and so on. Now we can really mirror the structure of our software with this. We have a span hierarchy, but we need to remember which span is a child of which span. To do that, we're going to need something to identify each span. So we will assign an ID to each span as we create them. We also need to save the parent span ID. So let's create a space inside each span so it can hold data like its ID and its parent ID.

3. Enhancing Trace Data and Distributing Tracing

Short description:

In addition to capturing basic data about spans, we can also keep tags and a status field to provide more context and enable better analysis. By introducing a finish method, we can calculate the duration of spans and identify performance bottlenecks. To distribute the trace on the backend, we create a trace context that concatenates the trace ID and the ID of the last span into a string. This string can be easily transferred and parsed by different components, allowing for continued tracing.

But why stop there? We got space for more data. Let's keep a set of tags so we can search and aggregate and group them later. Let's also keep a status field that's going to indicate whether the spans work, finish successfully or not. We can basically keep any kind of data that can be useful later on.

Since we know when we create them, let's introduce a finish method that'll write down when the spans finished. So now we can calculate how long the spans took. We have enough info to chart them now. And if we do, we're going to be able to identify performance bottlenecks easily. I mean, there'll be obvious that span should not take that much.

But still, how do we distribute this now? How can we continue this trace on the backend? We had the trace and its ID. We also have a bunch of spans attached to it. Let's create a trace context that's going to concatenate the trace ID and the ID of the last span into a string. We can now transfer this string so our backend or the next processing units can parse it and continue tracing, starting from the last span. Since it's going to be a string, we can easily transfer it, whether it's a client, a microservice, a cron job, or it's in JavaScript or Python or PHP, as long as it can parse and read a string, it can continue our trace. And that's distributed tracing.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Advanced Conference 2022React Advanced Conference 2022
25 min
A Guide to React Rendering Behavior
React is a library for "rendering" UI from components, but many users find themselves confused about how React rendering actually works. What do terms like "rendering", "reconciliation", "Fibers", and "committing" actually mean? When do renders happen? How does Context affect rendering, and how do libraries like Redux cause updates? In this talk, we'll clear up the confusion and provide a solid foundation for understanding when, why, and how React renders. We'll look at: - What "rendering" actually is - How React queues renders and the standard rendering behavior - How keys and component types are used in rendering - Techniques for optimizing render performance - How context usage affects rendering behavior| - How external libraries tie into React rendering
React Summit 2023React Summit 2023
32 min
Speeding Up Your React App With Less JavaScript
Too much JavaScript is getting you down? New frameworks promising no JavaScript look interesting, but you have an existing React application to maintain. What if Qwik React is your answer for faster applications startup and better user experience? Qwik React allows you to easily turn your React application into a collection of islands, which can be SSRed and delayed hydrated, and in some instances, hydration skipped altogether. And all of this in an incremental way without a rewrite.
React Summit 2023React Summit 2023
23 min
React Concurrency, Explained
React 18! Concurrent features! You might’ve already tried the new APIs like useTransition, or you might’ve just heard of them. But do you know how React 18 achieves the performance wins it brings with itself? In this talk, let’s peek under the hood of React 18’s performance features: - How React 18 lowers the time your page stays frozen (aka TBT) - What exactly happens in the main thread when you run useTransition() - What’s the catch with the improvements (there’s no free cake!), and why Vue.js and Preact straight refused to ship anything similar
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Our understanding of performance & user-experience has heavily evolved over the years. Web Developer Tooling needs to similarly evolve to make sure it is user-centric, actionable and contextual where modern experiences are concerned. In this talk, Addy will walk you through Chrome and others have been thinking about this problem and what updates they've been making to performance tools to lower the friction for building great experiences on the web.
JSNation 2023JSNation 2023
26 min
When Optimizations Backfire
Ever loaded a font from the Google Fonts CDN? Or added the loading=lazy attribute onto an image? These optimizations are recommended all over the web – but, sometimes, they make your app not faster but slower.
In this talk, Ivan will show when some common performance optimizations backfire – and what we need to do to avoid that.

Workshops on related topic

React Summit 2023React Summit 2023
170 min
React Performance Debugging Masterclass
Featured WorkshopFree
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
JSNation 2023JSNation 2023
170 min
Building WebApps That Light Up the Internet with QwikCity
Featured WorkshopFree
Building instant-on web applications at scale have been elusive. Real-world sites need tracking, analytics, and complex user interfaces and interactions. We always start with the best intentions but end up with a less-than-ideal site.
QwikCity is a new meta-framework that allows you to build large-scale applications with constant startup-up performance. We will look at how to build a QwikCity application and what makes it unique. The workshop will show you how to set up a QwikCitp project. How routing works with layout. The demo application will fetch data and present it to the user in an editable form. And finally, how one can use authentication. All of the basic parts for any large-scale applications.
Along the way, we will also look at what makes Qwik unique, and how resumability enables constant startup performance no matter the application complexity.
React Day Berlin 2022React Day Berlin 2022
53 min
Next.js 13: Data Fetching Strategies
WorkshopFree
- Introduction- Prerequisites for the workshop- Fetching strategies: fundamentals- Fetching strategies – hands-on: fetch API, cache (static VS dynamic), revalidate, suspense (parallel data fetching)- Test your build and serve it on Vercel- Future: Server components VS Client components- Workshop easter egg (unrelated to the topic, calling out accessibility)- Wrapping up
React Advanced Conference 2023React Advanced Conference 2023
148 min
React Performance Debugging
Workshop
Ivan’s first attempts at performance debugging were chaotic. He would see a slow interaction, try a random optimization, see that it didn't help, and keep trying other optimizations until he found the right one (or gave up).
Back then, Ivan didn’t know how to use performance devtools well. He would do a recording in Chrome DevTools or React Profiler, poke around it, try clicking random things, and then close it in frustration a few minutes later. Now, Ivan knows exactly where and what to look for. And in this workshop, Ivan will teach you that too.
Here’s how this is going to work. We’ll take a slow app → debug it (using tools like Chrome DevTools, React Profiler, and why-did-you-render) → pinpoint the bottleneck → and then repeat, several times more. We won’t talk about the solutions (in 90% of the cases, it’s just the ol’ regular useMemo() or memo()). But we’ll talk about everything that comes before – and learn how to analyze any React performance problem, step by step.
(Note: This workshop is best suited for engineers who are already familiar with how useMemo() and memo() work – but want to get better at using the performance tools around React. Also, we’ll be covering interaction performance, not load speed, so you won’t hear a word about Lighthouse 🤐)
Vue.js London 2023Vue.js London 2023
49 min
Maximize App Performance by Optimizing Web Fonts
WorkshopFree
You've just landed on a web page and you try to click a certain element, but just before you do, an ad loads on top of it and you end up clicking that thing instead.
That…that’s a layout shift. Everyone, developers and users alike, know that layout shifts are bad. And the later they happen, the more disruptive they are to users. In this workshop we're going to look into how web fonts cause layout shifts and explore a few strategies of loading web fonts without causing big layout shifts.
Table of Contents:What’s CLS and how it’s calculated?How fonts can cause CLS?Font loading strategies for minimizing CLSRecap and conclusion
React Summit 2022React Summit 2022
50 min
High-performance Next.js
Workshop
Next.js is a compelling framework that makes many tasks effortless by providing many out-of-the-box solutions. But as soon as our app needs to scale, it is essential to maintain high performance without compromising maintenance and server costs. In this workshop, we will see how to analyze Next.js performances, resources usage, how to scale it, and how to make the right decisions while writing the application architecture.