1. Introduction to Node.js Diagnostics
This talk is about getting diagnostics information out of Node.js without using a lot of third-party tools. Over the years, a lot of work has been put into diagnostics specifically and how we can improve them in Node. For most use cases, you can do a lot of debugging and getting information out of Node, just using the Node executable and Chrome.
♪ Hi, everyone. Thanks for coming to my talk titled Out of the Box Node.js Diagnostics. So, this talk is about getting diagnostics information out of Node.js without using a lot of third-party tools.
So, just for a little bit of background, getting diagnostics out of Node.js used to be pretty hard. Node.js used to follow a small core philosophy, which meant that a lot of functionality was left up to NPM modules. So, we had things like Node.js inspector for debugging, Node.js heap dump for capturing heap snapshots, Long John for asynchronous stack traces and things like that. But, over the years, a lot of work has been put into diagnostics specifically and how we can improve them in Node. But a lot of people might not be aware that these things exist yet. And for most use cases, you can do a lot of debugging and getting information out of Node, just using the Node executable and Chrome.
A lot of the content in this talk is actually from the official CLI documentation, if you want to look that up on your own. And this talk also assumes the latest version of Node 16.
2. Debugging with Environment Variables
3. Warnings and Deprecations in Node.js
Warnings in Node.js are used to notify users of potentially risky actions. There are several flags available, such as --no-warnings to disable all warnings, --redirect-warnings to save warnings to a file, and --trace-warnings to print a stack trace. For example, when requiring the sys module, a deprecation warning is shown, and using --trace-warnings reveals the source location. Deprecations in Node are identified by an ID, which can be looked up in the documentation for more details.
So, the next thing I wanted to talk about was warnings in Node.js. So, warnings are used to let the user know about that they're doing things that are potentially risky and we kind of advise against. So, there are a handful of flags that you can use here. The dash dash no warnings flag will disable all warnings. You might not want to do that. The dash dash redirect warnings flag can be used to take all warnings and dump them into a file somewhere so that you can look at them later. And then the dash dash trace warnings will actually print a stack trace whenever a warning is encountered. And this can be really useful for, you know, a warning is omitted, but you're not really sure why, it could be coming from deep inside of your node modules folder. So, the stack trace will actually kind of give you more insight into what's going on. And so, an example of that is shown here. So, if you were to require the sys module, sys is a, you know, an ancient module from node's core that has been long deprecated, but is still there. We don't want people to use it, so whenever you require it, you'll see this deprecation warning is printed, and then you can use the dash dash trace deprecation flag, as I mentioned before, to, I'm sorry, the trace warnings flag to see where the where in your code that's coming from. So, if you look at the arrow I have in the stack trace here, you will actually see it's coming from example.js on line one, and so it's very easy to kind of tell what's going on. So, one thing to note here is you'll see in the message dep 0025, so all deprecations in node are given an ID, so you can actually go into node core's documentation, we have a specific site page just for deprecations, you can look up that deprecation ID and kind of get more information than what's available in the stack trace here.
4. Deprecations in Node.js
Deprecations in Node.js are similar to warnings and have flags to control their behavior. The node deprecation flag hides deprecation warnings, while the trace deprecation flag prints a stack trace. The throw deprecation flag throws an exception when a deprecated feature is used. This can be useful for identifying and addressing deprecated code during testing or CI.
So, the next thing I want to talk about are deprecations, which are similar to warnings. It's a specific class of warning. Again, we have similar flags for dealing with them, so we have the node deprecation flag, if you don't want to see any deprecation warnings in your code, then if you know what you're doing, then you might want to use that. We also have dash dash trace deprecation, which prints a stack trace, similar to the warnings flag that I showed a little bit ago. And then we also have throw deprecation, which actually will throw an exception anytime you use a deprecated feature. Now, this isn't something you probably want to run in production, but in your CI code or when you're doing testing, it's very helpful for kind of finding things that you need to address and kind of move out of your code.
So we have a similar example here. Again, this is require sys because a deprecation warning is just another warning. So here I've used the dash dash throw deprecation for the same code as before. And in this case, you'll actually see a thrown exception, which is nice because it does some highlighting of the stack trace. So you'll notice that a lot of the stack frames here are grayed out. Those are things that come from like Node's core, for example. And the line that is still in black with the arrow pointing to it is the line in your code where the deprecation is coming from.
5. Synchronous I.O. and Event Loop Blocking
Synchronous I.O. blocks the event loop, which can negatively impact performance, especially in server applications with multiple requests. Node's trace sync I.O. flag can be used to report synchronous I.O. after the first turn of the event loop. An example is synchronously reading a file, which results in individual warnings for each operation. The stack trace shows the synchronous I.O. operations called after the first turn of the event loop.
So the next thing I wanted to talk about was synchronous I.O. and how you can find it in your code. So synchronous I.O. blocks the event loop. This used to be a real performance killer. It still is to an extent, but now we have worker threads which can kind of mitigate things a bit. But either way, you probably don't want to block your event loop if you don't need to. Especially if you're running an application like a server where you have multiple requests being handled at the same time. If you block the event loop on one of the requests, then all the other requests will suffer as well.
So when you're setting up your server, it's fine if you're doing synchronous I.O. during the startup phase, but once you start moving to serving traffic, you probably don't want to allow that. And so you can use Node's trace sync I.O. flag to report any synchronous I.O. that happens after the first turn of the event loop. So an example, which I have here is we're using a set immediate, so that we know that we're now moving on past the first turn of the event loop, and then we're synchronously reading a file from the file system. And so read file sync does a couple of operations under the hood. It opens the file, it will also stat the file, read the data from it and close it. And because all of these are happening synchronously, they'll all result in individual warnings. And so you can actually see in the stack trace that's provided here, open sync, try stat sync, read sync and close sync are all being called out as synchronous IO after the first turn of the event loop. And of course, if I were to remove the set immediate here, this would not report any warnings because we would still be on the first event loop turn.
6. Tracing Uncaught Exceptions and Process Exit
Next, let's discuss tracing uncaught exceptions and process exit in Node.js. The trace uncaught flag helps locate exceptions in your application code, providing the path where they were thrown. Similarly, the trace exit flag shows where the process exit call originated. Additionally, we'll talk about unhandled promise rejections and how Node.js 15 changed their behavior to throw exceptions. You can configure this behavior using the unhandled rejections flag with different modes: throw, strict, warning, warn with error code, and none.
Next, I wanted to talk about tracing uncaught exceptions. Sometimes, an exception is thrown in your application code, and it can be challenging to locate where it originated, especially in large applications or when it comes from the node modules directory. The stack trace on the error itself might not correspond to the actual location of the exception. To address this, you can use the trace uncaught flag when running your application. This flag not only prints out the error and its stack trace but also provides the path within your application where the exception was thrown from.
There is a similar flag for tracing process exit. If a node module calls process exit, it can be challenging to determine what is happening in your application. By using the trace exit flag, you can see where the exit call originated in the code and the exit code. For example, the output might indicate that the environment exited with code zero.
7. Handling Promise Rejections in Node.js
The new behavior in Node.js makes it more explicit and allows better handling of promise rejections. You can configure this behavior from the CLI using the unhandled rejections flag. There are five modes available: throw, strict, warning, warn with error code, and none. An example with unhandled rejections set to strict is shown, where a promise rejection becomes an uncaught exception.
8. Introduction to the Tick Processor in v8
9. Introduction to v8 Inspector
Node and v8 now have first-class support for the v8 inspector, which is Chrome's dev tools used with Node applications. To set it up, start Node with the --inspect flag and specify the host and port. Be cautious when binding to a public address, as it can be a security vulnerability.
So starting a few years ago Node and v8 actually started having first class support for the v8 inspector which is essentially just Chrome's dev tools used with Node applications. So if you've done any type of browser development or even Node development in the past few years, there's a good chance you've seen Chrome's dev tools. It includes things like a step debugger, profilers with GUI interfaces as opposed to the CLI based one I showed before. And so the way that you actually set this up is you start Node with the dash dash inspect flag and then you can actually specify the host and port that you want it to listen on on your machine. By default it's going to listen for on 127.0 to 0.1 port 9229. And then you can also set it up so that you as soon as you start the application it will set a breakpoint for you so you don't have to worry about doing that by hand and that's with the dash dash inspect dash BRK flag which works other than the breakpoint in the same fashion and then one security tip is be careful if you're going to bind to address 0.0.0.0 or anything else that's really a public address on your machine. This used to be the default for node and you know it was reported as a security vulnerability because if you have a server that can be reached publicly and you're bound to a publicly exposed address like that then technically it's possible for an attacker to actually connect to the debugger and start kind of messing with your code and run time. So that's just something to be aware of.
10. Using the v8 Inspector and Dev Tools
Continuing with the v8 inspector example, running Node with the --inspect flag connects to the debugger via a WebSocket URL. The dev tools provide a high-level view of CPU activity, including function calls and memory usage. Heap snapshots are useful for detecting memory leaks, allowing you to compare objects between snapshots.
So next I wanted to talk or I'm sorry just continuing with the v8 inspector example, you know the text at the top kind of shows you how you would run it node dash dash inspect break example.js. It will print out some information so it tells you where the debugger is listening you know via this WebSocket URL and then point you back to the documentation if you have you know more questions or need to do additional reading. But you can you know the picture at the bottom here just kind of shows you what it's like whenever you're dropped into the dev tools. Again a lot of people have probably seen this before but if you haven't this is kind of what you can expect to see.
11. CPU Profiler and Heap Snapshots in Chrome DevTools
Chrome DevTools provides a CPU profiler and heap snapshots for diagnostics. The CPU profiler allows you to collect profiles using command line flags or manually through the DevTools UI. The DevTools view of a CPU profile shows a high-level view of CPU activity over time and the functions being called. Heap snapshots are useful for debugging memory leaks, comparing snapshots to identify objects not being cleaned up. Heap snapshots can be captured through CLI signals or automatically near heap limit. They can also be collected manually via the DevTools UI.
One of the nice tools that is available in Chrome DevTools is a CPU profiler. It shows you what functions are executing over time. I want to point out that this is not the same as a flame graph. A flame graph actually takes stack traces over time and kind of consolidates them into into one larger stack trace so you can see where your application is generally spending its time. If you're looking for flame graphs, there's an excellent tool on MPM called zero x that you can play around with.
But as far as CPU profiling goes, Node actually has a couple of command line flags that allow you to manipulate the CPU profiler without actually needing to go in and do this by hand. So if you pass the dash dash CPU prof flag, it will collect a CPU profile for you. It will start the profiler when the application starts up, and then write out the profile when your application shuts down. The CPU prof dir flag allows you to specify where you want your profiles to be written to, if you need to change from, you know, the default location. Similarly, the CPU prof name flag allows you to give your profile a different file name. And then the dash dash CPU prof interval flag allows you to define the sampling interval of the profiler. So, if you needed to, you know, sample the stack more frequently or less frequently, you can control it there. And then, of course, you can also go into DevTools in the UI and collect profiles by hand, if need be.
And so, this is what the DevTools view of a CPU profile looks like. So, there is a kind of a region at the top that shows a high-level view over time of the CPU activity. And then the highlighted window is shown at the bottom, with the colored stack traces. You can actually see what functions were being called. So, you'll see a lot of, in this example, module.run main, as well as require. So, you can, you know, from looking at this, you can kind of understand that this is the startup phase of your application, where a lot of modules are being required and configured. And then the behavior kind of changes after that to be more, you know, more dependent on the traffic that you're serving.
12. Diagnostics and Debugging Techniques
It shows a list of every type of object broken down by its constructor. You can track down memory leaks. TLS connection tracing provides information directly from Node.js. Post-mortem debugging creates a core file of your application, but has downsides. Diagnostic reports offer a simpler alternative to postmortem debugging.
It shows a list of every type of object broken down by its constructor. And then it'll show things like the object count. In this specific example, there are four event emitters in the code. The shallow size tells you how much memory that actual object is holding onto. And then the retained size follows the pointers from those objects to tell you kind of the cascading effect of holding this object in memory. And so you can do a lot of useful things here while you're trying to track down memory leaks.
The next thing I wanted to talk about was TLS connection tracing. It used to be if you wanted to diagnose TLS issues, you had to have the open SSL client set up and pass some command line flags and things like that. But now you can get that information directly from Node.js. So the dash dash trace TLS flag from the CLI will dump all the same information for all TLS connections, just so you know this will be very noisy, so you definitely don't want to enable this in production. You can also set it on the individual socket level with TLS socket dot enable trace. And then you can also set it whenever you're setting up a socket or whenever you're setting up a server with the enable trace option pass to create server and TLS connect. Again, this will dump a ton of information. So just be prepared to sift through it.
13. Generating Diagnostic Reports in Node.js
This section covers generating diagnostic reports in Node.js using CLI flags. These reports contain information about the operating system, process, thread pool, stacks, and more. Care must be taken with sensitive information. The process.report.getreport API can be used to create reports, which include versions, event triggers, file names, process and thread IDs, and more. It's important to handle these reports with care and redact any sensitive information before sharing. The talk concludes with gratitude for attending and an acknowledgment of the value of learning about diagnostics.
So that's everything that I had for today. Again, thanks for coming to my talk, and feel free to reach out to me on Twitter, GitHub, anywhere. Thanks, everyone. I hope everyone is, like, writing lots of stuff on the dependent paper. So you asked the question, have you ever had a bug in your application but couldn't obtain the proper metrics to fix the problem? And 82% have answered yes. Yes, so I'm hoping that some of the information that got out of the talk today might help with that in the future. I think if, you know, if the same poll was asked five or six years ago, the answer would have probably been a lot closer to 100%. It used to be a really tough thing to do, and yes, the project has just made a ton of progress since the old days of Node. Yes, yes. I mean, even 84% is a lot, but yes, most of the people don't, I mean, know about the new things coming in and all. So I see lots of good comments in the channel that was lots of knowledge, amazing talk and I did not know anything that he said. So, I mean, people learn a lot today. So that was a cool session.
Learning and Contributing to Node.js Diagnostics
Lots of stuff that I didn't know about. Wow. Yes. Awesome. One follow-up question on your talk would be, where can I learn about the things that you mentioned? Two places: the official Node.js documentation, specifically the CLI documentation, and the Node.js website's Guides section. For those interested in working on Node.js diagnostics, there is the option to contribute to the project on GitHub or join the diagnostics working group. Areas for improvement in Node.js diagnostics include postmortem debugging and integrating flame graphs into Node.
Lots of stuff that I didn't know about. Wow. Yes. Awesome. So, okay. So, one follow-up question on your talk would be, where can I learn about the things that you mentioned in the talk? Yes. So, two places really. First, if you go to the official Node.js documentation, and then on the left-hand side there's all the different, you know, subsystems inside of Node. If you scroll down to the command line, I think it's CLI or Command Line Interface documentation, this talk came almost exclusively from that page and all the information there. And then if you also go onto the Node.js website, there's a section called Guides, and we have guides on different things like, you know, getting diagnostic information, creating flame graphs, running your application in Docker, and just a whole variety of different things like that. Okay. So, yeah, do check it out. For anyone who is listening here, go check out all the documentation and learn more about it.
So, after this talk, like, lots of people have learned new things, and they would be interested more in diagnostics. So, let's say someone is interested in working on Node.js diagnostics. So, is there a way that they can get involved with it? Absolutely. So, if you're interested in actually working on the project, you can come to, you know, github.com slash Node.js slash Node. That's where the project, you know, really lives. But we also have a diagnostics working group, which is a team of people who kind of specifically dedicate time to improving diagnostics in Node. And I believe the URL for that is github.com slash Node.js slash diagnostics. And, you know, you can go there and if you're really interested, join the working group and contribute back or just kind of follow along and see what people are talking about. Awesome.
So, there's a question by Azentyl1990. What are some of the areas you still see improvements could be made in diagnostics with Node.js? Um, that's a good one. So, I touched on postmortem debugging a little bit. I would love to see postmortem debugging get better. Unfortunately, that requires kind of a lot of cooperation with V8. And, you know, not that they're they don't cooperate, but I don't know that they see as much value in it. And then I would also like to see flame graphs have a kind of more first class citizenship inside of Node. So, right now, that's one of the few things I briefly touched on in the talk where you would actually still have to go outside of Node and dev tools to be able to look at.
Node.js Diagnostics Q&A
Johta developer asks about Node.js applications or libraries for diagnostics. Dev tools and Clinic by NearForm are recommended. In serverless environments, diagnostics may be limited to the provider's offerings. The speaker is excited about a small improvement to log visibility in dev tools. The talk transitions from errors to diagnostics in Node.js.
So, I think that would be a great addition. Yeah, that definitely looks like a great addition. Uh, Johta developer asks, any Node.js application or library that helps to visualize or collect all those diagnostics locally that you could recommend like flame graphs, aside from Chrome dev tools. Yeah. So, dev tools will be the main one. I'm trying to think. So, I know Node source has a distribution of Node that has a lot of like diagnostics things built into it. That's going to be more proprietary. I think you may have to pay for it or they might have a free trial. I think there's a, a module out of NearForm called Clinic that has a lot of this stuff integrated with a nice little UI. So, I would look at those two things first.
Okay. Thank you. And a CC Chris asks, which diagnostic techniques do you recommend having Node.js in a serverless environment? In a serverless environment? So, it's kind of tricky there because you, you probably don't have access to all of the same things. Like you can't just, you know, create a CPU profile and easily get it out. For a serverless, you're kind of more at the whim of the provider, I think. So, for example, I work at AWS, and Lambda has pretty good diagnostics information. You can run Node.js there, have logs go into CloudFormation, I'm sorry, not CloudFormation. I can't remember off the top of my head, pressure. There's a logging system there you can have the logs dumped to, and just, you know, other integrations with AWS services that you can take advantage of. But yeah, in a serverless environment, some of these things from this talk may not apply.
Okay, so I have a question for you because you are a part of Node.js TS committee. What are some of the features that are something which is coming up or you are looking forward to, excited about, anything like that to share? So this is a really little thing, but you and I were actually talking about this backstage. Sometimes when you would log things from your node application when you are connected to dev tools, it would show up in the console instead of in dev tools. And there was actually a pull request opened recently to try to improve that. So I think that's one of those, you know, a small thing but a nice UX improvement. Yeah, I'm really excited about that. That would really be a great addition, small, but still it will be impactful I guess. So, okay. You touched lots of points on diagnostics and actually before your talk, we had a talk on errors in Node.js, right? So it was a very nice transition from errors to showing all the diagnostics and in Node.js and how you could do stuff like that.
Responsibilities of TS Committee
The Technical Steering Committee (TSC) is the last resort for resolving conflicts and making technical decisions in the TS community. Ideally, all decisions should be made on GitHub, but when strong opinions hinder progress, the TSC steps in to make a decision or vote on the issue. People are encouraged to explore further on Google and GitHub after the talk.
I have a last question for you for sure. What are some of the responsibilities for you as a member of TS committee? I'm really interested in that.
So the Technical Steering Committee is kind of the last resort as far as resolving any types of conflicts and technical decisions. Ideally, nothing should come to the TSC. Everything should be decided on GitHub amongst the collaborators. Sometimes we get to a point where people have strong opinions and an issue just can't progress and the TSCO often then be pulled in to make a decision and occasionally even have to vote on it. But yeah, I would say that's the biggest thing.
Yeah, that's interesting. So I do not see any more questions but I'm sure people got a lot to explore after this talk. So they are going to do lots of Googling and GitHub and stuff. And so, everyone... Let me just... I'm still seeing the poll, so yes. If you have any other questions for Colin, you can still talk to him in the special chat. So, Colin will be available in his speaker room. And thanks a lot, Colin, for such an in-depth and wonderful talk. It was great, and people have learned a lot. Thank you so much for being with us here today. Thanks for having me. Bye.