Instrumenting Node.js Internals

Rate this content
Bookmark

When troubleshooting complex Node.js applications recollecting information about the network, filesystem or external dependencies is always useful. Even when working with a new codebase insights on how the application is built, which asynchronous calls are being used or how often. In this talk I want to introduce "instrument", a tool I've created for this which is also Open Source.

7 min
24 Jun, 2021

Video Summary and Transcription

The Talk is about troubleshooting Node.js applications and using a tool for improved visibility and information. The tool can collect information about URLs and module imports in a server. However, it is not recommended for use in non-local environments due to performance overhead and potential bugs.

Available in Español

1. Troubleshooting Node.js Applications

Short description:

My talk is about troubleshooting Node.js applications. I would try to tear apart the parts of the application to pinpoint the problem. I developed a tool that provides improved visibility and useful information. It can be used to collect information about URLs used by a Node.js application and identify module imports in a server. However, it's not recommended to use this tool in non-local environments due to performance overhead and potential bugs.

Hey, everyone. Before we continue, I wanted first to thank all the organizers and volunteers that made this happen, dedicate their time, and also gave me the opportunity to be presenting to you today. My talk is about troubleshooting Node.js applications.

Before that, a little introduction about myself. I am Alejandro. I'm from Buenos Aires, Argentina. I work as a back-end engineer at X-Team and I also contribute with a couple of communities.

So, into the topic of this talk, say you have a Node.js application that is running, but something is not working. There is no exception. You don't have logs. Where would you start? What I would try to do is to tear apart the parts of the application, so it's easier to pinpoint where exactly the problem is. Say it's a server, I would check the bootstrap part of the process. A breakpoint right after it initializes to see if the error happens before that or after that.

Oftentimes I would be in the process of troubleshooting an application, and I would think I have no idea why that is happening. I would be just like this dog. So a lot of times, this would also happen on my development environment, where I have full control, full access, and also should have a good observability about what is happening.

This last part was one of the motivations that I had to work on a tool that would practically no changes to your application could allow improved visibility and potentially turn yourself into Ironman, like in this scene. Viewing a lot of useful information at once. I will go over a few specific examples that I could think of on a short demo.

On the scenario number one, imagine you have a script or an application that is running through Node. We want to be able to tell if it's going over the network and if that's the case, to collect more information about what URLs it's using and possibly more information on that. So we'll go on to my demo and for this let's say we want to try to do that with NPM. And we will try to retrieve the latest version for a package, in this case Mocha. We get the result 802.1. Say we wanted to do that with this tool with NPM we can require instrument pass the same arguments that we were passing before and first of all we will see the same output that before 802.1 and and after that a short summary for the tool.

In this case for these two native modules that are enabled HTTP and HTTPS. Those two are specifically configured here in the modules list and you can add another ones like FS, Require, and others as well. This configuration is added on the instrument.config.js part. So we can see that this is going to registry.npmjs.org. We see that the method is get on the headers user agent npm in CI and whatnot. So all I had to do to require that tool is to add as a development dependency.

For another example, say you're a refactoring HTTP server, and you want to identify which files are importing module A.js. The module is only being loaded dynamically by a specific probe in your server. And in this case, it would be practically impossible to do this task with static analysis. You would have to instrument your application or your server to get that information. So by adding the required module in the list we previously saw, you could get that information using this tool as well.

A little caveat on this and a recommendation if you should be running these on other environments that are not your local environments and the answer is no, definitely no. There is a performance overhead on adding this instrumentation and I'm sure there are a few bugs that you will find if you test this on your own, so it's not recommended to do so. You can find this tool on github and see the code, see an open issue, pull request, and if you have further questions on how this is being done or other technical questions, we will be holding a Q&A session or feel free to send them over via social media, twitter. That's everything that I had for you today.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Top Content
Do you know what’s really going on in your node_modules folder? Software supply chain attacks have exploded over the past 12 months and they’re only accelerating in 2022 and beyond. We’ll dive into examples of recent supply chain attacks and what concrete steps you can take to protect your team from this emerging threat.
You can check the slides for Feross' talk here.
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
In the early years of Node.js, diagnostics and debugging were considerable pain points. Modern versions of Node have improved considerably in these areas. Features like async stack traces, heap snapshots, and CPU profiling no longer require third party modules or modifications to application source code. This talk explores the various diagnostic features that have recently been built into Node.
You can check the slides for Colin's talk here. 
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring.
While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.
JSNation Live 2021JSNation Live 2021
19 min
Multithreaded Logging with Pino
Top Content
Almost every developer thinks that adding one more log line would not decrease the performance of their server... until logging becomes the biggest bottleneck for their systems! We created one of the fastest JSON loggers for Node.js: pino. One of our key decisions was to remove all "transport" to another process (or infrastructure): it reduced both CPU and memory consumption, removing any bottleneck from logging. However, this created friction and lowered the developer experience of using Pino and in-process transports is the most asked feature our user.In the upcoming version 7, we will solve this problem and increase throughput at the same time: we are introducing pino.transport() to start a worker thread that you can use to transfer your logs safely to other destinations, without sacrificing neither performance nor the developer experience.

Workshops on related topic

Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Top Content
Workshop
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.