In this talk, we will have fun trying to tamper with a running Node.js process to change its behavior at runtime. Without changing the code or restarting the process, we will find a way to inject our own logic into it and start to do the things we want. What are the limitations of such an approach? Is there part of it that can be used in real life scenarios? Come and find out! Yes, there will be some live demo.
Can You Change the Behavior of a Running Node.js Process From the Outside?
AI Generated Video Summary
This talk explores how to remotely change the behavior of a Node.js process at runtime and inject a logger using the Chrome DevTool protocol. It demonstrates the power of dev tools and encourages their usage. Remote debugging is useful for debugging memory leaks in production. The method requires local machine access and has security implications, but it requires significant access and indicates a major breach. The talk emphasizes the importance of having awareness and monitoring in place for application protection.
1. Introduction to Changing Node.js Process Behavior
Hello everyone and welcome to my talk entitled Can you change the behavior of a running Node.js process from the outside? Today, we will explore how to remotely change the behavior of a Node.js process at runtime and inject a logger. I will start with a live demo to demonstrate the process. Let's open WebStorm and start our simple server. By running another node.js process named injector, we can see logs in the server and even customize the logging.
Hello everyone and welcome to my talk entitled Can you change the behavior of a running Node.js process from the outside? The thing is it's a very weird talk and there will be a lot of things that don't make any sense inside it. So please stay till the end. It will all come clean at one point and I wrote this sober that's a promise i make.
So before we start a couple words about myself. I'm Vladimir Dutourkem you can call me Vlad. I'm a Node.js collaborator. I've been having Node.js commit writes for more than three years now but at my day job I am a customer engineer at a startup named Screen. We do application security. Basically we give you a way to secure your production servers without changing your code. So if you have any web application running live on the internet you probably want to check what we do and don't worry we have a free plan.
So what's the plan today? What's the plan of this talk? Let's take a simple hello world server in Node.js. So we require HTTP and we answer it with OK to each request. So far so good. But it has no logs. Well yes it has no logs because the developer was too lazy to add them and the developer in this case is me. There is an easy solution for that. You change the code and you redeploy. And that's what normal people would do. So on line three you can see now that there are logs that have been added to the source code of the application, which is great. But that's not what we're going to do today. Because we are a bit crazy and we are a bit experienced with stuff, we are changing the behavior of the process at runtime and inject a logger in the Node.js process at runtime. So how to do that remotely? And because I am brave, I will start by a live demo and then I will explain what happened in this demo. So let's open WebStorm, here we've got our simple server and let's start it. Let's go to first shell, node server.js, and you can see I did not pass any flag to the process, it's really just started. So if I do a couple requests on it, curl HTTP local host 8080 slash, it responds with OK and there is nothing logged here. Now let's do a bit of magic that I will explain later, cqsl1 pid, we check the logs, it's passed in debug mode as you can see. So that's the first thing I will need to explain later. And now we just run another process, another node.js process named injector, nothing changed on the log here, but when I do curls on this side, well we've got logs in the server. Now it logs stuff. I can even make it log whatever I want because I decided to log the URL.
2. Changing Process State and Remote Debugging
So now it really logs the URL of the request we do. And that would work for any URL because what we injected is actually console.log, rec.method, rec.url. Okay, now I guess the legitimate question is how does that happen? How did that work? And that's what we will spend the next 15 minutes going through together.
So, we've got this running process. It's a simple hello world server, as I said, and it just answers with okay. So, first thing we need to do is to change its state to debug mode, so we can connect with a debugger. So, there's this cool thing in Node.js is that if you send the system signal sigusr1, then it enables the debugger on the process. And then you can connect with that through a debugger on a web circuit. So, the only change I do to the code now is I make it log the PID of the process. That's not necessary. We could use the PS command on the top command to find it, but in that case, I am a bit lazy. I mean, I was already too lazy to write logs, so I am too lazy to find the PID of a running process, so we just log that. Then when we do the kill-usr1 and the PID, it changes the state of the process. So, the kill command on Unix systems is not only to kill processes, but can also be used to send message signals. In our case, we send the sig-usr1 message signal and we pass the PID of the target process as a last argument.
That's what happened when we started to log process running and when we did this command in another shell, we logged DebuggerListName on websocket and then the url. This can also be done programmatically from another node.js process using, you guessed it, process.kill. Process.kill has a syntax where it accepts a signal and a PID and sends a signal to another process. So, ta-da, we actually remotely started the debugger, which is actually very, very useful, but I will explain that later. So, if we go to chrome __inspect, we actually see that the target is available to be debugged. That's the way you can check how many nodejs and chrome tabs are available for debug locally or remotely by going on chrome //inspect on chromium or chrome. Okay.
3. Using the Chrome DevTool Protocol
When using debug tools for Node.js, the underlying protocol is the Chrome DevTool protocol. This protocol is well-documented and can be found on chromedevtools.github.io. We will run a script that utilizes the chrome remote interface module to interface with the Chrome Devub protocol. We enable the runtime domain and evaluate an arbitrary script on the remote Node.js process using the runtime.evaluate method.
And, of course, when you use debug tools for Node.js, that's the protocol that is used under the hood. It's actually pretty well documented, and you can find the documentation on chromedevtools.github.io. It's a cool reading, because it teaches you about how things work under the hood, but that's actually the goal of this talk.
So, let's use that, because it sounds awesome, and it actually is. So, we will just run this script, and I'm pretty sure that in the few seconds I showed it, you are all able to know what it does. That was a joke, this picture is way too small. Let's go back to it just quickly.
On line two, I require a module that is named the chrome remote interface, which is actually the module we will use to interface with the Chrome Devub protocol. But once again, this is a way too small image for us to read it. Let's go with zooming in. On this slide, we start on line one by using the Chrome DevTool protocol and connect a client to the port 9 to 29. That's the default port for Node.js debugging and that's where by default the process will start. That can be configured on the Node.js side, but in that case, let's go with that. On line three, we do something really important. We enable the runtime domain. So in the Chrome DevTool protocol, there are multiple domains. There is a domain called runtime, there is a domain called debugger, there is a domain called CPU, there is a domain called memory and I'm sure you can guess what they do by their name. Before you use the method of a domain, you need to enable it to tell the V8 engine that you are going to use methods from this domain. So we start by doing that.
4. Retrieving Object IDs and Evaluating Scripts
The remote Node.js process returns an object ID, which is a memory address for the prototype of the server class. It creates a new variable pointing to it and gives us a way to retrieve this variable. By querying objects and passing the result, we get a pointer to an array where each element is an instance of HTTP.SERVER. We can access the single instance of HTTP.SERVER in this process. We evaluate another script by putting the function in the module and asking Node to load it on the process.
The thing we do on line 9 is we do query objects. That's a method that queries all the objects in the heap that have the prototype past as parameter. So on line 10, we've got a field prototyped object ID, and we pass the result of the previous call. In the previous call, we've got a pointer to the prototype of the server class. On this call, we take this pointer, give it back to V8, and tell it, give us all the objects that have this as a prototype. And we get a result of it, which is actually a pointer to an array, to a list. Once again, we've got an object ID pointing to a list for which each element is actually an instance of server, of HTTP.SERVER.
So on line 12, what we do is we ask for the list of properties of this list. It will have properties such as length, includes, but it will also have properties named 0, 1, 2, 3 that are actually the elements in this array, in this list. Now, because we know that in this code, there is only one instance of HTTP.SERVER, we know it will be on the first element of the list at the zero part. However, if we add more than that, we could introspect to find the one we want. So, we put SERVER instance which contains the object ID of our instance of HTTPSERVER. And that's already pretty cool. We have access to the single instance of HTTP.SERVER in this process.
5. Injecting Logs into Node.js Process
That's as easy as that. We call the method call function on line 7, passing it an object ID as a parameter. This method is powerful as it calls the function declared in function declaration with the value of this bound to the object ID. The function replaces the listeners for the request event with a small function that logs the method and URL of each request. We then put back the modified listeners on the event emitter. Finally, we clean up the added method and disable the debugger.
That's as easy as that. I will show you this function on the next slide. On line 7, we call the method call function on. This method is actually really powerful. First, you pass it an object ID, a pointer as parameter, and it will call the function declared in function declaration with the value of this bound to the object ID, the object pointed by the object ID. So basically, since server instance contains a pointer to instance of HTTP.server, the function declared on line nine is actually called with this being the value of the server.
Okay, so far we are basically just calling a method on the instance of HTTP server. So, what is this method? What's inside to inject? Well, it's just a small function that does the following. It takes an emitter as a parameter, so it's expected to be an event emitter. It gets all the listeners for the request event, that is the event fired by HTTP.server each time there is a new HTTP request in node. Then on line five, it removes all of them. So, there are no listeners on HTTP server anymore at this point. And for all of them, it replaces them by the function defined on line 11. So, function on line 11 takes requested responses parameters, logs request.method, request.URL, and then calls the original method. That is all of the listeners that have been removed before. So, what we do is we take all listeners and one by one, we wrap them with this function that logs the method at the URL. Then on line 16, we put back this listener in the same order that were on the event emitter. So, we just replace all the listener of the request event by a function that calls the original listener but logs what's getting it. Okay. And that's actually how it works. That's how the demo I did worked. Then we did a bit of cleanup. We did also to release the pointers to the object. I did not put that in this code but we did to do that, there are methods for that. But we can also quickly just disable everything. So first thing we do, we clean up the method we added on process by doing delete process of batch listeners. Then we evaluate require inspector.close which will disable the debugger on the process. And we close our session to the debugger. And that's it. We did it, that's how we dynamically injected logs.
6. Remote Debugger for Node.js
You can remotely enable the debugger for Node.js, which is really useful for debugging memory leaks in production. Visit the Stream blog for more information on this topic.
And that's so cool. Okay, that's cool because I say it's cool because I did it and I'm very not modest. So what did we learn today? You can remotely enable the debugger for Node.js. That's actually really useful. If you want to learn more of that, go to the Stream blog. I wrote an article on how this is useful to debug memory leaks in production because you can enable debugger in production. Then you can tune all the ports between the debugger and your local host and start to collect memory heap dumps remotely on a running production process. Same thing for CPU, which is the next article I write on my Stream blog.
7. The Power of Dev Tools
With the DevTool protocol, you can do whatever you want and change pretty much everything within a Node.js process. This requires local access to the machine running the process. You can build your own tools if the existing ones don't meet your needs. The talk emphasizes the power of dev tools and encourages their usage.
So really, really, it's really useful to debug things when it's hard to change the content of the process but you still have an SSH access to where the process is. With the DevTool protocol, you can do whatever you want. You don't need to be limited by the debugging tools. You can use the DevTool protocols to change pretty much everything within a Node.js process.
This requires, of course, local access to the machine running the process. So it doesn't add any security threat because you already have administrative access to the machine running the process. And sometimes the tools you need are not exposed in the big tools, in the Chrome interface or in your IDE or in VS Code. But you can build your own. If you need a very precise CPU profile, you can build your own. If you need to do some special stuff, like I did in this talk, you can build your own. That was not a talk about code injection. It was a talk to show you awesome and powerful the dev tools are and to encourage you to use them.
8. Q&A and Production Instrumentation
Thanks for your attention. Follow me on Twitter at Paul Defeats for the slides. Contact me at vladatscreen.com. We provide visibility into the process and block bad actors. No easy way to lock down debugger mode in production. Method requires local machine access. Remote debugging is used by some companies and for debugging memory leaks on Heroku. Chrome Debug Protocol not widely used for production instrumentation.
Thanks so much for your attention. You've been an amazing crowd. I will share the slides on Twitter today. So probably the easiest way to keep in touch and get the slides is to follow me on Twitter, at Paul Defeats. And you can obviously contact me directly through my professional email address, vladatscreen.com. Once again, if you have a running application, you want to check what we do, we will give you visibility about what happens in the process and we will block bad actors for you. It's really exciting technology. Thanks a lot and have a great day.
Hey. Hello, hello. How you doing today Vladimir? I'm good. That was intense, but excited about the Q&A now. Yeah, lots of stuff going on. And we actually have tons of people interacting in the chat, asking questions. I did see a good one from the conco. They asked, is there an easy way to lock down debugger mode in production to prevent attacks? Not that I am aware of. What you actually, in order to use what I've described as an attack, you need what's called an adjacent local access. So you will need to have access to the local machine to send the signal. So first of all, you want to prevent people to get SSH access to your production machine. And if they already have SSH access to your production machine, they can do much bigger things than what's described in this talk. So I would say that the first thing to do is, well, to have a good firewall and to make sure that nobody has access to your production machines. The method shown here are not actually available remotely because the first part of the method is to send a local system signal from another process to the node process.
Interesting, interesting. And Vlad, somebody else asked, is this method already used in order to instrument production applications? That's actually a great question. I'm aware of a company doing remote debugging and actually their feature is they give you a remote debugger over the internet in your production application. Also, this method is also used if you follow the screen blogs, as a blog post where I explain how to use part of this method to debug memory leaks in production on Heroku. And I'm aware that a few projects have been using this method to do that because they sent me messages to follow up on that. Regarding real production instrumentation, well, out of developer experience, you don't want it to be remote. And as far as I know, no actor currently on the market use the Chrome Debug Protocol to instrument applications. This might change in the close future because I'm still exploring that deeply because this talk is like cutting edge experiments.
9. Running and Security Implications
Nobody has been trained to go into that direction before. I'm still working on it and maybe in six months. In the Discord chat, Dan G asks if they can run this themselves and if there is a step-by-step blog article. I published the same content as the conference talk on ScreenBlog one month ago. There's also the question of performance impact, which should not be significant as you connect to the process, monkey patch it, and disconnect. However, the impact on the process state and the performance of Node.js instrumentation with ESM loader needs further investigation. Crowd-sourcing performance measurements is encouraged. ZeroCool asks about security implications, but the injection requires access to the server, which should be prevented.
And as far as I know, nobody has been trained to go into that direction before. So I'm still working on that and maybe in six months. Yeah. We'll keep an eye out for that. That sounds really exciting.
In the Discord chat, Dan G asks, first of all, he says great talk and applause. They ask, can we run this ourselves? Is there a blog article we can go through step-by-step in our own time? That's a great question. And actually, if you go into ScreenBlog, I published exactly the same content as the conference talk on the blog one month ago. So yeah, just go to ScreenBlog and check the latest Node.js article. It has the same content globally. And I just shared the repository for this one in the Q&A channel on Discord.
That's awesome. I love that. Like crowdsource help but you know, Vladimir does step one, y'all jump in. We also have a question. ZeroCool was asking, this was funny because they asked it and then I think they might have responded themselves but I'm curious to see what you'll say. ZeroCool asks, so does this have any security implications? And then they said, I guess not really because for injecting something into my note process someone would first need to gain access to my server which I want to prevent anyway, right? Exactly.
10. Debugging Techniques and Slido Results
The method can be used for malicious purposes, but it requires significant access and indicates a major breach. It's important to have knowledge of debugging techniques, even if you hope you never need to use them. Understanding the protocol and having the tools to level up can save time and provide a starting point for solving big problems. The series of talks aims to provide this knowledge. CPU profiling will be the topic of the next talk. The Slido results were unexpected, with only 41% of people using logs to detect security attacks. This highlights the importance of having awareness and monitoring in place.
So that's the beauty of this method is that if someone can use it for malicious purpose well that means that they are already enough access and you are anyway breached big time. So that's like a cherry on a big cake of hacking but it's still only the cherry. Love it, love it.
Is there anything else you think I don't know you'd like to add or you think would help folks if they're gonna start exploring this on their own? So, yeah, I saw that people shared in the Discord my fork of the Chrome remote debug interface. That's actually a really good module to start exploring remote debugging of Node.js or of Chrome in general. That's pretty much how Pupyter works under the hood. And actually I'm pretty sure that 90% of people won't need it. And it's my main point of evangelization regarding the debugging techniques is that when I expose them, I hope you never need them. So when I did a talk last year about how to remotely debug memory leaks, I just hope you never need to know it. And this one is even the next level. It's how do you debug stuff when the debug tools provided by Chrome don't give you what you need and you have to understand the protocol. And just like it's important that you know that it exists. It's important that if you want to hack with it, you hack with it. But in a daily life of a usual web developer, hopefully you won't have to do that. But when you have big problems, it's important that you have the first point of direction that you know that it is possible because that will save you big time. That will save you the first full week of trying to understand if what you want to do is doable. Yes it is, the tools needs you to level up on it. But at least you have the first pointer where to go. And that's pretty much where I'm going with this series of talk.
Okay, okay. Yeah, it's one of those things that you hope you never have to know, but it's always good to have in your back pocket. Exactly, so I'll be working on CPU profiling after that in term of a talk. And that's once again, something you need to know somehow to do, but hopefully you don't need to dig into that. Right, right. Hey Vlad, before we let you go, what did you think of the Slido results? It was, what was it? 41% of people use logs to know if their security was attacked. That's interesting. I was not expecting that. I was expecting either 80% people saying that they use log, either 20%. I was expecting like a big shift. So I think what it shows is that more than half of the people, 60% of the people, 50% because I remove the 9% of I have a security tool don't have any idea of their server being attacked.
11. Closing Remarks and Gratitude
Maybe you need to take a look at different solutions for protecting and monitoring applications in production. We've run out of time, but it was an amazing talk. Thank you for your time.
So maybe you need to take a look at different solutions about protecting applications and monitoring applications in production. Right, right. Maybe they need to re-watch your talk. That was awesome. It was awesome. I'm sorry about that Vlad, I was listening to the voices in my ear. I have the same, don't worry.
It looks like we've ran out of time, but we are so, so, so glad to have you. And that was an amazing talk. Thank you so much for all your time. Thanks so much for having me.