Can You Change the Behavior of a Running Node.js Process From the Outside?

Hello everyone and welcome to my talk entitled Can you change the behavior of a running Node.js process from the outside? The thing is it's a very weird talk and there will be a lot of things that don't make any sense inside it. So please stay till the end. It will all come clean at one point and I wrote this sober that's a promise i make.

So before we start a couple words about myself. I'm Vladimir Dutourkem you can call me Vlad. I'm a Node.js collaborator. I've been having Node.js commit writes for more than three years now but at my day job I am a customer engineer at a startup named Screen. We do application security. Basically we give you a way to secure your production servers without changing your code. So if you have any web application running live on the internet you probably want to check what we do and don't worry we have a free plan.

So what's the plan today? What's the plan of this talk? Let's take a simple hello world server in Node.js. So we require HTTP and we answer it with OK to each request. So far so good. But it has no logs. Well yes it has no logs because the developer was too lazy to add them and the developer in this case is me. There is an easy solution for that. You change the code and you redeploy. And that's what normal people would do. So on line three you can see now that there are logs that have been added to the source code of the application, which is great. But that's not what we're going to do today. Because we are a bit crazy and we are a bit experienced with stuff, we are changing the behavior of the process at runtime and inject a logger in the Node.js process at runtime. So how to do that remotely? And because I am brave, I will start by a live demo and then I will explain what happened in this demo. So let's open WebStorm, here we've got our simple server and let's start it. Let's go to first shell, node server.js, and you can see I did not pass any flag to the process, it's really just started. So if I do a couple requests on it, curl HTTP local host 8080 slash, it responds with OK and there is nothing logged here. Now let's do a bit of magic that I will explain later, cqsl1 pid, we check the logs, it's passed in debug mode as you can see. So that's the first thing I will need to explain later. And now we just run another process, another node.js process named injector, nothing changed on the log here, but when I do curls on this side, well we've got logs in the server. Now it logs stuff. I can even make it log whatever I want because I decided to log the URL.

So now it really logs the URL of the request we do. And that would work for any URL because what we injected is actually console.log, rec.method, rec.url. Okay, now I guess the legitimate question is how does that happen? How did that work? And that's what we will spend the next 15 minutes going through together.

So, we've got this running process. It's a simple hello world server, as I said, and it just answers with okay. So, first thing we need to do is to change its state to debug mode, so we can connect with a debugger. So, there's this cool thing in Node.js is that if you send the system signal sigusr1, then it enables the debugger on the process. And then you can connect with that through a debugger on a web circuit. So, the only change I do to the code now is I make it log the PID of the process. That's not necessary. We could use the PS command on the top command to find it, but in that case, I am a bit lazy. I mean, I was already too lazy to write logs, so I am too lazy to find the PID of a running process, so we just log that. Then when we do the kill-usr1 and the PID, it changes the state of the process. So, the kill command on Unix systems is not only to kill processes, but can also be used to send message signals. In our case, we send the sig-usr1 message signal and we pass the PID of the target process as a last argument.

That's what happened when we started to log process running and when we did this command in another shell, we logged DebuggerListName on websocket and then the url. This can also be done programmatically from another node.js process using, you guessed it, process.kill. Process.kill has a syntax where it accepts a signal and a PID and sends a signal to another process. So, ta-da, we actually remotely started the debugger, which is actually very, very useful, but I will explain that later. So, if we go to chrome __inspect, we actually see that the target is available to be debugged. That's the way you can check how many nodejs and chrome tabs are available for debug locally or remotely by going on chrome //inspect on chromium or chrome. Okay.

So, let's not use that. We could be a normal person and use the chrome dev tool and only play with JavaScript stuff. We would, for instance, monkey patch method on the event emitter to identify an event emitter that sends the request event and based on that, we will monkey patch this emitter because we identified an instance of HTTP dot server and that's actually totally doable with the JavaScript shell of the dev tools. But let's not do that. Let's actually learn to use lower level tools and perform that magic of the JavaScript code. So, have you heard of the dev tool protocol? It's actually what is used by any Node.js debugger and also by Chrome debuggers, whether they are remote or local. For instance, when you debug Chrome on an Android phone, that's the protocol you use. When you use a Pupator module to have a headless Chrome control, that's the protocol you use.

And, of course, when you use debug tools for Node.js, that's the protocol that is used under the hood. It's actually pretty well documented, and you can find the documentation on chromedevtools.github.io. It's a cool reading, because it teaches you about how things work under the hood, but that's actually the goal of this talk.

So, let's use that, because it sounds awesome, and it actually is. So, we will just run this script, and I'm pretty sure that in the few seconds I showed it, you are all able to know what it does. That was a joke, this picture is way too small. Let's go back to it just quickly.

On line two, I require a module that is named the chrome remote interface, which is actually the module we will use to interface with the Chrome Devub protocol. But once again, this is a way too small image for us to read it. Let's go with zooming in. On this slide, we start on line one by using the Chrome DevTool protocol and connect a client to the port 9 to 29. That's the default port for Node.js debugging and that's where by default the process will start. That can be configured on the Node.js side, but in that case, let's go with that. On line three, we do something really important. We enable the runtime domain. So in the Chrome DevTool protocol, there are multiple domains. There is a domain called runtime, there is a domain called debugger, there is a domain called CPU, there is a domain called memory and I'm sure you can guess what they do by their name. Before you use the method of a domain, you need to enable it to tell the V8 engine that you are going to use methods from this domain. So we start by doing that.

Now on line four, what we do is we evaluate an arbitrary script on the remote Node.js process. Okay, so let's go through that. We call a method named runtime.evaluate, that gives us the ability to run arbitrary JS code remotely. We pass a first argument that is expression. Expression actually contains JavaScript code that will be executed. In our case, we execute what's on line five, require HTTP.server.prototype. So we want this instruction to return the prototype of the HTTP.server class. We pass a parameter that is important on line six, that is include command line API. That actually tells the debugger that we want to have access to everything that is accessible on the REPL level. Otherwise, the require method won't be available because it's available in Node.js in certain scopes. So that's why we really, really, really need this argument to be passed.

The thing it returns is not actually the prototype object. It's an object ID. It's a pointer. It's a string. That is a memory address for the prototype of the server class. Meaning that it would be impossible for this remote Node.js process to return us with a real JavaScript object. It just creates a new variable pointing to it and gives us a way to retrieve this variable. I will call it a pointer or an object ID in this talk.

The thing we do on line 9 is we do query objects. That's a method that queries all the objects in the heap that have the prototype past as parameter. So on line 10, we've got a field prototyped object ID, and we pass the result of the previous call. In the previous call, we've got a pointer to the prototype of the server class. On this call, we take this pointer, give it back to V8, and tell it, give us all the objects that have this as a prototype. And we get a result of it, which is actually a pointer to an array, to a list. Once again, we've got an object ID pointing to a list for which each element is actually an instance of server, of HTTP.SERVER.

So on line 12, what we do is we ask for the list of properties of this list. It will have properties such as length, includes, but it will also have properties named 0, 1, 2, 3 that are actually the elements in this array, in this list. Now, because we know that in this code, there is only one instance of HTTP.SERVER, we know it will be on the first element of the list at the zero part. However, if we add more than that, we could introspect to find the one we want. So, we put SERVER instance which contains the object ID of our instance of HTTPSERVER. And that's already pretty cool. We have access to the single instance of HTTP.SERVER in this process.

Okay. So now let's evaluate another script. That's what we have in line 3. process the patch listeners equals require to inject the JS patch listeners. Basically, I was too lazy to write the whole function there. So what I did is I put the function in the module and I asked Node to load it. I put that on process because on our next call there is no way to have access to require, but there is access to process because process is available everywhere in JavaScript in a Node.js process. So what we do is we just put a global function that we will use in our next call.

That's as easy as that. I will show you this function on the next slide. On line 7, we call the method call function on. This method is actually really powerful. First, you pass it an object ID, a pointer as parameter, and it will call the function declared in function declaration with the value of this bound to the object ID, the object pointed by the object ID. So basically, since server instance contains a pointer to instance of HTTP.server, the function declared on line nine is actually called with this being the value of the server.

Okay, so far we are basically just calling a method on the instance of HTTP server. So, what is this method? What's inside to inject? Well, it's just a small function that does the following. It takes an emitter as a parameter, so it's expected to be an event emitter. It gets all the listeners for the request event, that is the event fired by HTTP.server each time there is a new HTTP request in node. Then on line five, it removes all of them. So, there are no listeners on HTTP server anymore at this point. And for all of them, it replaces them by the function defined on line 11. So, function on line 11 takes requested responses parameters, logs request.method, request.URL, and then calls the original method. That is all of the listeners that have been removed before. So, what we do is we take all listeners and one by one, we wrap them with this function that logs the method at the URL. Then on line 16, we put back this listener in the same order that were on the event emitter. So, we just replace all the listener of the request event by a function that calls the original listener but logs what's getting it. Okay. And that's actually how it works. That's how the demo I did worked. Then we did a bit of cleanup. We did also to release the pointers to the object. I did not put that in this code but we did to do that, there are methods for that. But we can also quickly just disable everything. So first thing we do, we clean up the method we added on process by doing delete process of batch listeners. Then we evaluate require inspector.close which will disable the debugger on the process. And we close our session to the debugger. And that's it. We did it, that's how we dynamically injected logs.

And that's so cool. Okay, that's cool because I say it's cool because I did it and I'm very not modest. So what did we learn today? You can remotely enable the debugger for Node.js. That's actually really useful. If you want to learn more of that, go to the Stream blog. I wrote an article on how this is useful to debug memory leaks in production because you can enable debugger in production. Then you can tune all the ports between the debugger and your local host and start to collect memory heap dumps remotely on a running production process. Same thing for CPU, which is the next article I write on my Stream blog.

So really, really, it's really useful to debug things when it's hard to change the content of the process but you still have an SSH access to where the process is. With the DevTool protocol, you can do whatever you want. You don't need to be limited by the debugging tools. You can use the DevTool protocols to change pretty much everything within a Node.js process.

This requires, of course, local access to the machine running the process. So it doesn't add any security threat because you already have administrative access to the machine running the process. And sometimes the tools you need are not exposed in the big tools, in the Chrome interface or in your IDE or in VS Code. But you can build your own. If you need a very precise CPU profile, you can build your own. If you need to do some special stuff, like I did in this talk, you can build your own. That was not a talk about code injection. It was a talk to show you awesome and powerful the dev tools are and to encourage you to use them.

Thanks so much for your attention. You've been an amazing crowd. I will share the slides on Twitter today. So probably the easiest way to keep in touch and get the slides is to follow me on Twitter, at Paul Defeats. And you can obviously contact me directly through my professional email address, vladatscreen.com. Once again, if you have a running application, you want to check what we do, we will give you visibility about what happens in the process and we will block bad actors for you. It's really exciting technology. Thanks a lot and have a great day.

Hey. Hello, hello. How you doing today Vladimir? I'm good. That was intense, but excited about the Q&A now. Yeah, lots of stuff going on. And we actually have tons of people interacting in the chat, asking questions. I did see a good one from the conco. They asked, is there an easy way to lock down debugger mode in production to prevent attacks? Not that I am aware of. What you actually, in order to use what I've described as an attack, you need what's called an adjacent local access. So you will need to have access to the local machine to send the signal. So first of all, you want to prevent people to get SSH access to your production machine. And if they already have SSH access to your production machine, they can do much bigger things than what's described in this talk. So I would say that the first thing to do is, well, to have a good firewall and to make sure that nobody has access to your production machines. The method shown here are not actually available remotely because the first part of the method is to send a local system signal from another process to the node process.

Interesting, interesting. And Vlad, somebody else asked, is this method already used in order to instrument production applications? That's actually a great question. I'm aware of a company doing remote debugging and actually their feature is they give you a remote debugger over the internet in your production application. Also, this method is also used if you follow the screen blogs, as a blog post where I explain how to use part of this method to debug memory leaks in production on Heroku. And I'm aware that a few projects have been using this method to do that because they sent me messages to follow up on that. Regarding real production instrumentation, well, out of developer experience, you don't want it to be remote. And as far as I know, no actor currently on the market use the Chrome Debug Protocol to instrument applications. This might change in the close future because I'm still exploring that deeply because this talk is like cutting edge experiments.

And as far as I know, nobody has been trained to go into that direction before. So I'm still working on that and maybe in six months. Yeah. We'll keep an eye out for that. That sounds really exciting.

In the Discord chat, Dan G asks, first of all, he says great talk and applause. They ask, can we run this ourselves? Is there a blog article we can go through step-by-step in our own time? That's a great question. And actually, if you go into ScreenBlog, I published exactly the same content as the conference talk on the blog one month ago. So yeah, just go to ScreenBlog and check the latest Node.js article. It has the same content globally. And I just shared the repository for this one in the Q&A channel on Discord.

Yeah, awesome. Yeah, I did see a few people asking for the repo. You're sure to get some stars right now. Vladimir, there's also the question, does this have a performance impact? That's a good question. And the answer is pretty much it should not because you connect to the process. You go, you monkey patch what you need to monkey patch into it. And then you get away from it, you disconnect and you let the JavaScript VM run its life. And I'm not 100% sure that a process state is not impacted by having been debugged. And that's pretty much what I want to dig on in the close future to see if that's an acceptable method of instrumentation because that's maybe a solution to circle around the issue related to Node.js instrumentation when people use ESM loader, when you use ESM imports because right now as a loader interface might not be the exact best way to do that so that's why I'm exploring that. But I need to compute the performance impact and I don't have numbers yet. So if you have a bit of time to dig into that, circle it back with me. I'd be happy to have your numbers. In terms of performance crowdsourcing number is the best way to get accurate measurements.

That's awesome. I love that. Like crowdsource help but you know, Vladimir does step one, y'all jump in. We also have a question. ZeroCool was asking, this was funny because they asked it and then I think they might have responded themselves but I'm curious to see what you'll say. ZeroCool asks, so does this have any security implications? And then they said, I guess not really because for injecting something into my note process someone would first need to gain access to my server which I want to prevent anyway, right? Exactly.

So that's the beauty of this method is that if someone can use it for malicious purpose well that means that they are already enough access and you are anyway breached big time. So that's like a cherry on a big cake of hacking but it's still only the cherry. Love it, love it.

Is there anything else you think I don't know you'd like to add or you think would help folks if they're gonna start exploring this on their own? So, yeah, I saw that people shared in the Discord my fork of the Chrome remote debug interface. That's actually a really good module to start exploring remote debugging of Node.js or of Chrome in general. That's pretty much how Pupyter works under the hood. And actually I'm pretty sure that 90% of people won't need it. And it's my main point of evangelization regarding the debugging techniques is that when I expose them, I hope you never need them. So when I did a talk last year about how to remotely debug memory leaks, I just hope you never need to know it. And this one is even the next level. It's how do you debug stuff when the debug tools provided by Chrome don't give you what you need and you have to understand the protocol. And just like it's important that you know that it exists. It's important that if you want to hack with it, you hack with it. But in a daily life of a usual web developer, hopefully you won't have to do that. But when you have big problems, it's important that you have the first point of direction that you know that it is possible because that will save you big time. That will save you the first full week of trying to understand if what you want to do is doable. Yes it is, the tools needs you to level up on it. But at least you have the first pointer where to go. And that's pretty much where I'm going with this series of talk.

Okay, okay. Yeah, it's one of those things that you hope you never have to know, but it's always good to have in your back pocket. Exactly, so I'll be working on CPU profiling after that in term of a talk. And that's once again, something you need to know somehow to do, but hopefully you don't need to dig into that. Right, right. Hey Vlad, before we let you go, what did you think of the Slido results? It was, what was it? 41% of people use logs to know if their security was attacked. That's interesting. I was not expecting that. I was expecting either 80% people saying that they use log, either 20%. I was expecting like a big shift. So I think what it shows is that more than half of the people, 60% of the people, 50% because I remove the 9% of I have a security tool don't have any idea of their server being attacked.

So maybe you need to take a look at different solutions about protecting applications and monitoring applications in production. Right, right. Maybe they need to re-watch your talk. That was awesome. It was awesome. I'm sorry about that Vlad, I was listening to the voices in my ear. I have the same, don't worry.

It looks like we've ran out of time, but we are so, so, so glad to have you. And that was an amazing talk. Thank you so much for all your time. Thanks so much for having me.