Monitoring, Alerting, And Visualizing your Node.JS server infrastructure with Open Source tools
AI Generated Video Summary
1. Introduction to Node.js Server Monitoring
Awesome. Thank you so much for the introduction. I have to admit, the video kind of startled me, I didn't expect the noise. So we're going to go ahead and get started here. I'm going to be going over monitoring, alerting, and visualizing your Node.js server infrastructure with open source tooling.
So let's go ahead. We already got a pretty good introduction. But if you guys want to add me on LinkedIn, if you don't want to ask questions here, which I don't know why you wouldn't, because it's all very, what's the word, you can just ask them and not have to attach your name to them. Yeah, so it's really quite great. But otherwise, you're welcome to connect with me there.
2. Time Series Applications and Server Monitoring
You can think of your Fitbit or Health Tracker. That's a time series device that you just happen to wear. Your thermostat at home is most likely streaming time series data somewhere, solar panels on your house. Nowadays in this world we have a lot more time series applications, and nowadays also server monitoring has become a lot more important.
They say the average user will only wait about three seconds for a website to load. I swear I only wait two. We expect everything to work perfectly, and we don't like when things are down. Nowadays, server monitoring has become a lot more important.
One thing to take note, which I already kind of explained here, but that doesn't mean that you can't use other DBs for your server monitoring data. That's not the case at all. There's plenty of databases that you can choose from, but for the most part, we see other DBs as good for other types of data.
This is just a few listed examples here. As you guys know, you're probably using Elastic if you have any searching on your website. You might use Mongo for documents. Of course, we all use SQL for, I would say, about 90% of data in the World Wide Web is actually stored in, yeah, a SQL-type DB. These are all DBs that offer some form of open source as well.
3. Platform Features and Data Acquisition
The platform allows you to collect, transform, downsample, trigger, and alert. Telegraph is a YAML file that runs with input and output plugins. Client libraries function like a REST API for reading, writing, deleting, and analyzing data. Visualizations can be done in a dashboard format. The task engine allows for scheduled actions and downsampling aggregates data points.
And then finally, the actual platform and what it allows us to do, which is things like collection, transformation, which we're going to briefly touch on, but not go too in-depth on, downsampling, triggering, and alerting. Now I'll kind of show some of these, but we're not going to, we don't have enough time to go into all of them, but there is tons of resources online.
So basically the idea with this as you can get the project going, you can get your data stored and then you can kind of build upon it. So these are current data acquisition methods. I've already talked pretty much about the big two. And what I'm really going to mention here is the difference between Telegraph and the difference between our client libraries. So Telegraph is basically a YAML file that runs. We have like 300 plus input plugins, a few different output plugins, say about 20 of those. But basically the reason you use it is because it's a no-code. It's basically you just change the parameters inside the YAML file and then you can go ahead and start uploading. I'll show a visualization of some of these in our UI. So it'll make a little bit more sense.
And then the client libraries, which you can just think of them almost as like a REST API. They allow you to read, to write, to delete and to just in general look at your data. And that I'll actually go into a little bit more because, at least for my project and for most people's projects, you're going to use the client libraries to actually get your data in. You could use Telegraph. There's no reason not to. I just chose to use the client library because it was what just found the easiest for my use case. These are also some of the things that you can do. So you can go ahead and do visualizations, which I'm going to show in a dashboard format. You can take action. So this task engine up here is basically like a CronJob engine. It basically allows you to say something like, every hour of the day, run this CronJob. What do is things like downsampling, which when we talk about downsampling, what we're actually referring to is the ability to take your data and aggregate it down. So that's something like the day is over. You've got all your server monitoring data, but your server has sent you data in for like every second. So you have a ton of data points. You just have millions of them basically. But the thing is, the day was pretty good. It was calm day.
Nothing really exciting happened. So what you're going to do now is you're going to run this cron job that says, give me the aggregate average for every hour of the day. So instead of having the CPU load be in this in the second range, it would give you 24 data points back of the average throughout the day. So you could still see your high times and your low times and such, but overall, it's a much smaller data packet to store, which is always good, because the smaller data packet that you store, the more in theory, if you were you know, a big client you're spending money to store it, it does save you money in the long run.
The other thing that you can see here is the check and notifications at the bottom. I know visually this can be a little hard to see. I'll show it in the UI and it will look a little bit clearer, but basically what you're doing here is you're saying, I'm going to set this threshold if the CPU load goes above this amount, go ahead and send me a Slack notification or a PagerDuty notification. You have a few different options here. We also have a HTTP endpoint, so really the Rolls Royce here with that one, you can send it really wherever you want. I for example, have a plant one at home that I monitor and I send myself a text via Twilio when it needs to be watered, which I then probably ignore of course, because why actually take advantage of it?
5. Writing Data and Retrieving with Flux Query
Basically, once you get it installed, you're good to go. When it comes to the write API, we're setting up authentication. The writing is straightforward. We're a schema-less DB, so you can start writing immediately. We're sending up a point of temperature with a tag and a value. Influx will automatically put a timestamp on it. To retrieve data, you can use a Flux query.
Basically, once you get it installed, you're good to go. I'm not going to go too much into the install. We all know how to install packages here.
So when it comes to the write API, basically, what we're doing here is we are setting up our authentication. So you need a URL, a token, or an organization and a bucket, which a bucket in this case is just another word for database. It's basically what you name your DB. And I will show how this actually looks. And I'm going to be a bad person and just show you my tokens and everything else. So it's fine, though, because I can easily ... This is all my local hosts. You can't do anything with my token anyways. But basically, I'm going to show you how you set this all up. Just keep in mind, in the future, please do keep it in a separate file when you put in these environment variables. As you can see here up at the top, it's got them in an environment MJS file instead. But basically, you get all of your authentication up and running. And then you can finally actually start writing.
So the writing is pretty straightforward here. So as you can see, one thing to note real quick here is we are a schema-less DB, which means you don't have to tell us what you intend to put in there. You can just start writing immediately, which is a little bit different because most people have to tell their database what they expect to receive in it. That being said, that has a double edged sword, the double edged sword being you could send bad data and we would write it. We do have an explicit schema option, so you could upload a schema and say, I only expect this type of data to come up, don't allow anything that doesn't match the schema. But one thing to note here is we're going ahead and we're sending up a point, a point of temperature. We have a tag for it, which in this case, the tag is called example and it's a string called write.ts. And then finally we have a value, which for this one is like a random math number because it's just an example of how to do this. And basically because we're not sending a timestamp with this, Influx will automatically put a timestamp on it for when it arrives. So you can put a timestamp, you can even put a timestamp that's in theory, like historical, like something like from two days ago if you needed to, obviously some people need to upload old data, things from CSVs and such. So that's when you're going to actually have to tell us the timestamp, but otherwise you can just start writing up data and we'll attach it ourselves.
So this one, I'm not going to go too much into because we don't need it for this project. But if you actually want to get your data back out, you can go ahead and use a Flux query to retrieve it back out. So the way Flux works is pretty, it's pretty straightforward.
6. Basic Query and Data Retrieval
This is a basic query that retrieves a range of data from a bucket. We can filter the data based on specific measurements, such as temperature. The retrieved data can be transformed into rows for use in visualization libraries like Grafana or Plotly.js.
This is just a very basic query. Basically we're saying from my bucket, give me a range of the past day. You could do it, you could give it no range. You could just say from bucket, but most people tend to give it some type of reasonable range to get their data out. And then for this one, we're filtering down on the measurement of temperature. So we're saying I maybe gave you some light values, I gave you some humidity values, but right now I just want my temperature values so I can graph just those instead. And what that's doing down here is it's actually turning them into rows. And those rows could be used inside of a visualization library. We also allow things like data frames and other such ways to get your data out. It's all meant to be compatible with other open source plotting libraries, things like Grafana or Plotly.js, et cetera.
7. Deleting Data and Monitoring Template
And then finally being able to delete data. So this one we're not again going to use in our example, but I just wanted you guys to be aware, deleting in a time series DB is not how you're used to doing it normally. You're not doing it based off a value or based off an ID. It's based off a range. So that also means that if you have data from this that you want to keep, do keep that in mind. If it's within that range, it will be deleted. So that's how you end up deleting your data. Maybe you had a bad first couple of tries. You can go ahead and say, delete for the last 30 minutes. And that's going to be that range, that start and stop.
8. Demo of Fake Express Server and Client Library
Let's go ahead and get into the demo because that's honestly the real meat of everything and that will kind of help us understand a bit better. This is my little fake express server, it's sending me back some data. I'm using the client library to get this uploaded. These are some of those telegraph plugins that I was talking about.
Let's go ahead and get into the demo because that's honestly the real meat of everything and that will kind of help us understand a bit better.
So please excuse my slightly messy code here, but basically this is my little fake express server, it's sending me back some data, it's not super exciting because it's not hosting anything. But as you can see up here, I've got my token, I've got my org ID, my bucket is called iot Center because that's how I downloaded it, and then I've got a URL because this is all running on my local host.
So as you can see in here, I've got this Get Right API, I've given it default tags, which basically automatically makes the service to be iot Center, and the host currently is basically my laptop that's what it will say inside the table. You don't have to have these default tags, but the dashboard won't work without them basically. And from there I'm using those, that point code that we talked about before, the one that's got your CPU usage, your memory usage, and your resource usage, and the overreaching one that creates the actual point value. So these are all kind of living here together.
For my Point Express, I was fighting with it too much so I just ended up doing some hard-coded values, but it will still help get the point across and we can kind of see how this ends up looking. And from there it's just running on the port 3000. This is a very, very basic little project. This is just to help us really get going. Just want to show what this looked like. So this is inside the local host. So this is what it actually looks like inside InfluxDB. Let me see if I can make this a little bit bigger. This might cause some things to get a little weird, but we're going to go with it. So let's go ahead into sources. So we are using, I'm using the client library to get this uploaded obviously. There are other client libraries that you can pick from.
And if you come here to this website and this is available, this is the open source. This is, oh dear, this is not working. That's okay, it's probably not working because we're on the open source, but it is available online. Let me really quick just find it. So it normally takes you here to the InfluxDB client JS, which has examples on how to get started as well as all of the other details. It's basically everything I already went through. But normally it would link for other ones as well. These are some of those telegraph plugins that I was talking about. I'm actually gonna make this a little smaller so they come up. Obviously we have a lot of different ones for a lot of different monitoring and just in general technologies. They're not all here in this UI, because as people add them, we don't always add them to the UI.
9. Using Community Templates and Node.js Dashboard
It takes some time to get everything in. You can go to the full list if you just type in Telegraph. This is where you actually get your community template. It's got a lot of different variety here. So we're just going to go on down here to Node.js. This is going to show the average response time, the maximum, the current heap usage, and CPU usage. This dashboard was used on one of our customers' Node.js servers, so it actually has real data.
It takes some time to get everything in. So you can go to the full list if you just type in Telegraph. And yes, this does work also with other DBs if you would prefer to use a different time series database.
Plenty of our competitors love Telegraph just as much as we do and that's what makes it so beautiful in open source.
The other thing here. So I have my bucket IoT Center and I could create other buckets here, but actually really quick. Sorry, guys, I don't mean to be so scatterbrained, but one thing to take note of is this is where you actually get your community template. So this is a page called community templates. It's got a lot of different variety here. As you can see, this one, for example, is for Apex Legends, which is like a game. So this allows people who are playing this game to kind of get their data from it and display it in a really pretty dashboard. But the one that we're using, not quite game. Let's see here. Oh, no. Oh, dear. I think I just crashed my window. Great. It wouldn't be live if I didn't just destroy everything. Sorry, guys. Let's go back. Luckily it will still reopen. Cool. So we're just going to go on down here to Node.js. So this is the one that we're going to be using, and I'm going to zoom into this so we can actually see a little bit better. So as you can see, and I'll show this also inside the UI. I'm trying to make this as like big as possible for us to kind of read it. So this is going to show the average response time, the maximum, the current heap usage. You can see some CPU usage. This dashboard was used on like one of our customers' Node.js servers, so it actually has real data that doesn't look as sad as mine does, which is mainly just a straight line because it's not doing anything. But as you can see, you can really get started on just this graph here, and basically how you do the install is you come up and you grab...
10. Node.js YAML File and Setup
You grab this Node.js YAML file, throw it in, and install it. The project comes with a monitor.js file for setup, including write process usage and setup instructions. You can change default tags and add multiple Node servers. The monitoring function is included, and there's an environment .js file for URL token and organization.
Sorry, I've got to make this a little smaller. You grab this Node.js YAML file, which basically you throw into here, you say, look up my template, please, and go ahead and install it. This might get a little bit weird because I've already got this template installed, but we're going to run with it.
All right. So the one I modified an hour ago is probably working. Yes. This is the stuff that you can see from my very own laptop. I started running it about 15 minutes ago or so. This is probably when I first got started. It had some weird spike. But as you can see, I can go ahead and just get this going automatically because I followed the instructions.
So one thing to note is that this project comes with a monitor.js file, which basically is how I got everything set up. As I said before, you can see this write process usage. You can see the setup. Like I said, this is expecting the default tags of service and host. You can change the name of these but do be aware this right here, this is where it's looking for it. So there's my MacBook. There's IoT Center. So if I had multiple Node servers running, I could go ahead and change the default tags and change it out as needed, basically.
And this one also added an if loop. Apparently, that's for an older version of Node, basically. And down here, what it's doing is it's doing the monitoring function down here in a function that you can export out. Again, I just did mine hardcoded because this is just a very small little project. But obviously, this is very helpful.
And then one thing to note. They do it properly. They make an environment .js file. This is where you should normally put your URL token and organization so you don't accidentally end up sending them up to GitHub because I've never done that. I'm responsible. And also this is where you find your URL.
11. Setting Up Project and Token Creation
To get this project up and running, set the localhost to 8086. For token creation, use an all-access token for local development, but for a more serious setup, create a custom token with specific bucket permissions.
For me, it's localhost 8086. Your org is the ID right here after the org. You can find it within the UI, but I just find it fast to just get it from here. And then for token creation, you just come right here. You can do an all-access token, which gives you all-access to all of your buckets. Because I'm just on my local host, I don't care, so I just did an all-access. But if you're using this in a more serious manner, please do make sure that you actually do a custom one and you just give it the permissions for just the buckets that it should have so we don't end up with random permissions everywhere. But, yeah. So, this is basically how you get this project up and going.
Resources, Q&A, and Visualization Options
And let me really quick pull back up my Node deck so that way we can grab the links here. These are all the resources that I've already kind of mentioned, but really quick, the Slack. You can also check out our docs as well. We also have blogs. InfluxCBUniversity is a learn at your own pace platform. The JS Client Library and the Node.js Server Template are available. Now, let's move on to the Q&A. What other options do we have in a visualization? One other thing that you could do is you could hook us up with Grafana. Grafana has their own version of a more advanced Node.js Monitor dashboard. You could use this instead. We have a Telegraph output plugin directly to Grafana. And if you want to edit things, you can use the configure toggle and add a brand new cell here.
And let me really quick pull back up my Node deck so that way we can grab the links here. Let's scroll on down. Great. So, these are going to be where I'm going to leave this as we go into the Q&A.
As I said before, these are all the resources that I've already kind of mentioned, but really quick, the Slack. If you have any further questions or you're getting started on your own project, please feel free to come to our Slack community. We're super active. The devrels are there, our engineers are there. You can also check out our docs as well. They're pretty well-written if I do say so myself. We also have blogs. Some of the projects from that inspired some of our customers are from those blogs. And then finally, InfluxCBUniversity is a learn at your own pace platform, so if you want to learn a little bit more but you want to do it on your own, you're welcome to use that service. It's completely free. And then, obviously, the JS Client Library and the Node.js Server Template, which you can follow word for word to get this all set up.
And now I will go ahead and go to the Q&A. All right. Thanks a lot, Zoe. Awesome. Yeah, please take a seat. We are not sure how much questions we will have, but at least this one. What other options do we have in a visualization? Actually, can I stay near my laptop for one sec? So one other thing that you could do is you could hook us up with, for example, Grafana. I had the tab open, but basically Grafana has their own version of a little bit more slightly, I'm going to say advance version of this with a Node.js Monitor dashboard, very similar concept. And so you could use this instead. We have a Telegraph output plugin directly to Grafana. So it's super, super easy for setup. And so you could do that. And then the other thing that I kind of really quick forgot to show is if you want to go ahead and edit things in here, this is that toggle that I was talking about. So you can go ahead and go to the configure and you can either use for these ones they're all hard coded into Flux because that's just how they are built. But if you wanted to, you could add a brand new cell here.
Customizing and Rearranging Dashboard Graphs
You can customize and rearrange the graphs on the dashboard. Add, delete, or move them as needed. The layout can be adjusted to meet your criteria and needs.
I can go to IOT center. Let's check out our CPU usage. Go ahead and hit submit and I can get a new graph and I'm going to go ahead and name this Zoe's new cell because why not? And then I can just go ahead and add it. And you can move these around because obviously now this one's kind of a little bit weird in the way it looks. So I might go ahead and I'd reorganize a little bit better obviously. But there we go. So you can kind of change these as you'd like. You can go ahead and obviously delete or move them if they don't meet your criteria or needs. But this can definitely be expanded upon too.
Dashboard Layout and CloudWatch Integration
How can you define the layout of a dashboard? Let's see if we have CloudWatch integration. We offer a wide list of AWS-based metrics. Check out the documentation for AWS CloudWatch statistics and authentication requirements. The YAML file provides configuration options, including access key, secret key, and token setup.
While we are here, maybe another question. How can you define the layout of a dashboard? If you can show that maybe? With CloudWatch. Let's see. Let's go to Telegraph. Let's see what it's got. And let's see if we have CloudWatch because I think that is one of the integrations that we offer. And that would be very good to see.
That's right. I think the full list is somewhere down below. Sorry, guys. I'm going to pull this back a little. Actually, we'll go to the docs because they're a little bit. Man, where's the full list? I should have had this up in a tab. But then I would have deleted it anyway. Yeah. So, here's all of our output plugins, somewhere our list is. You know what, I have an even better idea. I'm just going to put in Telegraph CloudWatch. That's a great solution to my problems.
Another question, is there an integration plugin between CloudWatch and Integrate? Yeah. So, it would appear that we have an integration here and it allows you to do statistic metrics use monitoring. And as you can see, we have a wide list here that you can use to pull metrics from. Most of these are AWS-based. So, you can check out the documentation here, which will tell you the AWS CloudWatch statistics. And really, it's just going to link to the GitHub, so we'll just go there. And so, with this, you can see all the authentication you're going to need, the global configuration options. And again, this is what that YAML file looks like. So, as you can see, most of it's commented out, but you would need to, obviously, start on commenting some of these. I think, for example, all of these are double commented, so they're meant to be comments, but you're going to need to give it things like the access key, the secret key, the token. And normally, these are pretty good about telling you how to get this all set up.
Dashboard Layout and Customization
The documentation for Telegraph is pretty in-depth. There is support for metrics, traces, and events in the cloud version, with plans to bring it to the open source in the future. The layout of the dashboards can be customized by adding, deleting, or rearranging cells. It can be a bit tricky to move cells around, especially with a large number of cells. Most dashboards consist of long graphs and a few additional elements.
This one also has an example on how it would be normally set up for people. And so, obviously, these docs are pretty large. For example, this CloudWatch one, at least, is very large. Most of the docs for Telegraph are pretty in-depth, I would use the word for. So, definitely go ahead and check it out. And I'm sorry, this is what I was looking for, is the full plugin directory. So, up here you could go ahead and change depending on what you're looking for, you could also, obviously, just do a full text search like I did.
Awesome. Thanks for that. Yeah, there is another question. Do you support anything else than metrics? So, in the open source, we normally just suggest metrics. In our cloud versioning, we now have unlimited cardinality, so we do support metrics, traces and events. It will eventually be coming to the open source in the next couple of months. It's just not available just yet. Alright. Good to know. Maybe to be back on that one, yeah.
How do we define the layout for the dashboards? So, by layout, I'm going to assume that they're talking about like, so, obviously, this was a prebuilt dashboard, it came prebuilt with these cells and everything. So, when you do the download here, you can't really do any edits. You could edit from here though. So, you could start to reconfigure these or like I did, you could add some new cells or you could start to delete them. And as for moving stuff around, it tends to be kind of like a drag and drop kind of deal. It can be, I'm not going to lie, it can be a little unruly depending on how many cells you have. And in this case, because my cell was created right at the top, it kind of shifted everything a little weird. Most dashboards I see tend to be a little bit more, like they're just a lot of the long graphs basically and a few of these up at the top. I could show some other examples in the community templates, but for the most part with this one, it's just going to be right out of the box the way it is. And then if you want to go ahead and change it, you're going to have to do that just inside. So, for example, this Docker one is what I would call a little bit more on the straightforward side where all the graphs are kind of just in one four by four block, I guess you could call it. And for example, down here, they have two longer graphs. If that makes sense.
Q&A: TypeScript Support and Awesome Socks
Will you add TypeScript support? I am not sure on that question, but you should be able to use our library even if you have TypeScript installed. Thanks a lot. What are the socks on the table? I brought some really awesome socks for you guys to grab. They have a cool little database design and are great hiking socks that fit most feet.
Absolutely. I think we will have one more question. Actually, they are raising quite a lot. So, yeah, maybe the top one. Will you add TypeScript support? For example, export types? I am not sure on that question. So, with that one, I would ask it, unfortunately, in our Slack channel because I'm not sure what our friends and devs are planning. From what I understand though, you should be able to use our library even if you have TypeScript installed. It should be relatively compatible because I've used it in a few TypeScript projects with no issue.
Well, thanks a lot. Again, I have one last question. What are the socks on the table? All right. So, for everybody here who's in person, I brought some really awesome socks for you guys to grab. They're sitting up here on the table. We have two different colors, but I've got these blue ones here. Look at this awesome little database on them. These are great hiking socks and they fit on most feet, as we like to say. Great.