Vue.js Live Conference

Vue.js London Live 2021

Local State and Server Cache: Finding a Balance

Natalia Tepluhina

GraphQL Galaxy

GraphQL Galaxy 2021

GraphQL Caching Demystified

Hi everyone, I am Matteo Collina and today I'm going to talk to you about GraphQL caching. Before we start, please follow me on Twitter at Matteo Collina, you can find it on the slide. So hey, here I am. I talk a lot about Node.js, JavaScript, GraphQL, open source, all the things. So I don't know, you might find it interesting. So today we are going to talk about GraphQL. Before we start though, oh, one more thing. I follow my newsletter, Adventures in Node-land, node-land.dev. Who I am, I'm Matteo, I'm part of the Node.js Technical Steering Committee and Chief Sourcer Architect at a company called NearForm. Check us out, we are hiring, doing a lot for GraphQL. So if you want, it's a good company. Anyway, going a little bit further, back when I was a kid in the 90s, yes, I am telling how old I am. I was really, really, really impressed by this show by David Copperfield. And I don't know about you, but I always wanted to be a bit fascinated by magic, right? And how to make things disappear, how to make things fly, whatever. It's very, very interesting. And I found them very entertaining, the shows. And in fact, there is a lot of hard work behind magic, right? So in this talk, we are going to talk about magic and we are going to make things disappear. So in fact, we are going to apply magic to GraphQL. So we are going to show how to improve the performance of your GraphQL gateway by four times. How? By making things disappear. So how? Well, you know, let's talk a little bit about the tool of the craft. We need tools, right? You know, we need things that we're going to use for this little demonstration. So first of all, we're going to use Fastify. Fastify is one of the fastest web framework for Node.js. It's very easy to use. It's similar to Express, but more modern, faster. It has more features, all things that you would need. It's great. Check it out. We're going to use Mercurius. Mercurius is the GraphQL adapter that you can run on top of Fastify. It's cool. Mercurius offers a few interesting features that make it unique. First of all, it integrates with the GraphQL JIT library, producing so that we can take your query and do just-in-time compilation for your query so that it can execute way faster and so on. It also does a little bit more things like that for performance and speed reasons. It's great. So check it out. Oh, it also supports the full federation, both as a gateway or as a microservice. So last tool of the craft is a library called Autocannon. Autocannon is a tool that I wrote long ago to do some load testing. And you can use this to script things in JavaScript. So it's, you know, to script load tests in JavaScript. It's great. I use that a lot of times. So these are our tools, right? So we're going to use these. Ah, okay. Three things. So let's talk a little bit about our experiment. We have two services that are federated by a gateway. And one offers the user object and the other one offers the post object. And we are going to use the tool to configure the just-in-time compiler. And we will enable the cache for the resolver, depending on our argument. So we can run multiple experiments, right? You can see it here that you can run multiple experiments. And see the impact of this cache module. What does this module look like? So let's see where things disappear or reappear. So okay. What's the service? Well, this is an example of the service. It's a user object that has an ID and a name. Very simple, okay? It's nothing special here. So it's time for some magic. Are you ready for the magic? Let's make things disappear. So how? Well, let's go back into our terminal. So this is connected to my server. So it's running on dedicated hardware. So I'm going to do live load testing. Oh, wow. So let's look at my repo. All of this is in the Mercurius Cache repo. And we can see that we have our benchmarks. And this is the gateway that I just showed you. And we have our gateway services. This is the user, and this is the post. Note that these services are all serving the data from memory. So there's no database involved. They're really fast. And this is our bench. So how do we benchmark things using AutoCanon? So basically, we require AutoCanon. And then we have our query. And we send our query as the body with 100 concurrent connections. That's it. And then we do several experiments using our bench script. So in our bench script, you see that we are running all the stuff with multiple. We're running all the services plus multiple stuff. Multiple example, one with no cache, one with a zero second time to live, and one with one second time to live, and one with 10 seconds time to live. So let's run this script. So first of all, this is our basic. This is our control check, right? It's a gateway mode, something where we are not going to cache anything. So we have done this. And whoa, it does 3,000 requests per second. OK, seems fast or not. Depends on what you want to do. Latency is good, though. So I'm pretty happy with the results. Now with zero second TTL, whoa, how? It just bumped 4x. I'm not doing any caching. It's zero second time to live. I'm just turning it on. And it just does 4x. And whoa, still 4x. Like, how come it is possible? How does this work? OK, let's leave this running. Let's explain this a little bit in a second. So our baseline has a P99 latency. That's what you want to measure for latency of 80 milliseconds. While request per second, it gives you 3,000 more or less, 3,000 requests per second. However, I can also create flame graphs. What is a flame graph? Well, this is a representation of where our CPU time is being spent. More specifically, all that time is being spent in doing HTTP requests. By the way, if you have not seen my talks about Undici and Node.js, please check it out, because you can speed up quite a lot your HTTP calls. But the result is that the vast majority of the time is spent doing HTTP. So well, what can we do? We need to reduce the HTTP, right? Yeah, how can we improve this? Well, just by making it a 0 second time to live, we can just reduce the latency to 18 milliseconds and multiply by 4 the number of requests per second and the throughput. Whoa, this is quite an improvement for not having any caching at all. Zero caching. It's not caching at all. We just enable the cache. Well, and if we enable the cache, it does not improve much. So how come it's possible? Well, this is the flame graph of our gateway now. And as you can see in the middle, the HTTP request that was there before is gone. And now we have in the middle a huge block of time being spent doing the caching. So literally, now the bottom is the caching system. But where did the HTTP call go? Where did it disappear? Well, what we are doing, we are doing deduplication, which is a clear strategy that will make things incredibly faster, especially on the graphical side. So let's go back and talk a little bit about the Node.js event loop. You probably have seen this diagram about Node. This is great because you have seen that a request comes in and you have an event, goes into the event queue, goes processing, and then this generates more asynchronous activity. What you have not seen is this diagram. This diagram, it's a different presentation of the exact same event loop. However, it shows it from the point of view of the JavaScript function being executed. So when the event loop is running, is waiting for something, it's waiting. This is on the left and the right, the event loop is waiting. And then when an event happens, it calls into C++, it calls into JavaScript, which typically schedules some next tick or some promises, then go back to C++, which in turn kick off the promises and next tick execution. And finally, once all of that is done and settled, it goes into the event loop. It goes back and relinquishes control to the event loop. All the time it takes in between, from this starting point of the C++ to the end of it, it's the time where the event loop is blocked. So in order to do the deduplication of the request, what we are doing is, when we are receiving a resolver being executed, we can compute a cache key. And with that cache key, we can create a matching promise. So and then we can complete our execution. However, when a follow up execution comes in, we can compute the exact same cache key and get the promise that we put there before, which might be still pending. However, we don't need to execute the same resolver two times, we can only execute them once. It's pretty great. You can avoid a lot of computations this way. This is what this module does. It's called async cache dedupe. You create a new cache where you go and define some methods on it and that are asynchronous. And then automatically it caches the results. And you can have a TTL, but it automatically dedupes and caches the result. It's phenomenal. And it's really fast. So you can use this in all the other places where you want to use the system. Well, first of all, but the question now is how can we compute a good caching key? Because we want to, in order to use this module, we need to provide a cache key. And how can we do a good cache key? Well, let's look at the anatomy of our solver. When you implement a resolver in Node.js, you can have four arguments. You have the root, you have the arguments, you have the context, which the root is the current object, but then you have the arguments for the resolver, the context, which can include your Node.js request, response, the database connections, all things, and then the info object, which includes the definition of the query that you are computing. Well, take that into your mind and just wait for a second. Now, what you can do now is use this. You can create, you can take an arbitrary object and JSONify it, right? You can call JSON stringify it. If you do that, the order of, depending on the order of the properties, you will get different JSON. However, there is a module called safe-stable-stringify, which independently of the ordering of the properties, it will always generate the same JSON. So what we can do is we can use this module and combine it with the resolver anatomy, the info, the data on the info object, the root, and all those things to create a cache, a hash key for an arbitrary complex, a hash key for that specific resolver. Now, how is it implemented? Well, what you can do, as you can see here, is pretty simple. We navigate the info object to get the selection, the current field selection, and then we create an object including the current resolved object, the arguments, the fields, and some more parameters. It's pretty great, you see? We can compute a cache key for each GraphQL resolver. And this is what we call the zero-second TTF. We are deduplicating all the resolver access to your data. Adding some caching is not improving much here because the target services are mostly very easy. They are not very, they don't require a lot of traffic. They don't require a lot of CPUs to compute. They don't have a database. They don't have nothing. However, these adding more caching will change in case you need more. Adding more time here will improve your performance if the target services are not fast enough or slow or something like that. Well, all of this is very good, but in-process caches are problematic. So we can't really increase the time too much because it's all in process. So if it's all in process, if the data expires on my node, it's not expiring on the other node. So how can we implement that? Well, one of the good solutions is to use something like Redis to implement a shared state, a shared cache between all the nodes. Yeah, but we tried that. We tried that and we implemented it and it did not work. And it did not work, well, mainly because we have in our benchmark 100 graphical query per second, which each one were invoking 20 resolvers. And this turns around that if you want to fetch that data from the cache, this is 2,000 Redis get per second. And unfortunately, the round trip time of Redis, the round trip time is 0.5 milliseconds, but the actual round trip time is 15. So can't do much. We need to parallelize these Redis gets, right? So maybe we can use a connection pool or I don't know. Well, there is something better. Well, I actually solved this problem already. Yes, this was this figure. Anyway, check out this talk. This talk that I did at RedisConf 2021 explain how to solve the problem of head-of-line blocking with autopipelining. So basically, it's a technique that we have invented to, well, we have applied to the IoRedis client that enable batching of multiple commands that happens in the same event loop iteration into one single Redis pipeline. So that we are sending them as a batch, making sure that we actually cut down the round trip time, the network round trip time on the server. It's great and it works beautifully and you can have to really speed up your Redis access. But this is actually the same thing that we were doing before with the sync actually, right? So it's turtles all the way down. Happy days. So we have all of this logic in production. So it's important to say that these code in production is giving us an improvement of 100 times in terms of number of requests per second. And it's having 15 times expansion factor on Redis. So for each complex query that we receive on average, we are doing 15 Redis gets with different cache keys to verify if things works as we would like to. It's pretty great, right? But it's also quite scary. By the way, Redis is not even flicking an eye. It's not even blinking an eye. It's perfectly fine with all this traffic. So we don't care. Redis is amazing, by the way. Go use Redis. More Redis for everybody. So those are our real life stuff. This technique has been a phenomenal life saver recently. So we were able to handle a huge peak of traffic without even blinking. So yeah, check it out. It's great. So however, there are two things in computer science, right? One is naming things, and the other is caching validations. Oh, come on. OK, that's really bad, right? Because we haven't talked about how we validate the cache. And this is one of the fundamental topics. However, we are almost at the 20 minutes mark. So I've run out of time. So I'm not going to cover them in this talk. I'm joking. We have not finished working. We have not finished the implementation of this module. But we are actually working on this. So in reality, you will be able to have will be adding them to a sync-hgdup soon. So you'll be able to invalidate the cache both locally and on Redis sooner rather than later. So check it out, because we are going to watch on my news on Twitter and on my newsletter, because there will be some good announcements in the coming weeks. So with that, I just wanted to say thank you. As I said, I am Matteo Collina. I am Chief Software Architect at Mirafone. You can find me on Twitter at Matteo Collina. Please ask me any question you want on Twitter. And I will be very happy to respond as soon as I can. So thank you for watching this talk. Bye. Music


#server cache