Learn how running your TS in Temporal's runtime gets you extreme reliability
How to build distributed systems in TypeScript
AI Generated Video Summary
This Talk discusses building distributed systems in TypeScript, focusing on an e-commerce store and implementing a create order endpoint. It covers handling retry and error cases, saving order state, handling crashes and duplicate orders, and the process order function and architecture. The Talk also introduces Temporal, a durable code execution framework that simplifies fault-tolerant software development.
1. Building Distributed Systems in TypeScript
This is how to build distributed systems in TypeScript. We will work on an e-commerce store and implement a create order endpoint. We will go through all the failure modes and try to make it more reliable. We reserve the inventory, charge the payment, and send the package. If any step fails, we handle it accordingly and consider retrying.
Hi, folks. This is how to build distributed systems in TypeScript. My name is Lauren, and I wrote a book on GraphQL called the GraphQL Guide, and currently I am working as a language runtime engineer at Temporal working on our TypeScript SDK.
So when working in a monolith, we might not have very many distributed systems concerns to be worried about. If we just have a single app server layer and a database that does transactions, then we might just be getting a request from a user, doing a single operation, and then if that fails, then we say to our user, try again. If we have, if we're talking to external APIs and they're an essential part of the business application, like I talk to Stripe and say charge this, and then I update my database, then I have some more problems, like I might not be able to reach Stripe. So I need to retry that. When I retry, I need to do so idempotently, so that I'm not double charging the user. And there are cases in which my database might be out of sync with Stripe's understanding of the world. So Stripe returns success and I can't reach the database and that operation fails or my process dies before I get to that step, then my data is inconsistent.
In this talk, we will be working on an e-commerce store and implementing a create order endpoint. And we will go through all the failure modes and try to make it more reliable, and we won't get to full reliability, but we'll see how much simpler we can make it if we write it in temporal. To get started, we have a serverless function hosted on Vercel, and we get out of the body the item ID, the quantity, and the address that you go to, and we get the user ID out of the jar. And then we take three steps whenever we have a new order. We reserve the inventory with the inventory service. Then we talk to the payment service to charge, and then we talk to the fulfilment service to send the package. And we respond success.
So in a successful case, this logic works fine, but there are a number of things that can go wrong. If we fail to reserve, then we don't want to continue charging and sending packages. If we successfully reserve and then fail to charge, then we not only don't want to send the package, but we also want to unreserve from the inventory so that someone else can buy those items. And then lastly, if reserving and charging is successful, and we get that same package, then we want to refund the charge and unreserve. So let's see what that logic looks like. Get down to here, reserve. If the reservation failed, then we say status 400. Now, when we charge and it fails, we want to talk to the inventory service again and say unreserve this amount of this item. And then finally, send package if that fails, then we want to first refund, then unreserve. Right now, when we have a failure, we send status 400 to the client. But ideally, we would be retrying each of these steps in case the failure was transient, like a network error or the service was temporarily down. But if we retry it the second time, maybe it'll go through. So let's add retrying to each of these steps.
2. Handling Retry and Error Cases
We have a retry function that handles service calls and retries them with exponential back off. We need an idempotency token to prevent duplicate calls. We also retry failure calls and catch errors in the response object.
We have a retry function that takes parameters, max attempts, a timeout in intervals. And so for each of these attempts, it tries calling the service, it races between that and the timeout. So if 30 seconds passes and we haven't heard back from the service, then we retry. And each time we retry, we are doing exponential back off. And here we're wrapping each of these service calls in the retry function.
One issue we have now as we are retrying is that we might have multiple successful calls and we need some kind of idempotency token so that the service knows not to do the second time. And the best way to do this is to get it from the client. So we'll add that. Get a request ID out of the body and pass it to each of the services.
We also want to retry these failure calls, like if the refund throws, then we'll neither refund nor will unreserve. So the customer will wind up still with the charge and the inventory is taken. So let's add those. So we're wrapping unreserve with the retry and also refund. Also, when we're speaking of throwing, we're not actually catching throwing, we're just seeing whether the response object had a failure property. So let's also be catching these errors.
3. Saving Order State and Handling Errors
We need to save the state of the order processing to a durable store and disk. We use Mongo Client to connect to the DB and update the order record at each step. We also need to handle database errors and compensation logic errors. Additionally, a worker pool is required.
So we get a reservation object if we successfully talked to the service. But if we weren't able to reach the service, then that'll throw and we catch that, save it here and then send it to the user. But what happens if this process crashes? So at this point with all these retries, we might be retrying for hours and we no longer fit in the time window of a Vercel serverless function.
And so as we have these long running processes, at some point, some of them will fail, crash because the server loses power or whatever. And we want to be able to pick back up at the same state we were in so that we can continue processing the order. And right now, the only awareness of what step we're on is in-memory of this code.
So at each step, we want to save to some durable store and a disk somewhere what state we're on, what we've done and what we have left to do. So let's do that. We've got Mongo Client and the different states that an order can be in. We connect to the DB, we insert one in the created state, we get the ID and then we use that to update the same order record after every state. Failed to reserve or reserved, failed to charge, et cetera. There are a number of other problems that we don't have time to fix. We want to make sure that the database client is retrying and we want to handle database errors and we want to handle these compensation logic errors. And we need to have a worker pool.
4. Handling Crashes and Duplicate Orders
We need a process to handle crashes and continue processing non-terminal records in the database. Temporal is a system that automatically persists and retries our code, eliminating the need for manual handling. In the Temporal system, we start a process order function using the Temporal SDK. If the request is already started, we inform the user that their order has already been submitted.
So right now, we're inside of a HTTP handler, but if this crashes, we need some other process somewhere to notice that a record in the database is in a non-terminal state. The last time it was updated was long enough ago that I know no one's still working on it, and the worker needs to pick that up and continue. We also probably want to return to the user earlier than the end of this function because all of this retry stuff might be taking minutes or hours.
I don't have time to code this. You probably don't have time to code this, and we shouldn't have to. We should be working at a level of abstraction where everything we do is automatically persisted and retried, and we don't have to worry about it. Unfortunately, that system exists, and it's called Temporal. Let's see what our app looks like in the Temporal system. Here's the new serverless function. We still get the same data out. We do only one step though, and that's client.start. This is using the Temporal SDK up here. We're starting a process order function, and we're providing it the order arguments and the request ID as sort of an ID of the function execution and also an id of potency token, and then respond that we've submitted this. If this has happened, we know that it's durably started and saved, and if we get this already started error, then we know the request was a duplicate. So we say, hey, the user's order has already been submitted.
5. Process Order Function and Architecture
We have a simple process order function with steps like reserve, charge, unreserve, and send package. These steps are implemented as activities, which handle potential failures. Each activity function is retried and timed out based on the retry policy. The process order function is persisted, allowing for continuation in the same state if the process dies. The system also supports additional features like timers and sleep. The architecture involves a temporal server, language runtimes, and SDKs.
Now let's look at this process order function. We have each step, reserve, charge. If charge fails, then we unreserve, and then we send package, and if that fails, we refund and then unreserve. So that's very simple. That's all of our logic and it works.
You might be thinking, okay, but these reserve, charge functions, et cetera, must be really complicated. Let's look at them. So these are called activities, and we use activities for anything that might fail. So all this logic doesn't have a risk of failure, but everything it calls does. This is the service reserve, unreserve. You can see it's just the basic service calls. Each of these activity functions is retried and timed out according to my retry policy and whatever options I put here, there's lots of different options. That's just all taken care of by the system.
Further, each step of this process order function is persisted. So if the process dies after this, then a new worker will continue execution in the exact same spot, in the exact same state, and any local variables. We only have order up here, but we could have lots of other ones that we change during the function. They'll all have the same values. You can also do other cool things like sleep for 30 minutes. We're doing the Amazon one click buy. We say reserve and then we set a 30 minute timer, and then we charge. And if we get canceled during those 30 minutes, then we don't charge our send package. We can also sleep for 30 days in a loop if we're trying to monthly charge someone on a subscription.
And the way this works is we have a temporal server, which is open source, and you can host it anywhere. And that does persistence. And we have language runtimes, or SDKs. And the SDK is put up into a client and a worker. And then you have a client that can be used anywhere. Right now we're using it on a serverless function. And the client talks to the temporal server, and the worker talks to the server. And the client says, start this workflow, which is this process order function.
6. Temporal: A Durable Code Execution Framework
Temporal is a durable code execution framework that allows you to write code oblivious to faults. It's a huge step forward in software development, working at a higher level of abstraction. Learn more at temporal.io.ts or contact me via email at lauren.temporal.io or on Twitter at laurenDSR.
And then the workers are constantly pulling and picking up tasks from the server. Like, okay, start this function, start this activity, et cetera. And the worker is reporting back to the server after it completes each step so the server can persist it.
So when people ask me, what's Temporal? I say it's a durable code execution framework. So we run your code in our workers and we do so durably, such that if anything goes wrong, we pick it back up in the exact same state. And so you can write your code in a way that's oblivious to faults. So that's Temporal.
I think it's a huge step forward in software development, working at a higher level of abstraction in which you don't have to worry about liability. If you'd like to learn more, check out temporal.io.ts. That's the TypeScript SDK docs. Feel free to email me, lauren.temporal.io. Hit me up on Twitter, at laurenDSR.