How to build distributed systems in TypeScript

Rate this content
Bookmark

Learn how running your TS in Temporal's runtime gets you extreme reliability

10 min
29 Apr, 2022

Video Summary and Transcription

This Talk discusses building distributed systems in TypeScript, focusing on an e-commerce store and implementing a create order endpoint. It covers handling retry and error cases, saving order state, handling crashes and duplicate orders, and the process order function and architecture. The Talk also introduces Temporal, a durable code execution framework that simplifies fault-tolerant software development.

Available in Español

1. Building Distributed Systems in TypeScript

Short description:

This is how to build distributed systems in TypeScript. We will work on an e-commerce store and implement a create order endpoint. We will go through all the failure modes and try to make it more reliable. We reserve the inventory, charge the payment, and send the package. If any step fails, we handle it accordingly and consider retrying.

Hi, folks. This is how to build distributed systems in TypeScript. My name is Lauren, and I wrote a book on GraphQL called the GraphQL Guide, and currently I am working as a language runtime engineer at Temporal working on our TypeScript SDK.

So when working in a monolith, we might not have very many distributed systems concerns to be worried about. If we just have a single app server layer and a database that does transactions, then we might just be getting a request from a user, doing a single operation, and then if that fails, then we say to our user, try again. If we have, if we're talking to external APIs and they're an essential part of the business application, like I talk to Stripe and say charge this, and then I update my database, then I have some more problems, like I might not be able to reach Stripe. So I need to retry that. When I retry, I need to do so idempotently, so that I'm not double charging the user. And there are cases in which my database might be out of sync with Stripe's understanding of the world. So Stripe returns success and I can't reach the database and that operation fails or my process dies before I get to that step, then my data is inconsistent.

In this talk, we will be working on an e-commerce store and implementing a create order endpoint. And we will go through all the failure modes and try to make it more reliable, and we won't get to full reliability, but we'll see how much simpler we can make it if we write it in temporal. To get started, we have a serverless function hosted on Vercel, and we get out of the body the item ID, the quantity, and the address that you go to, and we get the user ID out of the jar. And then we take three steps whenever we have a new order. We reserve the inventory with the inventory service. Then we talk to the payment service to charge, and then we talk to the fulfilment service to send the package. And we respond success.

So in a successful case, this logic works fine, but there are a number of things that can go wrong. If we fail to reserve, then we don't want to continue charging and sending packages. If we successfully reserve and then fail to charge, then we not only don't want to send the package, but we also want to unreserve from the inventory so that someone else can buy those items. And then lastly, if reserving and charging is successful, and we get that same package, then we want to refund the charge and unreserve. So let's see what that logic looks like. Get down to here, reserve. If the reservation failed, then we say status 400. Now, when we charge and it fails, we want to talk to the inventory service again and say unreserve this amount of this item. And then finally, send package if that fails, then we want to first refund, then unreserve. Right now, when we have a failure, we send status 400 to the client. But ideally, we would be retrying each of these steps in case the failure was transient, like a network error or the service was temporarily down. But if we retry it the second time, maybe it'll go through. So let's add retrying to each of these steps.

2. Handling Retry and Error Cases

Short description:

We have a retry function that handles service calls and retries them with exponential back off. We need an idempotency token to prevent duplicate calls. We also retry failure calls and catch errors in the response object.

We have a retry function that takes parameters, max attempts, a timeout in intervals. And so for each of these attempts, it tries calling the service, it races between that and the timeout. So if 30 seconds passes and we haven't heard back from the service, then we retry. And each time we retry, we are doing exponential back off. And here we're wrapping each of these service calls in the retry function.

One issue we have now as we are retrying is that we might have multiple successful calls and we need some kind of idempotency token so that the service knows not to do the second time. And the best way to do this is to get it from the client. So we'll add that. Get a request ID out of the body and pass it to each of the services.

We also want to retry these failure calls, like if the refund throws, then we'll neither refund nor will unreserve. So the customer will wind up still with the charge and the inventory is taken. So let's add those. So we're wrapping unreserve with the retry and also refund. Also, when we're speaking of throwing, we're not actually catching throwing, we're just seeing whether the response object had a failure property. So let's also be catching these errors.

3. Saving Order State and Handling Errors

Short description:

We need to save the state of the order processing to a durable store and disk. We use Mongo Client to connect to the DB and update the order record at each step. We also need to handle database errors and compensation logic errors. Additionally, a worker pool is required.

So we get a reservation object if we successfully talked to the service. But if we weren't able to reach the service, then that'll throw and we catch that, save it here and then send it to the user. But what happens if this process crashes? So at this point with all these retries, we might be retrying for hours and we no longer fit in the time window of a Vercel serverless function.

And so as we have these long running processes, at some point, some of them will fail, crash because the server loses power or whatever. And we want to be able to pick back up at the same state we were in so that we can continue processing the order. And right now, the only awareness of what step we're on is in-memory of this code.

So at each step, we want to save to some durable store and a disk somewhere what state we're on, what we've done and what we have left to do. So let's do that. We've got Mongo Client and the different states that an order can be in. We connect to the DB, we insert one in the created state, we get the ID and then we use that to update the same order record after every state. Failed to reserve or reserved, failed to charge, et cetera. There are a number of other problems that we don't have time to fix. We want to make sure that the database client is retrying and we want to handle database errors and we want to handle these compensation logic errors. And we need to have a worker pool.

4. Handling Crashes and Duplicate Orders

Short description:

We need a process to handle crashes and continue processing non-terminal records in the database. Temporal is a system that automatically persists and retries our code, eliminating the need for manual handling. In the Temporal system, we start a process order function using the Temporal SDK. If the request is already started, we inform the user that their order has already been submitted.

So right now, we're inside of a HTTP handler, but if this crashes, we need some other process somewhere to notice that a record in the database is in a non-terminal state. The last time it was updated was long enough ago that I know no one's still working on it, and the worker needs to pick that up and continue. We also probably want to return to the user earlier than the end of this function because all of this retry stuff might be taking minutes or hours.

I don't have time to code this. You probably don't have time to code this, and we shouldn't have to. We should be working at a level of abstraction where everything we do is automatically persisted and retried, and we don't have to worry about it. Unfortunately, that system exists, and it's called Temporal. Let's see what our app looks like in the Temporal system. Here's the new serverless function. We still get the same data out. We do only one step though, and that's client.start. This is using the Temporal SDK up here. We're starting a process order function, and we're providing it the order arguments and the request ID as sort of an ID of the function execution and also an id of potency token, and then respond that we've submitted this. If this has happened, we know that it's durably started and saved, and if we get this already started error, then we know the request was a duplicate. So we say, hey, the user's order has already been submitted.

5. Process Order Function and Architecture

Short description:

We have a simple process order function with steps like reserve, charge, unreserve, and send package. These steps are implemented as activities, which handle potential failures. Each activity function is retried and timed out based on the retry policy. The process order function is persisted, allowing for continuation in the same state if the process dies. The system also supports additional features like timers and sleep. The architecture involves a temporal server, language runtimes, and SDKs.

Now let's look at this process order function. We have each step, reserve, charge. If charge fails, then we unreserve, and then we send package, and if that fails, we refund and then unreserve. So that's very simple. That's all of our logic and it works.

You might be thinking, okay, but these reserve, charge functions, et cetera, must be really complicated. Let's look at them. So these are called activities, and we use activities for anything that might fail. So all this logic doesn't have a risk of failure, but everything it calls does. This is the service reserve, unreserve. You can see it's just the basic service calls. Each of these activity functions is retried and timed out according to my retry policy and whatever options I put here, there's lots of different options. That's just all taken care of by the system.

Further, each step of this process order function is persisted. So if the process dies after this, then a new worker will continue execution in the exact same spot, in the exact same state, and any local variables. We only have order up here, but we could have lots of other ones that we change during the function. They'll all have the same values. You can also do other cool things like sleep for 30 minutes. We're doing the Amazon one click buy. We say reserve and then we set a 30 minute timer, and then we charge. And if we get canceled during those 30 minutes, then we don't charge our send package. We can also sleep for 30 days in a loop if we're trying to monthly charge someone on a subscription.

And the way this works is we have a temporal server, which is open source, and you can host it anywhere. And that does persistence. And we have language runtimes, or SDKs. And the SDK is put up into a client and a worker. And then you have a client that can be used anywhere. Right now we're using it on a serverless function. And the client talks to the temporal server, and the worker talks to the server. And the client says, start this workflow, which is this process order function.

6. Temporal: A Durable Code Execution Framework

Short description:

Temporal is a durable code execution framework that allows you to write code oblivious to faults. It's a huge step forward in software development, working at a higher level of abstraction. Learn more at temporal.io.ts or contact me via email at lauren.temporal.io or on Twitter at laurenDSR.

And then the workers are constantly pulling and picking up tasks from the server. Like, okay, start this function, start this activity, et cetera. And the worker is reporting back to the server after it completes each step so the server can persist it.

So when people ask me, what's Temporal? I say it's a durable code execution framework. So we run your code in our workers and we do so durably, such that if anything goes wrong, we pick it back up in the exact same state. And so you can write your code in a way that's oblivious to faults. So that's Temporal.

I think it's a huge step forward in software development, working at a higher level of abstraction in which you don't have to worry about liability. If you'd like to learn more, check out temporal.io.ts. That's the TypeScript SDK docs. Feel free to email me, lauren.temporal.io. Hit me up on Twitter, at laurenDSR.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

React Day Berlin 2023React Day Berlin 2023
21 min
React's Most Useful Types
We don't think of React as shipping its own types. But React's types are a core part of the framework - overseen by the React team, and co-ordinated with React's major releases.In this live coding talk, we'll look at all the types you've been missing out on. How do you get the props type from a component? How do you know what ref a component takes? Should you use React.FC? And what's the deal with JSX.Element?You'll walk away with a bunch of exciting ideas to take to your React applications, and hopefully a new appreciation for the wonders of React and TypeScript working together.
Vue.js London 2023Vue.js London 2023
30 min
Stop Writing Your Routes
The more you keep working on an application, the more complicated its routing becomes, and the easier it is to make a mistake. ""Was the route named users or was it user?"", ""Did it have an id param or was it userId?"". If only TypeScript could tell you what are the possible names and params. If only you didn't have to write a single route anymore and let a plugin do it for you. In this talk we will go through what it took to bring automatically typed routes for Vue Router.
TypeScript Congress 2023TypeScript Congress 2023
31 min
Making Magic: Building a TypeScript-First Framework
I'll dive into the internals of Nuxt to describe how we've built a TypeScript-first framework that is deeply integrated with the user's IDE and type checking setup to offer end-to-end full-stack type safety, hints for layouts, middleware and more, typed runtime configuration options and even typed routing. Plus, I'll highlight what I'm most excited about doing in the days to come and how TypeScript makes that possible not just for us but for any library author.
React Advanced Conference 2021React Advanced Conference 2021
6 min
Full-stack & typesafe React (+Native) apps with tRPC.io
Top Content
Why are we devs so obsessed with decoupling things that are coupled nature? tRPC is a library that replaces the need for GraphQL or REST for internal APIs. When using it, you simply write backend functions whose input and output shapes are instantly inferred in your frontend without any code generation; making writing API schemas a thing of the past. It's lightweight, not tied to React, HTTP-cacheable, and can be incrementally adopted. In this talk, I'll give a glimpse of the DX you can get from tRPC and how (and why) to get started.
TypeScript Congress 2022TypeScript Congress 2022
10 min
How to properly handle URL slug changes in Next.js
Top Content
If you're using a headless CMS for storing content, you also work with URL slugs, the last parts of any URL. The problem is, content editors are able to freely change the slugs which can cause 404 errors, lost page ranks, broken links, and in the end confused visitors on your site. In this talk, I will present a solution for keeping a history of URL slugs in the CMS and explain how to implement a proper redirect mechanism (using TypeScript!) for dynamically generated pages on a Next.js website.

Add to the talk notes: https://github.com/ondrabus/kontent-boilerplate-next-js-ts-congress-2022 

Workshops on related topic

React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2022React Advanced Conference 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Top Content
Featured Workshop
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Node Congress 2024Node Congress 2024
83 min
Deep TypeScript Tips & Tricks
Workshop
TypeScript has a powerful type system with all sorts of fancy features for representing wild and wacky JavaScript states. But the syntax to do so isn't always straightforward, and the error messages aren't always precise in telling you what's wrong. Let's dive into how many of TypeScript's more powerful features really work, what kinds of real-world problems they solve, and how to wrestle the type system into submission so you can write truly excellent TypeScript code.
TypeScript Congress 2023TypeScript Congress 2023
131 min
Practice TypeScript Techniques Building React Server Components App
Workshop
In this hands-on workshop, Maurice will personally guide you through a series of exercises designed to empower you with a deep understanding of React Server Components and the power of TypeScript. Discover how to optimize your applications, improve performance, and unlock new possibilities.
 
During the workshop, you will:
- Maximize code maintainability and scalability with advanced TypeScript practices
- Unleash the performance benefits of React Server Components, surpassing traditional approaches
- Turbocharge your TypeScript with the power of Mapped Types
- Make your TypeScript types more secure with Opaque Types
- Explore the power of Template Literal Types when using Mapped Types
 
Maurice will virtually be by your side, offering comprehensive guidance and answering your questions as you navigate each exercise. By the end of the workshop, you'll have mastered React Server Components, armed with a newfound arsenal of TypeScript knowledge to supercharge your React applications.
 
Don't miss this opportunity to elevate your React expertise to new heights. Join our workshop and unlock the potential of React Server Components with TypeScript. Your apps will thank you.
TypeScript Congress 2022TypeScript Congress 2022
116 min
Advanced TypeScript types for fun and reliability
Workshop
If you're looking to get the most out of TypeScript, this workshop is for you! In this interactive workshop, we will explore the use of advanced types to improve the safety and predictability of your TypeScript code. You will learn when to use types like unknown or never. We will explore the use of type predicates, guards and exhaustive checking to make your TypeScript code more reliable both at compile and run-time. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Are you familiar with the basics of TypeScript and want to dive deeper? Then please join me with your laptop in this advanced and interactive workshop to learn all these topics and more.
You can find the slides, with links, here: http://theproblemsolver.nl/docs/ts-advanced-workshop.pdf
And the repository we will be using is here: https://github.com/mauricedb/ts-advanced
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.