Safely Handling Dynamic Data with TypeScript
AI Generated Video Summary
This Talk discusses the safe handling of dynamic data with TypeScript using JSON Schema and TypeBox. Fastify, a web framework, allows developers to validate incoming data using JSON schema, providing type safety and error handling. TypeBox is a powerful library that allows developers to define JSON schemas and derive static types in TypeScript. The combination of JSON schema, TypeBox, and Fastify provides powerful tools for type safety and validation of dynamic data.
1. Introduction to Handling Dynamic Data
Hello, everybody. Today I'm going to be talking to you all about safely handling dynamic data with TypeScript. We use data in various ways as software developers, such as API routes, forms, authentication payloads, and communication between large systems. Let's look at an example of a JSON object representing a person. JSON is a verbose way of representing data and is widely used in APIs. Now, let's explore a Fastify route and the challenges of handling unknown request bodies.
So, handling data. What is data? As software developers, we use a lot of it and in a lot of different ways. Some good examples, at least of how I use it is within API routes, when building a back end service, dealing with forms in the front end, and also authentication payloads and all of the surrounding things that go with authentication in an entire full stack application. And this is just a short list. You can only imagine how long this can get when you start dealing with databases or data science and just general, any sort of communication between large systems.
Talking about backend APIs, let's take a look at a Fastify route. In this case, we're defining a post route. The path is add user, and the request handler here has two arguments, request and response, and we're destructuring the body from that request object. Does anyone know what type body might be? Is it a record, an object, is it any type? Trick question. It's unknown. The body property of that request object, taking a peek again at the code, the Fastify route has no idea what it is because in context of Fastify as a framework, we're not sure what the developer intends to be coming in through their request. And there's no way for Fastify to know that when you're right when the code is being written or even compiled. Well, maybe not when it's compiled. We'll get to that later. So let's take another look here.
2. Handling Unknown Request Bodies with JSON Schema
Looking at the previous JSON object or take a slice of look at the ID and the name, there are string keys and string values. TypeScript will throw an error. Object is of type unknown. So there's some patterns we can use. You can use basic typecasting where we can say body.name as string. But in that case, there is no verification. That's where I want to introduce JSON Schema. JSON Schema is a super powerful API, lets you define a JSON object with more JSON. JSON Schema uses a standard or a specification to allow a developer to define the shape of a JSON object, given things such as the type, listing the properties, saying what properties are required or not, listing if there are additional properties or not, and even being able to define more complex types. JSON schema, and this only scratches the surface of JSON schema. You can use regular expressions, you can use references, you can use logical operations like all of or any and or some, and it's just so powerful when you leverage JSON schema to define your JSON objects.
3. Type Safety with JSON Schema and TypeBox
You give that to a validator and you say make sure that my JSON object is actually what I'm saying it's supposed to be via the schema. Throw an error or give me some special output if it's not. So with that, we're we have our JSON object, that payload from before in one hand. We now know that we can define a schema for it in the other hand. So, let me introduce you to TypeBox. TypeBox is a fantastic library that allows you to not only define JSON schemas using a fluent-like API, but also derive a static type of that JSON schema in your TypeScript code. And so at a more complex example, let's redefine our body schema using type box. And then in the final block of this code, Fastify.post, you can see we're now using a generic parameter. It's a object, a name, I like to call them name generic parameters. And we've assigned that T body schema type to the body property of that generic parameter. The Fastify type system will forward that T body schema type through to the body property of the request object. And so that's great. Now we've typed the body property of the request object.
You give that to a validator and you say make sure that my JSON object is actually what I'm saying it's supposed to be via the schema. Throw an error or give me some special output if it's not. So with that, we're we have our JSON object, that payload from before in one hand. We now know that we can define a schema for it in the other hand.
So going back to the Fastify application and that route, are we able to provide any type safety yet? Not really. All we can do is validate that that incoming body is what we want it to be using JSON schema. But come on now. It's 2021. There's some pretty cool things we can do.
So, let me introduce you to TypeBox. TypeBox is a fantastic library that allows you to not only define JSON schemas using a fluent-like API, but also derive a static type of that JSON schema in your TypeScript code. So, given this very basic example, you can see on the second line, constant T equals type.string. That is going to return type.string, an object with one property, type, that is set to string. This is a valid JSON schema object. We are saying that the variable T is the JSON schema type string. And we can use that JSON schema that is assigned to variable T in anywhere we would use JSON schema, in any validation or serialization level.
But on that last line of the sample, you can see type T equals static and then the generic parameter type of T, which is that constant declared on the second line. And now that type T is a string. The type box library is smart enough to derive the type from a JSON schema into that static shape. And so at a more complex example, let's redefine our body schema using type box. You can see we started with type dot object and then we've listed our six properties, id and name, our type string, and then employed company age and projects are all just type optional. And then within those optional types, we define type Boolean for employed, string for company, number for age. And in projects, you can see we're doing type string inside of type array, which is the type box way of saying an array of strings. And then we're defining a type T body schema is going to equal the static resolution of the type of that type box body schema. And then in the final block of this code, Fastify.post, you can see we're now using a generic parameter. It's a object, a name, I like to call them name generic parameters. And we've assigned that T body schema type to the body property of that generic parameter. The Fastify type system will forward that T body schema type through to the body property of the request object. And so that's great. Now we've typed the body property of the request object.
4. Validation Step and Error Handling
But we're still missing a key step here, but this code already solves it. Is the validation step. Fastify has this schema option which allows the developers to verify the incoming data of a request and the outgoing response shape using JSON schema. That validation step happens before that handler function executes. So if the validation step fails, then the entire route will error and return an error to the user saying invalid body.
But we're still missing a key step here, but this code already solves it. Is the validation step. As you can see underneath the add user string, there's another argument added to this route. And it's schema and then the object body and then it's the body schema. And don't get confused. That body schema is the actual JSON defined, returned by type boxes API. Fastify has this schema option which allows the developers to verify the incoming data of a request and the outgoing response shape using JSON schema. Under the hood, we pass that to another JSON validator and we validate the content based on the schemas provided. That validation step happens before that handler function executes. So if the validation step fails, if the incoming request body doesn't match the schema that we've set it to in that second function argument, then the entire route will error and return an error to the user saying invalid body. And then I think it also includes the JSON validator error as well.
5. Type Safety and Validation with Fastify
Inside the function handler, Fastify validates the body property using the defined schema. This provides type safety for accessing properties like body.name. However, TypeScript's compile-time type safety is not runtime-based. With Fastify's validation layer, we can get close to type safety for dynamic data. JSON schema, TypeBox, and Fastify together are powerful tools.
And with that all together, inside of our function handler now, by the time Fastify gets there, we can think, okay. Fastify has validated the body property of the request object using the schema we've defined using typebox. We've also provided the type for that body property using the generic parameters that is a type that is derived directly from the same JSON schema used to validate that property. And so, now, inside of our function handler, that body property from the request is going to have the tbody schema type. And accessing something like body.name will, in fact, return a type safe string. And if you're accessing one of the optional properties, it will be that type. In this case, body.number or undefined.
So all in all, JSON schema plus TypeBox plus Fastify is super, super powerful. And I briefly want to jump over to VSCode and show you all this in sort of real-time so that you can get an understanding of what it actually will look like in a real Fastify application. So in my Fastify app here, I have this basic run function that's going to create a new Fastify app. We're going to register our create server plugin. We're going to do a wait, app.ready, get the address, and then print some nice things out to the console. Over here on the server, you can see we've imported that type box API. We've defined our body schema. And you can see it has this really interesting shape.
6. Type Safety and Validation with Fastify
The type of body schema is a type box T object with a generic of this named generic parameters. Generics are crazy powerful nowadays. Inside of this function handler, you can now most confidently rely on the type safety of your dynamic data that is all based on the same JSON schema that is being used to validate it at the same time.
The type of body schema is a type box T object with a generic of this named generic parameters. Where the first property ID is of type T string. Name is of type T string. Employed is of type T Optional with the generic parameter T Boolean and so on and so forth. In fact, the projects one gets cut off, but it's going to be T array, T string. And we can sort of hover over that by seeing what would this projects be. You can see in the IntelliSense it's T Optional, T array, T string. Generics are crazy powerful nowadays.
You can see down here this type, in the type T body schema, it's kind of looking kind of funky. We have empty object and empty object and object, optional employed, optional company, optional age, optional projects, and then and another object ID and name. Going down into our Fastify route, you can see we're passing that body schema that is defined here. It's an actual JSON schema to the body property of our schema parameter here for Fastify. And in this post generic, we're passing that type T body schema. And if we hover over body inside of the function handler you can see we're getting that static object type. And it passes that same generic name generic parameter that comes from body schema. And now when you do body.name, you get type string. And if you do body.age, you're going to get number or undefined. And so inside of this function handler, you can now most confidently rely on the type safety of your dynamic data that is all based on the same JSON schema that is being used to validate it at the same time.
So jumping back over, I want to say a big shoutout to the Undraw Graphics illustrations collection. Without it, my presentation would be way, way more boring. And also this whole thing was built with HighlightJS and the TMCW Big Library. I highly recommend it if you yourself are working on a presentation. I want to say a big thank you for attending my talk today. I hope you learned something. And I hope to answer your questions in a little bit. If you'd like to follow me on Twitter, there is my thing. You can also follow me at GitHub as well. I love to connect with folks. I love talking about Node.js and TypeScript. And yeah, I hope to see you all around.
7. Reflection on Poll Results
The poll results show that many people are familiar with the topic, but there are still some who have never used it. The goal is for attendees to try it out after the talk and Q&A.
Thanks again. Hello. Wow. Hey. So what do you think of the results of the poll? I think that was great. I think that was the answer I was kind of hoping for where it seems like a good part of the crowd at least knows what it is, has used it before at different levels of experience. And there's still a slice of folks who have never touched it. And hopefully, after this talk and maybe this Q&A, they'll give it a shot. Yeah. That would be awesome, of course. Well, that's why people come to these events, right, to get to know new stuff and hopefully play around with them and use them at their companies in production.
Express Validation and Fastify
Express doesn't have built-in JSON schema validation like Fastify, but you can use similar JSON schema validators as middleware. Fastify uses AJV as its validator, which is fast and spec compliant.
The first question from one of our audience members is from Walker MAA, and he says, great talk. Is Express also able to validate body against schema as Fastify? Yeah. So, Express, I believe, so I don't use Express as a Fastify maintainer. I tend to just use Fastify. I believe that it's not built into Express, but there are very similar JSON schema validators that will work just the same as Fastify with Express that you can use as like a middleware. And the thing with Fastify is that it's built in, so it's like by default, you don't have to load any special plugin. There's no, there's nothing you have to like pass to the server. It will use a schema as provided to the routes. And under the hood, we use a validator called AJV, which is another JSON validator, and I think there's another question in here that asks about one of the other options there. So most folks are familiar with something like Joy, which comes from the Happy web framework set of tools. But for Fastify we use AJV, it proves to be the fastest, and it was most spec compliant with how fast your JSON schema iterates. Awesome.
Comparison between Joy and Typebox
Joy and Typebox have a similar API for building JSON schemas, but Joy also provides validation and serialization features. Typebox focuses solely on building JSON schemas and can be used with any JSON validator.
Next question is from Christina, is Typebox the TypeScript version of Joy? Yeah, so Joy is a big library. It does a lot of things. Firstly, it does have a similar API to Typebox, where it lets you build your JSON schemas using the Joy API. That part of it is very similar to Typebox, but where they differ is that I believe Joy will still do validation on the...has validation and serialization features for you based on the schemas that are output by the Joy API. Typebox doesn't have any validation, doesn't have any serialization. It's just the JSON schema building part. It will play nicely with any JSON validator.
Using Swagger OpenAPI and JSON Schema
Swagger and OpenAPI are implemented on JSON schema. You can use their JSON schemas to make your Fastify routes type safe when implementing an API server.
Okay, awesome. Next question is from Zero Carol. Okay, cool, but why not use Swagger OpenAPI? This is a great question. This comes up frequently all the time. The answer is sort of the same questions. Why not? Go for it. Swagger and OpenAPI, I believe, is all implemented on JSON schema. Everything that they export has their own blops of JSON schemas that you can use and plug those into other things, other tools, such as like the automatic swagger websites that you see. Those are all powered by the JSON schemas and that is the exact kind of thing that you can pass to your routes here, where if you use the OpenAPI format to define your REST API, you could use those same JSON schemas to make your Fastify routes type safe when you're implementing that API server.
Typebox and TypeScript Interfaces
Typebox is a tool that outputs TypeScript types and JSON schema, eliminating the need to write and maintain separate schemas and interfaces. By relying on a tool like Typebox, you ensure that the translation between schema and type is accurate and reliable, reducing the risk of errors and inconsistencies.
Cool. Next talk is after you, but first we have a question from Johnny Gat. Isn't Typebox the same as TypeScript interfaces? Yes, it is. Typebox is a tool that will output TypeScript types as well as the JSON schema. You could do it yourself by hand. You can write your interface that implements your own JSON schema. The idea here is classic programmer like, don't repeat yourself. Why write two schemas? Plus, if you make a change on the schema, you have to make sure you make that change on the interface. There's not a lot of great ways to validate that the interface that you wrote is actually correct. The correct implementation of the JSON schema. Users make errors all the time. So by leaving it to a tool to make that translation and convert the schema into a type or vice versa or both at the same time, that is a lot more reliable. Yeah, it adds a bit of a security layer that things don't go out of date and out of sync. Awesome.
Using JSON Schema for Type Security
The reason why I strongly encourage folks to use JSON schema rather than just passing a type interface to their route is because Fastify will validate the thing that you're providing a type for and it's that validation step that actually provides the type security that you're using TypeScript for in the first place. Another question from David Lime, could TypeBox be used to generate OpenAPI also? I believe you could find some level of interop there where you could define your OpenAPI objects or schemas using TypeBox and have that JSON schema to use wherever you might use your OpenAPI schema. We have another question from Marnice asking if there's anything like TypeBox for the front end. TypeBox or similar APIs should work on the front end. It's all about what other JSON tooling you have.
Next question is from Max Zdzadorov. Why should I use JSON schema and not just declare TypeScript interfaces? Well, I think we kind of just touched on that, but do you have some more insights that you want to share? Yeah, I'll share a little bit more. So, the reason why I strongly encourage folks to use JSON schema rather than just passing a type interface to their route is because Fastify will validate the thing that you're providing a type for and it's that validation step that actually provides the type security that you're using TypeScript for in the first place and this kind of gets into the whole ethos of like, what is type safety? Is it actually type safe if it is going to be something that occurs at runtime and it's that level of like, security and safety and if we can get as close to that as possible that is sort of the best case scenario and getting to that point, getting to as close to safety as possible or type safety as possible is having to rely on a tool like this where it's JSON schema for validation and the type is inferred by that same schema. Yeah, cool.
So, another question from David Lime, could TypeBox be used to generate OpenAPI also? Great question. I'm not super familiar with OpenAPI, but I believe so. I believe you could find some level of interop there where you could maybe define your OpenAPI objects or schemas using TypeBox and have that JSON schema to use wherever you might use your OpenAPI schema and have the resulting types. The only thing I'll say though is I believe with OpenAPI, a big part of the goal is to get that JSON object as its own file and then pass that file around. Where TypeBox, it only returns it during run time. It's in the code. With that said, it's JSON in a variable. You can 100% stringify it and write that to a file. But that second operation step might be a little bit different than what many OpenAPI users are used to. Okay. I believe that that is the last question that we have at this moment.
Conclusion of Q&A Session
Great talk, Ethan, my favorite of the day so far. Thanks a lot for your great talk and this informative Q&A session. Don't forget Ethan is going to be in his spatial chat if you have more questions for him.
I think he's a DC fan. Great stuff. Great talk, Ethan, my favorite of the day so far. You can take that home and tell your mom that you had a big fan.
And with that, I would like to end this Q&A session. Thanks a lot for your great talk and this informative Q&A session and hope to see you again soon.
Of course. Thank you all. Have a good day. Don't forget Ethan is going to be in his spatial chat if you have more questions for him. Bye.