Safely Handling Dynamic Data with TypeScript

Hello, everybody. My name is Ethan Erewood. I'm a software engineer for Microsoft. And today I'm going to be talking to you all about safely handling dynamic data with TypeScript. So handling data. What is data? As software developers, we use a lot of it. And in a lot of different ways. Some good examples, at least of how I use it, is, you know, within API routes, when building a backend service, dealing with forms and the frontend. And also authentication payloads and all of the surrounding things that go with authentication in an entire full stack application. And this is just a short list. You can only imagine how long this can get when you start dealing with databases or data science. And just general any sort of communication between large systems. So let's take a peek at an example of a record or a chunk of data. In this case, I'm using a JSON object. We got a bunch of keys here. ID, name, employee, company, age, and projects. We're representing a person. Maybe this is an employee directory. Or maybe it's a user directory for a site such as LinkedIn. Where we have a user. We want to get their name. We want to know are they employed or not. Which is a Boolean value. We want to know what company they work for. And we want to know how old they are. And we also might want to list their projects. And as many folks know, JSON is a very verbose way of representing data. There are it has a lot of great primitives that are all based in JavaScript. And it can be quite extensive. In fact, entire APIs are powered by JSON objects alone through the open schema format. So talking about, you know, back end APIs, let's take a look at a Fastify route. In this case, we're defining a post route. The path is add user. And the request handler here has two arguments. Request and response. And we're destructuring the body from that request object. Does anyone know what type body might be? Is it a record? An object? Is it any type? Trick question. It's unknown. The body property of that request object, taking a peek again at the code. The Fastify route has no idea what it is. Because in context of Fastify as a framework, we're not sure what the developer intends to be coming in through their request. And there's no way for Fastify to know that when you're writing when the code is being written or even compiled. Well, maybe not when it's compiled. We'll get to that later. So let's take another look here. Looking at the previous JSON object, or take a slice of it, just look at the ID and the name. There are strings, string keys and string values. And then looking at the post route again, we know that body is unknown. That we're destructuring from the request object. So what would be body.name? In this case, imagine that the JSON object is being passed to this is being sent to this route as the body in the post request. So what would name be? Probably string, right? Well, trick question again. TypeScript will throw an error. Object is of type unknown. Why is that? Well, it's because the body property is coming from that request object because it's unknown. No other types can be derived from it in safely in TypeScript. TypeScript goes, no, no, no, stop here. I don't want you to keep going and using properties on this object because as the TypeScript compiler, I don't know what it is and I can't provide you the type safety that you're looking for. So even though as a user, we might think, ah, the name property. It's always going to be a string. There's no ifs, ands or buts about it. In this case, TypeScript is like, well, you didn't tell me that. I have no way of assuring that. So there's some patterns we can use. You can use basic type casting where we can say body.name as string and we will tell TypeScript it's a string. But in that case, there's no verification. There's no way of saying that assuring that that name property is actually a string because TypeScript is a compile time only type safety. During runtime, it's all just JavaScript. There is no type safety at the runtime. So what are some other solutions? That's where I want to introduce JSON schema. JSON schema is a super powerful API. It lets you define a JSON object with more JSON. Isn't that just wonderful? Kidding aside, JSON schema is actually incredibly verbose, even more verbose than the JSON objects that it's probably defining. JSON schema uses a standard or a specification to allow a developer to define the shape of a JSON object, given things such as the type, listing the properties, saying what properties are required or not, listing if there are additional properties or not, and even being able to define more complex types. As you can see in the code sample, the projects property is of type array. And then we get to go even further and see the items of that array are of type string. JSON schema, and this only scratches the surface of JSON schema. You can use regular expressions. You can use references. You can use logical operations like all of or any, and or some, and it's just so powerful when you leverage JSON schema to define your JSON objects. So all that said, though, JSON schema, I think it's even in their spec, is intended for validation. The validation in a sense of here's a schema and here's a JSON object. You give that to a validator and you say make sure that my JSON object is actually what I'm saying it's supposed to be via the schema. Throw an error or give me some special output if it's not. So with that, we're we have our JSON object, that payload from before in one hand. We now know that we can define a schema for it in the other hand. So going back to the Fastify application and that route, are we able to provide any type safety yet? Not really. All we can do is validate that that incoming body is what we want it to be using JSON schema. But come on now. It's 2021. There's some pretty cool things we can do. So let me introduce you to Typebox. Typebox is a fantastic library that allows you to not only define JSON schemas using a fluent like API, but also derive a static type of that JSON schema in your TypeScript code. So given this very basic example, you can see on the second line, constant T equals type.string. That is going to return type.string an object with one property type that is set to string. This is a valid JSON schema object. We are saying that the variable T is the JSON schema type string. And we can use that JSON schema that is assigned to variable T in anywhere that we would use JSON schema, in any sort of validation or serialization level. But then on that last line of the sample, you can see type T equals static. And then the generic parameter type of T, which is that constant declared on the second line. And now that type T is a string. The Typebox library is smart enough to derive the type from a JSON schema into that static shape. And so at a more complex example, let's redefine our body schema using Typebox. You can see we started with type.object, and then we've listed our six properties, id and name are type.string. And then employed company age and projects are all just type.optional. And then within those optional types, we define type.boolean for employed, string for company, number for age. And in projects, you can see we're doing type.string inside of type.array, which is the Typebox way of saying an array of strings. And then we're defining a type, T body schema, is going to equal the static resolution of the type of that Typebox body schema. And then in the final block of this code, Fastify.post, you can see we've now used a generic parameter. It's an object, a name, I like to call them name generic parameters. And we've assigned that T body schema type to the body property of that generic parameter. The Fastify type system will forward that T body schema type through to the body property of the request object. And so that's great. Now we've typed the body property of the request object. But we're still missing a key step here. But this code already solves it, is the validation step. As you can see underneath the add user string, there's another argument added to this route. And it's schema, and then the object body, and then it's the body schema. And don't get confused. That body schema is the actual JSON schema defined, returned by Typebox's API. Fastify has this schema option, which allows the developers to verify the incoming data of a request and the outgoing response shape using JSON schema. Under the hood, we pass that to AJV, another JSON validator, and we validate the content based on the schemas provided. That validation step happens before that handler function executes. And so if the validation step fails, if the incoming request body doesn't match the schema that we've set it to in that second function argument, then the entire route will error and return an error to the user saying, you know, invalid body. And then I think it also includes the JSON validator error as well. And with that all together, inside of our function handler now, by the time Fastify gets there, we can think, okay, Fastify has validated the body property of the request object using the schema we've defined using Typebox. We've also provided the type for that body property using the generic parameters that is a type that is derived directly from the same JSON schema used to validate that property. And so now inside of our function handler, that body property from the request is going to have the T body schema type. And then accessing something like body.name will in fact return a type safe string. And if you're accessing one of the optional properties, it'll be that type, in this case body.age would be number, or it's undefined. And this is like my favorite part about all of this. But it also sort of breaks down a little bit of the whole kind of problem in the first place where TypeScript is a compile time only type safety, not runtime based. And so when you're inside of a route handler, and you're taking that body from the request object, if we forget about the fact that Fastify is going to validate our incoming data for us, you have to remember that you can unsafely just say body is this type, and then go about your business in your route. And that's kind of it. There's no actual true type safety there. But with the validation layer that Fastify adds, we can get as close as possible to type safety for highly dynamic data. Because if we put a hard line, or if we trust Fastify enough to be like, this route handler will not execute if the validation step doesn't succeed, then we can tell TypeScript confidently that the type of that property inside of that route handler is going to be what we want it to be, what we think it's going to be, because we trust in that validation step. If that validation step logically is incorrect, and we'll let through something that we aren't prepared for, then we're going to have a problem. And unfortunately, even a JavaScript application wouldn't be able to sort of respond to that. It will probably have to error out. If you're trying to access a property that doesn't exist because it somehow gets around the validation step. So all in all, JSON schema plus Typebox plus Fastify is super, super powerful. And I briefly want to jump over to VS Code and show you all this in sort of real time so that you can get an understanding of what it actually will look like in a real Fastify application. So in my Fastify app here, I have this basic run function that's going to create a new Fastify app. We're going to register our create server plugin. We're going to do a wait app.ready, get the address, and then print some nice things out to the console. Over here on the server, you can see we've imported that Typebox API. We've defined our body schema. And you can see it has this really interesting shape. The type of body schema is a Typebox T object with a generic of this named generic parameters, where the first property ID is of type T string, name is of type T string, employed is of type T optional with the generic parameter T boolean, and so on and so forth. In fact, the projects one gets cut off, but it's going to be T array T string. And we can sort of hover over that by seeing, okay, what would this projects be? You can see in the IntelliSense, it's T optional, T array, T string. Generics are crazy powerful nowadays. And you can see down here in the type T body schema, it's kind of looking kind of funky. We have empty object and empty object and empty object and object, optional employed, optional company, optional age, optional projects, and then, and another object ID and name. Going down into our Fastify route, you can see we're passing that body schema that is defined here. It's an actual JSON schema to the body property of our schema parameter here for Fastify. And in this post generic, we're passing that type T body schema. And if we hover over body inside of the function handler itself, you can see we're getting that static object type and it passes that same generic name, generic parameter that comes from body schema. And now when you do body.name, you get type string. And if you do body.age, you're going to get number or undefined. And so inside of this function handler, you can now most confidently rely on the type safety of your dynamic data that is all based on the same JSON schema that is being used to validate it at the same time. So jumping back over, I want to say a big shout out to the Undraw Graphics Illustrations collection. Without it, my presentation would be way more boring. And also, this whole thing was built with HighlightJS and the TMCW Big Library. I highly recommend it if you yourself are working on a presentation. And I want to say a big thank you for attending my talk today. I hope you learned something. And I hope to answer your questions in a little bit. If you'd like to follow me on Twitter, there is my thing. You can also follow me at GitHub as well. I love to connect with folks. I love talking about Node.js and TypeScript. And yeah, I hope to see you all around. Thanks again. Hello. Wow. Hey. So what do you think of the results of the poll? I think that was great. I think that was the answer I was kind of hoping for, where it seems like a good part of the crowd at least knows what it is, has used it before at different levels of experience. And there's still a slice of folks who have never touched it. And hopefully, after this talk and maybe after this Q&A, they'll give it a shot. Yeah, that would be awesome, of course. Well, that's why people come to these events, right? To get to know new stuff and hopefully play around with them and use them at their companies in production. The first question from one of our audience members is from Walker MAA. And he says, great talk. Is Express also able to validate body against schema as Fastify? Yeah. So Express, I believe. So I don't use Express as a Fastify maintainer. I tend to just use Fastify. I believe that it's not built in to Express, but there are very similar JSON schema validators that will work just the same as Fastify with Express that you can use as a middleware. The thing with Fastify is that it's built in. So it's like by default, you don't have to load any special plugin. There's nothing you have to pass to the server. It will use a schema as provided to the routes. And under the hood, we use a validator called AJV, which is another JSON validator. And I think there's another question in here that asks about one of the other options there. So most folks are familiar with something like Joy, which comes from the Happy web framework set of tools. But for Fastify, we use AJV. It proves to be the fastest. And it was most spec compliant with how fast JSON schema iterates. Awesome. Next question is from Christina. Is Typebox the TypeScript version of Joy? Yeah. So Joy is a big library. It does a lot of things. Firstly, it does have a similar API to Typebox, where it lets you build your JSON schemas using the Joy API. That part of it is very similar to Typebox. But where they differ is that I believe Joy will still do validation on the dual, like has validation and serialization features for you based on the schemas that are output by the Joy API. Typebox doesn't have any validation, doesn't have any serialization. It's just the JSON schema building part. And it will play nicely with any JSON validator. Okay. Awesome. Next question is from Zero Carroll. Okay, cool. But why not use Swagger slash OpenAPI? Yeah, this is a great question. This comes up frequently all the time. And the answer is sort of the same questions. Like, why not? Go for it. Swagger and OpenAPI, I believe, is all implemented on JSON schema. Everything that they export has their own lopset of JSON schemas that you can use and plug those into other things, other tools, such as like the automatic Swagger websites that you see. Those are all powered by the JSON schemas. And that is the exact kind of thing that you can pass to your routes here, where if you use the OpenAPI format to define your REST API, you could use those same JSON schemas to make your Fastify routes type safe when you're implementing that API server. Next talk is after you. But first we have a question from Johnny Gat. Isn't Typebox the same as TypeScript interfaces? Yes, it is. Typebox is a tool that will output TypeScript types as well as the JSON schema. You could do it yourself by hand. You can write your own interface that implements your own JSON schema. But the idea here is classic programmer, like, don't repeat yourself. Why write two schemas? And plus, if you make a change on the schema, you'd have to go and make sure you make that change on the interface. And there's not a lot of great ways to validate that the interface that you wrote is actually correct, the correct implementation of the JSON schema. Users make errors all the time. So by leaving it to a tool to make that translation and convert the schema into a type or vice versa or both at the same time, that is a lot more reliable. Yeah, it adds a bit of a security layer that things don't go out of date and out of sync. Next question is from Max Zierzadorff. Why should I use JSON schema and not just declare TypeScript interfaces? Well, I think we kind of just touched on that, but do you have some more insights that you want to share? Yeah, I'll share a little bit more. So the reason I really, like, I strongly encourage folks to use JSON schema rather than just passing a type interface to their route is because Fastify will validate the thing that you're providing a type for, and it's that validation step that actually provides the type security that you're using TypeScript for in the first place. And this kind of gets into the whole ethos of, like, what is type safety? Is it actually type safe if it is going to be something that occurs at runtime? And it's that level of, like, security and safety, and if we can get as close to that as possible, that is sort of the best case scenario. And getting to that point, getting to as close to safety as possible, or type safety as possible, is having to rely on a tool like this, where it's JSON schema for validation and the type is inferred by that same schema. Yeah. Cool. So another question from David Lime. Could Typebox be used to generate OpenAPI also? Great question. I'm not super familiar with OpenAPI, but I believe so. I believe you could find some level of interop there, where you could maybe define your OpenAPI objects or schemas using Typebox and then again have that JSON schema to use wherever you might use your OpenAPI schema and have the resulting types. The only thing I'll say, though, is I believe with OpenAPI, a big part of the goal is to get that JSON object as, like, its own file and then, like, pass that file around, where Typebox, it only returns it, like, during runtime. It's, like, in the code. With that said, it's JSON in a variable. You could 100% stringify it and write that to a file, but that second operation step might be a little bit different than what many OpenAPI users are used to. Okay. I believe that that is the last question that we have at this moment. Oh, no, we have another one. This question is by Mar Nice. Do you know anything like Typebox, but then for the frontend? So Typebox or similar APIs should work on the frontend. It's all about what other JSON modeling you have. Typebox is just a JavaScript module that sort of has a very easy to use functional API that, like, returns a JSON object. That is all, like, that's no I don't think there's any special Node.js things that it's using, and if there are, I don't I believe the interop with the browser would be okay. So off the top of my head, I don't know if Typebox can just plug and play with a browser, but it should be fine, and if not, there's definitely alternative tools that could be. Or PRs welcome. Yeah, exactly. So, Mar Nice, if you have nothing to do over the weekend, you can try it out on your frontend. We have a comment from DCboyCM, who has a Batman avatar, so I think he's a DC fan. Great stuff. Great talk, Ethan. My favourite of the day so far. So you can take that home and tell your mom that you had a big fan. And with that, I would like to end this Q&A session. Thanks a lot for your great talk and this informative Q&A session, and hope to see you again soon. Of course. Thank you all. Have a good day. Don't forget, Ethan is going to be in his spatial chat if you have more questions for him. Bye, Ethan. Bye.