Generating types without climbing a tree

Rate this content
Bookmark

How do you generate types dynamically? How do you write a script that creates some typescript code? The approach most people would recommend is to use Abstract Syntax Tree manipulations. I was working on a deadline to implement types for our OpenAPI client, and I would have missed our release window. I needed something different and easier to build. Luckily, a friend recommended me a library I didn't know: code-block-writer. I fall in love with it at first sight.

30 min
21 Sep, 2023

Comments

Sign in or register to post your comment.

AI Generated Video Summary

This talk explores the challenges and benefits of generating types for APIs. The speaker discusses the need for a better client experience and the popularity of generating clients. They also explain the use of OpenAPI for generating REST API clients and the use of Cold Block Writer for code generation. The talk covers the process of defining types for parameters and responses, generating the client and request, and using the generated client. The speaker also touches on validation in production and the initial challenges with TypeScript.

1. Introduction to Generating Types

Short description:

Hi, everyone. I'm Matteo Collina, Platformatic Co-founder and CTO, and today I'm going to talk about something that makes me a little bit nervous, generating types. Entries. Why? Let's dig into it. So, the first question for all of you is, how many ropes do you need to climb a tree? This is a fundamental question and we are going to try to answer that question in this talk.

Hi, everyone. I'm Matteo Collina, Platformatic Co-founder and CTO, and today I'm going to talk about something that makes me a little bit nervous, generating types. Entries. Why? Let's dig into it. So, the first question for all of you is, how many ropes do you need to climb a tree? This is a fundamental question and we are going to try to answer that question in this talk. So, let's go.

First thing, a couple of things about me. I am Matteo Collina, part of the Node.js Technical Steering Committee, also board member of the Open Source Open JS Foundation. I created a bunch of libraries, help maintaining nodes, probably using some of my software given it's been downloaded 17 billion times last year, so yeah, maybe. So, some of my decisions impact a few people, so hopefully I don't break too many of you during this talk or otherwise.

So anyway, let's go and let's jump. So, I wrote a lot of modules, I maintain a lot of modules on NPM, a lot of downloads, whatever that matters. By the way, if you ever use one of those, if you want to send me a GitHub sponsor, everything is appreciated. I have a little bit of a small confession to make. What? What? Well, before 2022, I almost never used TypeScript. And here I'm speaking at TypeScript Congress. So what is that? It's really fascinating. I completely skimmed and avoid this technology for as long as I could. What happened in 2022? Well, I started a new company, and it was a developer experience for companies. So yeah, I kind of had to dig into TypeScript. In fact, it's all based on something I'd previously done, which is Fastify, which is a project inside the Open Source Foundation too. And it's all about building APIs. I'm a back-end engineer by trade and what I want to help people and how to help people with my tool, you know, to build and use APIs better and make it faster and make them easier to develop, maintain, scale, whatever you name it. So yeah, typically the typical flow of when one of your team members or somebody else is starting writing your APIs, you have an API on one end and you are typically a back-end engineer writing some complex SQL or some complex MongoDB queries, whatever you want to like, then they generate, write some documentation from that API and then your client, your engineer, code their client. So, okay, how does it make to generating the types? Yes, let me introduce the problem first. So, this is more or less what Plasformatic wants to simplify and improve how we create back-end software. Typically what we want to do is we want to move, allow you to move very quickly from A to B using as much rails as possible, guard rails for all of you to move very fast. And then, you know, you can get your old SUV to go where exactly you want to go. So, we want to help developers get rid of all the undifferentiated heavy lifting of building your JS apps. So, as part of that, we, at some point, we started focusing after we worked a lot on APIs and improve on two APIs that had been written.

2. The Need for a Better Client Experience

Short description:

What we found out from talking to a lot of people was that having a client for APIs was fundamental for a good development experience. Generating clients has become popular recently with libraries like tRPC and TS Rest. However, these libraries introduce tight coupling between the client and server code, making it difficult to scale and evolve. We need a better client experience that allows for flexibility and the use of different versions of TypeScript.

What we found out from talking to a lot of people was that, you know, oh, but having a client for those APIs was actually fundamental for providing a good experience and streamlining the development. So, what happened was that, you know, most people really like this flow much better. So, instead of having to call the client and prepare documentation for the API, I just want to hit a button and generate a client.

In fact, this has been so popular for ages, and kind of we got it lost a little bit along the way, because, you know, I remember generating clients for my APIs when I was doing WSDL and SOAP a long time ago. So, you know, probably not a great comparison, but it is. So, anyway, this has been extremely popular recently. In fact, this has been popularized a lot by a few new libraries. One is tRPC, which is fantastic, provides a fantastic developer experience. It allows you to expose your... creates an RPC call from the server to the... from the client to the server in a completely type-safe way. It's great, okay? It has only one significant drawback, which is introduced type coupling within your... the code of your frontend with the code of your backend, it also uses a proprietary protocol to communicate. So yeah, it's very tight. It's a very tight coupling between client and server. There is another library called TS Rest, which is also great, fantastic library, but it has another small... still another small problem, which is you need to specify the contract between your client and your server at its own typescape definition. Again, in order to consume the library, you need to have a source code dependency on the server. So again, for me, it provides introduced tight coupling between those two.

In fact, the problem here is the concept of you know, what happens in most teams that I have worked and developed with and so on, it's that you have basically some API that is developed by more or less a team and there is another group of people that you know, creates and consumes or that consumes those APIs. And those groups typically are more or less separated in most companies and in most organizations. So, integrating everything together, create a tight coupling between everything that makes it very hard to break down and scale and keep evolving things. And by the way, there is nothing to be taken by those libraries, which are fantastic. I'm just talking about what I am not, some of the reasons why I think we need a better client experience for APIs. So what, where are we left? Well, something that we want to provide some fantastic developer experience but without any tight coupling. So, without any code being shared, essentially, between the server and the client, so that you can essentially use and consume other APIs. And so, that one can use TypeScript 4.9 and the other one can use TypeScript 6 in the future. And they can evolve and have different versions, different things. Everything is fun and perfect. And so, how can we generate this developer experience? I started this journey, one of my journey types of work with this question.

3. Automatically Generating REST API Clients

Short description:

And as part of this journey, a survey showed that 80% of developers prefer REST to GraphQL. My journey started with the question of how to automatically generate a minimal and simple client for REST APIs. OpenAPI, a standard, provided a fantastic solution. Using Fastify or platform, we can easily generate OpenAPI documentation and spec definitions. The client had specific requirements: easy consumption on the front end, no runtime data validation, fully typed for catching developer mistakes, and no runtime dependencies. My initial attempt at code generation proved complex and difficult to debug.

And as part of this journey was the little bit of a survey that shows that more or less 80% of developers prefer REST to GraphQL, which sounds very much unintuitive to me, but that's what it is. And if you have better data than this, please let me know, because this is actually something that I'm very curious about.

So my journey started where how can we automatically generate a client for our REST APIs? And the generated code should be either minimal and simple. So how can we get that? Okay? Well, something very fantastic was that OpenAPI is a standard, okay? And I built this framework called Fastify, which is a really good framework. It's taking over more or less a lot of it's getting a lot of attention these days on how to build APIs in Node.js. And you can easily get an OpenAPI or swagger, as it was previously called, create generated very, very quickly on top of your routes. So we can get the documentation, but also the spec definition for OpenAPI. So we have all the, we can get these mostly for free using Fastify or using platform.

So the client, in fact, had a few requirements for me. And one was, well, I have my routes defining using OpenAPI, and I want to consume them very easily on the front end. I really don't want any data validation happening at runtime, or maybe it should be disabled, okay, whatever, because I don't want to waste any time on the client. I want my code to be as snappy as possible. But I also want to be fully typed to catch developer mistakes on the editor. And I want to use fetch, so I have no dependency whatsoever at runtime. So I can just get my generated file, and that's it.

So my first try to generate this code was to use. So I started using this code generation. How can I get the code that generated my code generated? And I read a good article on DevTool. I started doing a lot of things and I wanted to do. Let's do an experiment. Let's just generate this very, very simple function and this code on the right. It's what it takes to generate that function. Now look at the code. It's actually complex. You can read what it's doing. I create an identifier, I create the function, it does all the things, but it's very, very hard to... If there is a problem there, it's actually very, very hard to debug. It also takes a lot of code to generate very few lines. So I was not necessarily very... Oh, I was...

4. Generating TypeScript and Using Cold Block Writer

Short description:

I thought I could generate TypeScript easily, but it seems like a massive tree to climb. However, I found a library called Cold Block Writer that makes code generation easier. It has a nice block property that allows for automatic insertion of braces and manages indentation. The drawback is that the generated code may not be valid, but it's a smaller and more convenient option. We can use the OpenAPI spec to generate function names based on the path and operation ID.

Oh, ouch. How can I... I thought I could generate TypeScript easily, but how can I get it? It seems this is a massive tree that I have to climb and I couldn't get it done in my timeline, so I was like, maybe not. I didn't want to climb any trees, but this seems a massive tree to climb, so, ouch, how can I get it done?

So I did another try. I did some scavenging, and I did a second try and this library that I found is called Cold Block Writer, which is absolutely something that it's really, really nice. And I think you should be taking a look at Cold Block Writer if you are trying to generate code dynamically. Why? Because it has a very, very good property here. If you look at the code on the right, it's essentially homomorphic in the sense that the code, in how we write the function, is very, very similar to the generated code that we wanted to have. Because these are homomorphism between those two things, we can actually find bugs and fix them very very easily because, oh, these are the three lines that generate the same code. I can see these three lines and those lines. So, it's actually very useful, very easy to write because it has this very nice block property that allows us to create insert braces automatically. Hey, nice! And it also generates and manages the indentation for us automatically. I really like this library.

It has only one big drawback. The generated code is not guaranteed to be valid. It's also way, way smaller, so that if I wanted to bring down the TypeScript compiler, like 50 something, whatever megabytes, they're reducing it. So, maybe it's now 30. I don't know. Anyway, it's a massive dependency, the TypeScript compiler, but this one is actually very tiny too. So, it's actually very nice and cool. So, it seems like a lazy option here. And I am a lazy developer, so I started going, let's do that. Let's do lazy.

Okay. And let's take a look at the OpenAPI spec and how it generates the code. What code we want to generate. So, look, the OpenAPI spec, it has a path property, and it says that there is a slash movies, for example. And each path has an operation ID. And this operation ID, we can use to generate our function names. So, when we, somebody wants to call a wait get movies, for example, we can call the slash movies endpoint automatically.

5. Defining Types for Parameters and Responses

Short description:

We can define and generate types for each parameter and response in the OpenAPI specification. This allows us to generate interfaces and use them in our API.

Okay? Also, it has defined its parameter and responses. Yeah, it's great. Okay? And then we can do the exact same thing for each of all the other operations that are defined in my OpenAPI specification. In fact, we can even, you know, we can even, oops, sorry. We can even, you know, check and define the types and generate the types for each one of our parameter and responses. If you look at it, it's actually pretty great. Okay? And we have all the parameters defined and then we have all our responses defined. So we can generate interfaces for both of them and use them as part of our API.

6. Generating the Client and Request

Short description:

So we have create movie and there is a create movie request that returns a response. We have split the types of our APIs with the actual interfaces, allowing for better manipulation. The generated code is very simple and can be quickly generated. Let's do a quick demo to see how easy it is. We create a small API, start it up, and generate the client using TypeScript. The client is fetch-based and has no dependencies. We have a get movie request in the api.types file.

So we have create movie and there is a create movie request that returns a response. Seems good. So I think this is, you know, for me, one of the interesting points here is that you see here we have split, oops, sorry. We have split here the types of our APIs with the actual interfaces. So in this way, things can be manipulated better, we think. So anyway.

Some of the good, there's a few good ideas here, but the generated code is actually very, very simple. Like it's you can actually map with one to one to what you need. And I think it was, we could generate this client very quickly. So let's do a little bit of a quick demo. And so that we can see how easy it is to do stuff. So let's create a little bit of a small API, so we can do create platformmatic. And here we go. We want to be here, new SQLite, but we could use everything. Oops, sorry. And yes, yes. And no for now, because I don't want to do NPM install. So I am creating very quickly an API automatically. And you know right now I don't need anything. So when I start this up, I could here do a platformmatic start. And if we open this up, okay, and we go back in here, sorry, here we go, I can see exactly what's happening. So this is what gets loaded and I can open my OpenMPI documentation, and I see that I have movies with all my parameters, which we can automatically get for creating everything. So how do we use, how do we, oop, sorry, how do we generate the client? Okay, oop, sorry, I am… So how do we generate the client? So let's go back cd client, okay, and here we have, I prepared a very simple thing with a package json, vcsnode, and typescript so we cannot waste time with npm install, so now I can do plt so-clear, and plt frontend, and I need to put in the url of my app, and then I want to generate it as typescript, and it has generated us two files api-types.b.ts and api.ts, and we have two of them, okay, and let's take a look. So this client is actually very simple to use, and we have uh we have here our fetch. So I need rts.config.json here, and I need the compiler options okay, and I need the target, and okay, otherwise things will be awkward. Here we go, and comment back, okay, good. So this will be awkward, okay. So anyway, what we have here, we have our our fetch. So it's all fetch based and all standard based. So we have actually no dependencies whatsoever, and then in api.types, as I showed, we have our get movie request here that gets things through.

7. Using the Generated Client and Running the Test

Short description:

So let's try and use this and see how the experience looks. We import from api and use setBaseURL, create movie, and get movies. We encounter some missing elements in the createMovieRequest. After creating the movie, we retrieve the movies and log them. We run the test and everything is running smoothly. We use the functions run, await, and Start Wars. Let's run it again.

So yeah, this seems pretty good. So let's try and use this and how the experience would look like. So if I open test yes, I could import from api. And now I could get setBaseURL and for example, create movie and get movies. Now I could do setBaseURL 0.3.0.4.2. And then we want to create here, does not, as you see here, does not exist here, okay?

And we, does not exist in the createMovieRequest. So in fact, if we look at the types here, if we go in createMovie, sorry. Okay, createMovieRequest, you see that it has an ID, optional ID, and then a required title. So here it's fine, but then we need to, we have created our movie and we don't need anything here. So okay, and then we could just get movies here, and do console log, and I need specified some parameters, but in this case are empty. So, and we have generated, we have getting these movies. Okay. And let's try this. Okay. So if I do TS node test. Now it's running. Cool. Okay. I'm running price. Why it's running price? Because these two are, you know, let's be nice. Let's be nice. So function run. Here we go. And I think that they were executed in parallel, right? So we can do await. And here we do Start Wars. And here we do press await. Yes. And then run. Now we can run this again. And here we go.

QnA

Generating the Client and Handling Internal Issues

Short description:

We have Star Wars. We got the exact same result. This is a pretty good way to generate a client. It works with any openAPI compatible with any openAPI system. Try our open source tool at docs.platformatic.dev. Thank you for having me at the S-Congress.

You see we have the two that we had before. And now we have Star Wars. We can also verify it very quickly, if everything was correctly. And it's here, we could try this out and see if this works correctly. As you can see, we got the exact same result. And this is the code that will be executed and generated. So no, this is, I think this is pretty nice. It's a pretty good way to generate a client. As you said, here I'm just going to let me do a tree. Here we have no dependencies whatsoever apart from ts-node and stuff, like if you can, here we have only ts-node and types. So there's no other dependencies in those files. It's super small and all API based. So the benefit of this approach of using openAPI is that it works with any openAPI compatible with any openAPI system, so you can get a very good developer experience just by generating some code for those openAPI definitions that are already available for a lot of APIs, so you don't need to generate some new ones or create new libraries or anything to consume them. So I don't know, I think this was an approach that was very, very interesting to try and verify if everything would have helped us somehow.

So the question for you is, did we climb the tree? So hey, did we climb the tree? I don't know. Anyway, thank you very much for watching. Try our open source tool at docs.platformatic.dev. And I don't know, thank you very much. And thank everybody for having me at the S-Congress. Thank you. Bye.

What's up, Matteo? How are you doing? Hi everybody. Hello. I love your voice man. You've got such a great presentation style. That's awesome. Why don't we jump to some questions? We've got some real crackers here. We were looking at these just before we came on and we were talking about this one. What should you do if you have internal issues and your front-end team can't trust your back-end team? You mentioned that in your talk you don't think you should be using like validating with Zod on the front-end or anything like Zod. What should you do in those situations? There are two bits, okay.

Validation in Production and TypeScript Conversion

Short description:

Doing validation in production can slow down your code and increase bundle size. It's not recommended to validate all responses in production. However, adding a development mode to pinpoint response discrepancies can be useful. TypeScript's ability to run code in the developer editor creates a unique developer experience that convinced me of its value.

You don't, you really don't want to do those kinds of validation in production. Mostly because doing any validation does, or having any validation library in the front-end does more or less two effects. First, it makes your code slow down. This is the first one. And the second one, it increases your bundle size. So as small as it is, as small as your library is, it still needs to be parsed, evaluated. Whenever you're going and calling, you're validating that response and it takes quite a lot of effort in it, okay. So it's actually, to be honest, not something good for production usage in validating all responses.

Having said that, adding that some kind of development mode or something to be able to pinpoint your team and your client team and say, look, your response is not what you said it should be. With a little finger, you do something different. You've been in that position, so I'm pretty aware, so everybody. I think we'll be adding that, in fact, maybe we already did, but in the time I recorded the talk. So it's coming, or if it's not there, it's coming soon. It's useful, but not so useful in that sense, because if you have this problem of, you don't want to be running it in production, right? So otherwise, you lose a lot of things. Yeah. I mean, you don't necessarily want your development code to be different from your production code as well. So there's that kind of in the mix, too.

I love what you said at the start. You're kind of famous, kind of recent TypeScript convert. I remember, I've been following you for a while on Twitter, and a few years ago, you said, you know, TypeScript sucks, why are you all doing this? And now you're here. So what happened? What changed your mind? What's the story of how you felt? OK. I realized something very, very interesting about TypeScript. It's TypeScript is code that you run in the developer editor. And that's probably the biggest truth about TypeScript that you can think about. And that's what allows TypeScript to create a certain level of developer experience that are not possible otherwise. And I love that. OK, to be honest, this is pretty phenomenal and amazing. So that's probably the reason. I haven't changed my mind. It took a while to convince you though, right? I mean, you weren't on board straight away.

Initial Challenges with TypeScript

Short description:

At the beginning, I had a bad experience with the TypeScript community. They criticized me for not writing in TypeScript, which was difficult and not a direct mapping to JavaScript.

No, I wasn't. I had a very bad impact at the beginning with the TypeScript community, and they basically completely flushed me out as a person. I received hundreds of issues telling me to write this in TypeScript, and you don't really want anybody to come to your house and tell you to change the paint. Or change the foundations as well. I mean, that's a big deal. Change the foundations, okay? And in some cases, not even possible. Like when this happened, TypeScript was very, very... It was not even a 1-on-1 mapping with JavaScript, and it was pretty bad, to be honest.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
23 min
Scaling Up with Remix and Micro Frontends
Do you have a large product built by many teams? Are you struggling to release often? Did your frontend turn into a massive unmaintainable monolith? If, like me, you’ve answered yes to any of those questions, this talk is for you! I’ll show you exactly how you can build a micro frontend architecture with Remix to solve those challenges.
Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
JSNation Live 2021JSNation Live 2021
29 min
Making JavaScript on WebAssembly Fast
JavaScript in the browser runs many times faster than it did two decades ago. And that happened because the browser vendors spent that time working on intensive performance optimizations in their JavaScript engines.Because of this optimization work, JavaScript is now running in many places besides the browser. But there are still some environments where the JS engines can’t apply those optimizations in the right way to make things fast.We’re working to solve this, beginning a whole new wave of JavaScript optimization work. We’re improving JavaScript performance for entirely different environments, where different rules apply. And this is possible because of WebAssembly. In this talk, I'll explain how this all works and what's coming next.
React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.

Workshops on related topic

React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2022React Advanced Conference 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Featured Workshop
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
WorkshopFree
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
TestJS Summit - January, 2021TestJS Summit - January, 2021
173 min
Testing Web Applications Using Cypress
WorkshopFree
This workshop will teach you the basics of writing useful end-to-end tests using Cypress Test Runner.
We will cover writing tests, covering every application feature, structuring tests, intercepting network requests, and setting up the backend data.
Anyone who knows JavaScript programming language and has NPM installed would be able to follow along.
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
TypeScript Congress 2023TypeScript Congress 2023
131 min
Practice TypeScript Techniques Building React Server Components App
Workshop
In this hands-on workshop, Maurice will personally guide you through a series of exercises designed to empower you with a deep understanding of React Server Components and the power of TypeScript. Discover how to optimize your applications, improve performance, and unlock new possibilities.
 
During the workshop, you will:
- Maximize code maintainability and scalability with advanced TypeScript practices
- Unleash the performance benefits of React Server Components, surpassing traditional approaches
- Turbocharge your TypeScript with the power of Mapped Types
- Make your TypeScript types more secure with Opaque Types
- Explore the power of Template Literal Types when using Mapped Types
 
Maurice will virtually be by your side, offering comprehensive guidance and answering your questions as you navigate each exercise. By the end of the workshop, you'll have mastered React Server Components, armed with a newfound arsenal of TypeScript knowledge to supercharge your React applications.
 
Don't miss this opportunity to elevate your React expertise to new heights. Join our workshop and unlock the potential of React Server Components with TypeScript. Your apps will thank you.