Trees aren't just Foliage: ASTs and Practical Usage

Rate this content
Bookmark

ASTs are prevalent in everything we do. ESLint, Typescript, etc allow us to dive into our source code in a way we might not be familiar with. Let's take a look at how some modern tools use ASTs to improve our lives as developers. Some of the tools to be covered include: TS-Morph, Typescript, TreeSitter, and ESlint. Depending on time, I can dig into various tools and patterns for working with ASTs and how they can fit into any modern developer workflow.

Chris Griffing
Chris Griffing
9 min
15 Nov, 2023

Comments

Sign in or register to post your comment.

Video Summary and Transcription

ASTs, or abstract syntax trees, are used by popular tools like Babel, TypeScript, ESlint, and Prettier to improve the developer experience. They have various use cases including compiling and code analysis. Editor tooling and writing tools can be enhanced using ASTs, with examples including formatting with Prettier, type annotations in JetBrains editors, and code mods for framework upgrades. Ts-morph is a useful tool for code transformations, while Tree Sitter is a portable tool that supports many languages and can be used to build IDEs or text editors in the browser.

Available in Español

1. Introduction to ASTs

Short description:

Today I'll be talking about ASTs and how trees aren't just foliage. ASTs, or abstract syntax trees, represent the important bits of our code. Popular tools like Babel, TypeScript, ESlint, and Prettier use ASTs to improve the developer experience. ASTs have various use cases including compiling and code analysis. ESLint, an AST-based tool, is widely used for code analysis.

Hi, everyone. My name's Chris, and today I'll be talking about ASTs and how trees aren't just foliage. First, who am I? I stream programming way too much on Twitch, I'm a lazy YouTuber, and I'm a senior front end engineer at Fairwinds. At Fairwinds, we can help you optimize and manage your Kubernetes clusters at scale.

So first, what is an AST? It's a type of syntax tree. Concrete syntax trees represent every detail of our code. That means the parentheses, the brackets, the semicolons, the indentation, all of it. Abstract syntax trees actually represent the important bits of our code. So if we think about an if statement, the clause as well as the block are separate nodes but still related and linked together. Let's take a look at an image that explains it a bit more. So here we have an example of a statement sequence. Within that statement sequence, we see a while loop and a return statement. Within the while loop, we see the condition and the body. And within the condition, we see a comparison operator and the things being compared. All of these things kind of branch and tie together, almost like a family tree, which is kind of interesting.

Some popular tools that most of us are using day to day are Babel, TypeScript, ESlint, and Prettier. All of these use ASTs to make our developer experience better. So what are they doing? What are some common use cases for them? Why would you want to use ASTs to write dev tools? Well, some use cases are compiling, code analysis, editor tooling, and code mods. For compiling, we could think maybe about TypeScript being turned into JavaScript via TSC, which stands for TypeScript Compiler. Back in the day, turning ES6 into ES5 via Babel is one way to go. Now we use more modern versions of JavaScript in browsers that might not support them yet. That's what Babel is doing. And technically, Babel used to be called six-to-five when it was limited to ES6 and ES5. We also have JSX being turned into react.createElement. That's another compiler step. And compilers in general for other languages, you know, to turning C into assembly. That's another way of compiling using ASTs. When we think about code analysis, ESLint is probably the first thing we think of. But there's some older tools like JSHint that were not AST-based. The reason the ESLint is the one we all care about and think about these days is because it's easier to write a rule or an extension to ESLint via ASTs rather than the way JSHint was doing it back in the day.

2. Editor Tooling and Writing Tools with ASTs

Short description:

For Editor tooling, we could consider formatting with Prettier, type annotations in JetBrains editors, syntax highlighting with semantic highlighting in VS Code, and code mods for framework upgrades. Writing tools using ASTs involves using TypeScript (ts), unist, tsmorph, and tree sitter. Unist provides a baseline for other tools like HAST, MDEST, ZAST, SAS, SCS, and less. Ts-morph is useful for code transformations like react-code-mod.

For Editor tooling, we could consider formatting. Prettier is breaking our code down into an AST and building it back up again with the standard indentation, where the brackets go, all those things in a pure standard formatted way that we don't have to care about or manually do by hand, which is awesome. Type annotations in JetBrains editors are really cool. So you could get the names of arguments to a function annotated. If you're not passing an object with props, if you're passing ordered arguments, having those named is really nice as a developer experience. They can also annotate inferred types when you're assigning a variable or a return value.

Another really interesting Editor tooling aspect is syntax highlighting. Keep in mind that most syntax highlighting is text-made grammar-based, which is fundamentally regular expressions under the hood, but VS Code ships a feature called semantic highlighting that uses knowledge from the language servers for various languages to give us a little bit more accurate highlighting of what's going on in our code.

And finally, code mods. So think about framework upgrades. Nowadays, we don't have to do import React from React, and if you have a codebase that is doing it, if you've updated your bundlers or compilers, you might not need to do that anymore. So instead, we could have React code mod remove that for us automatically throughout the entire codebase instead of going file by file, manually removing those things. Another option is turning function dot bind this into an arrow function, which is what a dot bind this for a function is doing under the hood, but it's just syntactic sugar. All of these things are just nice for framework and library authors to implement as things that help us upgrade and stay up to date, because they don't want to treat and deal with bugs for legacy versions of their framework anyway.

So how could you write some tools using ASTs? What are some tools or write tools? Well, let's look at ts, that is TypeScript. Let's look at unist, tsmorph, and tree sitter. Those are the ones that I'm really going to dig into. So ts is literally TypeScript. You would just import TypeScript, do ts dot create source file from the text snippet that you have, and then you can iterate every node within that tree using for-each-child, looking for the node type that you're caring about. Here, I'm looking for an A class declaration. Unist is very interesting because there is no standard for how ASTs should specify props and how we can understand what parts of the code they're from and what type of node it is, etc. One thing to remember, though, is that unist is not intended to be self-sufficient, it's a baseline for other things to use. So, HAST for HTML, MDEST for Markdown, ZAST for XML, or SAS for CSS, SCS, and less. All of these tools add their own props to that node type that help give us a little more information about what the code is doing and more metadata that we can use to manipulate that code or understand it better. One important thing is there's something like 32 utility functions written for unist, and there's probably even more. All of these utility functions can be used for any unist compatible syntax tree, which means you don't have to write a lot of those like visitor pattern functions by hand, which is really cool. ts-morph is very interesting because it is useful for code transformations. Think code mods like react-code-mod. I believe react-code-mod uses JS code shift, but I could be wrong about that. But still, ts-morph is something I have experience with from my own usage, and it's quite useful.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Modern Web Debugging
JSNation 2023JSNation 2023
29 min
Modern Web Debugging
Top Content
Few developers enjoy debugging, and debugging can be complex for modern web apps because of the multiple frameworks, languages, and libraries used. But, developer tools have come a long way in making the process easier. In this talk, Jecelyn will dig into the modern state of debugging, improvements in DevTools, and how you can use them to reliably debug your apps.
The Future of Performance Tooling
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Top Content
Our understanding of performance & user-experience has heavily evolved over the years. Web Developer Tooling needs to similarly evolve to make sure it is user-centric, actionable and contextual where modern experiences are concerned. In this talk, Addy will walk you through Chrome and others have been thinking about this problem and what updates they've been making to performance tools to lower the friction for building great experiences on the web.
TypeScript and React: Secrets of a Happy Marriage
React Advanced Conference 2022React Advanced Conference 2022
21 min
TypeScript and React: Secrets of a Happy Marriage
Top Content
TypeScript and React are inseparable. What's the secret to their successful union? Quite a lot of surprisingly strange code. Learn why useRef always feels weird, how to wrangle generics in custom hooks, and how union types can transform your components.
React's Most Useful Types
React Day Berlin 2023React Day Berlin 2023
21 min
React's Most Useful Types
Top Content
We don't think of React as shipping its own types. But React's types are a core part of the framework - overseen by the React team, and co-ordinated with React's major releases.In this live coding talk, we'll look at all the types you've been missing out on. How do you get the props type from a component? How do you know what ref a component takes? Should you use React.FC? And what's the deal with JSX.Element?You'll walk away with a bunch of exciting ideas to take to your React applications, and hopefully a new appreciation for the wonders of React and TypeScript working together.
Stop Writing Your Routes
Vue.js London 2023Vue.js London 2023
30 min
Stop Writing Your Routes
The more you keep working on an application, the more complicated its routing becomes, and the easier it is to make a mistake. ""Was the route named users or was it user?"", ""Did it have an id param or was it userId?"". If only TypeScript could tell you what are the possible names and params. If only you didn't have to write a single route anymore and let a plugin do it for you. In this talk we will go through what it took to bring automatically typed routes for Vue Router.
Making Magic: Building a TypeScript-First Framework
TypeScript Congress 2023TypeScript Congress 2023
31 min
Making Magic: Building a TypeScript-First Framework
I'll dive into the internals of Nuxt to describe how we've built a TypeScript-first framework that is deeply integrated with the user's IDE and type checking setup to offer end-to-end full-stack type safety, hints for layouts, middleware and more, typed runtime configuration options and even typed routing. Plus, I'll highlight what I'm most excited about doing in the days to come and how TypeScript makes that possible not just for us but for any library author.

Workshops on related topic

React, TypeScript, and TDD
React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
Paul Everitt
Paul Everitt
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
Best Practices and Advanced TypeScript Tips for React Developers
React Advanced Conference 2022React Advanced Conference 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Top Content
Featured Workshop
Maurice de Beijer
Maurice de Beijer
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Deep TypeScript Tips & Tricks
Node Congress 2024Node Congress 2024
83 min
Deep TypeScript Tips & Tricks
Top Content
Workshop
Josh Goldberg
Josh Goldberg
TypeScript has a powerful type system with all sorts of fancy features for representing wild and wacky JavaScript states. But the syntax to do so isn't always straightforward, and the error messages aren't always precise in telling you what's wrong. Let's dive into how many of TypeScript's more powerful features really work, what kinds of real-world problems they solve, and how to wrestle the type system into submission so you can write truly excellent TypeScript code.
Practice TypeScript Techniques Building React Server Components App
TypeScript Congress 2023TypeScript Congress 2023
131 min
Practice TypeScript Techniques Building React Server Components App
Workshop
Maurice de Beijer
Maurice de Beijer
In this hands-on workshop, Maurice will personally guide you through a series of exercises designed to empower you with a deep understanding of React Server Components and the power of TypeScript. Discover how to optimize your applications, improve performance, and unlock new possibilities.
 
During the workshop, you will:
- Maximize code maintainability and scalability with advanced TypeScript practices
- Unleash the performance benefits of React Server Components, surpassing traditional approaches
- Turbocharge your TypeScript with the power of Mapped Types
- Make your TypeScript types more secure with Opaque Types
- Explore the power of Template Literal Types when using Mapped Types
 
Maurice will virtually be by your side, offering comprehensive guidance and answering your questions as you navigate each exercise. By the end of the workshop, you'll have mastered React Server Components, armed with a newfound arsenal of TypeScript knowledge to supercharge your React applications.
 
Don't miss this opportunity to elevate your React expertise to new heights. Join our workshop and unlock the potential of React Server Components with TypeScript. Your apps will thank you.
Advanced TypeScript types for fun and reliability
TypeScript Congress 2022TypeScript Congress 2022
116 min
Advanced TypeScript types for fun and reliability
Workshop
Maurice de Beijer
Maurice de Beijer
If you're looking to get the most out of TypeScript, this workshop is for you! In this interactive workshop, we will explore the use of advanced types to improve the safety and predictability of your TypeScript code. You will learn when to use types like unknown or never. We will explore the use of type predicates, guards and exhaustive checking to make your TypeScript code more reliable both at compile and run-time. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Are you familiar with the basics of TypeScript and want to dive deeper? Then please join me with your laptop in this advanced and interactive workshop to learn all these topics and more.
You can find the slides, with links, here: http://theproblemsolver.nl/docs/ts-advanced-workshop.pdf
And the repository we will be using is here: https://github.com/mauricedb/ts-advanced
Mastering Node.js Test Runner
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Marco Ippolito
Marco Ippolito
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.