Trees aren't just Foliage: ASTs and Practical Usage

Rate this content
Bookmark

ASTs are prevalent in everything we do. ESLint, Typescript, etc allow us to dive into our source code in a way we might not be familiar with. Let's take a look at how some modern tools use ASTs to improve our lives as developers. Some of the tools to be covered include: TS-Morph, Typescript, TreeSitter, and ESlint. Depending on time, I can dig into various tools and patterns for working with ASTs and how they can fit into any modern developer workflow.

9 min
15 Nov, 2023

AI Generated Video Summary

ASTs, or abstract syntax trees, are used by popular tools like Babel, TypeScript, ESlint, and Prettier to improve the developer experience. They have various use cases including compiling and code analysis. Editor tooling and writing tools can be enhanced using ASTs, with examples including formatting with Prettier, type annotations in JetBrains editors, and code mods for framework upgrades. Ts-morph is a useful tool for code transformations, while Tree Sitter is a portable tool that supports many languages and can be used to build IDEs or text editors in the browser.

1. Introduction to ASTs

Short description:

Today I'll be talking about ASTs and how trees aren't just foliage. ASTs, or abstract syntax trees, represent the important bits of our code. Popular tools like Babel, TypeScript, ESlint, and Prettier use ASTs to improve the developer experience. ASTs have various use cases including compiling and code analysis. ESLint, an AST-based tool, is widely used for code analysis.

Hi, everyone. My name's Chris, and today I'll be talking about ASTs and how trees aren't just foliage. First, who am I? I stream programming way too much on Twitch, I'm a lazy YouTuber, and I'm a senior front end engineer at Fairwinds. At Fairwinds, we can help you optimize and manage your Kubernetes clusters at scale.

So first, what is an AST? It's a type of syntax tree. Concrete syntax trees represent every detail of our code. That means the parentheses, the brackets, the semicolons, the indentation, all of it. Abstract syntax trees actually represent the important bits of our code. So if we think about an if statement, the clause as well as the block are separate nodes but still related and linked together. Let's take a look at an image that explains it a bit more. So here we have an example of a statement sequence. Within that statement sequence, we see a while loop and a return statement. Within the while loop, we see the condition and the body. And within the condition, we see a comparison operator and the things being compared. All of these things kind of branch and tie together, almost like a family tree, which is kind of interesting.

Some popular tools that most of us are using day to day are Babel, TypeScript, ESlint, and Prettier. All of these use ASTs to make our developer experience better. So what are they doing? What are some common use cases for them? Why would you want to use ASTs to write dev tools? Well, some use cases are compiling, code analysis, editor tooling, and code mods. For compiling, we could think maybe about TypeScript being turned into JavaScript via TSC, which stands for TypeScript Compiler. Back in the day, turning ES6 into ES5 via Babel is one way to go. Now we use more modern versions of JavaScript in browsers that might not support them yet. That's what Babel is doing. And technically, Babel used to be called six-to-five when it was limited to ES6 and ES5. We also have JSX being turned into react.createElement. That's another compiler step. And compilers in general for other languages, you know, to turning C into assembly. That's another way of compiling using ASTs. When we think about code analysis, ESLint is probably the first thing we think of. But there's some older tools like JSHint that were not AST-based. The reason the ESLint is the one we all care about and think about these days is because it's easier to write a rule or an extension to ESLint via ASTs rather than the way JSHint was doing it back in the day.

2. Editor Tooling and Writing Tools with ASTs

Short description:

For Editor tooling, we could consider formatting with Prettier, type annotations in JetBrains editors, syntax highlighting with semantic highlighting in VS Code, and code mods for framework upgrades. Writing tools using ASTs involves using TypeScript (ts), unist, tsmorph, and tree sitter. Unist provides a baseline for other tools like HAST, MDEST, ZAST, SAS, SCS, and less. Ts-morph is useful for code transformations like react-code-mod.

For Editor tooling, we could consider formatting. Prettier is breaking our code down into an AST and building it back up again with the standard indentation, where the brackets go, all those things in a pure standard formatted way that we don't have to care about or manually do by hand, which is awesome. Type annotations in JetBrains editors are really cool. So you could get the names of arguments to a function annotated. If you're not passing an object with props, if you're passing ordered arguments, having those named is really nice as a developer experience. They can also annotate inferred types when you're assigning a variable or a return value.

Another really interesting Editor tooling aspect is syntax highlighting. Keep in mind that most syntax highlighting is text-made grammar-based, which is fundamentally regular expressions under the hood, but VS Code ships a feature called semantic highlighting that uses knowledge from the language servers for various languages to give us a little bit more accurate highlighting of what's going on in our code.

And finally, code mods. So think about framework upgrades. Nowadays, we don't have to do import React from React, and if you have a codebase that is doing it, if you've updated your bundlers or compilers, you might not need to do that anymore. So instead, we could have React code mod remove that for us automatically throughout the entire codebase instead of going file by file, manually removing those things. Another option is turning function dot bind this into an arrow function, which is what a dot bind this for a function is doing under the hood, but it's just syntactic sugar. All of these things are just nice for framework and library authors to implement as things that help us upgrade and stay up to date, because they don't want to treat and deal with bugs for legacy versions of their framework anyway.

So how could you write some tools using ASTs? What are some tools or write tools? Well, let's look at ts, that is TypeScript. Let's look at unist, tsmorph, and tree sitter. Those are the ones that I'm really going to dig into. So ts is literally TypeScript. You would just import TypeScript, do ts dot create source file from the text snippet that you have, and then you can iterate every node within that tree using for-each-child, looking for the node type that you're caring about. Here, I'm looking for an A class declaration. Unist is very interesting because there is no standard for how ASTs should specify props and how we can understand what parts of the code they're from and what type of node it is, etc. One thing to remember, though, is that unist is not intended to be self-sufficient, it's a baseline for other things to use. So, HAST for HTML, MDEST for Markdown, ZAST for XML, or SAS for CSS, SCS, and less. All of these tools add their own props to that node type that help give us a little more information about what the code is doing and more metadata that we can use to manipulate that code or understand it better. One important thing is there's something like 32 utility functions written for unist, and there's probably even more. All of these utility functions can be used for any unist compatible syntax tree, which means you don't have to write a lot of those like visitor pattern functions by hand, which is really cool. ts-morph is very interesting because it is useful for code transformations. Think code mods like react-code-mod. I believe react-code-mod uses JS code shift, but I could be wrong about that. But still, ts-morph is something I have experience with from my own usage, and it's quite useful.

3. AST Tools and Tree Sitter

Short description:

One thing to remember is that ts-morph has functions for manipulating code, but you may need to write custom functions for traversing your code. ESLint allows you to create your own AST-based tools and enforce coding standards. Tree sitter is a portable tool written in pure C that supports many languages and can be compiled to wasm. It can be used in the browser to build IDEs or text editors without a server. Some tree sitter parsers may not compile to wasm properly, but you can still support over 40 languages in the browser.

One thing to remember, though, is it is not unist compatible. I believe it's using a very similar node to TypeScript, so you're not going to be able to use any of those utility functions from the unist ecosystem. ts-morph does have some really good functions for manipulating code, but as far as traversing your code, you may want to write some custom functions for that yourself.

And eslint is another interesting way of creating your own AST-based tools. You could write your own rules that enforce coding standards at your organization and that's awesome! One thing to remember though is it's not the same AST structure that you've become familiar with with unist or TypeScript. The reason for that is very simple. It existed before those things were prevalent enough to, you know, build on top of.

And another interesting part of writing a rule for ESLint is that you pass an object to ESLint that has a prop that is a selector and a function for that prop that is a callback function that ESLint is going to call as it traverses the tree. Why is this important? Well, if every rule traversed the tree from top to bottom on its own, ESLint would be really slow. So instead it can do a single traversal, call your callback function when you hit a node that you're looking for, and then from there you can navigate relative to that node that you found. So you can still crawl the tree a bit, but you don't want to do an entire traversal most of the time.

And last but not least, tree sitter is very interesting. It ships with NeoVim by default, I believe, and it supports a lot of languages. It's written in pure C, which makes it really portable. So it's usable in many languages, and that means that it's also compilable to wasm, which is cool. Like you can use it in your browser now. So you could write basically an IDE or a text editor of some kind with you know parser specific knowledge right there in the browser with no server necessary. One thing to remember about the wasm output is that some of the parsers for tree sitter don't necessarily compile to wasm very well. I ran into it a couple years ago, maybe a year ago, where I think like eight of the 40 plus parsers that I had available did not compile to wasm properly which is fine. But maybe they've improved it and there's always room for more to be created. So yeah you could support 40 plus languages in the browser writing your own tool. So yes, I know I kind of glossed over a few things, I jumped a little quickly, but hopefully you have enough keywords and knowledge to go research some of these things and write some tools that will help improve your life and the developers on your team's lives around you.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

JSNation 2023JSNation 2023
29 min
Modern Web Debugging
Few developers enjoy debugging, and debugging can be complex for modern web apps because of the multiple frameworks, languages, and libraries used. But, developer tools have come a long way in making the process easier. In this talk, Jecelyn will dig into the modern state of debugging, improvements in DevTools, and how you can use them to reliably debug your apps.
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Our understanding of performance & user-experience has heavily evolved over the years. Web Developer Tooling needs to similarly evolve to make sure it is user-centric, actionable and contextual where modern experiences are concerned. In this talk, Addy will walk you through Chrome and others have been thinking about this problem and what updates they've been making to performance tools to lower the friction for building great experiences on the web.
Vue.js London 2023Vue.js London 2023
30 min
Stop Writing Your Routes
The more you keep working on an application, the more complicated its routing becomes, and the easier it is to make a mistake. ""Was the route named users or was it user?"", ""Did it have an id param or was it userId?"". If only TypeScript could tell you what are the possible names and params. If only you didn't have to write a single route anymore and let a plugin do it for you. In this talk we will go through what it took to bring automatically typed routes for Vue Router.
React Day Berlin 2023React Day Berlin 2023
21 min
React's Most Useful Types
We don't think of React as shipping its own types. But React's types are a core part of the framework - overseen by the React team, and co-ordinated with React's major releases.In this live coding talk, we'll look at all the types you've been missing out on. How do you get the props type from a component? How do you know what ref a component takes? Should you use React.FC? And what's the deal with JSX.Element?You'll walk away with a bunch of exciting ideas to take to your React applications, and hopefully a new appreciation for the wonders of React and TypeScript working together.
TypeScript Congress 2023TypeScript Congress 2023
31 min
Making Magic: Building a TypeScript-First Framework
I'll dive into the internals of Nuxt to describe how we've built a TypeScript-first framework that is deeply integrated with the user's IDE and type checking setup to offer end-to-end full-stack type safety, hints for layouts, middleware and more, typed runtime configuration options and even typed routing. Plus, I'll highlight what I'm most excited about doing in the days to come and how TypeScript makes that possible not just for us but for any library author.

Workshops on related topic

React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2022React Advanced Conference 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Featured Workshop
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
TypeScript Congress 2023TypeScript Congress 2023
131 min
Practice TypeScript Techniques Building React Server Components App
Workshop
In this hands-on workshop, Maurice will personally guide you through a series of exercises designed to empower you with a deep understanding of React Server Components and the power of TypeScript. Discover how to optimize your applications, improve performance, and unlock new possibilities.
 
During the workshop, you will:
- Maximize code maintainability and scalability with advanced TypeScript practices
- Unleash the performance benefits of React Server Components, surpassing traditional approaches
- Turbocharge your TypeScript with the power of Mapped Types
- Make your TypeScript types more secure with Opaque Types
- Explore the power of Template Literal Types when using Mapped Types
 
Maurice will virtually be by your side, offering comprehensive guidance and answering your questions as you navigate each exercise. By the end of the workshop, you'll have mastered React Server Components, armed with a newfound arsenal of TypeScript knowledge to supercharge your React applications.
 
Don't miss this opportunity to elevate your React expertise to new heights. Join our workshop and unlock the potential of React Server Components with TypeScript. Your apps will thank you.
TypeScript Congress 2022TypeScript Congress 2022
116 min
Advanced TypeScript types for fun and reliability
Workshop
If you're looking to get the most out of TypeScript, this workshop is for you! In this interactive workshop, we will explore the use of advanced types to improve the safety and predictability of your TypeScript code. You will learn when to use types like unknown or never. We will explore the use of type predicates, guards and exhaustive checking to make your TypeScript code more reliable both at compile and run-time. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
Are you familiar with the basics of TypeScript and want to dive deeper? Then please join me with your laptop in this advanced and interactive workshop to learn all these topics and more.
You can find the slides, with links, here: http://theproblemsolver.nl/docs/ts-advanced-workshop.pdf
And the repository we will be using is here: https://github.com/mauricedb/ts-advanced
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.
Node Congress 2021Node Congress 2021
245 min
Building Serverless Applications on AWS with TypeScript
Workshop
This workshop teaches you the basics of serverless application development with TypeScript. We'll start with a simple Lambda function, set up the project and the infrastructure-as-a-code (AWS CDK), and learn how to organize, test, and debug a more complex serverless application.
Table of contents:        - How to set up a serverless project with TypeScript and CDK        - How to write a testable Lambda function with hexagonal architecture        - How to connect a function to a DynamoDB table        - How to create a serverless API        - How to debug and test a serverless function        - How to organize and grow a serverless application


Materials referred to in the workshop:
https://excalidraw.com/#room=57b84e0df9bdb7ea5675,HYgVepLIpfxrK4EQNclQ9w
DynamoDB blog Alex DeBrie: https://www.dynamodbguide.com/
Excellent book for the DynamoDB: https://www.dynamodbbook.com/
https://slobodan.me/workshops/nodecongress/prerequisites.html