Start Building Your Own JavaScript Tools

Rate this content
Bookmark

Your first JavaScript tool might not be the next Babel or ESLint, but it can be built on them! Let's demystify the secret art of JavaScript tools, how they work, and how to build our own. We'll discover the opportunities in our everyday work to apply these techniques, writing our own ESLint rules to prevent mistakes and code transforms to make breaking changes easy to apply. We’ll walk through the fundamentals of working with an abstract syntax tree, and develop our understanding through a live-code. You will be amazed at what you can build, and together we’ll explore how to get started.

22 min
05 Jun, 2023

AI Generated Video Summary

[♪ music ♪ by The Illuminati plays)] I see a common thread across any project I work on. Different developers are making the same mistake and we have preferred ways of doing things. Preventing mistakes and sharing best practices are great reasons to look at tools like linters and in particular ESLint. Let's write our first rule together. We're just scratching the surface of building our own tools, which can have a massive impact on improving the developer experience.

1. Introduction to Building Custom Tools with ESLint

Short description:

[♪ music ♪ by The Illuminati plays)] I see a common thread across any project I work on. Different developers are making the same mistake and we have preferred ways of doing things. Preventing mistakes and sharing best practices are great reasons to look at tools like linters and in particular ESLint. Let's write our first rule together. I'm going to share a little about what I work on and a problem I like to solve where I think would be really helpful. Right now, I work on a design system called Canvas and we provide a component library for UI developers to use. We invest a lot into making our components accessible and useful across a wide range of use cases. Let's dive in.

[♪ music ♪ by The Illuminati plays)] [♪ music ♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ by The Illuminati plays♪ show you how. What if we could get started with building our own tools? Where would we begin? What if the tools were specific to our project? Uniquely helpful and tailored to our needs? What if we could learn how to do this in the next 20 minutes learning what we need to know and how to take our first steps? In the next 20 minutes, let's write our first printing rule, write our first code mod, look at our first AST and build our understanding of them, and learn the fundamentals of how our tools work. Let's dive in.

I see a common thread across any project I work on. Different developers are making the same mistake and we have preferred ways of doing things while it's hard to communicate that to everyone else on our project or using our library. Preventing mistakes and sharing best practices are great reasons to look at tools like linters and in particular ESLint. Not only can we use ESLint's existing rules to handle many problems, but it has a rich ecosystem of plugins specific to different areas and we can write our own rules and plugins, which I'll show you how. ESLint rules are great for problems or mistakes that keep coming up, creating a better feedback loop for developers as they're writing their code, automating code review feedback, sharing best practices, and preventing anti-patterns in our code bases. Rules can be for any project, maybe just for your app, helping developers on your immediate team or others in a shared code base. If you maintain a library, they can help anyone using your library. Anyone can write these rules, not just project maintainers or expert developers.

All right, let's write our first rule together. I'm going to share a little about what I work on and a problem I like to solve where I think would be really helpful. Right now, I work on a design system called Canvas and we provide a component library for UI developers to use. We invest a lot into making our components accessible and useful across a wide range of use cases. As part of that effort, we maintain three different packages for our components, main, preview, and labs. We usually build new components in labs or preview, and as we prove out the API, refine design and make sure accessibility is perfected, we'll promote things from labs and preview up to main. This presents a somewhat unique challenge though. We have some old components, usually in main, that are going to be replaced by newer ones in labs or preview. When a developer is choosing between the two versions though, which way do they go? Sometimes, we really want to tell developers to use the newer one in labs or preview. We already plan to promote it to main in an upcoming release. The component is ready to use and has some benefits we don't want our users to miss out on. How do we communicate this? We do what we can with our docs and sending out announcements, but the best way is to tell them as they write their code. I'm pulling up an online tool that I use to test out new rules. Usually I'll develop with a test suite writing, Lint rules is a really good use case for test driven development. This tool is great to explore the problem, get a first draft, and often that's enough. This is astexplorer.net. There are a few things going on here, but I'll explain how each part works. On the left, there are two editors placed for the code we're working on on the top left and editor for writing an AS Lint rule down below and other tools as well. On the right, we have an AST viewer up above and the output window for debugging below that.

2. Understanding AST and Sample Code

Short description:

In the header, we can configure the parser and transform type. The AST viewer is amazing and we're going to explore it. An AST is just a tree that stores the syntax of our code. We can tell ESLint what we care about by specifying node types. Looking at this gives us everything we need to detect patterns. Let's look at some sample code that includes imports for our components. We have a not preferred situation with a segmented control component. When we selected the transform for ESLint V8, we got some sample code with a create function that returns an object literal.

In the header, we can also configure the parser and transform type which I've already set to Babel AS Lint parser and AS Lint v8. Besides giving us a really useful development environment, this AST viewer in the top right is amazing and we're going to explore it in a bit.

AST is an acronym for abstract syntaxtry. If you didn't study computer science at university though, the terminology for an AST might seem intimidating. I promise you though, it's just a tree and if you've worked with a DOM for a webpage, you know how a tree works. An AST just has the syntax for our code stored in it. We can explore this tree in the AST Explorer. It's also worth mentioning that the tree is basically a JavaScript object. There's nothing really new here. Like an HTML tag at the top of a webpage, that being the top node of the tree in the DOM, the top of the AST is the program node. This program node has child nodes such as the ones under body. Here we've got multiple import declarations and these import declarations have children of their own and so on. There's another type of property on a node that's very important, the type. This is how we're going to tell ESLint what we care about. We're going to tell it this for this node type. We want to hear about it and ESLint is going to handle walking the tree and any time it node that we care about it's going to pass it into a function that we can control and define. There are also some properties to describe each node, like the source type being module on the program. There are other values that we can look at that can be really helpful depending on the node we're working with. Looking at this gives us everything we need to detect patterns which is what an ESLint rule is doing.

Let's look at some sample code that includes imports for our components. On the top left we've got a couple different examples, a few actually. First there's the not preferred situation where we have a segmented control component that is coming from our main package CanvasKit React here. That is not preferred because we have a newer one in CanvasKit Preview React. This is our preview package. We want to direct developers to using that over our main package at some point. When we selected the transform for ESLint V8, we got some sample code in the bottom left. This is awesome because it's going to give us a place to kick off. I've cleaned it up a little bit down to its bare essentials. What we have is a create function. This create function returns an object literal.

3. Analyzing Import Declarations

Short description:

This object it returns can have any number of properties. We care about import declarations and want to narrow down the conditions for reporting. We want to look for when we're importing from the main package and when the component name is segmented control.

This object it returns can have any number of properties. These propertys are node types that we care about. In this case, the example has a template literal. Anytime that there's a template literal it's going to get that node. And immediately it's going to report that we have a template literal. We don't care about template literals though.

We care about something else here. Let's click on the import statement in our code. This is going to highlight something in the tree view. And here we see an import declaration. That's what we care about. We want to look at these import declarations and take a closer look.

So let's change this to import declaration. And now we have a rule firing on all these import declarations. All right. Let's narrow down now. We don't want to just report any import declaration. We want to do it conditionally. So I'm going to create a couple conditions that we want to look for. So we want to look for when the, is, when we're importing from the main package. So is main package. I'm just going to put true for that for a moment. Then we also want to look for the component name, being segmented control. So we're going to say, is segmented control. Cool. Also going to say that's true. And then I'm going to wrap this context.report in those two conditions. And then we're going to make those conditions functional. These two conditions are true. That's what we want to know.

4. Analyzing JavaScript Code for Specific Conditions

Short description:

Let's make them smart now. We want to check if the value is happening on a part of the node. We can look at the node.source.value and compare it against the main package. The other condition is if it's the segmented control. We look at the node.imported.name off of the node.specifiers array to find one segmented control.

And there we go. All right. Let's make them smart now. OK, looking at the main package, that would be this right here. We want to check to see when this value is happening on a part of the node. So if we look at the node, we got the import declaration. We see we're dealing with a source. And the value is under the source.value. So we can look at the node.source.value and compare it against this value here, which is the main package right there. All right, let me indent this a little bit so it's a little bit more readable. It's a little bit bigger here. Cool. So that's the first condition. And already we can see it narrowing down to just one situation here. That's good.

The other condition we want to look at is if it's the segmented control. So if we click on segmented control, this will focus the appropriate nodes in the tree view. And we can see there's an imported and a local. If we look around at a different one, we can see that the local is what we alias something to, if that's different. And the imported is the name that the package is giving us. So back over here in this example, they actually are one and the same, but it's the imported.name that we care about off of the node. So let's look at that. We're going to look at the node.imported. Sorry, actually, I missed something here. This is under the specifiers. All right, so we need to look at the specifiers. And this is an array, so we need to find something in the array. We need to find one segmented control to know that we're dealing with something we care about. node.specifiers. We can find within here, and then that will give us a specifier.

5. Testing ESLint Rules for Code Quality Control

Short description:

And we want to look for when specifier.imported.name matches segmented control. This is our first ESLint rule. We have a return of an object here, and we're defining the node type that we care about, the import declaration. We have some conditions looking at different children and different properties on that node, like looking at the source and looking at its value, looking through its specifiers, iterating through them, and finding one that matches a name that we care about. There is a preferred version in preview. We're giving a helpful message to developers using our project. If I was to package up this rule and get it ready for consumers, I'd write out a test suite and add support for other duplicate components we have in our next release. There's also the option to implement a fixture in ESLint, which is essentially a code mod. Let's look at code mods next. A code mod automates a code refactor or change in syntax.

And we want to look for when specifier.imported.name matches segmented control. All right, move this into its own line. Format this a little bit so it's easy to read. Cool. And it's still finding just that one example. Let me take off, or let's see. We are filtering based on that, or filtering based on that. OK, let's try changing some things. We take this and make it just a little bit different. The rule's not fired, so we're filtering on that name properly. And if we change this to just something else, it's not firing on that. That's not even a valid name, but we'll just put this back. Cool.

This is our first ESLint rule. In review, we have a return of an object here, and we're defining the node type that we care about, the import declaration. We have some conditions looking at different children and different properties on that node, like looking at the source and looking at its value, looking through its specifiers, iterating through them, and finding one that matches a name that we care about. And when those things happen, oh, let's give a useful message. There is a preferred version in preview. There we go. Now, if we come over here on the right in the output, we can see that we're giving a helpful message to developers using our project. There's more we can do, like moving this caret indicating where we need to make a change over to where we have the package name. And we can do things like write tests and all that. But this is just a good, quick, little, ESLint rule that we wrote that would be very helpful to our users. So let's take a look at what else we would do.

Like I was saying, if I was to package up this rule and get it ready for consumers, I'd write out a test suite and add support for other duplicate components we have in our next release, recommend what is preferred for each, if there's one in preview, or labs that people should be using when they're ready. There's also the option to implement a fixture in ESLint, which is essentially a code mod. I didn't do that because there's another tool we'll look at that's dedicated to that purpose, especially when we're doing things like what we are with breaking API changes. Let's look at code mods next. A code mod automates a code refactor or change in syntax. They can be written by maintainers of a library, users of a library, or really any developer that has a bunch of changes to apply across a code base.

6. Using Code Mods and JS CodeShift

Short description:

Code mods are great for making quick upgrades and repetitive changes. They save time and reduce the risk of mistakes. They are especially valuable for library maintainers. My first experience with a code mod was in 2018 when we used one to migrate our library. Now, on my design system team, we use code mods to make upgrading easier. The most popular tool for writing code mods is JS CodeShift. Let's look at an example in AST Explorer, where we update import packages and use JS CodeShift to replace identifiers.

Code mods are great for improving how quickly consumers can upgrade, say, to a new major version that has breaking changes. They don't need to manually change their code and can just run a script. That's pretty easy. They're also great for making hundreds or even thousands of small repetitive changes. I'd rather spend an hour writing code for a code mod than hand editing hundreds of files, which can take longer and easily result in mistakes. In particular, if you're maintaining a library, it's something that could be really valuable to provide your users.

Let me share with you my first experience using a code mod. Back in 2018, I was working on a product team. We're changing our library for CSS and JS and styled components. We were moving from glamorous to emotion. And while considering our options and the effort required, we found a code mod from another speaker at this conference, Tejas Kumar. Tejas is fantastic, and so is his work. His code mod was perfect for helping us migrate. We ran it on our codebase, and it held most of the effort. Fast forward to 2021, and on my design system team, we were releasing a major version of our library that had some major changes to our APIs. So we decided to start including code mods to make it easier for teams to upgrade. The most popular tool for writing code mods is JS CodeShift.

Let's jump back into AST Explorer and look at an example. Revisiting my previous example, we often promote components from our Labs and Preview packages to Main. Let's look at a case where we want to update the import package from Preview, then to the Main one. Let's say we're promoting segmented control in our next release, and we've already gone ahead and set the parser to Babel and the transform to JS CodeShift. Again, this is going to give us some really handy boilerplate in the bottom left. We still have our tree view in the top right, and we have some output and debugging on the bottom right. The default example's a little bit fun. It's going and finding identifiers, like segmented control and menu here, and just reversing the strings. Cute. There are some differences now with how you write a JS CodeShift code mod versus an ESLint rule. There are also some similarities. Like before, we want to match on a certain identifier to make sure a certain thing, like a node type. And then for a given node type, we want to look for situations that we care about, like replacing all the identifiers in this example.

7. Modifying Import Declarations

Short description:

What's different here is that we're doing things like to source and do we have this chaining API going on. We want to change the preview in the package name to import from the main package. There's another component from menu that's still in preview. Let's modify the import declaration by filtering the ones that match the path we care about.

What's different here is that we're doing things like to source and do we have this chaining API going on. There's some things we can do to break that apart, but this is fine for now. There's also this j API. That's api.js code shift, and the convention is to alias it to j, so we can easily access things like j.identifier, which creates a new identifier node. So we can create new nodes and manipulate existing ones and really mutate what we have to change it for our needs.

So back to our example, we have this string here in the preview in the package name. And that's what we want to change. We want to take that preview and take that out and make it import from the main package, because we're promoting this to the main package. We don't want to mutate this one. There's another component from menu. That's still in preview, and that's not getting promoted yet. So let's see what it takes to modify this.

All right. Like before, we're working with an import declaration. So let's grab that. And let's find all of those. OK, we're going to find all of the import declarations. And what do we do next? Let's filter down to just the ones that match the path we care about. So we can do a filter. And instead of just a node, we actually have a path which contains a node. So we're going to take the path here and pass that into the function. And we care about path dot, what is this again? Again, I'll use the tree view. I'll click on the string. And we are looking at the import declarations source dot value. Cool. So path dot node dot source dot value. And we're going to check if it matches the current value. And I'm going to take for a moment, I'm going to take this whole string and make a variable for that. That's very handy. That's old import.

8. Creating Code Mods to Automate Code Refactoring

Short description:

We're going to make a new import without the preview bit. We'll compare the node source value to the old import and replace it with an import declaration. We can reuse the specifiers from the current import and create a new import using J.string literal. Let's take a look at the result: import segmented control.

And we're going to make a new import as well. We'll add it. And that's going to be the same thing without that preview bit. Cool. All right.

So we're going to compare that node source value. When it matches the old import, that's what we care about. And next, for each of those cases that we do find, we want to replace it. Not with an identifier, but with an import declaration. And we're going to create it like this. Let me get rid of all of that. And this is nice. We've got a little bit of code suggestion here saying that first we need the specifiers. And then the second thing we need is the source value. So the cool thing is we can reuse the specifiers from the current import. And we can go path. What was this called again? That's these things. Path.node.specifiers. That's it. Path.node.specifiers. Node.specifiers. And the second thing would be the new import. We can't just give it a string value. We actually have to give it an AST node type. And we can create that using J.string literal. Giving it that string value will give us something that we can use here. And oh my gosh, I think it works. Let's take a look. So import segmented control. There it is.

9. Updating Imports and Code Mods

Short description:

We've updated the import effectively without affecting other imports. Code mods can be challenging to write and can accidentally modify unintended code. Extracting utilities and having a set of test cases can help address these challenges. Codeshiftcommunity.com provides examples, best practices, and helpful tools for writing code mods. Our team recently released a major version with breaking changes, and we received positive feedback on the code mod's ease of use and the seamless upgrade process.

We don't have the preview on there anymore. We've updated this effectively and we haven't touched the other import that we didn't want to change. Going back to the source to just quickly review. We're finding by a certain node type. We're filtering based on some criteria like the old import is present here. And then we're replacing a node that we care about, particularly this import declaration. We're giving a new import declaration. Using the specifiers here and then we're passing in the new import value that is the main package. And we're promoting our components. This is what we do when we promote things in our new major versions. We have some richer code mods that handle a bunch of other edge cases, but this gives you the basic idea of what's going on.

Cool. Live code, fun. OK. I've heard this a lot from a lot of folks I talk to and even seen some of the challenges within my own team. It's not always easy to write good code mods that work in all use cases. Code mods can be non-trivial to write, and it's easy to accidentally target code that we didn't intend to modify. We will need good tests. We can work around this by extracting utilities to tackle common problems in our domain, having a set of test cases that we always apply to a certain type of code mod. And look at Codeshiftcommunity.com. It's a community project that offers really great examples, best practices, and some helpers you can even import to make your code mods easier to write.

This month, our team released a new major version with breaking changes. And we had several component APIs change, which could bother our users, but instead, this is one of our reactions. Hey, team. Longtime listener, first time caller. Just wanted to parachute in and mention how awesome the V9 code mod was to use. It made upgrading so, so easy. I love to see this. I've never met Simon, but he carries a lot of respect in our community, so this meant a lot coming from him. And this kind of feedback is actually kind of typical when you're automating all the breaking changes and making upgrades seamless as possible.

10. Building Your Own Tools and Conclusion

Short description:

We're just scratching the surface of building our own tools, which can have a massive impact on improving the developer experience. Building tools to solve problems is what our career is about. Get started with building your own JavaScript tools and check out the ESLint docs and CodeShift Community website for more information. Visit astsareawesome.com for a quick review and additional resources. Thank you for listening!

We're still only scratching the surface of what we can do with building our own tools. Whenever you do this, you're improving the developer experience for yourself and every other developer in your community. The impact can be massive, maybe 10x, maybe 1,000x. More importantly, the satisfaction is immeasurable.

Developers appreciate when their work becomes easier. Building tools that solve a problem is what our whole career is about. Doing this for each other is one of my favorite ways to work. I hope you have some ideas now about how to get started with building your own JavaScript tools.

To learn more about working with ESLint, I recommend their docs at eslint.org. If you're working with codemods, the best place to go is the CodeShift Community website. And for a quick review of everything I just talked about, links to even more resources, check out a site I built, astsareawesome.com. Thank you, everyone, for listening. Please send me your questions, ideas, and thoughts on this topic any time. Thank you.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

JSNation Live 2021JSNation Live 2021
31 min
Vite: Rethinking Frontend Tooling
Vite is a new build tool that intends to provide a leaner, faster, and more friction-less workflow for building modern web apps. This talk will dive into the project's background, rationale, technical details and design decisions: what problem does it solve, what makes it fast, and how does it fit into the JS tooling landscape.
JSNation 2022JSNation 2022
21 min
The Future of Performance Tooling
Our understanding of performance & user-experience has heavily evolved over the years. Web Developer Tooling needs to similarly evolve to make sure it is user-centric, actionable and contextual where modern experiences are concerned. In this talk, Addy will walk you through Chrome and others have been thinking about this problem and what updates they've been making to performance tools to lower the friction for building great experiences on the web.

Workshops on related topic

React Day Berlin 2022React Day Berlin 2022
86 min
Using CodeMirror to Build a JavaScript Editor with Linting and AutoComplete
WorkshopFree
Using a library might seem easy at first glance, but how do you choose the right library? How do you upgrade an existing one? And how do you wade through the documentation to find what you want?
In this workshop, we’ll discuss all these finer points while going through a general example of building a code editor using CodeMirror in React. All while sharing some of the nuances our team learned about using this library and some problems we encountered.
JSNation 2023JSNation 2023
44 min
Solve 100% Of Your Errors: How to Root Cause Issues Faster With Session Replay
WorkshopFree
You know that annoying bug? The one that doesn’t show up locally? And no matter how many times you try to recreate the environment you can’t reproduce it? You’ve gone through the breadcrumbs, read through the stack trace, and are now playing detective to piece together support tickets to make sure it’s real.
Join Sentry developer Ryan Albrecht in this talk to learn how developers can use Session Replay - a tool that provides video-like reproductions of user interactions - to identify, reproduce, and resolve errors and performance issues faster (without rolling your head on your keyboard).
JSNation Live 2021JSNation Live 2021
86 min
Build React-like apps for internal tooling 10x faster with Retool
Workshop
Most businesses have to build custom software and bespoke interfaces to their data in order to power internal processes like user trial extensions, refunds, inventory management, user administration, etc. These applications have unique requirements and often, solving the problem quickly is more important than appearance. Retool makes it easy for js developers to rapidly build React-like apps for internal tools using prebuilt API and database interfaces as well as reusable UI components. In this workshop, we’ll walk through how some of the fastest growing businesses are doing internal tooling and build out some simple apps to explain how Retool works off of your existing JavaScript and ReactJS knowledge to enable rapid tool building.
Prerequisites:A free Retool.com trial accountSome minimal JavaScript and SQL/NoSQL database experience
Retool useful link: https://docs.retool.com/docs