Plug-in architecture: how TypeScript let us paint-by-numbers

Rate this content
Bookmark

Adding a new feature to an unfamiliar codebase can be painfully slow. What if you could lean on TypeScript to guide you?

At Snyk plugin architecture is a common pattern to extend language support across different products and codebases. Adding TypeScript to the mix has allowed us to add new features quickly and it can be as simple as painting by numbers when it comes to extending the code.

Join me as we create a simple library using plugin architecture and follow the trail of TypeScript hints to add a new feature. 

15 min
29 Apr, 2022

Video Summary and Transcription

When faced with challenges in supporting multiple package managers and keeping up with growth, implementing a plugin architecture can help. Extending a CLI for source control management systems like GitHub and GitLab can be done using TypeScript and Oclef CLI. TypeScript errors can be resolved by adding missing properties and implementing required functions for plugins. Supporting multiple repositories by following TypeScript errors and having the right setup can reduce time to production and onboarding. Plugin architecture with TypeScript can be a valuable tool for faster development and onboarding onto repositories.

1. Introduction to Challenges and Plugin Architecture

Short description:

Hi. My name is Lilia and I'm an engineering manager at Snyk. When I started at Snyk, I had little understanding of the challenges ahead. We needed to support multiple package managers and keep up with our own growth. To speed up support, we implemented plugin architecture. With TypeScript, we could follow the trail of errors to help us.

Hi. My name is Lilia and I'm an engineering manager at Snyk leading the technical services team. When I started at Snyk about four years ago now, I had very little understanding of the challenges that would lie ahead of me. Having just led an engineering team building cookie cutter websites at a small agency, my expectations were that Snyk would be similar, but maybe more challenging. And I was certainly not prepared for having suddenly to understand every single package manager, like a registry and source control management system that has ever been built. And all this because I've joined the ecosystems team.

I remember one of the very first features was adding support for npm6. This is the first time that npm introduced log files. So this was adding a new flavor of npm into the platform. Then came Yarn, Yarn Workspaces, Gradle, Go, different flavors of Go, Kotlin, SPT, Cocoapods. It just seemed to be a never-ending stream of package managers and tools that needed to be supported. And sometimes the new versions were completely incompatible with the previous versions and required rewriting the support entirely from scratch.

Not only did we need to extend support and keep up with customers migrating to all of these different tools and versions, we also needed to keep up with our own growth. New team members are joining all the time, and onboarding onto the codebase can take some time, as it is complex. Once we've added support for the package manager once or twice, we kind of start seeing some patterns, and then you can speed up. However, as a new team member joins, they have to start from scratch, and then it takes them another 2 or 3 goes before they start seeing those same patterns as you. So we needed to think about how we can speed up. As some of the support was taking longer, sometimes certain areas of the product were being missed where support needed to be extended. So we needed to speed up, but we also needed to ensure that we were covering every area of the product where support needed to be extended. We needed plugin architecture.

Plugin architecture consists of two main components, the core system and the plugin modules themselves. And the plugins interact with the core system via a predefined interface. At the time we've had some of that in place, but not to the extent we wanted. With some time, we were able to start seeing what is the responsibility of the core system and what is the responsibility of the plugins. However, some of these behaviors can still be quite unclear. So you have a little bit of a challenge there as well. So as an opportunity presented itself to build a new library, using plugin architecture, we could use what we've learned from introducing TypeScript really early on into our application, as well as what we've learned and the bits and pieces that we liked and didn't like from our existing plugin architecture. And we could put all of our learnings together and create something new and try again. And we found that with some setup upfront, we could lean on TypeScript and follow the trail of TypeScript errors to help us. So let's have a look at an example to understand how you can follow the TypeScript errors as hints.

2. Extending CLI for Source Control Management

Short description:

We'll be extending a CLI called the Source Control Management Helper (SEM Helper) to work with different source control management systems like GitHub and GitLab. The CLI has a command called Describe GitHub, which provides simple information about a repository, such as whether it is forked or archived. The code is set up using TypeScript and Oclef CLI, with separate directories for commands and plugins. The describe GitHub command loads the GitHub plugin and calls its describe function. The plugin uses the OctoKit rest library to make API calls to GitHub. We also have an index and types file that allow us to support other SCMs like GitLab.

We'll be extending a CLI that provides some simple functionality relevant for working with different source control management systems, tools like GitHub and GitLab. So, let's dive in.

So our example today is a really simple CLI called the Source Control Management Helper. We're gonna call it SEM Helper for short. So let's have a look here and we essentially have a command called Describe and we're calling Describe GitHub and then it's expecting a couple of parameters, the owner and the repo. So we're trying to describe a repository. So if we give it some owner and repo as well, we're returning back just really simple pieces of information, whether the repository is forked and whether it's archived as well.

So if we have a look conceptually, we have essentially got a core system that is able to say, repo.describe for example. And then we have a set of plugins that can be called upon, in this case, it's GitHub specifically. And we returned some very simple metadata, for example, for true, archived, false. So let's jump into the code and have a look at what we have set up here.

So it is a really simple, basic, out of the box, follow the TypeScript documentation TypeScript set up. You'll see here that we have a source index and we're using Oclef CLI. So we don't have to worry about parsing of parameters or anything like that. You'll see that there is a command directory as well as a lib plugins directory. In the commands here, we have describe GitHub, which is what you have just seen running. And in describe GitHub, it's got some arguments, it's got an example, some description, and the main functionality here is it loads a plugin, loads a plugin GitHub, and then we're calling plugin dot describe and sending it the repository and on that. And that's it. And then in lib plugins, there is a GitHub directory in which there is a little bit of code, which is get credentials and then it describe API call itself as well. So we need to be able to grab the token and then we're calling in this case, OctoKit rest, which is a GitHub library and calling the specific repos get to get the information that we want. So let's have a look again inside the plugins. We also have an index and types file. So in the index file, you'll see that there is that our load plugin function and it takes an SCM type, which is all supported SCMs. This is an enum of all the supported SCMs. And we want to be able to support GitLab as well. So let's do that. So let's add and register GitHub as well as GitLab as a supported plugin here. So we're going to just say, GitLab equals GitLab. Okay, and let's see what happens. So now that we've done that, you'll see there are in our load plugin file in that index.

3. TypeScript Errors and GitLab Plugin Implementation

Short description:

We have a couple of TypeScript errors related to missing properties in the type. We need to add the GitLab repo and organization equivalents, as well as register GitLab as a plugin. However, we also need to implement two required functions, get credentials and describe, for the plugin. Once we complete these missing pieces, we can check if the errors are resolved.

We have a couple of TypeScript errors. And you'll see the same if you run npm run build. So we've got three places where we've now got an error. And it's saying that Property GitLab is missing in type but is required. So you can see here that we're expected to populate the name of the entity that we're dealing with. So for GitHub it's a repo, for GitLab, it's also a repo. So let's add that in, GitLab repo.

Okay, then the parent entity type for GitHub with the organization. So for GitLab we also need the equivalent which is also organization. And the last one is plug-ins. So we have a type here that is basically saying that every supported SCM must be registered here as a plug-in. So let's register our new GitLab plug-in. And it looks like we now have to actually create a file for GitLab to have plug-ins.

Okay, let's do that. So GitLab index, let's export nothing. Just an empty object for now. And let's register it here. And let's see what we get. Okay, that's all done. Oh wait, what an error here. What does it not like? Oh, okay, perfect. So it's telling us that we are missing two functions that are required for every plug-in. One called get credentials and another one called describe. And you can see that they're actually defined here. So each one of these needs to be a performance to the interface SCM client, where we need to have a function called get credentials that returns credentials of a certain shape, as well as the describe function that will return archived and fork. And until we have completed this and returned the correct type, this error will not go away. So essentially, by just registering a new type plugin that is to be supported, TypeScript has told us about the three places where we needed to register or have GitLab specific implementation. And even when we've added just a basic plugin, it wasn't good enough because we have defined an interface. Therefore, we're expected to return the correct information. So let's go and implement all the missing bits and pieces, and then have a look if all the errors are now gone.

4. Supporting GitHub and GitLab with TypeScript

Short description:

Now we support both GitHub repositories and GitLab repositories by following TypeScript errors and having the right setup. Registering necessary functionality with types allows for easy addition of new plugins. You only need to know where to register the support for the new plugin. By doing these simple things, you can reduce the time to production and onboard onto the repository faster. Try plugin architecture with TypeScript in your next project.

Now that we've added all the code necessary for GitLab to work as well by calling the relevant GitLab APIs, we can run a CLI with describe-gitlab command as you can see, and we're getting the same results back. So we now support both GitHub repositories and GitLab repositories. And all this by following a couple of errors, really from TypeScript.

So how are we able to do that? The trick was really in having the right setup, where any variable, any functionality that needs to come from the plugin needs to be registered as required. So we have our supported SCMs. And you'll see that we are using this everywhere where we're expecting the functionality to be extended for the SCM. So here in entity name, here in parent entity name, here in the plugin itself. And for the plugin, we have typed what is expected. So we have the entire interface typed out. We have the expected functions that it must implement, which is the get credentials, and describe. We can also extend and add a new function that it must implement. Maybe it's a delete to delete the repo. And as soon as we do that, we're going to get TypeScript errors in all of the plugins, because now these plugins no longer conform to the defined interface.

So by having the right setup upfront and making sure that you register all of the necessary functionality with types, you can register a new plugin and then follow the trail of TypeScript errors to fill in the blanks. And that means that you only need to know the one place where you need to add the new plugin as registered, as supported. And you don't need to read the rest of the code because then you have defined interfaces that you can go and implement and fill in the blanks. So there's no need to understand how the GitHub plugin works, there's no need to understand how the main system code works as well. As long as you know where to register the support for the new plugin. By doing those simple things, we were able to add the plugin without diving too deep into the inner workings of the main code. So you can reduce the time to production because if you know exactly in which parameters you need to write the code, so the functions that you need to implement, what do they return, where those functions are being used, in which case here TypeScript will tell you. Then you don't need to get familiar with the core code at all. Prepping all of this in advance when you are setting up the architecture of the repository has huge benefits later, and you can reduce time to onboard onto the repository. There's no need to learn how the entire code base works all together before being able to contribute to it. You can learn just the areas that are necessary if you lean on TypeScript, and if you ever actually need to change the core system code, you can get familiar with it there and then. So my ask from you is to save time for yourself in the future. Try plugging architecture with TypeScript in your very next project. Thank you.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Remix Conf Europe 2022Remix Conf Europe 2022
23 min
Scaling Up with Remix and Micro Frontends
Top Content
Do you have a large product built by many teams? Are you struggling to release often? Did your frontend turn into a massive unmaintainable monolith? If, like me, you’ve answered yes to any of those questions, this talk is for you! I’ll show you exactly how you can build a micro frontend architecture with Remix to solve those challenges.
Remix Conf Europe 2022Remix Conf Europe 2022
37 min
Full Stack Components
Top Content
Remix is a web framework that gives you the simple mental model of a Multi-Page App (MPA) but the power and capabilities of a Single-Page App (SPA). One of the big challenges of SPAs is network management resulting in a great deal of indirection and buggy code. This is especially noticeable in application state which Remix completely eliminates, but it's also an issue in individual components that communicate with a single-purpose backend endpoint (like a combobox search for example).
In this talk, Kent will demonstrate how Remix enables you to build complex UI components that are connected to a backend in the simplest and most powerful way you've ever seen. Leaving you time to chill with your family or whatever else you do for fun.
React Advanced Conference 2022React Advanced Conference 2022
29 min
Understanding React’s Fiber Architecture
Top Content
We've heard a lot about React's Fiber Architecture, but it feels like few of us understand it in depth (or have the time to). In this talk, Tejas will go over his best attempt at understanding Fiber (reviewed by other experts), and present it in an 'explain-like-I'm-five years old' way.
TechLead Conference 2023TechLead Conference 2023
35 min
A Framework for Managing Technical Debt
Let’s face it: technical debt is inevitable and rewriting your code every 6 months is not an option. Refactoring is a complex topic that doesn't have a one-size-fits-all solution. Frontend applications are particularly sensitive because of frequent requirements and user flows changes. New abstractions, updated patterns and cleaning up those old functions - it all sounds great on paper, but it often fails in practice: todos accumulate, tickets end up rotting in the backlog and legacy code crops up in every corner of your codebase. So a process of continuous refactoring is the only weapon you have against tech debt.In the past three years, I’ve been exploring different strategies and processes for refactoring code. In this talk I will describe the key components of a framework for tackling refactoring and I will share some of the learnings accumulated along the way. Hopefully, this will help you in your quest of improving the code quality of your codebases.

React Summit 2023React Summit 2023
24 min
Debugging JS
As developers, we spend much of our time debugging apps - often code we didn't even write. Sadly, few developers have ever been taught how to approach debugging - it's something most of us learn through painful experience.  The good news is you _can_ learn how to debug effectively, and there's several key techniques and tools you can use for debugging JS and React apps.

Workshops on related topic

React Advanced Conference 2021React Advanced Conference 2021
174 min
React, TypeScript, and TDD
Top Content
Featured WorkshopFree
ReactJS is wildly popular and thus wildly supported. TypeScript is increasingly popular, and thus increasingly supported.

The two together? Not as much. Given that they both change quickly, it's hard to find accurate learning materials.

React+TypeScript, with JetBrains IDEs? That three-part combination is the topic of this series. We'll show a little about a lot. Meaning, the key steps to getting productive, in the IDE, for React projects using TypeScript. Along the way we'll show test-driven development and emphasize tips-and-tricks in the IDE.
React Advanced Conference 2022React Advanced Conference 2022
148 min
Best Practices and Advanced TypeScript Tips for React Developers
Top Content
Featured Workshop
Are you a React developer trying to get the most benefits from TypeScript? Then this is the workshop for you.In this interactive workshop, we will start at the basics and examine the pros and cons of different ways you can declare React components using TypeScript. After that we will move to more advanced concepts where we will go beyond the strict setting of TypeScript. You will learn when to use types like any, unknown and never. We will explore the use of type predicates, guards and exhaustive checking. You will learn about the built-in mapped types as well as how to create your own new type map utilities. And we will start programming in the TypeScript type system using conditional types and type inferring.
DevOps.js Conf 2024DevOps.js Conf 2024
163 min
AI on Demand: Serverless AI
Featured WorkshopFree
In this workshop, we discuss the merits of serverless architecture and how it can be applied to the AI space. We'll explore options around building serverless RAG applications for a more lambda-esque approach to AI. Next, we'll get hands on and build a sample CRUD app that allows you to store information and query it using an LLM with Workers AI, Vectorize, D1, and Cloudflare Workers.
React Summit Remote Edition 2021React Summit Remote Edition 2021
87 min
Building a Shopify App with React & Node
Top Content
WorkshopFree
Shopify merchants have a diverse set of needs, and developers have a unique opportunity to meet those needs building apps. Building an app can be tough work but Shopify has created a set of tools and resources to help you build out a seamless app experience as quickly as possible. Get hands on experience building an embedded Shopify app using the Shopify App CLI, Polaris and Shopify App Bridge.We’ll show you how to create an app that accesses information from a development store and can run in your local environment.
JSNation 2022JSNation 2022
41 min
Build a chat room with Appwrite and React
WorkshopFree
API's/Backends are difficult and we need websockets. You will be using VS Code as your editor, Parcel.js, Chakra-ui, React, React Icons, and Appwrite. By the end of this workshop, you will have the knowledge to build a real-time app using Appwrite and zero API development. Follow along and you'll have an awesome chat app to show off!
GraphQL Galaxy 2021GraphQL Galaxy 2021
164 min
Hard GraphQL Problems at Shopify
WorkshopFree
At Shopify scale, we solve some pretty hard problems. In this workshop, five different speakers will outline some of the challenges we’ve faced, and how we’ve overcome them.

Table of contents:
1 - The infamous "N+1" problem: Jonathan Baker - Let's talk about what it is, why it is a problem, and how Shopify handles it at scale across several GraphQL APIs.
2 - Contextualizing GraphQL APIs: Alex Ackerman - How and why we decided to use directives. I’ll share what directives are, which directives are available out of the box, and how to create custom directives.
3 - Faster GraphQL queries for mobile clients: Theo Ben Hassen - As your mobile app grows, so will your GraphQL queries. In this talk, I will go over diverse strategies to make your queries faster and more effective.
4 - Building tomorrow’s product today: Greg MacWilliam - How Shopify adopts future features in today’s code.
5 - Managing large APIs effectively: Rebecca Friedman - We have thousands of developers at Shopify. Let’s take a look at how we’re ensuring the quality and consistency of our GraphQL APIs with so many contributors.