Publishing TS Libraries for Fun and Profit

Rate this content

Publishing libraries to NPM is easy - just `tsc && npm publish` and you're done, right?

Whoops, you forgot proper ESM compat. And a user is asking for a UMD build. And it doesn't work in Webpack 4. And `moduleResolution: "node16"` can't find the types.

Publishing libraries today is _complicated_. We'll take a look at the many problems and questions you should consider when publishing a package, and some hard-earned possible answers to those questions.type

31 min
21 Sep, 2023


Sign in or register to post your comment.

AI Generated Video Summary

Mark Erickson discusses the complexities of publishing TypeScript libraries, including considerations like build artifact file formats, package exports, and different user environments. He shares his experiences with ESM support and interop with other module formats, and the challenges faced in migrating Redux to TypeScript. Erickson highlights the importance of understanding file formats and module types, and the insights gained from discussions with the TypeScript team. He also emphasizes the need for better tools and documentation in the ecosystem for publishing and maintaining TypeScript libraries.

1. Introduction to Publishing TypeScript Libraries

Short description:

Hi, my name is Mark Erickson and today I am very excited to talk to you about publishing TypeScript libraries for fun and profit. Publishing packages is not as simple as running TSC and npm publish. There are many considerations to keep in mind, such as build artifact file formats, package exports, different user environments, bundler behavior differences, and more. Maintaining Redux and other libraries has given me insight into the complexities of the process, including the challenges of ESM support and interop with other module formats.

♪ Hi, my name is Mark Erickson, and today I am very excited to talk to you about publishing TypeScript libraries for fun and profit. Mostly excited? Somewhat excited? Okay, look, it's been a really difficult year. There hasn't been a whole lot of fun. And actually, trust me, there has not been any profit at all. We're gonna go through the details.

A couple quick things about myself. I am a senior front-end engineer at Replay, where we're building a time-traveling debugger for JavaScript. Please check it out. I will answer questions pretty much anywhere there is a text box on the Internet. I collect all kinds of interesting links. I write extremely long blog posts. I am a Redux maintainer, but most people know me as that guy with the Simpsons avatar.

So, publishing packages is really simple, right? You just run TSC and npm publish, and you're done. Thank you. Oh boy, wow, I wish it were that easy. This would be a whole lot shorter talk, if it was. Earlier this year I got kind of annoyed and published a tweet where I listed some of the things you have to keep in mind when publishing packages stay. Build artifact file formats, whether to bundle or keep individual JavaScript files, package exports, WebPack 4, Typescript module resolution, different user environments, bundler behavior differences, Node ESM versus CGS, whether to bundle your TypeScript types, and now React's use client. There's no guides. Everybody's borrowing from everybody else, and it's a miracle this ecosystem works at all.

So, how did I get involved in this process? Well, I've been maintaining Redux for the last several years, and as of the start of this year, I maintained five different libraries. Redux core, React Redux, Redux Slunk, Reselect, and Redux Toolkit. And each of these had a somewhat different build setup, but in general, there was a mixture of ESM, CommonJS, and UMD build artifacts. Everything used a .js extension. Everything was being compiled to ES5 for IE11 compatibility. Most of the packages used rollup plus babble, except Redux Toolkit, which used es-build. None of the packages defined the exports field in package.json, and there were a variety of different folders being used for the build output.

So, what does ESM support even mean, anyway? And the problem here is that the ES2015 language spec defined the syntax for importing and exporting, and some of the expected behaviors, but it didn't define how runtime environments, like the browser or node, are actually supposed to handle loading these, or how they're supposed to interop with other module formats like CommonJS. Now, most of us have been writing ESM syntax for years, but when you publish a library, you normally convert it to CommonJS before you publish. And you also usually compile your syntax to ES5, so it works in IE11, unfortunately.

2. Understanding File Formats and Module Types

Short description:

So packaged JSON has different fields that tools look for to find the right file. Node took years to add support for ES modules. There's a new field called exports, but it's a breaking change. Node understands file type through file extension or the type module field. We decided to modernize Redux packages and encountered import issues in Node-ESM environment. We migrated Redux to TypeScript but didn't ship it. We wanted modern build output and smaller bundle sizes.

So packaged JSON has a bunch of different fields that different tools look for to try to find the right file. Node looks at the main field for CommonJS files, bundlers often look at the module field for ESM, CDNs and tools like unpackage look for different keys, TypeScript looks for its TypeScript types, and all these different tools have different expectations. On top of that, it took Node years to add decent support for ES modules because they were trying to figure out how it would work with CommonJS.

So there's a relatively new field called exports, and it's supposed to be the fully definitive one-stop shop for where you tell tools how to find your different entry points and different file formats. So you can find a whole bunch of different entry points, you can have nested conditions, like here's where to find an ESM file versus a CommonJS file, you can define conditions like development and production. But the problem is that adding exports is really a breaking change for your package, which means you can only do it in a major version.

So how does Node understand whether a given file is ESM or CommonJS? There's two different ways. One is it now allows you to use a .mjs or a .cjs file extension to declare what type of module it is, or you can add the type module field at the top level, or type CommonJS, and every file with a .js extension will be treated as if it's that type of module. So at the start of the year, we decided to modernize all the Redux packages. We had gotten some bug reports that you couldn't import them properly in a Node-ESM environment. We'd actually migrated the Redux score to Typescript back in 2019 and then never actually shipped it. Version 4 worked fine and we had concerns about shipping a new major version. And we wanted to modernize all the build output and ship modern JS syntax for smaller bundle sizes.

3. Challenges with Type Module and Exports

Short description:

Based on my research, I thought all I have to do is add type module and an exports field and everything will just work. But when I tried it, everything exploded, especially under Jest. Switching from Jest to VTest for testing provided better ESM support.

So, based on my research, I thought all I have to do is add type module and an exports field and everything will just work. Right? Right? No. No, not at all the case. I put up a pull request that tried to modernize Redux toolkit as my first attempt. I added type module and exports, I modernized all the build output. I also had to make changes to a bunch of scripts that we had that were common JS files, because with type module those are now treated as ESM files. But I had this and I tried it, and then everything exploded. Well, okay. Mostly things exploded under Jest. Redux Thunk and Emmer both have default exports. And Jest was getting really confused about how to interpret default exports versus named exports. I ended up trying to publish an alpha version of Redux Thunk to work around this, that didn't really help much. I was able to switch to Emmer's named exports and that sort of helped a little bit. I actually got fed up and I actually switched all of our testing from using Jest to VTest because it was supposed to have better ESM support. And that actually went pretty well.

4. Challenges and Research

Short description:

I published the first alphas of the Redux core and Redux toolkit in January and encountered issues with module resolution and node ESM. I received contradictory advice from different people, so I decided to do some research.

Most of it was just some search in place plus a different config file. So I published the first alphas of the Redux core and Redux toolkit in January, and people probably started telling me how I was doing it wrong. Matusz Brzezinski, who is an expert on this sort of thing, had a whole bunch of different suggestions. People filed issues saying that it didn't work when module resolution was set to node 16 for TypeScript. And someone else pointed out that, well, actually the node ESM thing you're trying to fix just actually really doesn't even work. And I was not a happy camper. I felt like I was getting contradictory advice from everybody. So it was time to do research.

5. Insights on ESM and TypeScript

Short description:

I had calls with Matusz Brzezinski and Andrew Branch from the TypeScript team. They provided insights into shipping ESM and TypeScript's module support. The right file extension or type module determines if a file is CommonJS or ESM. Using the .mjs file extension simplifies the project.

I had one call with Matusz Brzezinski, where he gave his thoughts on what I ought to try. He even suggested, like, is it actually even worth shipping the ESM? Maybe you should just, like, do CommonJS only. I also had a call with Andrew Branch from the TypeScript team, who's been doing a lot of work on TypeScript's module support. And he gave me a lot of details about how TypeScript interprets module files and all the different instructions, as well as some key pieces of information on how do TypeScript and Node know whether a given file is CommonJS or ESM. And it really is about either using the right file extension, like .mjs, or having that type module. And as much as I don't like the .mjs file extension, it turns out that using that actually does simplify a number of things within the project.

6. CI Checks and Tools

Short description:

I realized I needed to set up CI checks to see how different tools interpreted the package definitions. I wrote an example application and tested it with various bundlers and build tools. React Aria and Are The Types Wrong? were helpful resources. I even created my own CLI tool for Are The Types Wrong? Later, an official CLI tool was added.

So it was very clear that I was in over my head with this stuff and just trying to look at a package definition to figure out if it was going to work, wasn't going to scale. So I needed to set up CI checks to actually see how different tools interpreted things. The problem is there is at least a half dozen different bundlers and build tools, each with their own quirks. So I ended up writing one little example application and then building it with create React app 4 and 5 and V and Next and a couple of different node setups and running that all on every pull request just to see how it was going to interpret the packaging setup. Turns out that React Aria is doing something pretty similar. I also discovered that Andrew Branch had been building a tool called Are The Types Wrong? as an outgrowth of his work on TypeScript's module definition and documentation. Now, originally, this was just a website. So you would either type in a module name or you could upload a packaged tarball, and it will scan it and try to tell you, here's how TypeScript will interpret all the different entry points and type definitions. Now, at the time, there was no CLI for this tool, so I actually ended up writing my own and trying to use that in our CI checks. Later, an official CLI tool for Are The Types Wrong? was added, and I actually still need to switch over and make use of that. But it's been extremely helpful in both local development and in CI to verify that all the package definitions are set up hopefully the right way.

7. Working on Redux Thunk and Imer-10 Beta

Short description:

A few months later, I decided to work on the Redux Thunk library. I switched to using ESBuild with a wrapper called TsUp. I dropped UMD files and added a pre-compiled ESM build for browsers. Webpack 4 caused issues, so I shipped an additional build for Webpack 4. The typedefs generated by tsup had a .d.ts extension, causing false CJS warnings. I still need to solve this issue. After the second attempt, the packages improved, but some tweaks are still needed. I tested Imer-10 Beta and found that the bundle size increased due to the use of the older build tool TSDX.

So, a few months later I was ready to give this another shot, and I decided to try working on the Redux Thunk library first because it's really small. But it was still being built with Babel and Rollup, and I wanted to switch over to using ESBuild, so I found a really nice wrapper called TsUp. And I was able to get that set up pretty quickly, and it works pretty well. It's a wrapper around ESBuild aimed at TypeScript libraries. It also has the ability to generate a bundled ts TypeDefs file for you. So I ended up with this package, and it's better, although it probably still needs a bit of work from there.

Now, I mentioned earlier that we shipped ES modules, CommonJS and UMD file formats. What's a UMD file? Universal Module Definition is a really bizarre module format that can simultaneously be used as an AMD file, a CommonJS file, or a global script tag. And it's not that much more effort to maintain, but it felt legacy, and I didn't know if we should keep it. So I looked around, kept asking for advice. And the best advice I could find was a couple people saying, eh, you probably don't need it anymore. Even for something like CodePen, it has support for ES modules these days, you probably don't need it. So I made the decision to drop UMD files from our packages, although I did replace that with a special ESM build that's been pre-compiled to production mode, so it ought to work okay in browsers.

Then I found out that Webpack 4 didn't like the setup, because number one, it doesn't support the exports field, it also doesn't support parsing ES 2018 objects with spread syntax, or optional chaining syntax. And for that matter, you can't have a .mjs file in the main field, it'll choke on that too, so you have to switch to a .js extension just for that. And the problem is that Webpack 4 is still pretty widely used by a number of older build setups. So in order to try to keep from breaking the ecosystem, I reluctantly decided that I'm going to ship an additional build artifact, ESM format, but compiled to ES 2017 in a .js extension, just to keep Webpack 4 happy.

What about typedefs? Well, the version of tsup that I was using at the time will generate a bundled typedef file, but it always gives it a .d.ts extension. And this is a problem, because it turns out that, are the types wrong, reports that as a false CJS warning when you're using Module Resolution Node 16. And talking to Andrew Branch, it turns out that you really should have separate files with actually a .d.mts and a .d.cts extension, so that TypeScript fully knows here's what the types look like when you're running in ESM mode versus common JS mode, because there can actually be some differences. I decided to punt on solving that problem for now. It's still a thing I need to go back and look into. Andrew Branch did go and file a PR for .tsup, so that it will try to generate different bundled TypeScript definition files. That came out in a later version of .tsup, and I still need to try that out myself.

So, here's what some of the packages look like after the second attempt, and this is better, I think, but it probably still needs a couple more tweaks. I probably need to do some nesting for the import and default conditions to specify different type definition files for each of those. Now, Michel Westray, author of the Imer library, had been working on developing Imer-10 during the spring, and he was also trying to modernize that package and dropping some backwards compatibility stuff, and Redux Toolkit relies heavily on Imer. So, I was very eager to try out Imer-10 Beta, but I noticed that the bundle size actually went up a little bit. That seemed weird. So, I actually pulled down, cloned Imer-Repo, pulled down the Beta branch, and was looking at it, and I found that it was using an older build tool called TSDX, and it was still generating a lot of ES5 build syntax and a lot of weird cruft in there.

8. React Server Components and Redux

Short description:

I found a PR that switched to use TSUp and dropped the bundle size by 40%. Next 13.4 came out with React server components and people reported issues with Redux. My co-maintainer at Apollo filed a React issue and we faced complications. We think React Server Components are useful, but the rollout is making it harder to maintain libraries. We have betas and alphas of Redux packages with fixes and are looking for feedback. Trying to publish typedefs complicates things, and there is no definitive guide to package publishing.

So, I had learned a lot, hopefully, about trying to package things, and so I actually found a PR that switched it to use TSUp and applied all the same packaging changes that I was using on our Betas, and that worked, and it actually dropped the bundle size significantly by about 40%. And Michelle actually shipped those changes as part of Emerton Final, so that is actually available in the wild.

And then things got more complicated. So, Next 13.4 came out in May, and it ships with React server components and the new app router as the default. And people have been trying to use Redux with this, and unfortunately, things keep breaking, and so we've been getting a steady stream of bug reports from people saying that React Redux or Redux toolkit don't work in a server component environment.

Now, my Redux co-maintainer, Lensweber, actually has been working at Apollo, on Apollo Client, since the start of the year, and he's been doing a lot of research on how libraries on the client side can interact with a server component environment. And so he'd filed a React issue, asking for some advice, wrote an RFC about how to make Apollo and Next work together, and even published an experimental package. Where things got really complicated was that a specific Next Canary release broke Apollo, briefly.

This was fixed pretty fast, but it did spawn a very long and detailed and kind of argumentative discussion thread. One of the Next developers left a comment, saying that packages really need to publish an additional build artifact with a React server package definition inside. And that just seemed like it was really going to confuse things. So Mark Baga said, you need to make sure that the client code gets stripped out of there. Lens and I complained that this was making our job as maintainers way harder, and I got pretty frustrated. And I put in a lot of work. This feels really, really demoralizing.

So a few weeks later, Lens actually put up a blog post titled My Take on the Current React Server Components Controversy. And there had been a lot of arguments and debates floating around online, and a lot of confusion about what was happening. And he wanted to give our thoughts as library maintainers. And he said that we think React Server Components are a really useful technology, but the way this has been rolled out is making it a lot harder for us to help our users, and they're filing a lot more bug reports. There's a lot more about React and Next that we have to understand, and this is making it a lot harder to maintain and publish a library that works with React. On top of that, it really feels like there's been very little communication from the React team about how these sorts of things are going to affect the ecosystem.

So where do things stand today? We have betas of the Redux core and Redux toolkit that are out and live with these changes, and we would like people to try those out. We've got alphas of reselect and Redux slunk, and I have a pull request up to try to update React Redux's packaging. I still need to go back and finish that. We do have a few fixes to try to keep React Redux from breaking in a server component environment, and the packages generally pass the are-the-type-wrong checks, except for that false CJS warning that I still need to go back and look at. So, what have I taken away from all the effort this year? I have packaged configs that mostly seem to work. Trying to publish typedefs definitely makes things more complicated. Bundling my JavaScript ahead of time helps in some cases, but is harder in others. It is almost impossible to keep up with all the different build tools and their combinations and their environments and the unique needs that each one has. And unfortunately, there is no fully definitive guide to how to publish a package the right way.

9. Challenges and Conclusion

Short description:

I've been begging for someone to write a guide on publishing TypeScript libraries, but no one has done it yet. The ecosystem needs better tools for publishing and understanding how different build tools work. React Server components are useful but disruptive, and there is no documentation for library authors on how to deal with them. The CommonJS ESM transition has been a long-standing nightmare. If you want more information, check out my blog post titled My Experience, Modernizing Packages to ESM.

I've been begging for years for someone who actually knows what they're doing to please write such a guide. No one's done it yet. I've written a long blog post with the same, a lot of the lessons from this talk. Trust me, it's not a definitive guide. It's a here's all the painful things I've run into and I'm opposed.

The ecosystem desperately needs better tools to help with publishing. TS up is pretty useful, although that's just the build step. We could really use some kind of service that would take an example app and a library and build it with the half dozen build tools and tell you how each one works or fails.

And React Server components are a very useful tool but they're also really disrupting the ecosystem. It's a lot more things for both users and library maintainers to keep in mind. And there's no documentation for library authors to know how to deal with this. And finally, the CommonJS ESM transition has been going on for years and it is a nightmare that shows no signs of stopping any time soon.

So if you'd like more information, this is based on a much longer blog post I wrote titled My Experience, Modernizing Packages to ESM. You can find a lot more details there as well as resources to a lot of the things that I've looked at in trying to learn how to try to publish a package correctly. Hopefully this information has been helpful for you, and if you are trying to publish package in today's environment, I'm sorry, you have my sympathies.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Vue.js London 2023Vue.js London 2023
30 min
Stop Writing Your Routes
The more you keep working on an application, the more complicated its routing becomes, and the easier it is to make a mistake. ""Was the route named users or was it user?"", ""Did it have an id param or was it userId?"". If only TypeScript could tell you what are the possible names and params. If only you didn't have to write a single route anymore and let a plugin do it for you. In this talk we will go through what it took to bring automatically typed routes for Vue Router.
TypeScript Congress 2022TypeScript Congress 2022
27 min
TypeScript and the Database: Who Owns the Types?
We all love writing types in TypeScript, but we often find ourselves having to write types in another language as well: SQL. This talk will present the choose-your-own-adventure story that you face when combining TypeScript and SQL and will walk you through the tradeoffs between the various options. Combined poorly, TypeScript and SQL can be duplicative and a source of headaches, but done well they can complement one another by addressing each other's weaknesses.
TypeScript Congress 2022TypeScript Congress 2022
30 min
Lessons from Maintaining TypeScript Libraries
Maintaining widely-used JS libraries is already complicated, and TypeScript adds an additional set of challenges.
Join Redux maintainer Mark Erikson for a look at some of the unique problems TS library maintainers face, and how the Redux team has handled those problems. We'll cover:
- Tradeoffs of different ways to define TS types for a library
- How to target different versions of TS, and considerations for determining the supported version range
- Migrating existing JS libraries to TS
- Differences between writing "app" types and "library" types
- Managing and versioning public types APIs
- Tips and tricks used by types from the Redux libraries
- TS limitations and possible language-level improvements

Workshops on related topic

JSNation 2022JSNation 2022
99 min
Finding, Hacking and fixing your NodeJS Vulnerabilities with Snyk
npm and security, how much do you know about your dependencies?
Hack-along, live hacking of a vulnerable Node app, Vulnerabilities from both Open source and written code. Encouraged to download the application and hack along with us.
Fixing the issues and an introduction to Snyk with a demo.
Open questions.
React Summit 2022React Summit 2022
51 min
Build Web3 apps with React
The workshop is designed to help Web2 developers start building for Web3 using the Hyperverse. The Hyperverse is an open marketplace of community-built, audited, easy to discover smart modules. Our goal - to make it easy for React developers to build Web3 apps without writing a single line of smart contract code. Think “npm for smart contracts.”
Learn more about the Hyperverse
We will go over all the blockchain/crypto basics you need to know to start building on the Hyperverse, so you do not need to have any previous knowledge about the Web3 space. You just need to have React experience.