Publishing libraries to NPM is easy - just `tsc && npm publish` and you're done, right?
Whoops, you forgot proper ESM compat. And a user is asking for a UMD build. And it doesn't work in Webpack 4. And `moduleResolution: "node16"` can't find the types.
Publishing libraries today is _complicated_. We'll take a look at the many problems and questions you should consider when publishing a package, and some hard-earned possible answers to those questions.type
Publishing TS Libraries for Fun and Profit
Publishing libraries to NPM is easy - just `tsc && npm publish` and you're done, right?
AI Generated Video Summary
Mark Erickson discusses the complexities of publishing TypeScript libraries, including considerations like build artifact file formats, package exports, and different user environments. He shares his experiences with ESM support and interop with other module formats, and the challenges faced in migrating Redux to TypeScript. Erickson highlights the importance of understanding file formats and module types, and the insights gained from discussions with the TypeScript team. He also emphasizes the need for better tools and documentation in the ecosystem for publishing and maintaining TypeScript libraries.
1. Introduction to Publishing TypeScript Libraries
Hi, my name is Mark Erickson and today I am very excited to talk to you about publishing TypeScript libraries for fun and profit. Publishing packages is not as simple as running TSC and npm publish. There are many considerations to keep in mind, such as build artifact file formats, package exports, different user environments, bundler behavior differences, and more. Maintaining Redux and other libraries has given me insight into the complexities of the process, including the challenges of ESM support and interop with other module formats.
♪ Hi, my name is Mark Erickson, and today I am very excited to talk to you about publishing TypeScript libraries for fun and profit. Mostly excited? Somewhat excited? Okay, look, it's been a really difficult year. There hasn't been a whole lot of fun. And actually, trust me, there has not been any profit at all. We're gonna go through the details.
So, how did I get involved in this process? Well, I've been maintaining Redux for the last several years, and as of the start of this year, I maintained five different libraries. Redux core, React Redux, Redux Slunk, Reselect, and Redux Toolkit. And each of these had a somewhat different build setup, but in general, there was a mixture of ESM, CommonJS, and UMD build artifacts. Everything used a .js extension. Everything was being compiled to ES5 for IE11 compatibility. Most of the packages used rollup plus babble, except Redux Toolkit, which used es-build. None of the packages defined the exports field in package.json, and there were a variety of different folders being used for the build output.
So, what does ESM support even mean, anyway? And the problem here is that the ES2015 language spec defined the syntax for importing and exporting, and some of the expected behaviors, but it didn't define how runtime environments, like the browser or node, are actually supposed to handle loading these, or how they're supposed to interop with other module formats like CommonJS. Now, most of us have been writing ESM syntax for years, but when you publish a library, you normally convert it to CommonJS before you publish. And you also usually compile your syntax to ES5, so it works in IE11, unfortunately.
2. Understanding File Formats and Module Types
So packaged JSON has different fields that tools look for to find the right file. Node took years to add support for ES modules. There's a new field called exports, but it's a breaking change. Node understands file type through file extension or the type module field. We decided to modernize Redux packages and encountered import issues in Node-ESM environment. We migrated Redux to TypeScript but didn't ship it. We wanted modern build output and smaller bundle sizes.
So packaged JSON has a bunch of different fields that different tools look for to try to find the right file. Node looks at the main field for CommonJS files, bundlers often look at the module field for ESM, CDNs and tools like unpackage look for different keys, TypeScript looks for its TypeScript types, and all these different tools have different expectations. On top of that, it took Node years to add decent support for ES modules because they were trying to figure out how it would work with CommonJS.
So there's a relatively new field called exports, and it's supposed to be the fully definitive one-stop shop for where you tell tools how to find your different entry points and different file formats. So you can find a whole bunch of different entry points, you can have nested conditions, like here's where to find an ESM file versus a CommonJS file, you can define conditions like development and production. But the problem is that adding exports is really a breaking change for your package, which means you can only do it in a major version.
So how does Node understand whether a given file is ESM or CommonJS? There's two different ways. One is it now allows you to use a .mjs or a .cjs file extension to declare what type of module it is, or you can add the type module field at the top level, or type CommonJS, and every file with a .js extension will be treated as if it's that type of module. So at the start of the year, we decided to modernize all the Redux packages. We had gotten some bug reports that you couldn't import them properly in a Node-ESM environment. We'd actually migrated the Redux score to Typescript back in 2019 and then never actually shipped it. Version 4 worked fine and we had concerns about shipping a new major version. And we wanted to modernize all the build output and ship modern JS syntax for smaller bundle sizes.
3. Challenges with Type Module and Exports
Based on my research, I thought all I have to do is add type module and an exports field and everything will just work. But when I tried it, everything exploded, especially under Jest. Switching from Jest to VTest for testing provided better ESM support.
So, based on my research, I thought all I have to do is add type module and an exports field and everything will just work. Right? Right? No. No, not at all the case. I put up a pull request that tried to modernize Redux toolkit as my first attempt. I added type module and exports, I modernized all the build output. I also had to make changes to a bunch of scripts that we had that were common JS files, because with type module those are now treated as ESM files. But I had this and I tried it, and then everything exploded. Well, okay. Mostly things exploded under Jest. Redux Thunk and Emmer both have default exports. And Jest was getting really confused about how to interpret default exports versus named exports. I ended up trying to publish an alpha version of Redux Thunk to work around this, that didn't really help much. I was able to switch to Emmer's named exports and that sort of helped a little bit. I actually got fed up and I actually switched all of our testing from using Jest to VTest because it was supposed to have better ESM support. And that actually went pretty well.
4. Challenges and Research
I published the first alphas of the Redux core and Redux toolkit in January and encountered issues with module resolution and node ESM. I received contradictory advice from different people, so I decided to do some research.
Most of it was just some search in place plus a different config file. So I published the first alphas of the Redux core and Redux toolkit in January, and people probably started telling me how I was doing it wrong. Matusz Brzezinski, who is an expert on this sort of thing, had a whole bunch of different suggestions. People filed issues saying that it didn't work when module resolution was set to node 16 for TypeScript. And someone else pointed out that, well, actually the node ESM thing you're trying to fix just actually really doesn't even work. And I was not a happy camper. I felt like I was getting contradictory advice from everybody. So it was time to do research.
5. Insights on ESM and TypeScript
I had calls with Matusz Brzezinski and Andrew Branch from the TypeScript team. They provided insights into shipping ESM and TypeScript's module support. The right file extension or type module determines if a file is CommonJS or ESM. Using the .mjs file extension simplifies the project.
I had one call with Matusz Brzezinski, where he gave his thoughts on what I ought to try. He even suggested, like, is it actually even worth shipping the ESM? Maybe you should just, like, do CommonJS only. I also had a call with Andrew Branch from the TypeScript team, who's been doing a lot of work on TypeScript's module support. And he gave me a lot of details about how TypeScript interprets module files and all the different instructions, as well as some key pieces of information on how do TypeScript and Node know whether a given file is CommonJS or ESM. And it really is about either using the right file extension, like .mjs, or having that type module. And as much as I don't like the .mjs file extension, it turns out that using that actually does simplify a number of things within the project.
6. CI Checks and Tools
I realized I needed to set up CI checks to see how different tools interpreted the package definitions. I wrote an example application and tested it with various bundlers and build tools. React Aria and Are The Types Wrong? were helpful resources. I even created my own CLI tool for Are The Types Wrong? Later, an official CLI tool was added.
So it was very clear that I was in over my head with this stuff and just trying to look at a package definition to figure out if it was going to work, wasn't going to scale. So I needed to set up CI checks to actually see how different tools interpreted things. The problem is there is at least a half dozen different bundlers and build tools, each with their own quirks. So I ended up writing one little example application and then building it with create React app 4 and 5 and V and Next and a couple of different node setups and running that all on every pull request just to see how it was going to interpret the packaging setup. Turns out that React Aria is doing something pretty similar. I also discovered that Andrew Branch had been building a tool called Are The Types Wrong? as an outgrowth of his work on TypeScript's module definition and documentation. Now, originally, this was just a website. So you would either type in a module name or you could upload a packaged tarball, and it will scan it and try to tell you, here's how TypeScript will interpret all the different entry points and type definitions. Now, at the time, there was no CLI for this tool, so I actually ended up writing my own and trying to use that in our CI checks. Later, an official CLI tool for Are The Types Wrong? was added, and I actually still need to switch over and make use of that. But it's been extremely helpful in both local development and in CI to verify that all the package definitions are set up hopefully the right way.
7. Working on Redux Thunk and Imer-10 Beta
A few months later, I decided to work on the Redux Thunk library. I switched to using ESBuild with a wrapper called TsUp. I dropped UMD files and added a pre-compiled ESM build for browsers. Webpack 4 caused issues, so I shipped an additional build for Webpack 4. The typedefs generated by tsup had a .d.ts extension, causing false CJS warnings. I still need to solve this issue. After the second attempt, the packages improved, but some tweaks are still needed. I tested Imer-10 Beta and found that the bundle size increased due to the use of the older build tool TSDX.
So, a few months later I was ready to give this another shot, and I decided to try working on the Redux Thunk library first because it's really small. But it was still being built with Babel and Rollup, and I wanted to switch over to using ESBuild, so I found a really nice wrapper called TsUp. And I was able to get that set up pretty quickly, and it works pretty well. It's a wrapper around ESBuild aimed at TypeScript libraries. It also has the ability to generate a bundled ts TypeDefs file for you. So I ended up with this package, and it's better, although it probably still needs a bit of work from there.
Now, I mentioned earlier that we shipped ES modules, CommonJS and UMD file formats. What's a UMD file? Universal Module Definition is a really bizarre module format that can simultaneously be used as an AMD file, a CommonJS file, or a global script tag. And it's not that much more effort to maintain, but it felt legacy, and I didn't know if we should keep it. So I looked around, kept asking for advice. And the best advice I could find was a couple people saying, eh, you probably don't need it anymore. Even for something like CodePen, it has support for ES modules these days, you probably don't need it. So I made the decision to drop UMD files from our packages, although I did replace that with a special ESM build that's been pre-compiled to production mode, so it ought to work okay in browsers.
Then I found out that Webpack 4 didn't like the setup, because number one, it doesn't support the exports field, it also doesn't support parsing ES 2018 objects with spread syntax, or optional chaining syntax. And for that matter, you can't have a .mjs file in the main field, it'll choke on that too, so you have to switch to a .js extension just for that. And the problem is that Webpack 4 is still pretty widely used by a number of older build setups. So in order to try to keep from breaking the ecosystem, I reluctantly decided that I'm going to ship an additional build artifact, ESM format, but compiled to ES 2017 in a .js extension, just to keep Webpack 4 happy.
What about typedefs? Well, the version of tsup that I was using at the time will generate a bundled typedef file, but it always gives it a .d.ts extension. And this is a problem, because it turns out that, are the types wrong, reports that as a false CJS warning when you're using Module Resolution Node 16. And talking to Andrew Branch, it turns out that you really should have separate files with actually a .d.mts and a .d.cts extension, so that TypeScript fully knows here's what the types look like when you're running in ESM mode versus common JS mode, because there can actually be some differences. I decided to punt on solving that problem for now. It's still a thing I need to go back and look into. Andrew Branch did go and file a PR for .tsup, so that it will try to generate different bundled TypeScript definition files. That came out in a later version of .tsup, and I still need to try that out myself.
So, here's what some of the packages look like after the second attempt, and this is better, I think, but it probably still needs a couple more tweaks. I probably need to do some nesting for the import and default conditions to specify different type definition files for each of those. Now, Michel Westray, author of the Imer library, had been working on developing Imer-10 during the spring, and he was also trying to modernize that package and dropping some backwards compatibility stuff, and Redux Toolkit relies heavily on Imer. So, I was very eager to try out Imer-10 Beta, but I noticed that the bundle size actually went up a little bit. That seemed weird. So, I actually pulled down, cloned Imer-Repo, pulled down the Beta branch, and was looking at it, and I found that it was using an older build tool called TSDX, and it was still generating a lot of ES5 build syntax and a lot of weird cruft in there.
8. React Server Components and Redux
I found a PR that switched to use TSUp and dropped the bundle size by 40%. Next 13.4 came out with React server components and people reported issues with Redux. My co-maintainer at Apollo filed a React issue and we faced complications. We think React Server Components are useful, but the rollout is making it harder to maintain libraries. We have betas and alphas of Redux packages with fixes and are looking for feedback. Trying to publish typedefs complicates things, and there is no definitive guide to package publishing.
So, I had learned a lot, hopefully, about trying to package things, and so I actually found a PR that switched it to use TSUp and applied all the same packaging changes that I was using on our Betas, and that worked, and it actually dropped the bundle size significantly by about 40%. And Michelle actually shipped those changes as part of Emerton Final, so that is actually available in the wild.
And then things got more complicated. So, Next 13.4 came out in May, and it ships with React server components and the new app router as the default. And people have been trying to use Redux with this, and unfortunately, things keep breaking, and so we've been getting a steady stream of bug reports from people saying that React Redux or Redux toolkit don't work in a server component environment.
Now, my Redux co-maintainer, Lensweber, actually has been working at Apollo, on Apollo Client, since the start of the year, and he's been doing a lot of research on how libraries on the client side can interact with a server component environment. And so he'd filed a React issue, asking for some advice, wrote an RFC about how to make Apollo and Next work together, and even published an experimental package. Where things got really complicated was that a specific Next Canary release broke Apollo, briefly.
This was fixed pretty fast, but it did spawn a very long and detailed and kind of argumentative discussion thread. One of the Next developers left a comment, saying that packages really need to publish an additional build artifact with a React server package definition inside. And that just seemed like it was really going to confuse things. So Mark Baga said, you need to make sure that the client code gets stripped out of there. Lens and I complained that this was making our job as maintainers way harder, and I got pretty frustrated. And I put in a lot of work. This feels really, really demoralizing.
So a few weeks later, Lens actually put up a blog post titled My Take on the Current React Server Components Controversy. And there had been a lot of arguments and debates floating around online, and a lot of confusion about what was happening. And he wanted to give our thoughts as library maintainers. And he said that we think React Server Components are a really useful technology, but the way this has been rolled out is making it a lot harder for us to help our users, and they're filing a lot more bug reports. There's a lot more about React and Next that we have to understand, and this is making it a lot harder to maintain and publish a library that works with React. On top of that, it really feels like there's been very little communication from the React team about how these sorts of things are going to affect the ecosystem.
9. Challenges and Conclusion
I've been begging for someone to write a guide on publishing TypeScript libraries, but no one has done it yet. The ecosystem needs better tools for publishing and understanding how different build tools work. React Server components are useful but disruptive, and there is no documentation for library authors on how to deal with them. The CommonJS ESM transition has been a long-standing nightmare. If you want more information, check out my blog post titled My Experience, Modernizing Packages to ESM.
I've been begging for years for someone who actually knows what they're doing to please write such a guide. No one's done it yet. I've written a long blog post with the same, a lot of the lessons from this talk. Trust me, it's not a definitive guide. It's a here's all the painful things I've run into and I'm opposed.
The ecosystem desperately needs better tools to help with publishing. TS up is pretty useful, although that's just the build step. We could really use some kind of service that would take an example app and a library and build it with the half dozen build tools and tell you how each one works or fails.
And React Server components are a very useful tool but they're also really disrupting the ecosystem. It's a lot more things for both users and library maintainers to keep in mind. And there's no documentation for library authors to know how to deal with this. And finally, the CommonJS ESM transition has been going on for years and it is a nightmare that shows no signs of stopping any time soon.
So if you'd like more information, this is based on a much longer blog post I wrote titled My Experience, Modernizing Packages to ESM. You can find a lot more details there as well as resources to a lot of the things that I've looked at in trying to learn how to try to publish a package correctly. Hopefully this information has been helpful for you, and if you are trying to publish package in today's environment, I'm sorry, you have my sympathies.