You can check the slides for James' talk here.
AI Generated Video Summary
1. Introduction to Node.js API Standardization
Hello, Node Congress. I am James Snell, or JA Snell on GitHub or Twitter. And I am with CloudFlare. I'm also on the node technical steering committee. I've been contributing to Node for almost seven years now. Started back in 2015, right when NodeJS and IOJS were having their issues and got them back together. So they've been around for a while.
And in that time, there's been some themes that have developed over time. You know, we have all the node APIs that are out there. You know, we've added a bunch of new features to node. We added HTTP2 support and new URL parsing. Recently, we just added support for web streams, so the readable stream and writable stream API. I think two years ago, we implemented the Web Crypto API. So, you know, there's a lot that's happened. And over that time, you know, we've had this kind of ongoing debate and ongoing conversation about just how standardized Node's API's should be or how much of the, you know, web platform standards Node should be paying attention to.
Here's a puzzle. You know, base64 encoding is something that is very common to many applications. We see it everywhere. In Node, you know, we've always had this, you know, buffer from hello to string base64. We can, you know, take a base64 encoded string and get our buffer back out from it.
You know, and what is, you know, once you have the basic mechanism, you know, which API here is the correct one? Unfortunately, the answer is all of them. All of them are the correct way, you know, but having so many different ways to do something that is so fundamental it just adds friction, adds difficulty to developers who are writing code that runs on the web. Amazingly, this fourth option, this Base64.js module, you know, has over 30 million downloads per week from NPM and has over 1,300 dependents. And a lot of those are Node applications, despite the fact that Node has this built in, right? But also a large number of those dependencies are, you know, browsers, browser applications that, you know, that all they have to rely on built in is B2A. And, you know, B2A itself is lacking in quite a bit of functionality. So, you know, the answer is, you know, all of these are the right way, but none of them are the right way because there's always a different way of doing it.
3. Challenges with Encoding and Binary Operations
You know, what if we wanted to have Base64 URL encoding, right? Which is a variance of Base64. Well, it turns out, you know, even that, you know, from Base64 to Base64 URL, there are differences in APIs. Not all the options work. You know, sometimes, you know, we have to change a few things. Fortunately, both Deno and Node have this built in as part of their, you know, core libraries. And then at least those will work consistently with each other. But you still, you know, start having a lot of these options.
4. Challenges with Small Core Approach
5. The Idea of Small-Core
The idea of small-core is to provide minimal APIs and rely on external sources for additional functionality. This approach makes runtimes like Node smaller and easier to maintain, but it shifts the responsibility of finding missing functionality to developers.
6. Abuse of Small Core in Runtimes
7. Misalignment Between Node and Web Browsers
Node and web browsers have a lot of overlap in functionality, including working with streaming data, binary data, and crypto. However, these platforms do things in completely different ways, causing friction for developers. The conflict between Node and web browsers has been ongoing, with each side insisting on doing things their own way. This lack of alignment has led to the creation of Node-specific APIs that are widely used across different platforms.
There's another argument and this is really from day one of my involvement with Node, and I've been hearing this, this Node is not a web browser, it shouldn't act like one, it shouldn't have these APIs that exist in the browser. They do different things and honestly, they're a lot closer than it may appear.
Yeah, Node doesn't have a rendering process, it's not going to go out and parse HTML and render it in a window. There is a lot of stuff that Node does that web browsers don't, and there's a lot of stuff that web browsers do that Node doesn't. So, you know, yes, the statement is factually true, but it ignores a very fundamental concept that Node and web browsers do a lot of the exact same things. They both are used to implement applications on the web. They both need to work with streaming data. They both need to work with binary data. They both need to do crypto. There is a huge amount of overlap in functionality between these environments, and I'm not just talking just Node, you know, this includes Deno, this includes environments like Cloudflare Workers. There's a huge amount of overlap between these platforms. So why are these platforms doing things in completely different ways, you know, even when they're on the areas of overlap? It doesn't make any sense.
8. Challenges with API Standardization
Even if you've never touched Node, you probably use an application that is using the readable stream module. Every runtime's choice to do its own thing has very real costs and causes very real pain for developers. We shouldn't pick a winner is basically a concept within Node that says we shouldn't decide, we being the Node core developers, we shouldn't decide which of these modules out on npm people should use. It's a fundamental problem and it's not one that we should take lightly. We need to be opinionated. Being opinionated is good. In the runtime, being opinionated about what APIs are best is what you need to, it's something that the runtimes themselves need to do. We should absolutely pick a winner when it comes to APIs if we don't developers suffer.
Even if you've never touched Node, you probably use an application that is using the readable stream module. Every runtime's choice to do its own thing has very real costs and causes very real pain for developers. You know, so the readable stream module, right? When we decide to change an API in Node, that module has to change that cause, you know, and that trickles down to everyone else who's using it, even if they are not using Node, right?
The readable stream module, the decisions we make about what other dependencies in Node are required trickle down. So, you know, readable streams uses the buffer API, which means you can't just pull in the readable stream API into Node, or into Deno or any other environment. You also have to pull buffer in. Readable stream is also based on Node's idea of event emitter. Which means you have to pull in event emitter into Deno and, you know, workers and browsers. So, you know, this choice that Node made to have this API, and the fact that so many people are using it, even if they are never even if they've never touched Node, it drags on a huge amount of additional work and additional pain and additional effort for developers pretty much everywhere. Whether you're using Node or not.
So, there's another idea here that we shouldn't pick a winner. So, backing up what this means is, you know, so Node is a runtime and there's this ecosystem of developers who are creating stuff on that runtime. And they publish those things to npm. This idea of we shouldn't pick a winner is basically a concept within Node that says we shouldn't decide, we being the Node core developers, we shouldn't decide which of these modules out on npm people should use. If there are two different implementations of a web framework, for instance, you know, we shouldn't have an opinion about which one is better or we shouldn't, you know, advantage one over the other. Basically, just let the market decide, let the developers decide. All we're doing is providing a platform upon which these things will run.
The idea of this, I mean it's fine in concept, except when it just ends up adding friction to developers. And the friction I'm talking about is deciding which one of these you actually should use, which one of these is going to be maintained long term, which one of these is actually going to be the safest investment for your application. And how are you protected against making the wrong choice? How do you know you're not going to end up having to rewrite your application or things are going to stop working or you're going to be stuck on an older version of Node that might have security vulnerabilities because you can't update because it requires half your application to write, not to be rewritten. It's a fundamental problem and it's not one that we should take lightly. All right, so I'm going to say just right off, I'm sorry, this idea is wrong. It's something that we messed up on with the Node. It's something that I had, it's a philosophy I had bought into. But as time has gone, I've come to realize that this entire idea is wrong. We need to be opinionated. Being opinionated is good. In the runtime, being opinionated about what APIs are best is what you need to, it's something that the runtimes themselves need to do. When there is a common need for something, runtimes should favor a single consistent common API while encouraging competition in the implementation of it. We should absolutely pick a winner when it comes to APIs if we don't developers suffer. A case study of this is the nodeBuffer API.
9. Buffer API Standardization
10. Proposal for API Standardization
11. Runtime-Specific APIs and Node vs Web Streams
Runtime specific APIs that have to be polyfilled and other environments are a mistake. Node streams and web streams are both complex APIs. Node streams are widely used and complicated, but still work well. Web streams have a similar level of complexity and perform the same tasks. The decisions made by Node and the what WG impact developers worldwide.
It doesn't matter which runtime we're talking about. Runtime specific APIs that have to be polyfilled and other environments are a mistake right? If they are if they need to be in these other environments, if developers need this functionality in multiple places, then there should be a standard API. And it should be common.
Node streams versus web streams. Here's a great example of this. Node streams predate the what W3C stream standard. It's been around for years. There are three different versions of node streams in one implementation. It's one of the oldest most complicated APIs in node. And it's also one of the most broadly used across the entire ecosystem. Node streams are an overly complicated mess of code and APIs that are horrible to use, but still work really, really well.
12. Cooperation and Standard Streams API
13. The Deano Standard Library
The Deano standard library. I just gotta make a point. They call it a standard library. It's really just set of Deano-specific APIs. They call it a standard library for Deano itself. Brilliant idea. It'd be better if it wasn't specific to Deano. And if you look through the implementation, there are a number of places where it's very specific to Deano. You know, it's something that we can do better, right?
It's something that we can continue to iterate on. One key question here, what about backwards compatibility? Adding standard APIs does not mean changing or removing existing ones. A good example of this is URL parse and NewURL in Node. Both of these still exist in Node. NewURL is the one you should be using. But URL parse is not going anywhere. It's still there. It just has a bunch of bugs and some security concerns. And there's lots of reasons you shouldn't use it. But if you do use it, it'll continue to work. Adding these new things does not mean getting rid of the old stuff, right?
Standard Library and Node Core Contributors
Take what Deno has started with their standard library and expand it to a cross-platform, multi-runtime environment. The underlying implementations can be different, but the APIs should be clear, consistent, and common across all environments. The poll results show that a third of respondents are unsure about the APIs being used. Now, let's move to the questions. Are any Node core members part of W3C W3C, WG or TC39? Node itself cannot join these organizations, but individual Node core contributors are active in TC39 W3C.
So take basically what Deno has started with their standard library, which exists as a separate repo and separately installable pieces. Let's start there, perhaps. Expand that out to something that is truly cross-platform, truly a multi-runtime environment that works for Node. It works for Deno. It works for workers. And it will work for any environment that chooses to support the standard library APIs. The underlying implementations can be different. Nobody says that these have to be implemented the exact same way. But the APIs themselves should be clear, consistent, and common across all of these different environments.
Anyway, I'm out of time. So, that's my modest proposal. I'm looking forward to conversations around this and seeing where we can go. And I hope you all enjoy the rest of the conference. Bye.
So now, let's move to the questions. People can still ask questions in the Node Talk Q&A in the Discord channel. We have one from Com Frarial. Are any Node core members part of W3C W3C, WG or TC39? Yeah, absolutely. So the one thing I can describe is Node itself is part of the JS Foundation, which is a certain type of nonprofit organization in the United States. It cannot, because of the type of organization it is, Node itself cannot join TC39, and it can't join W3C as an organization. But individuals can, individuals that work for employers like IBM or Cloudflare or whatever, where these companies might be involved with these groups, then they can participate. So we have individual Node core contributors who are very active in TC39 W3C. And Whatwg is, you know, you don't have to be a member of a company or anything, anybody can participate in there.
Relevance of Browser APIs and Node Development
Browsers expose a large number of standardized APIs, many of which are relevant to server and edge developers. APIs like URL, compression stream, decompression stream, streams, crypto, WebSocket, File, and Blob are common to multiple platforms. While some web platform APIs may not be relevant, there is significant overlap. Front-end and Node.js development require different mindsets. The speaker is working on implementing the quick protocol for Node, which is expected to land halfway through the year.
We've been active in a lot of those conversations. Thanks. Thank you for your answer.
We have another question. Browsers expose a very large number of standardized APIs for users. How many of those APIs are actually relevant to server and edge developers running Node.js, Deno or Edge compute environments like Cloud for Workers? Quite a bit of them. And anybody who's been paying attention to Node for a while and Deno and Workers. You know, we've seen a lot of APIs like URL, right? Compression stream, decompression stream like Deno just rolled out decompression stream I think in 1.9 today I think. There's all of the streams APIs, all of the crypto APIs. There's the WebSocket APIs. There's you know File and Blob and you know there's a huge number of these things that are relevant. That represent functionality that is common to all these platforms. So you know, yes there's a lot of the web platform ones that just are never going to be relevant right? Like I don't think we're ever going to see you know like media rendering APIs there but yeah where there is overlap there's definitely a lot of APIs that we can look at.
Thank you. And we have a comment here from JsCars saying that he's certainly right, most of the time I need two mindsets. One I work front-end development and other when I'm working with Node.js. So yeah so thank you very much for your talk. Now that you work in Node core, can you tell us what is the most interesting topic that you're working on right now or any feature? For Node, I'm working on trying to get the quick protocol landed, so we're going to be working on that quite a bit more over the next couple of months and hopefully we'll get that landed probably about halfway through the year. Wow very exciting, very exciting. I hope it happens soon. Thank you very much James. I hope you enjoyed this conference and I hope the attendees learn a lot too.