Node Congress

Node Congress 2022

<p>JSON modules have been an important feature of the JavaScript ecosystem for a long time, and it’s started to take a new shape with there new ESM import syntax. Let’s review the history of JSON support in Node.js, its relationship with web compatibility, and how we can make (finally) it happen.</p><p>  <a href="https://www.froala.com/wysiwyg-editor?pb=1"> </a></p>

The Road to JSON Import Support in Node.js

Hello everyone, my name is Antoine Duhamel or aduh95 on GitHub. And I'm here to tell you about the road to JSON Import support in Node.js. So first I'm going to talk a bit about me. So I'm a Node.js Technical Steering Committee member since April 2021. And I'm also working at Transloadit. That's my day job. And they also pay me to tell you that if you want your work on Node.js, your contribution on Node.js Core to be sponsored by them, just send an email. It's a cool team. Anyway, so my talk is about JSON modules. So what are they? It's just a handy way for a JavaScript author to interact with JSON files. That could be for a configuration or to consume an API. And I'm going to go through the history of JSON modules in the JavaScript ecosystem. And then I will see how you can use them today. What's the next steps? So the history of JSON modules goes back to the beginning of Node.js. So first when Node.js was introduced, there was no module system, no standard module system in the JavaScript ecosystem. Mostly you were using flow boards. And so something has to be made up for this. So Node.js came up with the CommonJS, which is also called CGS. And they had support for JSON quite early. So on this screenshot, we can see it was in 2011. So I can say it was forever ago in the JavaScript world. And the one obvious drawback of CommonJS or CGS, it's not supported in browsers. So the next step is the ESM specification or the ECMAScript modules. So that was part of the ES6 or ES2015 spec. So the ES6 spec is a big, big spec jump where we went from old JavaScript and more modern JavaScript. And one of its additions was the import and export keywords. And so the module mechanism that would allow JavaScript code to interact with other JavaScript files. And that is currently supported by Node.js, browsers, Deno, TypeScript, most of the ecosystem. And so one notable thing I should add on that, it's browsers in particular, enforce that only JavaScript files can be loaded through this mechanism. And that's going to be important just later. So on Node.js side, the first implementation landed in the version 8.5.0. It was a very experimental stage at this point. And we make most of the CommonJS mechanisms. So one of its features was to be able to import JSON files as well. So when that landed, there's been discussion to add the JSON module support in browsers. So that was actually merged in 2019. So the idea was if, so on a browser, it's modules are fetched through HTTP. So if the HTTP response contains a MIME type for a JSON file, it's going to be loaded and parsed as a JSON file. If it's JavaScript, it's going to be parsed and loaded as JavaScript. But that was actually reverted a few months later because of security concerns raised by some of the implementers. So maybe we can go through the security, the details of this revert. So let's say you are consuming a weather API and you expect to get a JSON object with a different value inside. So you import it as a module because you can. And what if instead of returning a JSON file, the server, either because it's compromised or because it's malicious, returns the JavaScript files that does something nasty. So in this example, I'm adding a fetch call that would send all the local storage data to another server. The thing is it could be anything. It's undefined behavior at this point. And brother vendors decided it was not acceptable. So to work around this, a proposal was raised to instead change the JavaScript language to add another addition to the import syntax, which could ensure that this doesn't happen. So you can see here at the end of this import statement, there's an assert with type JSON. So that would mean for the JavaScript engine that if the response is not a JSON data, the import statement fails and the code is not executed. So if you're not familiar with how to change the JavaScript language, it would be interesting to talk a bit about that. So the JavaScript language is governed by the ECMAScript spec. And that spec is written by the TC39 committee. So that's a lot of acronyms, but bear with me. So the TC39 has a list of proposals that is publicly available that you can see. And each proposal goes through four different stage and the JSON module line posts. You go have a screenshot of that. So as you can see, the JSON modules and import assertions proposal are up there in the stage three bucket. So that means it's mostly done. It's ready to be implemented. And they are waiting for feedback before deciding if it will go in stage four, which is the proposal is integrated to the official spec. I wanted also to give a shout out to the temporal proposal, if you're not familiar. I'm pretty pumped about this one. So as you can see here on this screenshot, it's an API to replace the date API to work with time and date in JavaScript, which is supposed to be better than the current date API. Anyway, just wanted to mention that. So back to JSON modules. So now that the ECMAScript spec has a mechanism to ensure there won't be a security problems when loading JSON modules, that's back to the HTML spec, which have landed an update of the spec to allow JSON module when an assertion is present. And now it's time for NetJS to build on both spec to have their own JSON modules. So you might be wondering at this point, what the HTML spec has to do with the NetJS implementation. So I have an example for you. So in this very case, when the user imports twice the same module, but with a different assertion type, interestingly enough, the ECMAScript spec does not define what should happen then. It forbids us to resolve both. It says that at least at most one can resolve, the other one must be rejected. And it gives to the implementers how to deal with it. So in the case of NetJS, we decided it makes sense to follow the HTML spec definition for how to handle this. So we can ensure that a module can run exactly the same in the browser or in NetJS. So when you work on NetJS, you sometimes have to go through the HTML spec to be sure the implementation will be future proof. So when NetJS lands HTTPS imports, it can work as it was in the browser. So what's the status today? So currently, NetJS has the import assertion and JSON modules available on V17 and V16 will also come soon-ish. TypeScript also in 4.5, Deno 1.17, Chromium browsers also added. One missing in Firefox, Safari, and vendors, but hopefully that will be resolved soon. How to use them? So you might have seen already some examples in my talk, but I wanted to give a clear overview. So you import it as any module and you add the assertion at the end. And it also works with dynamic imports. One interesting caveat, you might be tempted to import only a part of the packages. So I think if you wanted to get the version of the package you're in, you might want to do this, but this will not work. You have to import it using the default import and then access the field you're looking for. So the reason for this is JSON files are not arrays objects. It could be strings or arrays or even number. So you have to... I mean, the JavaScript engine cannot guess which it is, so it has to treat it as a black box, if you will, and it's only the default import is available. So what's next? So the ball is now on your court. So if you're a JavaScript player, you can test it, play with it. I wouldn't recommend putting it on production just yet. Maybe you'd like to wait for it to reach stage four before that. But yeah, definitely if you have a toy project, you were wanting to experiment with that, definitely do that. You can open issues and PRs on the repos and participate to the discussion regarding what's the next step. So Node.js now has implemented the same web compatibility feature. Do we want to go further than that? So maybe more modules, more modules, TypeScript, why not CoffeeScript? That's definitely things we would like to add support to in Node core. We need some folks who have the will to make it happen. So if that's you, you know you. Yeah, so that's it for me. So thanks for tuning in and if you have any questions, that's the time. Hey, Ardaeen. Hey, Sabren. Thanks for such a great in-depth talk. So we asked the question which you asked to our audience and to see the results. So okay, so the majority, 73% says that they use it but it gets transpiled to common.js using TypeScript, Babel. Interesting. Yeah, I guess there's a lot of TypeScript lovers in the audience today. It's still nice to see that at least one in five is using it without translation. That's great. And everyone has heard of it at least. Yeah, I mean, just 0% I do not know what it is. At least I like this part of the result. Yeah, that would have been scary to find out no one heard of it. So were you expecting anything different or what did surprise you here? I'm still a bit surprised that so many people transpiled to common.js but I guess that makes sense because it's the classical way of doing things. So if you have a project that has a bit of a historic that's been in place for a while, maybe ESM wasn't around them using CGS, it still makes sense. It's still interesting to see. Yes. Okay, so let's take some Q&As or the questions, audience questions. So we have a question on slide number 18, you mentioned it is not possible to import version from a JSON in one line, but it works in two lines. What are the difference between two syntaxes? Okay, yeah. So yeah, it's a bit, indeed, maybe a bit tricky if you're not used to the ESM syntax. When you're working with JSON files with require calls, you can simply do that in one line, import one part, one chunk of the JSON file. That doesn't work in the ESM syntax. That's mainly for performance reasons. So when the JavaScript engine reads your import, it has to know if what you're importing actually exists, and it would be too costly for it to read the JSON file at this stage. So that's why it's forbidden. And it might feel arbitrary if you're not used to it, but just something to have in mind. Yeah. Yeah, so get everything and then get the version out of that. So okay, so we have another question. Do you think with TC39 adopting types as comments and native ESM bundling over time, ESM will gain more users? Yeah, so it's hard to predict how the JavaScript ecosystem will adapt to the thing. I certainly hope it will happen because I think ESM is really awesome and is the future of JavaScript. And certainly helping reduce the gap between what the users want to write as code should help. So yeah, I'd like to answer yes to the question, but it will depend on everyone else. Yeah, it's always tricky. Yeah, I would say. It can go either way, but yeah, let's see what future holds for that. So we have another question from Oswald. So there is a few concerns people have raised with ESM. For example, it's inability to do things like const some initialize module equal to require the module name and then calling it with some options. What do you think is an appropriate way to overcome this? Yeah, so it's a bit like the first question. Using the ESM syntax makes you sometimes use two lines instead of what could have been one in common GS. And yeah, I don't think there's really a walk around for it. And some people will prefer one syntax over the other, that's for sure. But yeah, maybe if you're not used to ESM and you still want to use common GS, by the way, it's still possible. I don't think it's going away anytime soon. But if you want to use ESM and get back those capabilities, yeah, I guess there's no good answer for that. You have to work around the syntax. So as I said, it's mostly chosen for performance reasons. Performance doesn't matter as much in common GS because it's only working in Node.js, which does not do a network call while ESM works everywhere. So if you have to do a network call every time you import something, it can add up quite a bit. So performance was more unfocused when designing this syntax. And what was the question exactly? Is there a way to overcome this? Yeah, to overcome, I don't think. So yeah, I guess the answer is I don't think it can be overcome. It's just a shift in perspective on how you import your APIs and how you export them as well. Okay. So you need to have a walk around to overcome that. So Veevers asks, what happens if you import invalid JSON and can such an error be cached? So if you import, so there are two types of imports when you use ESM syntax, the static imports. So that's the one you see at the top level, import name from the URL or the module name. So that's if you try to import a JSON file that's invalid this way, the whole module graph will crash. So if anywhere in your static imports, you have an invalid JSON import, it's going to crash and you cannot catch that. That's something to be aware of. Maybe something you want, maybe something you don't want. If you don't want it, you can use instead dynamic imports. So that's the other import with the parentheses and you put your model name and it returns a premise. And that's, you can catch the error if there is an error, just add a.catch or if you wait, you can have it in a.tray catch and it looks like a require call. Okay. Yeah. So, okay. We have another question. Will Node.js add support for CSS modules? I guess it depends if there's a good use case for that. And if someone champions to bring them, I don't, I think it's really a broader focus feature because I don't think there's a lot of reasons to have CSS modules in Node.js today. But if you have a use case, you can make it happen if you're willing enough. Yeah. Yeah. So you can make out some issue or some discussion thread on the GitHub repo for that maybe. And yeah. Yeah. Definitely make yourself known because if no one says they don't need it, no one will want to implement it. Great. Thank you so much. These were all the questions that we had. So thanks a lot for answering and to the audience, you can still get to ask questions to Antoine. Just move on to his speaker room. You can go in the timeline and see where his speaker room is. Join him and ask more questions or discuss if you want to discuss anything. Thanks a lot, Antoine, once again for joining us. It was really great having you here. Yeah. Thank you. Welcome. Thanks for having me.
