The Road to JSON Import Support in Node.js

Rate this content
Bookmark

JSON modules have been an important feature of the JavaScript ecosystem for a long time, and it’s started to take a new shape with there new ESM import syntax. Let’s review the history of JSON support in Node.js, its relationship with web compatibility, and how we can make (finally) it happen.

16 min
18 Feb, 2022

Video Summary and Transcription

This Talk discusses the road to JSON import support in NodeJS, covering the history and implementation of JSON modules. It explores security concerns and the proposal for import assertions. The Talk also explains how to use JSON modules in NetJS and the availability of support in different browsers. It mentions working with dynamic imports and future plans for adding more modules in Node Core. Additionally, it addresses the syntax differences between ESM and CommonJS and the challenges of overcoming them.

Available in Español

1. Introduction to JSON Modules

Short description:

Hello, everyone. My name is Anton Duhamel or aduh95 on GitHub. I'm here to tell you about the road to JSON import support in NodeJS. I'm gonna talk a bit about me, so I'm NodeJS Technical Steering Committee member since April, 2021. And I'm also working at Translordit, so that's my day job. My talk is about JSON modules, so what are they? It's just a handy way for a JavaScript author to interact with JSON files. The history of JSON modules goes back to the beginning of NodeJS. First, when Node.js was introduced, there was no module system, no standard module system in the JavaScript ecosystem. Node.js came up with the CommonJS, which is also called CGS, and they had support for JSON quite early. The next step is the ESM specification or the ECMAScript modules. That was part of the ES6 or ES2015 spec. It allows JavaScript code to interact with other JavaScript files. Currently supported by Node.js, browsers, Deno, TypeScript, most of the ecosystem. On Node.js side, the first implementation landed in version 8.5.0. It was a very experimental stage at this point and mimicked most of the common JS mechanisms. One of its features was to be able to import JSON files as well. There has been discussion to add the JSON module support in browsers. That was merged in 2019.

Hello, everyone. My name is Anton Duhamel or aduh95 on GitHub. I'm here to tell you about the road to JSON import support in NodeJS.

So first, I'm gonna talk a bit about me, so I'm NodeJS Technical Steering Committee member since April, 2021. And I'm also working at Translordit, so that's my day job. And they also pay me to tell you that if you want your work on NodeJS, your contribution on NodeJS core to be sponsored by them, just send an email. It's a good time.

Anyway, so my talk is about JSON modules, so what are they? So it's just a handy way for a JavaScript author to interact with JSON files. So that could be for configuration or to consume an API. And I'm gonna go through the history of JSON modules in the JavaScript ecosystem, and then I will see how we can use them today and what's the next steps.

So the history of JSON modules goes back to the beginning of NodeJS. So first, when Node.js was introduced, there was no module system, no standard module system in the JavaScript ecosystem. Mostly you were using flow boards. And so something has to be made up for this. So Node.js came up with the CommonJS, which is also called CGS, and they had support for JSON quite early. So on this screenshot, we can see it was in 2011. So you can say it was forever ago in the JavaScript world. And the one obvious drawback of CommonJS or CGS, it's not supported in browsers. So the next step is the ESM specification or the ECMAScript modules. So that was part of the ES6 or ES2015 spec. So it was, the ES6 spec is a big spec jump where we went from all JavaScript and more modern JavaScript. And one of its addition was the import and export keywords and the module mechanism that would allow JavaScript code to interact with other JavaScript files. And that is currently supported by Node.js, browsers, Deno, TypeScript, most of the ecosystem. And one notable thing I should add on that, it's browsers in particular enforce that only JavaScript files can be loaded through to this mechanism. And that's gonna be important just later. So on Node.js side, the first implementation landed in version 8.5.0. It was a very experimental stage at this point and mimicked most of the common JS mechanisms. So one of its feature was to be able to import JSON files as well. So when that landed, there has been discussion to add the JSON module support in browsers. So, and that was actually merged in 2019.

2. Security Concerns and Import Assertions

Short description:

The idea of loading modules fetched through HTTP was reverted due to security concerns. A proposal was raised to add an assert with type JSON to the import syntax, ensuring that only JSON data is loaded. The JavaScript language is governed by the ECMAScript spec, written by the TC39 committee. The JSON modules and import assertions proposal is currently in stage three and ready to be implemented.

So the idea was if, so on a browser, it's modules fetched through HTTP. So if the HTTP response contains a MIME type for a JSON file, it's gonna be loaded as a, and parsed as a JSON file. If it's JavaScript, it's gonna be parsed and loaded as JavaScript. But that was actually reverted a few months later because of security concerns raised by some of the implementers. So maybe we can go through the security in the details of this revert.

So let's say you are consuming a weather API and you expect to get a JSON object with a different value inside. So you import it as a module because you can. And what if instead of returning a JSON file, the server, either because it's compromised or because it's malicious, returns a JavaScript file that does something nasty. So in this example, I'm adding a fetch call that would send all the local storage data to another server. But the thing is it could be anything, it's undefined behavior at this point. And browser vendors decided it was not acceptable.

So to work around this, a proposal was raised to instead change to JavaScript language to add another addition to the server to the import syntax, which could ensure that this doesn't happen. So you can see here at the end of this import statement, there's an assert with type JSON, so that would mean for the JavaScript engine that if the response is not JSON data, JSON data, the import statement fails and the code is not executed. So if you're not familiar with what's, how to change the JavaScript language, it would be interesting to talk a bit about that. So the JavaScript language is governed by the ECMAScript spec and that spec is written by the TC39 committee. So that's a lot of acronyms, but bear with me. So the TC39 has a list of proposals that is publicly available that you can see. And each proposal goes through four different stage and the JSON module line posts, you go out and screenshot of that. So as you can see, the JSON modules and import assertions proposal are up there in the stage three packet. So that means it's mostly done. It's ready to be implemented. And they are waiting for feedback before deciding if it will go in stage four, which is the proposal is integrated to the official spec. I wanted to also to give a shout out to the template proposal. If you're not familiar. I'm pretty pumped about this one. So as you can see here on this screenshot, it's an API to replace the data API to work with time and date in JavaScript, which is supposed to be better than the current data API. Anyway, just wanted to mention that. So back to just modules. So now that the ECMAScript spec has a mechanism to ensure that there won't be a security problems when loading JSON modules.

3. Using JSON Modules in NetJS

Short description:

Now that the ECMAScript spec has a mechanism to ensure security when loading JSON modules, NetJS can build on both specs to have its own JSON modules. The HTML spec defines how to handle importing the same module with different assertion types, and NetJS follows this definition. NetJS currently has the import assertion and JSON modules available, and it's also supported in TypeScript, Deno, and Chromium browsers. However, it's still missing in Firefox, Safari, and Vendors. To use JSON modules, import them as any module and add the assertion at the end. Just remember that if you want to import only a part of the package, you need to use the default import and access the specific field you're looking for. While it's not ready for production yet, you can test and experiment with it in your toy projects.

So now that the ECMAScript spec has a mechanism to ensure that there won't be security problems when loading JSON modules. That's back to the HTML spec, which has landed an update of the spec to allow JSON modules when an assertion is present. And now it's time for NetJS to build on both specs to have their own JSON modules.

So you might be wondering at this point, what the HTML spec has to do with the NetJS implementation. So I have an example for you. In this case, when the user imports twice the same module but with a different assertion type, the HTML spec does not define what should happen. It forbids us to resolve both and leaves it to the implementers to decide. In the case of NetJS, we decided to follow the HTML spec definition for how to handle this, ensuring that a module can run the same in the browser or in NetJS.

Currently, NetJS has the import assertion and JSON modules available on v17, and v16 will also come soon-ish. TypeScript also has it in 4.5, Deno in 1.17, and Chromium browsers have added it. However, it is still missing on Firefox, Safari, and Vendors, but hopefully, that will be resolved soon. To use them, you import it as any module and add the assertion at the end. Dynamic imports are also supported. One caveat is that if you want to import only a part of the package, you need to import it using the default import and then access the specific field you're looking for. JSON files can be strings, arrays, or even numbers, so the JavaScript engine treats them as a black box, and only the default import is available.

The ball is now in your court. If you're a JavaScript player, you can test and play with it. However, I wouldn't recommend putting it into production just yet. Maybe wait for it to reach stage four before doing so. But if you have a toy project or want to experiment, definitely give it a try.

4. Working with Dynamic Imports and Future Plans

Short description:

And you can also work with dynamic imports. The reason for this is JSON files are not always objects. It could be strings or arrays or even number. So, if you're a JavaScript player, you can test it, play with it. Node.js now has implemented the same web compatibility feature. We want to go further than that, adding support for more modules in Node Core. Thanks for tuning in and if you have any questions, that's the time.

And you can also work with dynamic imports. One interesting caveat, you might be tempted to import only a part of the packages, so I think if you wanted to get the version of the package you're in, you might want to do this, but this will not work. You have to import it using the default import and then access the field you're looking for. So, the reason for this is JSON files are not always objects. It could be strings or arrays or even number. So, you have to, I mean the JavaScript engine cannot guess which it is, so it has to treat it as a black box, if you will. And it's only the default import is available.

Yeah, so what's next? So, the ball is now on your court, so if you're a JavaScript player, you can test it, play with it. I wouldn't recommend putting it on production just yet. Maybe you'd like to wait for it to reach stage four before that, but yeah, definitely, if you have a toy project, you were wanting to experiment with that, definitely do that. You can open issues and PRs on the repos and participate to the discussion regarding what's the next step. So, Node.js now has implemented the same web compatibility feature. Do we want to go further than that? So maybe, yeah, more modules, Tomo modules, TypeScript, why not Coffeescript? That's definitely things we would like to add support to in Node Core. We need some folks who have the will to make it happen. So if that's you, you know you. Yeah, so that's it for me. So thanks for tuning in and if you have any questions, that's the time.

Hey, Ardaeen. Thanks for such a great in-depth talk. So we asked the question which you asked to our audience and to see the results. So, okay, so the majority, 73% says that they use it but it gets transpiled to common JS using TypeScript, Mabel, et cetera. Interesting. Yeah, I guess there's a lot of TypeScript lovers in the audience today. It's still nice to see that at least one in five is using it without translation. So that's great. And everyone has heard of it at least. Yeah, I mean, just 0%, I do not know what it is. At least I liked this part of the research. Yeah, that would have been scary to find out knowing a little bit. So, what are you expecting, anything different or what did surprise you here? I'm still a bit surprised that so many people transpiles to common JS, but I guess that makes sense because it's the classical way of doing things.

QnA

Importing JSON in ESM Syntax

Short description:

If you have a project with a historic codebase using CommonJS, it can make sense to stick with it. When working with JSON files, importing a specific part in one line is not possible in ESM syntax due to performance reasons. The JavaScript engine needs to know if the imported file exists, and reading the JSON file at import time would be too costly. As for the future of ESM, it's hard to predict, but I believe it is the future of JavaScript and can help bridge the gap between user code and desired functionality. Concerns have been raised about ESM's inability to do certain things, like initializing modules with options, but there are workarounds.

So, if you have a project that has a bit of a historic that's been in place for a while, maybe ESM wasn't around them using CGS, it can make sense. Yeah, it's still interesting to see.

Yes, okay. So let's take some Q and A's of the questions, audience questions. So, okay, so we have a question on slide number 18, you mentioned it is not possible to import version from a JSON in one line, but it works in two lines. What are the difference between two syntaxes? Okay, yeah, so, yeah, it's a bit, maybe a bit tricky if you're not used to the ESM syntax. When you're working with JSON files with required calls, you can simply to that in one line, import one part, one chunk of the JSON file. And that doesn't work in the ESM syntax, that's mainly for performance reasons. So when the JavaScript engine reads your import, it has to know if this, what you're importing actually exists. And it would be too costly for it to read the JSON file at this stage. So that's why it's forbidden. And it might feel arbitrary if you're not used to it. But yeah, just something to have in mind on the. But yeah. Yeah, so get everything and then get the version out of that. So, okay.

So we have another question. Do you think with TC39 adopting types as comments and native ESM by bundling over time, ESM will gain more users? Yeah, so it's hard to predict. How the JavaScript ecosystem will adapt those things. I certainly hope it will happen because I think ESM is really awesome and is the future of JavaScript. And certainly helping reduce the gap between what the users want to write as code should help. So, yeah, I'd like to answer yes to the question, but it will depend on everyone else, you know. Yeah, it's always tricky. Yeah, I would say, but it can go either way. But yeah, let's see what future holds for that.

So, we have another question from Uswal. So, there is a few concerns people have raised with ESM. For example, it's inability to do things like const, some initialize module equal to require, then module name, and then calling it with some options. What do you think is an appropriate way to overcome this? Yeah, so it's a bit like the first question. Using the ESM syntax makes you sometimes use two lines instead of what could have been one incoming JS.

Syntax Differences and Overcoming Challenges

Short description:

There is no good answer for the syntax differences between ESM and CommonJS. It's mostly chosen for performance reasons, but there is no way to completely overcome it. It requires a shift in perspective on how APIs are imported and exported.

And yeah, I don't think there's really a workaround for it. And some people really prefer one syntax over the other. That's for sure. But yeah, maybe if you're not used to ESM and you still want to use a common JS. By the way, it's still possible. I don't think it's going away anytime soon, but if you want to use the ESM, get back those capabilities. Yeah, I guess that there's no good answer for that. You have to work around the syntax.

So as I said, it's mostly chosen for performance reasons. Performance doesn't matter as much in common JS because it's only working in Node.js which does not do a network call while ESM works everywhere. So if you have to do a network call every time you import something can add up quite a bit. So performance was more unfocused when designing this syntax. And what was the question exactly? Is there a way to overcome this? Yeah, to, well to overcome, I don't think. So yeah, I guess the answer is I don't think it can be overcome. It's just a shift in perspective on how you import your APIs and how you export them as well.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Node Congress 2022Node Congress 2022
26 min
It's a Jungle Out There: What's Really Going on Inside Your Node_Modules Folder
Top Content
Do you know what’s really going on in your node_modules folder? Software supply chain attacks have exploded over the past 12 months and they’re only accelerating in 2022 and beyond. We’ll dive into examples of recent supply chain attacks and what concrete steps you can take to protect your team from this emerging threat.
You can check the slides for Feross' talk here.
Node Congress 2022Node Congress 2022
34 min
Out of the Box Node.js Diagnostics
In the early years of Node.js, diagnostics and debugging were considerable pain points. Modern versions of Node have improved considerably in these areas. Features like async stack traces, heap snapshots, and CPU profiling no longer require third party modules or modifications to application source code. This talk explores the various diagnostic features that have recently been built into Node.
You can check the slides for Colin's talk here. 
JSNation 2023JSNation 2023
22 min
ESM Loaders: Enhancing Module Loading in Node.js
Native ESM support for Node.js was a chance for the Node.js project to release official support for enhancing the module loading experience, to enable use cases such as on the fly transpilation, module stubbing, support for loading modules from HTTP, and monitoring.
While CommonJS has support for all this, it was never officially supported and was done by hacking into the Node.js runtime code. ESM has fixed all this. We will look at the architecture of ESM loading in Node.js, and discuss the loader API that supports enhancing it. We will also look into advanced features such as loader chaining and off thread execution.
JSNation Live 2021JSNation Live 2021
19 min
Multithreaded Logging with Pino
Top Content
Almost every developer thinks that adding one more log line would not decrease the performance of their server... until logging becomes the biggest bottleneck for their systems! We created one of the fastest JSON loggers for Node.js: pino. One of our key decisions was to remove all "transport" to another process (or infrastructure): it reduced both CPU and memory consumption, removing any bottleneck from logging. However, this created friction and lowered the developer experience of using Pino and in-process transports is the most asked feature our user.In the upcoming version 7, we will solve this problem and increase throughput at the same time: we are introducing pino.transport() to start a worker thread that you can use to transfer your logs safely to other destinations, without sacrificing neither performance nor the developer experience.

Workshops on related topic

Node Congress 2023Node Congress 2023
109 min
Node.js Masterclass
Workshop
Have you ever struggled with designing and structuring your Node.js applications? Building applications that are well organised, testable and extendable is not always easy. It can often turn out to be a lot more complicated than you expect it to be. In this live event Matteo will show you how he builds Node.js applications from scratch. You’ll learn how he approaches application design, and the philosophies that he applies to create modular, maintainable and effective applications.

Level: intermediate
Node Congress 2023Node Congress 2023
63 min
0 to Auth in an Hour Using NodeJS SDK
WorkshopFree
Passwordless authentication may seem complex, but it is simple to add it to any app using the right tool.
We will enhance a full-stack JS application (Node.JS backend + React frontend) to authenticate users with OAuth (social login) and One Time Passwords (email), including:- User authentication - Managing user interactions, returning session / refresh JWTs- Session management and validation - Storing the session for subsequent client requests, validating / refreshing sessions
At the end of the workshop, we will also touch on another approach to code authentication using frontend Descope Flows (drag-and-drop workflows), while keeping only session validation in the backend. With this, we will also show how easy it is to enable biometrics and other passwordless authentication methods.
Table of contents- A quick intro to core authentication concepts- Coding- Why passwordless matters
Prerequisites- IDE for your choice- Node 18 or higher
JSNation 2023JSNation 2023
104 min
Build and Deploy a Backend With Fastify & Platformatic
WorkshopFree
Platformatic allows you to rapidly develop GraphQL and REST APIs with minimal effort. The best part is that it also allows you to unleash the full potential of Node.js and Fastify whenever you need to. You can fully customise a Platformatic application by writing your own additional features and plugins. In the workshop, we’ll cover both our Open Source modules and our Cloud offering:- Platformatic OSS (open-source software) — Tools and libraries for rapidly building robust applications with Node.js (https://oss.platformatic.dev/).- Platformatic Cloud (currently in beta) — Our hosting platform that includes features such as preview apps, built-in metrics and integration with your Git flow (https://platformatic.dev/). 
In this workshop you'll learn how to develop APIs with Fastify and deploy them to the Platformatic Cloud.
JSNation Live 2021JSNation Live 2021
156 min
Building a Hyper Fast Web Server with Deno
WorkshopFree
Deno 1.9 introduced a new web server API that takes advantage of Hyper, a fast and correct HTTP implementation for Rust. Using this API instead of the std/http implementation increases performance and provides support for HTTP2. In this workshop, learn how to create a web server utilizing Hyper under the hood and boost the performance for your web apps.
React Summit 2022React Summit 2022
164 min
GraphQL - From Zero to Hero in 3 hours
Workshop
How to build a fullstack GraphQL application (Postgres + NestJs + React) in the shortest time possible.
All beginnings are hard. Even harder than choosing the technology is often developing a suitable architecture. Especially when it comes to GraphQL.
In this workshop, you will get a variety of best practices that you would normally have to work through over a number of projects - all in just three hours.
If you've always wanted to participate in a hackathon to get something up and running in the shortest amount of time - then take an active part in this workshop, and participate in the thought processes of the trainer.
TestJS Summit 2023TestJS Summit 2023
78 min
Mastering Node.js Test Runner
Workshop
Node.js test runner is modern, fast, and doesn't require additional libraries, but understanding and using it well can be tricky. You will learn how to use Node.js test runner to its full potential. We'll show you how it compares to other tools, how to set it up, and how to run your tests effectively. During the workshop, we'll do exercises to help you get comfortable with filtering, using native assertions, running tests in parallel, using CLI, and more. We'll also talk about working with TypeScript, making custom reports, and code coverage.