Building a Node.js Runtime for the Browser

Rate this content
Bookmark

We'll walk through what it takes to build a Node.js runtime for the browser, the challenges we discovered along the way and solutions we built to those challenges.

Jasper De Moor
Jasper De Moor
13 min
15 Nov, 2023

Comments

Sign in or register to post your comment.

Video Summary and Transcription

This Talk discusses the challenges and collaboration involved in building Nodebox, a Node.js compatible runtime for the browser. It provides an overview of how Nodebooks works, including the main manager, previews, and web workers. The Talk also covers the simplicity and speed of reading from the filesystem in Nodebooks. It highlights the complexity of implementing HTTP support and WebSocket mocking in Nodebox. Lastly, it mentions the ability to build a web server using Nodebox and provides information on available templates.

Available in Español

1. Building Nodebox: Challenges and Collaboration

Short description:

Hey, I'm Jasper. I'll talk about how we built Nodebox, a Node.js compatible runtime for the browser. We needed to build a file system, HTTP server, websockets support, modules, and the ability to fetch NPM modules. We had help from existing libraries in the Browserify ecosystem.

Hey, I'm Jasper. I'm a staff engineer at Codesoundbox, and I'll give a talk about how we built our Node.js compatible runtime for the browser called Nodebox.

So why do we actually build a Node.js compatible runtime for the browser? We wanted to build a Node.js compatible runtime because we wanted to allow our playground library, Sandpack, to run small projects. Like for example a small Next.js example, a feed example, or Next.pressjs application. Obviously for the purposes of documentation.

And so what did we need to build to actually make this possible? There's a lot of stuff that goes into Node.js. For example, we had to build a file system, an HTTP server and websockets support, we had to build modules, we had to be able to fetch NPM modules. We also had to make sure that we had child process support, as most libraries and frameworks heavily on that. And there's also a lot of more smaller libraries and other standard libraries in Node.js that we had to support to make all this work.

So we actually had a lot of help from existing libraries, because there's a lot of stuff out there from the Browserify ecosystem. For example, there's assert, and there's like zlib. There's also a buffer and events, or path string decoder, like URL utils. Readable stream by the Node team, which helps us build stream support. And then there's also like crypto, and a bunch of other ones that are not listed on this slide.

2. Overview of Nodebooks and Filesystem

Short description:

So a general overview of how Nodebooks actually works. We have a main manager that controls everything in our virtual environment. It spawns processes, mimicking actual Node processes. We also have previews, which act like an HTTP server. Our Node processes are web workers with their own contained process. We use web workers to initialize each worker and do the initializing work of building the file system tree, loading web assembly files, and waiting for it to finish. Once the worker is ready, we send it back to main and can run commands like Next.js. We wrap the module in our own global to make it believe it's running in a node environment. Our filesystem works with the main process having the whole state and workers having eventual consistency. When writing to the filesystem, the write is synced and sent to the main process to synchronize it across the entire application state.

So a general overview of how Nodebooks actually works. So we have our main manager, which controls everything in the sort of our virtual environment. And that spawns processes, which are actual Node processes, or try to be Node processes, they mimic it. And then we also have previews, which is sort of like your HTTP server, which goes through our preview manager, which we built, which then starts like iframes to mimic this HTTP server behavior. And our Node processes are actually web workers, which then also have their own contained process similar to how Node works.

And so how does a Node process work in our environment? So as I said before, we use web workers. To do this, we initialize each worker by sending a file system buffer and environment variables and a bunch of other small config options we have in Node box. Well, once this happens, the worker spins up and starts doing its initializing work, which is building the file system tree, loading some web assembly files which we use for example, for transpiling our code or some things like probably compression, which doesn't really exist in the browser at the time. And we also have things like waiting for it to to finish at the end. And then we go into actually loading the rest of the stuff.

So, once the worker is ready, we send it back to main. And once it's in main, main knows our worker's ready. And now, we can do like running a command. For example, running Next.js is as simple as passing in the JSON command next and then it spins up a whole Next.js server. It does this by going into our node modules and resolving the next binary which is actually just a JavaScript file in the end. So, we resolve that which ends up being like .bin slash next and then it has a sim link to slash node module slash next slash CLI.MJS or something. And then we run that actual script as like node, the resolve file and then which we pass as args to our module. And then we use that to evaluate the module. How we do that is by wrapping our module in our own like global which we built which contains most globals, node modules expect like the module, global require there's a couple other ES module stuff and then global this which is all different from the browser and we try to make it believe that it's running in a node environment by this instead of the actual browser. So we also set certain browser globals to undefined or null. And then we run it in a function. We wrap it in a function with our global arguments. So the code believes these are new globals instead of the actual globals that the browser has. And that we also do apply where we override the disks to be our global disks instead of the browser's disks. So it really believes it's inside of a node environment while it's still running inside a browser.

So how does our filesystem work? Our filesystem works in a way that our main process has the whole filesystem state and all our workers have eventual consistency. For example, here we have an example of how it writes to a filesystem. So in the module, you have importfs and then you call fs.write. That write then gets sent to our filesystem state which instantly syncs it inside its own state and then it sends a message to our worker bus to the main process to say I've written some file. Can you synchronize it across our entire state of the application? And then the main filesystem state also receives that, updates its internal state and then emits it to all other workers.

Check out more articles and videos

We constantly think of articles and videos that might spark Git people interest / skill us up or help building a stellar career

Full Stack Documentation
JSNation 2022JSNation 2022
28 min
Full Stack Documentation
Top Content
Interactive web-based tutorials have become a staple of front end frameworks, and it's easy to see why — developers love being able to try out new tools without the hassle of installing packages or cloning repos.But in the age of full stack meta-frameworks like Next, Remix and SvelteKit, these tutorials only go so far. In this talk, we'll look at how we on the Svelte team are using cutting edge web technology to rethink how we teach each other the tools of our trade.
Gateway to React: The React.dev Story
React Summit US 2023React Summit US 2023
32 min
Gateway to React: The React.dev Story
A behind the scenes look at the design and development of the all-new React docs at react.dev. The new react.dev launched this year introducing new methodologies like challenges and interactive sandboxes and subtle inclusivity features, like "international tone" and culturally agnostic examples. Not only have the new docs changed how people learn React, they've inspired how we think about developer education as a community. In this talk, you will learn how the React team and some ambitious community members made the "React docs rock" for a generation of front end developers and how these new patterns and established techniques can be applied in your favorite projects.
Opensource Documentation—Tales from React and React Native
React Finland 2021React Finland 2021
27 min
Opensource Documentation—Tales from React and React Native
Documentation is often your community's first point of contact with your project and their daily companion at work. So why is documentation the last thing that gets done, and how can we do it better? This talk shares how important documentation is for React and React Native and how you can invest in or contribute to making your favourite project's docs to build a thriving community
Documenting components with stories
React Finland 2021React Finland 2021
18 min
Documenting components with stories
Most documentation systems focus on text content of one form or another: WYSIWYG editors, markdown, code comments, and so forth. Storybook, the industry-standard component workshop, takes a very different approach, focusing instead on component examples, or stories.
In this demo, I will introduce an open format called Component Story Format (CSF).
I will show how CSF can be used used to create interactive docs in Storybook, including auto-generated DocsPage and freeform MDX documentation. Storybook Docs is a convenient way to build a living production design system.
I will then show how CSF stories can be used create novel forms of documentation, such as multiplayer collaborative docs, interactive design prototypes, and even behavioral documentation via tests.
Finally, I will present the current status and outline a roadmap of improvements that are on their way in the coming months.
TypeScript for Library Authors: Harnessing the Power of TypeScript for DX
TypeScript Congress 2022TypeScript Congress 2022
25 min
TypeScript for Library Authors: Harnessing the Power of TypeScript for DX
Using real-life open-source examples, we'll explore the power of TypeScript to improve your users' experience. We'll cover best practices for library authors, as well as tips and tricks for how to take a library to the next level. This talk will cover: 
- how to leverage the type inference to provide help to your users; - using types to reduce the need and complexity of your documentation - for example, using function overloads, string literal types, and helper (no-op) functions; - setting up testing to ensure your library works (and your types do too!) with tools like tsd and expect-type; - treating types as an API and reducing breaking changes whilst shipping enhancements; - I'd draw on my experience with libraries like nuxt3, sanity-typed-queries and typed-vuex and show what we managed to do and what I'd do differently in future. 

The Legendary Fountain of Truth: Componentize Your Documentation!
React Advanced Conference 2021React Advanced Conference 2021
24 min
The Legendary Fountain of Truth: Componentize Your Documentation!
"In Space, No One Can Hear You Scream." The same goes for your super-brand-new-revolutionary project: Documentation is the key to get people speaking about it.Building well-fitted documentation can be tricky. Having it updated each time you release a new feature had to be a challenging part of your adventure. We tried many things to prevent the gap between doc and code: code-generated documentation, live examples a-la-Storybook, REPL...It's time for a new era of documentation where people-oriented content lives along with code examples: this talk will guide you from Documentation Best Practices – covered from years of FOSS collaborative documentation – to the new fancy world of Components in Markdown: MDX, MDJS, MD Vite, and all.Let's build shiny documentation for brilliant people!