This talk will teach you how to write performance-critical Node.js modules without the burden of distributing platform-dependent artifacts and using the C/C++ toolchain. You will discover how to smoothly integrate Rust code into your Node.js + TypeScript application using WebAssembly. You will also learn how to avoid the typical WebAssembly serialization issues, and understand when other alternatives like Neon or Napi.rs are preferable. Together, we will cross the language bridge between Rust and Node.js while preserving the familiar DX you're used to.
Type-safe bindings for Node.js with Rust and WebAssembly
AI Generated Video Summary
This is TypeScript Bindings for NodeJS with Rust and WebAssembly. We'll be looking for an alternative, easier approach for creating native NodeJS modules, while also automatically generating types for them.
Hi, everyone, and thanks to Node Congress for having me. This is TypeScript Bindings for NodeJS with Rust and WebAssembly. We'll be looking for an alternative, easier approach for creating native NodeJS modules, while also automatically generating types for them.
A little bit about me. I'm Alberto Schibel. I'm from Venice, Italy. I'm a software engineer at Prisma where I ported several Rust modules to WebAssembly. I'm also a consultant working with NodeJS, TypeScript, and Rust. You can find me online at j.com.io. You can also find the slides for this talk on my GitHub page.
2. WebAssembly and Rust for TypeScript Modules
WebAssembly, or WASM, is a low-level abstraction for the CPU your code is running on. It's a fast, compact bytecode designed for near-native speed and optimized for fast start-up time and small memory footprint. It's a portable compilation target for many languages, including Rust. Rust is consistently voted as the best language for WebAssembly. Let's see how we can create TypeScript modules from it.
So, elephant in the room, what is WebAssembly and how's that useful? Well, WebAssembly, or WASM, is basically a low-level abstraction for the CPU your code is running on. It's fast, compact bytecode in the sense that it's a portable binary format for a virtual machine that models loads and stores of numbers inside linear memory. It's designed for near-native speed and is also optimized for fast start-up time and small memory footprint. It was created by browser vendors to port C++ code to the web without performance degradation and now it's a portable compilation target for many languages including Rust, Go and many others.
That means you can compile your code to WASM once and then you can run the same compiled artifact on different platforms. For instance Node.js supports WebAssembly since version 8 and now you can import a WASM module exactly like you would import a standard npm package. And why is a single portable compilation target useful? Consider Node.js, a popular NAPI add-on that compiles SAS styles to CSS. To support multiple system configurations, architectures and even Node.js versions this library needs to be compiled separately for each of these configurations. This means 35 different compilation targets and it makes every new deployment a time and resource consuming task. Those of you who use Prisma will probably know that we're on a similar situation with TypeScript CLI and a library that downloads some compiled Rust binaries on demand. This is what made us consider WebAssembly and adopted as much as we to simplify our deployment process. And if you ever tried to write native Node.js add-ons yourself, you probably know that it's not a straightforward process. Sometimes, no jib fails with cryptic error messages and, frankly, the tooling necessary to build and import C++ modules isn't as human friendly as what Node.js developers are accustomed to. So, this is perhaps one of the major reasons why Rust is consistently voted as the best language for WebAssembly.
So, let's see how we can create the TypeScript modules from it. And for those of you new to Rust, let's define some baseline glossary, right? So, anything you use a package of json for, you would put in a cargo.toml file in a Rust project. What you usually call npm packages are crates in Rust and you operate on them via the cargo CLI. For instance, compiling Rust code, you would use the cargo build command and specify the completion target. In our case, it's Wasm 32 unknown, unknown. It uses a 32-bit addressing space and isn't tied to any particular OS vendor or CPU architecture. If we want to move around more than purely numeric data across the WebAssembly bridge, we're going to need a binding tool. It's both a CLI and Rust library. When you install it, you should specify a particular version, because it doesn't yet follow semantic versioning. This means that the version you specify in your cargo.toml file should match the version of Wasm you have installed on your machine. Moreover, to support WebAssembly, you need to mark your create type as c.lib. It will tell Rust to compile your code as a dynamic library that can be loaded by a C-compatible runtime, like Node.js. So, compiling Rust WebAssembly is a two-step process. First, you need to run cargo.build to create a compiled WebAssembly artifact, which will have the dot wasm extension, and then you will add wasm.bindgen to generate the Node.js and TypeScript bindings you will use to import the compiled wasm module. Of course, if wasm.bindgen supported all the commonly used Rust data structures and TypeScript conventions, this talk will already be over, and clearly that's not the case. So let's see how we can work around this.
3. Rust Functions and WebAssembly
Let's define Rust functions that take a number, duplicate it, and return it. Rust syntax can be overwhelming, but we can import the wasm.bindgen library and generate bindings for the functions. We have functions for 64-bit integers and floating point numbers. The generated TypeScript declaration preserves function names and maps number types accordingly.
So, for our first example, let's first see how we can define Rust functions that take a number, duplicate it and return it to the caller. I know that Rust syntax can be overwhelming, so bear with me. We first import the wasm.bindgen library. We then tell Rust to generate bindings for the function that follows, which will be compiled to WebAssembly. And whenever you see a code with a hashtag and a square bracket, that means it's a Rust macro, a special kind of function that expands to generated code at compile time. We define a public function, duplicate underscore U64 that takes an assigned 64-bit integer, multiplies it by two, and returns it. The other function is similar, but it uses floating point numbers instead. And wasm.bindgen generates the following type script declaration for us that you see at the bottom. We see that function names are preserved as is. U64 numbers are mapped to begins. And F32 numbers are plain numbers in TypeScript because it doesn't really have a dedicated floating point type.
4. String Conversion Functions
5. Using Functions in TypeScript
What happens when we try to use these functions in TypeScript? If we pass compatible types, they work as expected. But if we escape TypeScript validations and pass incompatible types, we'll get runtime errors. Complex data structures like structs have unexpected behavior when generating TypeScript bindings. The generated code leaks internal details and lacks a constructor. Wrapping strings in a struct causes compilation errors. Enums work one-to-one with TypeScript enums, but they can be problematic. Discriminated unions, or target unions, are not supported by WasBindgen.
What happens when we try to use these functions in TypeScript? Well, if we pass types compatible with the TypeScript declarations, they work as expected at runtime. But if we escape from the TypeScript validations by disguising a string as a big int, and we call n to string function with that? Well, in that case we'll get a runtime syntax error because the function expects a number but it's being called with a string.
What if we need more complex data structures? Here's an example with our scholars struct which wraps values like numbers, characters, or booleans. Say we want a function that extracts the value of one of the fields namely the letter. If we were to manually write the typescript bindings for this we will define scalars as a typed dictionary which we call construct in place and we type the letter field as a string because typescript doesn't really distinguish between single character and multi-character strings. However, this is not what WasBindsNGenerates and this is what it gets, what it creates and although the four struct members have the types we expect, we actually get a scalar class definition, not a dictionary type. Moreover, we see some internal details that are leaking out to the generated code and namely that's the free method which doesn't take any argument and doesn't return any value. This is not something we wrote in our Rast type. This is something that WasBindsNGenerates.
Finally, WasBindgen provides partial support for homogeneous vectors, but only for numeric types, which are translated to typed array instances. And they are essentially only useful when manipulating raw binary data. They are quite far from the standard general-purpose arrays we usually want. Also, vectors of non-primitive types, nested vectors, or tuples are not supported at all.
7. Seamless Integration with tspy
We discovered tspy, a magical tool that generates type-safe and ergonomic bindings for seamlessly integrating Rust functions into TypeScript apps with WebAssembly. It eliminates the need for manual casting and provides strong late type bindings. We'll demonstrate its usage with Enum variants and the serialization process. WebAssembly is ideal for CPU-intensive tasks and complex logic, but lacks input-output support. TSFI is the best solution for type-safe bindings, although it heavily relies on macro magic. Generic containers like vectors and hash maps require specifying the generic type and wrapping them in a struct or enum variant. Check out the example of TSFI usage in Lira, a TypeScript-based full-text search engine with WebAssembly support.
So it's typed as any in TypeScript and it's not really that useful. So we started this journey in an effort to seamlessly integrate Rust functions for data structures into TypeScript apps with WebAssembly. And it looks like we should give up. Unless maybe there is a magical tool that could help us by generating type safety and ergonomic bindings.