Opinionated Guide for Web Development with Rust Web Workers

I have a toy site that parses Rocket League replays using the rust crate boxcars. Parsing replays is done synchronously and shouldn’t block the browser’s UI thread, so parsing is offloaded to a web worker. Getting this side project to a point where it works + is maintainable (minimum configuration) has been an exercise in doggedness as I spent weeks exploring options.

I believe I’ve found a happy medium and here’s the recipe:

• Wasm-pack for converting rust to js + wasm
• Typescript. Refrain from anything special that can’t be compiled (eg: no css modules or importing images). jsx / tsx is fine. No need for babel
• Use a highly trimmed webpack config to output fingerprinted files
• Hugo to glue everything together and preprocess sass files and other assets

Here’s the philosophy: By minimizing configuration and dependencies while maintaining production quality, our project becomes easier to maintain

To give a sneak peek, the entire pipeline can be executed as

wasm-pack build crate && tsc && mkdir -p data && webpack --mode production && hugo

or an npm run build.

Other than wasm-pack everything else has basically been derived from process of elimination, so let’s go through this recipe in detail.

Sample Code

Here is some sample code to demonstrate the core interaction between our UI and web assembly. First we need a web worker so that one can pass messages between the UI and worker thread.

src/index.tsx

// unused, but an example node_modules import
import { render, h } from "preact";

const replayWorker = new Worker("worker.js", { type: "module" });
replayWorker.postMessage("LOAD");

While { type: "module" } is a valid option for a worker, no browser supports it, so typescript will leave it in without complaint and have webpack split the worker into a separate bundle.

“worker.js” is not a typo even though it is written in typescript as typescript will convert the code to js before they are passed to the webpack phase.

src/worker.ts

In our worker code we load the web assembly.

let parser: ReplayParser | null = null;
onmessage = async (e) => {
  switch (e.data) {
    case "LOAD":
      const module = await import("../crate/pkg/rl_wasm");
      parser = new ReplayParser(module); 
      break;
  }
}

src/ReplayParser.ts

And we can use a neat typescript trick in ReplayParser to take advantage of the typescript declarations written by wasm-pack to ensure that our javascript is adhering to the wasm return types.

import * as rl_mod from "../crate/pkg/rl_wasm";
type RLMod = typeof rl_mod;
export class ReplayParser {
  mod: RLMod;
  constructor(mod: RLMod) {
    this.mod = mod;
  }
}

Configuration Files

I hate configuration files. I’ve tried to whittle them down, but they are necessary so let’s get them out of the way

tsconfig.json (remove comments if copying + pasting)

{
  "compilerOptions": {
    "strict": true,              // strict code quality
    "module": "esnext",          // allow for dynamic import of wasm bundle
    "moduleResolution": "node",  // allow imports of npm dependnecies
    "target": "es2018",          // Allow modern features as we're using wasm anyways
    "jsx": "react",              // I like preact so these options are for writing tsx
    "jsxFactory": "h",
    "outDir": "./dist/"          // output directory for webpack to consume
  },
  "include": ["src/**/*"]
}

The typescript phase can be ran with npx tsc.

webpack.json

const path = require("path");
const WorkerPlugin = require("worker-plugin");
const AssetsPlugin = require("assets-webpack-plugin");

module.exports = {
  entry: "./dist/index.js", // the output of our typescript phase
  devtool: "source-map",
  plugins: [
    new WorkerPlugin(),
    new AssetsPlugin({ filename: "data/webpack.json" }),
  ],
  output: {
    filename: "[name].[contenthash].js",
    path: path.join(__dirname, "static", "js"),
    publicPath: "/js/",
  },
};

There are only two webpack plugins needed to accomplish this:

• worker-plugin so that webpack correctly splits the worker script into a fingerprinted file that also can correctly import a fingerprinted web assembly module
• assets-webpack-plugin which will output a file that contains a map of original js filenames to the fingerprinted ones for the downstream hugo process.

That’s it for webpack. This all can be accomplished with one of the shortest webpack config I’ve ever seen:

Couple things to note:

• The source maps are distributed into production. I have no problem with this and neither should you.
• The source maps will be of the compiled javascript (not original typescript source). Since typescript generates modern javascript that is quite close to the original typescript, one shouldn’t have a problem stepping through the code.

The webpack phase can be ran with npx webpack --mode production.

Directory structure

The project directory should be setup with these folders:

• assets: owned by hugo. Place files that will be processed by hugo here (images, sass, etc)
• crate: our rust crate that wasm-pack compiles to wasm
• cypress: (optional) used for end to end integration testing.
• data: owned by hugo but generated by the build process. We configure webpack to output a webpack.json here which contains a mapping of original source file names to fingerprinted ones.
• dev: (optional) misellaneous files (eg: nginx.conf used for static hosting) or files used in multiple places to be symlinked.
• layouts: owned by hugo. Place an index.html in this directory.
• src: typescript code
• static: owned by hugo. Where one sticks static content, so this is where we’ll put the resulting js bundles written by webpack as well as any static images
• tests: unit tests written with jest

Hugo

Let’s take a look at the most unorthodox piece of this puzzle – using the static site generator hugo (specifically hugo extended). Over time I’ve grown more sure of this decision of using hugo for a few reasons:

• Hugo’s philosophy of convention over configuration means one can introduce it with very little configuration.
• Hugo is distributed as a single binary for all platforms
• Hugo (extended) has a built in Sass preprocessor, so no additional dependencies or configuration for css processing
• Hugo can fingerprint assets (which ruled out the Rust alternative, zola)
• Hugo can work hand in hand with webpack through through the data/webpack.json file generated by assets-webpack-plugin so hugo can insert the fingerprinted js links
• A static site generator is purpose made to glue everything together to make a cohesive site through a built in templating language, which unlike webpack needs plugin after plugin and tons of configuration to try and replicate.

Aside: the reason why we need webpack to fingerprint the js (instead of having hugo do it) is that webpack also spits out multiple fingerprinted js files even with a single entry point (eg: web worker, wasm, extracted vendor dependencies)

Those familiar with the hugo ecosystem, may recognize this as a heavily simplified victor-hugo setup. In my opinion this guide is better due to it’s simplicity and being opinionated can drastically decrease the cognitive overhead. That is the goal here. Supporting everything under the sun leads to adding too much magic or configuration.

Yes it seems like overkill to introduce a static site generator for this project, but I believe it is the right approach. There is no magic glue between any of the components. Nothing really to go wrong.

I see people complain about the complexity of hugo and all the ceremony it takes to set it up but I only had to move around some files (see directory structure section described above) and add a single config.toml file. Brace yourself, it’s long:

disableKinds = ["taxonomy", "taxonomyTerm", "RSS", "sitemap"]

Even this isn’t required, but for a simple project I don’t want all those extra files generated.

To import our main javascript into app:

<script type="text/javascript" src={{ "{{ .Site.Data.webpack.main.js | relURL "}}}}></script>

Hugo fits nicely into the equation. While hearing that hugo is a static site generator may scare of SPA enthusiasts, don’t be, this works well for a SPA setup.

JS Bundlers

I’ve tried to love the parcel bundler. It’s zero configuration setup works well most of the time, and while it still has a place for standard projects that need to hit the ground running – I’ve grown disillusioned with it for a Rust WebAssembly purposes:

• Bizarre, baroque, and hard to reproduce errors
• Incorrect cache busting causing stale files to break the production web app (I’ve had to reset to a poor person’s implementation of an asset pipeline: sha256 + sed)
• And a whole host of other problems revolving around rust.
• That specifying typescript compilation target means nothing without a browserlist.

Such began the search for other bundlers. I dabbled in the “no bundler” snowpack, but it’s reliance on etags instead of fingerprinting assets relects poorly in lighthouse performance audits (though this may be changing). Also the potential for loading each source file as a separate request is frightening. I need fingerprinting.

Webpack is a natural replacement for parcel, except I have a big gripe with webpack: it’s configuration. I’m sure if you polled developers how they feel about webpack configuration it’d be one word: magic. I find that they are write once and cross fingers it’s perfect for eternity. I’m not ok with trading one type of magic (parcel’s one size fits all) for another (webpack’s unintelligible config).

It seems everyday there is a new article about how to configure webpack (the irony of me writing an article on the same topic isn’t lost on me). Here are some top results for webpack “How to configure Webpack 4 from scratch”, “A tale of Webpack 4 and how finally configure it”, “An annotated webpack 4 config” – these articles contain bloated dependencies and configs. The last example in particular tops out at 60 dependencies and a 130 line config just to build the project. It can feel like webpack is a hammer and everything is a nail. No you don’t need hot module replacement (HMR), css modules, mdx, babel, a separate dev / prod config. Keep it simple.

There are a ton of webpack intermediaries that claim a zero or simplified config: create-react-app, nwb, neutrino, poi but they all fell short and left a sour taste in my mouth as these intermediaries only service their use case and rust + webassembly isn’t in it. I spent way more time trying to simplify a webpack setup through these tools than if I had just written it myself.

Ts-loader

One can consolidate the typescript then webpack invocations into the webpack config if they want to add a dash of complexity to their webpack config and add the ts-loader dependency. Personally, I don’t see the need for this consolidation as the “simplest setup” is still far too long to be a replacement for a single tsc command. Here is what our config would look like if we added in ts-loader

Not recommended

const path = require("path");
const WorkerPlugin = require("worker-plugin");
const AssetsPlugin = require("assets-webpack-plugin");

module.exports = {
  entry: "./src/index.tsx",
  devtool: "source-map",
  module: {
    rules: [
      {
        test: /\.tsx?$/,
        loader: "ts-loader",
      },
    ],
  },
  resolve: {
    extensions: [".ts", ".tsx", ".js"],
  },
  plugins: [
    new WorkerPlugin(),
    new AssetsPlugin({ filename: "data/webpack.json" }),
  ],
  output: {
    filename: "[name].[contenthash].js",
    path: path.join(__dirname, "static", "js"),
    publicPath: "/js/",
  },
};

The benefits of ts-loader:

• Sourcemaps can now contain the original typescript code (may need to update tsconfig.json to enable source maps)

If it had been new TsLoaderPlugin() then it could maybe be worth considering, but the extra configuration seems a bit much.

Typescript

I write my projects in Typescript because I like static types.

SASS

Sass is not my first choice for a css preprocessor. I had gotten quite fond of using postcss in other projects, but I was not fond of adding the necessary configuration + multiple dependencies. If Hugo had an out of the box experience with postcss, I would have gone with that, but it doesn’t – it requires configuration and for the postcss dependency installed out of band.

So sass it is. At first I was grimacing at the thought of the work converting the postcss dialect chosen to scss, but it turns out that browsers have really good support for modern css features so there was actually a minimum amount of conversion necessary.

Sass usage in hugo is quite pleasant. I define an variable to hold a dictionary of options dictating we want to have includes from “node_modules” and that we want the result css compressed

{{ '{{ $opts := (dict "outputStyle" "compressed" "includePaths" (slice "node_modules")) '}}}}

Critical css can have it’s own stylesheet and inlined

{{ '{{ $critical := resources.Get "critical.scss" | toCSS $opts '}}}}
<style>{{ '{{$critical.Content | safeCSS '}}}}</style>

With the rest of the app’s styles coming later.

{{ '{{ $app_css := resources.Get "app.scss" | toCSS $opts | fingerprint '}}}}
<link rel="stylesheet" type="text/css" href="{{"{{ $app_css.RelPermalink "}}}}" />

One can also create a stylesheet that vendors all 3rd party style sheets so small app css updates don’t trigger a download of potentially heavy 3rd party stylesheet.

Testing

Cypress is used for end to end tests against the actual site. These tests confirm ability of the WebAssembly and js to function against real world input. Integration tests allows the project to skip component testing in favor of real life interactions. With no component tests, a project is free to swap out the internals (ie: move to a different framework or eschew all of them) and not invalidate any tests (easy maintenance). There are still unit tests, but only those that don’t use the DOM.

I’m not too opinionated with unit tests. Ideally js would have a built in test runner kinda like rust, but for now jest with ts-jest is fine (ts-jest necessary until jest natively supports es6 modules). Removing jest from the dependencies shrunk a package-lock.json from 12k to 6k lines, which is incredible, so there is still a lot of room left for improvement for unit testing.

FAQ

How to handle cross platform contributors

The command posted earlier:

wasm-pack build crate && tsc && mkdir -p data && webpack --mode production && hugo

Is not cross platform and leaves Window users out in the cold. This can be ok if this project of yours is for yourself and you don’t use windows. Don’t sweat the details. With WSL so popular these days, it’s not even a problem.

If native windows development is absolutely necessary, one can write cross platform node js scripts.

Development & Livereload

Rust and typescript aren’t known for their quick compilation times and these lags in response can be felt as even the tiniest change would necessitate an entire rebuild with npm run build. It would be nice when a typescript file changed to not need to execute the rust toolchain. When I edit frontend code, I like to see the browser refreshed quickly with my changes without effort on my part.

Fortunately, 3 out of the 4 tools used so far natively support a watch mode and they meld together nicely. Below is a script I have that is aliased to npm start

#!/bin/bash

npm run build
npx tsc --watch --preserveWatchOutput &
npx webpack --mode production --progress --watch &
hugo serve --renderToDisk

Modifying a typescript file causes a quick site reload. Perfect for quick iteration. Even though I exclusively use webpack in production mode, I’ve not found watch recompilation times to be an issue.

The one tool that doesn’t support a watch mode is wasm-pack, but it is a known issue with workarounds using fswatch, cargo-watch, or entr, etc.

while true; do
  ls -R -d crate/Cargo.toml crate/Cargo.lock crate/src/ \
    | entr -d wasm-pack build crate;
done

In practice, I have not needed to use the above workaround as it’s been intuitive enough for me to execute wasm-pack build after any rust modifications. After wasm-pack is done executing, webpack sees the wasm was modified and automatically picks it up and a new site is generated.

How to deploy

I prefer deploying sites by packaging them up into docker images. No exception here, just ensure that no generated files are copied into the dockerfile to aid reproducibility.

If instead you are uploading the public/ folder directly someplace, it’s probably a good to clean out the webpack generated static/js on every build with the clean-webpack-plugin. Since I don’t consider this essential, I’ve excluded this from posted webpack config. And when the additional configuration is approximately the same size of the equivalent command (rm -rf ./static/js), the configuration is preferred. Anyways, here is the config if one wants to clean directory on every build.

const path = require("path");
const WorkerPlugin = require("worker-plugin");
const AssetsPlugin = require("assets-webpack-plugin");
const { CleanWebpackPlugin } = require('clean-webpack-plugin');

module.exports = {
  entry: "./dist/index.js", // the output of our typescript phase
  devtool: "source-map",
  plugins: [
    new CleanWebpackPlugin(),
    new WorkerPlugin(),
    new AssetsPlugin({ filename: "data/webpack.json" }),
  ],
  output: {
    filename: "[name].[contenthash].js",
    path: path.join(__dirname, "static", "js"),
    publicPath: "/js/",
  },
};

Splitting Vendored JS

In the event that you are embedding heavy dependencies that don’t change at nearly the same rate as your application, one can split the dependencies out in order to ease the amount of data that users need to pull down when one deploys their app. One can do this with a optimization.splitChunks

  optimization: { splitChunks: { chunks: "all" } },

That’s it. Don’t get carried away by all the options. Full config shown below:

const path = require("path");
const WorkerPlugin = require("worker-plugin");
const AssetsPlugin = require("assets-webpack-plugin");

module.exports = {
  entry: "./dist/index.js",
  devtool: "source-map",
  plugins: [
    new WorkerPlugin(),
    new AssetsPlugin({ filename: "data/webpack.json" }),
  ],
  output: {
    filename: "[name].[contenthash].js",
    path: path.join(__dirname, "static", "js"),
    publicPath: "/js/",
  },
  optimization: { splitChunks: { chunks: "all" } },
};

Again, only use this optimization if dependencies significantly outweigh the app and one deploys often. I have a preact app where I don’t use this trick as there really isn’t any dependency baggage so there isn’t a reason to split dependencies off. Other times it’s a fat react + antd where I don’t want my users to download these again if I simply fix a typo in the app.

Proxy backend requests

When developing a frontend, it is beneficial to interact with the backend. While this can vary greatly from project to project, for my projects I’m able to reuse everything that is used for production. I have a docker container for housing the API and containers for S3 (minio), postgres, and redis. All of these are orchestrated by docker compose.

And since I’m already using nginx to host the frontend assets, I can simply bind mount /usr/share/nginx/html to the ./public directory to serve all the latest assets as they change. This is why the start.sh script shown earlier has hugo use --renderToDisk.

The last thing I do is have a traefik container which is the entrypoint for all requests route /api requests to the api server and / to the nginx server.

I’m quite satisfied with the result:

• Purpose built tools watching their domain for any changes
• Re-using production containers for development

Linting / Formatting / Other tools

For formatting, run prettier without installing as a dev dependency. Pinning a formatter to a specific version seems a tad asinine, as one should always be using the latest version. Just like how I don’t customize rustfmt through rustfmt.toml, don’t customize prettier, as it can be confusing where configuration is located (eg package.json, .prettierrc, prettier.config.js) and serves only to pollute the project directory.

I eschew linting in frontend projects. It’s a bit of a surprise considering how much I love clippy for rust projects, but I don’t have any love for eslint. Every guide on how to get started with eslint, typescript, prettier, react, etc has one installing a dozen linting dependencies and crafting a configuration that can reach hundreds of lines. This is not ok. My preference would be for an opinionated, self-contained tool that does not need to be added as a dev dependency to run lints. Rome can’t come fast enough. Until then, typescript type checking + prettier get me most of the way there.

As for other tools – when in doubt don’t use them:

• You don’t need css modules when you can write regular sass files or attach styles directly to the component
• You don’t need to Extract & Inline Critical-path CSS in HTML pages: split up your sass stylesheets and have hugo inline the style in the html.
• You don’t need to Remove unused CSS when you import stylesheets of just the components you need
• You don’t need dynamically loaded images fingerprinted (this is where etags do come in handy)
• You don’t need it. It’s not worth maintaining an extra dependency + configuration

CSS in JS

Thus far, I’ve been mainly advocating for using a separate sass stylesheet. And I believe that one should continue using a separate stylesheet. However I recognize arguments of the shortcomings of css. So with that said, I have some thoughts if one finds the use of css in js a necessity:

• Don’t use styled-components, as one “will have to do a little bit of configuration” to use with Typescript.
• Consider avoiding TypeStyle even though it is a css in js library that has a deep integration with Typescript, as developer mind share will be less due to the library targeting a specific language and the slow pace of development.
• styled-jsx requires configuration and doesn’t play nice with typescript.

The one css in js library that checked most the boxes is framework agnostic version of emotion mainly due to “no additional setup, babel plugin, or other config changes [required].” I can drop it into any project and expect it to work. The same can’t be said the @emotion/core package which needs the magic comment /** @jsx jsx */ at the head of the file in order to work (and was unable to witness the css typechecking promised by the @emotion/core package).

There is no pressure to jump on the css in js bandwagon. I had a pure preact project, migrated to css in js with emotion, and then migrated back to standalone stylesheet as the styles in that project hadn’t grown unruly enough that I felt the need for a css in js solution. Emotion is relatively lightweight, but for very lightweight sites it’s effect on the bundle size can be felt.

In Conclusion

It may seem like I’m bending over backwards to find pieces that fit the web development puzzle, but I’m left quite satisfied. Every tool chosen (wasm-pack, typescript, webpack, hugo, emotion (optional)) are industry standard and no one can say they are an obscure or bad choice. I like to lay out the investigative process, which means weeding out the tools that don’t work.

Some parting words of advice:

• Keep the project simple
• You don’t need that configuration
• You don’t need that dependency
• Use purpose built tools for the job

Yes there will be those who clearly shouldn’t follow this recipe, but I needed to reduce my overhead as a I switched between projects and reducing magic through dependencies and configuration has proven valuable.

Error: Webpack Compilation Error
Module parse failed: Unexpected token (1343:22)
You may need an appropriate loader to handle this file type, currently no loaders are configured to process this file. See https://webpack.js.org/concepts#loaders
| async function init(input) {
|     if (typeof input === 'undefined') {
>         input = import.meta.url.replace(/\.js$/, '_bg.wasm');
|     }
|     const imports = {};
@ ./src/lib/tests/sum.spec.ts 1:0-79 9:34-51 10:29-42 13:28-40 14:27-50