Keeping up with the fronted Joneses: streaming server side rendering

Published on:

Table of Contents

React server components. Streaming server side rendering. Edge compute. Web app architecture advancements are making classic SPAs appear long in the tooth.

Time to see what I’ve been missing, and try to keep up with the Joneses (negative connotation fully implied).

I might suffer from Perpetual Reengineering Syndrome as this is the umpteenth time a 100k LOC web app side project has been rearchitected / replatformed. Am I a perfectionist or just have an insatiable urge to tinker?

Whatever the answer, I love writing about it.

The Joneses might be onto something, after everything was said and done, the results were quite impressive:

  • Eliminated layout shift. A server rendered page can use session information to determine if the user is logged in and serve the appropriate layout, all without a significant regression to time to first byte.
  • Improved LCP as the database query can be kicked off at request time, an improvement from sending the request after the client has been hydrated.
  • Infrastructure consolidation: no more need for a split deployment where certain routes run on a VPS while other run on a hosting provider
  • Migration away from Next.js resolved edge runtime build issues, unlocking much needed SEO.
  • Database queries are cached

What does all this mean? Read on to learn more, but first it’s illustrative to see how the app has evolved over time.

A history lesson

The first iteration in 2020 had the backend written in Rust with warp and diesel, as the engine that powered the business logic was also in Rust. The frontend embedded this same engine via WebAssembly. I felt vindicated with my choice of Rust. Users could process files locally without uploading, and if they did want to share, the backend would validate and parse out information to persist in a database.

Even in this first iteration I knew the user base was going to be globally distributed, so I opted for (the now deprecated) Cloudflare Workers Sites to serve static assets and cache S3 file requests. But other than this, the site was a classic React SPA with API calls, as I wanted to necessitate only static assets for local analysis, so the backend could be on fire, but only a subset of users would care.

Enter Next.js (2022)

After 18 months with this setup, I found myself laboring too hard on the Rust backend, a glorified CRUD app. The ecosystem was (and probably still is) too young, too low level, and requires too many dependencies to wire up. It was hurting iteration speed and feature development.

I decided to rewrite the Rust backend into Next.js and embed the engine via NAPI-RS. The result was improved type safety across the network, much fewer dependencies, and less code.

The greatest downside was increased complexity when it came to provisioning a Next.js standalone deployment. I know Next.js recently buffed up their support with a self-hosting tutorial, but it’s always been a bit of a 2nd class citizen to their Vercel hosting.

In this iteration, I kept Cloudflare Workers Sites and classic React SPA architecture, but I was able to leverage Next.js page pre-rendering so Cloudflare could serve up fully rendered HTML instead of a shell.

Enter microservices and Vercel (2023)

The server running the API needed to be over provisioned. When a user uploaded compressed large files, up to 1GB would be needed to process this file. There’s no way around it, the operation is inherently expensive. A spike in uploads would hose the server.

Microservices to the rescue. By splitting the file processing from the CRUD API, they could scale independently. I surveyed the possible options and landed on Google Cloud Run. I wrapped the Rust engine in the smallest of axum servers, and haven’t looked back. I tend to be wary of microservices, but this move has been a slam dunk.

By this time, Cloudflare Workers Sites had been deprecated, and I got the feeling using it in front of Next.js was only complicating the architecture, as Next.js has the concept of running functions on the edge baked in. Vercel is the natural place to deploy Next.js to take advantage of all its features. But I hit a snag, as Vercel has a 4.5MB body size limit on Vercel serverless functions. That sank any hopes of hosting a file upload endpoint on Vercel.

I’ve thought about this problem a lot. Too many people would jump to something like presigned S3 uploads and event notifications. This would result in:

  • Worse user experience. That event notification can take a minute or more to be delivered
  • An increase in the number of moving infrastructure pieces
  • Vendor lock-in, though recently more S3 providers are supporting event notifications, I don’t believe they have been standardized.

I decided on a different approach. I enacted what I call “split deployments”, where the non-troublesome endpoints run on Vercel, with the rest proxied to an origin server.

The result was a mixed bag. While Next.js became the single source of truth, the simplification goal didn’t really pan out, as the API is still split amongst two providers. On the bright side, it became easier to share code and run more on the edge. Previously, logged in user requests had to hit the origin server, but now their presence is detected at the edge.

Enter next on pages (Q3 2024)

As the site grew, it started to encroach on Vercel usage limits, which were opaque enough that I could not understand actions I could take to mitigate usage.

In my investigation of Cloudflare’s Next on Pages, it was so much faster and cheaper than Vercel that I was compelled to adopt it. The speed improvement is likely due to Vercel soft-deprecating the edge runtime in a tweet:

So, @vercel reverted all edge rendering back to Node.js 😬

And Theo provided further commentary:

Vercel is no longer recommending shipping your Next apps to edge, and they are no longer doing it themselves.

The argument against edge deployments is that any more than a single database call will likely result in worse performance than a single server next to the database due to latency from ephemeral connections and repeated round trips to and from the database.

Understandable.

Edge compute isn’t for all use cases. Those that have optimized the number of SQL statements to reduce the number of roundtrips required, should be fine. I don’t understand the zeitgeist shift away from edge compute. It was never supposed to be a silver bullet.

The deprecation of the edge runtime is probably why Next.js bundling issues that involve the edge go unanswered. These issues prevent me from rolling out features and is quite unfortunate.

Thankfully, Cloudflare edge stays true to its name and I saw nearly a 10x latency reduction with Cloudflare.

However, this gem has rough edges:

Time for something drastic

Fast forward to the present day; there has been a flurry of activity in JS fullstack hosting.

My mind can’t help wandering – could this be a sign that I could consolidate everything into a single deployment?

This sign was cemented when a new Next on Pages deployment was met with Cloudflare errors of “no nodejs_compat compatibility flag set”. I can see the setting is correct, so I don’t know what the issue is. Luckily, Cloudflare allows previous deployment rollbacks, so the website is up – just no new features. I reported the error to the Cloudflare discord to see if I was missing something, but silence.

In truth, I could try harder to triage the issue, but I see the writing on the door: it does not make sense for Cloudflare to support both OpenNext and Next on Pages. I’d bet that Cloudflare’s long term bet is on OpenNext, as it works without the edge runtime, which we have already seen is being de-emphasized by Vercel.

OpenNext on Cloudflare sounds great. One gets the benefits of edge deployments without the edge runtime issues. The problem is that OpenNext does not support the Pages Router, nor does it look likely. I could migrate to App Router, but I don’t want React server components. I have an immersive and interactive web app, and Next.js forcing the "use client" directive everywhere rubs me the wrong way and I think they’ve made the wrong UX/DX decision.

That’s what made Tanstack Start so appealing, it’s a SPA-first framework. When I say “SPA”, it’s out of three options (SPA, server components, and Stateful servers) that Ryan Carniato outlines in the video “Are There Actually That Many Different Ways to Build Web Apps?”. So server side rendered React is considered part of the “SPA” category.

Time to stream

After porting 90% of the app to Tanstack Start, I hit some roadblocks and rough edges that will no doubt be ironed out in the upcoming months. I ended up abandoning the port, but I was able to play with server side rendering and streaming, as if I needed to rework every route due to switching frameworks, I might as well try this paradigm out.

Following the history, the app has always been a classic SPA where data fetches occur after the client side javascript is fetched, parsed, and executed. With server side rendering, no longer will there be a flash of skeletons as the app determines the appropriate layout for a given session. And streaming cuts down on LCP latency, as route data can be fed to the client as static assets are applied to the document.

Three downsides come to mind with the adoption of server side rendering:

  • The double data problem. The data is duplicated once in the rendered HTML and once again as embedded JSON to assist hydration.
  • Requirement of a healthy server. Previously the web app could limp quite far on only static assets. This downside matters more if the server is on a VPS where an errant sysadmin (read: me) could accidentally bring down the site. Things look different for serverless deployments, though the risk more likely shifts to misconfiguration.
  • If an OAuth2 / authentication callback wants to set a SameSite=strict cookie, instead of emitting an HTTP redirect, one needs to return a client side redirect.

Streaming performance

Let’s talk actual performance numbers. I deployed the web app onto Cloudflare twice, one classic SPA and the other server side rendered. I measured time to first byte (TTFB), layout finalization (LF), and largest contentful paint (LCP) and took the average of 5 runs.

SPA (ms) SSR (ms) Δ (ms) %
TTFB 48 62 14 30%
LF* 300 62 -238 -79%
LCP 601 435 -166 -28%

*LF: Layout finalization: a web vital of my own design. Measures the latency from the initial page load until the page layout is fully stable and finalized. Informally, it’s the length of time where the user sees layout skeletons. It’s analogous to Cumulative Layout Shift (CLS), but along a time axis, landing sometime after the first contentful paint.

The delta direction confirms my intuition: we should expect server rendering to have a worse time to first byte, as it needs to do more work to send a response. A 30% increase for TTFB sounds impactful, but at 14ms of difference, it will be imperceptible. I know Vercel is championing Partial Prerendering, where the shell is prerendered and served from a CDN, with data streamed in. My understanding is this achieves the same TTFB of a static site, but from these numbers, I don’t see how this scheme will achieve any meaningful improvement over plain server side rendering at the edge.

Then there is layout finalization where server rendering crushed the classic SPA. SSR is able to deliver the logged in UI without needing to fetch, parse, and execute Javascript in order to queue a data fetch (at time 250ms). A nearly 80% decrease in latency is incredible.

Finally, comes the largest contentful paint where streaming comes into play. The server side implementation kicked off a database query as soon as the request was received whereas the SPA was once again subjected to queuing a separate data fetch after the client side Javascript is executed. And these facts are reflected in the numbers with a 165ms improvement in LCP. Not bad! The database is on the west coast and I’m in Chicago.

Enter Remix

With Next.js a no-go and Tanstack Start too nascent, Remix was the logical choice (and is what was used in the performance benchmarking).

I’m confident that Remix on Cloudflare will be a first class citizen for the years to come, as Shopify acqui-hired the Remix team to support Hydrogen. Hydrogen, in turn, is built on Remix and hosted via Shopify’s Oxygen runtime, which is based on Cloudflare Workers. There are other edge-based Remix hosting providers like Netlify, but I didn’t evaluate them due to a lack of value proposition.

The biggest question is: where to deploy? There’s Cloudflare Workers or Cloudflare Pages. There’s a helpful compatibility matrix. These two runtimes are so close, I hope they merge one day.

For Cloudflare Workers, I’m interested in the newly released static assets support, which has the advantage over Workers Sites of not incurring KV usage. A Remix template transitioning to static assets was merged two weeks ago. Perfect timing. Despite its newness, static assets’ biggest sign of immaturity is the lack of an easy way to customize HTTP headers (for instance, Cache-Control). In practice, Cloudflare’s use of etags results in decent caching via 304’s Not Modified. It’s not optimal, but close enough.

For Pages, there were a few notable drawbacks:

  • Subpar Durable Objects support. I have ambitions for a feature that will make use of Websockets, and I’m planning on leveraging Durable Objects to coordinate them.
  • Lack of official support for Hyperdrive. I have a pre-existing Postgres database and I see Hyperdrive as an alternative to staging pgbouncer in front of a Postgres database so it’s not swarmed by ephemeral connections in a stampede of user requests. Unofficially, Pages supports Hyperdrive, according to a Cloudflare developer on Discord, perhaps incorrect docs?

I decided on Cloudflare Workers to be better suited for features like Durable Objects.

The migration

The migration to Next.js to Remix on Cloudflare was a lot of work but not too difficult.

Instead of databases, object stores, and anything that requires configuration directly referencing process.env, Cloudflare handlers pass down the environment as part of the handler. Those that relied on process.env needed to be restructured so the functions they expose are now behind an initializer. Not a bad change, makes it a bit easier for dependency injection now that the environment isn’t seeping into the nooks and crannies.

On the topic of dependency injection, as a lazy user of Tanstack React Query, I’d write code that referenced a globally available query client:

const queryClient = new QueryClient();
function useXyzMutation() {
  return useMutation({
    // ...
    onSuccess: () => {
      queryClient.invalidateQueries({ /* ... */ })
    }
  })
}

A classic SPA can get away with this, but it’s a vulnerability in a server side world where a single client would be used for all requests, leaking data. Thankfully, it was only a matter of deleting the global query client and updating the custom hooks to use useQueryClient. I’m not sure why the quickstart docs use a global query client; seems short-sighted.

Amusingly, I was a heavy user of synchronous require as a way to lazily reference potentially non-existent assets. Vite does not cooperate with require so I have a “pre-build” stage that creates a null object that can be statically imported.

Then came migrating next/link, next/router, next/head, next/image and next/dynamic. Most of these have analogous Remix or React implementations. For testing, next/jest.js was replaced with Vitest as nearly a drop in replacement.

What might be surprising is the lack of incompatible dependencies. The only one I can think of is sharp used for image transcoding, but that was easy enough to externalize. I had spun sharp out earlier to avoid CPU architecture issues that arose from developing on an x86 machine and deploying to aarch64. I had hoped specifying the Wasm runtime during NPM installation would make the architecture difference moot, but I guess NPM doesn’t record that information.

Wasm

If there is one thing I’ve learned in all my years of experience is that every framework, build tool, and hosting provider handles WebAssembly differently. I’ve probably wasted more years of my life trying to have a good production and development story with Wasm across all environments than with any feature development. Web workers are a close 2nd, but their story is equally egregious as web workers have been around for 14 years, while Wasm is half that.

And I’d consider myself a relative expert on WebAssembly. I’ve written about Wasm many times before, including recommendations when publishing a Wasm library, and I’ve made contributions to Rollup’s Wasm plugin.

Still, nothing guides you on how to deploy a production app that imports Wasm.

For Remix and Vite, I think I tried every build plugin under the sun without success. I’m not sure where wires are getting crossed, but I took matters into my own hands and structured server side code to initialize Wasm to look like:

// detect if on cloudflare to import the Wasm.Module directly
if (typeof WebSocketPair !== "undefined") {
  // Notice the import is not a relative import!
  const wasmApp = await import("wasm_app_bg.wasm");
  initSync(wasmApp.default);
} else {
  const wasmUrl = await import("./wasm/wasm_app_bg.wasm?url");
  const fs = await import("node:fs/promises");
  const data = await fs.readFile(`.${wasmUrl.default}`);
  await init(data);
}

With the following Vite config to turn the dummy package reference back into a relative import.

{
  build: {
    rollupOptions: {
      output: {
        paths: { "wasm_app_bg.wasm": "./wasm_app_bg.wasm" },
      },
      external: ["wasm_app_bg.wasm"],
    },
  },
}

To top everything off, the Wasm file is copied to the output directory.

cp src/app/app/server-lib/wasm/wasm_app_bg.wasm src/app/build/server/assets/.

Pretty? Absolutely not, but after a frustrating few hours, this solution is 100% tolerable.

Forbidden steam

This section got too long and was split into a separate article, but the takeaways are:

  • Steam blocks OpenID signature verification requests from Cloudflare based on the presence of “Cf-Worker” HTTP header and requires requests to go through a proxy that strips the header
  • Cloudflare has a non-spec compliant fetch implementation for URLSearchParams bodies that require a workaround

On Monorepos

Despite the repo containing several JS packages, they are all disparate with no workspace to collate them.

This didn’t last long, as Vite started complaining when assets from another package were referenced as they’re outside of an allow list:

The request url "/workspaces/pdx-tools/src/map/src/map-worker-bridge.ts" is outside of Vite serving allow list.

- /workspaces/pdx-tools/src/app

Refer to docs https://vite.dev/config/server-options.html#server-fs-allow for configurations and more details.

Ok, creating an NPM workspace in the root directory is easy enough.

It started out well, but soon puzzling Typescript errors appeared, like this one referencing some Sentry code.

error TS2607: JSX element class does not support attributes because it does not have a 'props' property.

<Sentry.ErrorBoundary fallback={fallback}>{children}</Sentry.ErrorBoundary>

My understanding is that Typescript started erroring because NPM hoists shared dependencies to the project root, but Typescript needs to be made aware of this. In this example, more than one project used Sentry, so Sentry was hoisted to node_modules/ instead of package-a/node_modules.

The question seems simple, how can I make Typescript aware of hoisted dependencies in an NPM workspace?

To this day, I have no clue. I couldn’t figure it out. Surveying 10 Typescript monorepos would give 10 different setups without rhyme or reason why a particular setup was used.

I resolved this by throwing the following into my Vite config:

{
  server: {
    fs: {
      allow: ['../..'],
    }
  }
}

I want to try out PNPM one day, but surveying their examples led to the same problem: no conformance on how to set up Typescript in a monorepo.

I came away with the impression that tooling support for NPM workspaces is clearly lacking. And Cloudflare’s wrangler is no exception.

Conclusion

Thank goodness I don’t take this propensity for rewrites to my day job.

But overall I’m happy with the adoption of Remix, streaming server side rendering, and Cloudflare Workers. It won’t be for everyone, but I’m seeing the performance and simplification benefits.

Comments

If you'd like to leave a comment, please email [email protected]