JWP Consulting GK

Migrating Projectify to use Render

Written by Justus Perlwitz on 2024-07-23

[Record Scratch] [Freeze Frame] Yep, my SPA is slow.

You’re probably wondering how I ended up in this situation.

The Projectify web app has recently migrated to use Render.com instead of Heroku. The reasons for that are complex. Figuring out which platform to migrate to is challenging by itself, but planning a safe migration with minimum downtime is even more challenging.

The Projectify web app has components that all need to work together, otherwise the app will not function correctly. There is a SvelteKit frontend, a Django backend, a Celery worker, a Redis cache and message queue, and the whole thing uses PostgreSQL as its RDBMS. Image assets are stored with Cloudinary so far, but even here I am looking to migrate the app away.

Projectify architecture until now

For a long time, the frontend has been served statically from Netlify. This caused an initial-load bottleneck.

The Projectify frontend is mostly a complex SPA, and only sometimes a slightly faster multi-page app. This is because most of the useful frontend functionality is hidden behind a user authentication check and can’t be pre-rendered.

SvelteKit has great support for pre-rendering parent layouts. But, if the whole parent can only be rendered if a user is loaded, then that page will stay blank until the SvelteKit, running in the frontend, has made sure that a user is authenticated, and that no redirect to the login page is necessary.

There are many ways to “optimize” this behavior. It probably all started with YouTube lazily loading its frontend and showing us these frustrating gray placeholders (sigh), instead of just investing more time in making the first render faster server-side.

Someone will probably interject big-tech-scalability and cutting-edge-frontend technology arguments here. And I can live with that. The (decidedly FOSS culture) guiding principle behind the development of the Projectify app is:

Practical solutions to practical problems

Developing the Projectify frontend as a solo developer doesn’t give me so much time to invent a whole secondary UI for those times when a user’s internet connection isn’t reliable or because someone decided that outsourcing a service provider’s compute time to their customers is fashionable now.

Where does that leave a project that is deeply invested in SvelteKit?

SvelteKit’ server side rendering

Everything old is new.

If you can somehow co-locate your frontend with your backend, and make it pre-render different things based on who is talking to it, you’ve essentially recreated old-school service side rendering. And I don’t like that I myself think of it as old-school. Some also call this pattern Backend for Frontend (BFF) and the SvelteKit developers are aware of this trend as well. Everything old is new, and HTMX seems to be well received. Even I spend some evenings staring into a corner thinking, if only I had used HTMX.

But then I remind myself of how pleasant it is to write even just HTML with SvelteKit. If only I could use Svelte components and pages in Django. Yes, some ideas around there exist. There is dxsvelte and it looks fun to use.

However, using SvelteKit itself is already straying from the norm, and I don’t want to spend my innovation tokens on even less common things. I also don’t intend to rewrite the backend in SvelteKit. Django’s batteries-included approach works well for me. Reviewing all the CSRF security options before the migration reminded me of how well this approach works. Switching to SvelteKit would require me to re-evaluate every single security issue from scratch.

Instead, I will just embrace SvelteKit’s server side rendering (SSR). Reviewing the Projectify source code you can see how many times I’ve disabled SSR, and how many opportunities for optimizing page load times present themselves.

Caddy

The crucial thing with serving an SSR-based frontend is that some requests will still go to the backend and that SvelteKit and its adapter-node themselves are not suited to handling incoming requests from the web.

The solution is to place a reverse proxy in front of frontend/backend and route requests accordingly. A reverse proxy was already needed just for running Django, and Heroku’s integrated routing has performed this role so far.

I’m quite familiar with Caddy, and given how easy it is to get started working with it, I’ve decided to provision Caddy as the “missing link” to kick off SvelteKit’s server side rendering. This allow requests to reach out to the backend when needed, based on the URL matched inside Caddy’s reverse proxy handler.

The advantage when working with this, is that cross-domain cookies become so much easier. All requests are served through the same domain. Cookies can be set to samesite=Strict.

Caddy matches and routes requests (see the Caddyfile) like so:

Caddy also adds security related headers, and Projectify now has a 105 out of 100 score (wow?) on the MDN HTTP Observatory.

Supervisor, Podman, and Docker

To be able to recreate a complex deployment scenario, I’ve spent a significant amount of time making sure that each part of the Projectify web app can be run as a single command using Nix.

I’ve also created container build Nix derivations to deploy the Projectify app inside a Podman environment. The relevant Nix flake files can be found here:

Since I prefer rootless containers over Docker’s approach, and generally prefer entirely Free Software over Docker’s intellectual property mishmash, I’ve embraced Podman and Podman-compose. Nix excels here: not only does it build Docker-compatible containers much, much faster, it also gives you streaming builds that use less CPU and memory on compression.

This is useful if a container is uploaded to a registry anyway. Skopeo is used to upload all Projectify containers to the GitHub container registry after a build is finished.

This is the ideal buikd [sic] process. You may not like it, but this is what peak container build performance looks like.

The fact that Nix, NixOS and nixpkgs are developed and maintained by volunteers, and that I am able to leverage the Nixiverse to this degree is humbling.

Now, anyone can spin up a complete Projectify deployment with just a few commands on their local machine. This includes a Redis compatible cache (keydb) and a PostgreSQL instance.

Since container build times can still be somewhat long, I’ve also added a Supervisor configuration and added the necessary binaries to the Projectify repository’s root folder flake file. Setting up and running the Projectify web app using Supervisor can be achieved with just a few commands, and all builds used are cached by Nix and therefore fast.

Configuration prep-work

The Projectify backend was recently changed to allow setting multiple security origins (CSRF and CORS origins) through the SECURITY_ORIGINS variable.

Of course, switching everything to be served through one reverse proxy is supposed to solve the issue of dealing with cross origin requests to work in the first place.

I still wanted to make sure I don’t miss anything and therefore set the new Render deployment’s SECURITY_ORIGINS to include multiple domains:

https://STAGING_HOST_NAME.onrender.com,https://www.projectifyapp.com

I updated ALLOWED_HOSTS in the Render deployment to match the above origins:

STAGING_HOST_NAME.onrender.com,www.projectifyapp.com,api.projectifyapp.com

I set FRONTEND_URL to be https://www.projectifyapp.com, as this environment variable only exists for cosmetic purposes. That is, it’s used for inserting full URLs into backend-sent emails and so on.

DB Migration steps

Since database migrations are always exciting, I am writing down the PostgreSQL migration steps here for future reference:

# Make sure app is in maintenance mode
# Run inside nix shell nixpkgs#heroku
heroku maintenance:on --app projectifyapp
heroku pg:backups:capture --app projectifyapp
# Ensure backups are ready
heroku pg:backups:info --app projectifyapp
# Download backup
heroku pg:backups:download --app projectifyapp

A file called latest.dump appears in the current working directory. I add my own workstation’s IP to the Render.com PostgreSQL instance access control list.

Next, following the migration tutorial from Render, I use pg_restore to upload the dump. The connection string can be retrieved from the PostgreSQL instance’s settings page.

I adjust the invocation by adding the --clean flag, making sure that previous DB migrations are wiped out.

Please: think carefully before running pg_restore with --clean.

read -l CONNECTION_STRING
pg_restore --verbose --no-acl --no-owner \
    --clean \
    -d $CONNECTION_STRING \
    latest.dump

Server migration

With the database restored, I first update the backend at api.projectifyapp.com as an intermediate step. Since ALLOWED_HOSTS has been set to allow both requests to www and api to go through, I can decouple migrating Netlify and Heroku.

After adding the www and api sub domains as custom domains in the Render app, I update the DNS records to point to the new domain one and wait for the changes to propagate.

I update the frontend hosted on Netlify to direct API requests to https://api.projectifyapp.com/api.

It works well and I follow up by updating the @ record (projectifyapp.com) first and waiting for the DNS records to propagate. An intermittent error I encountered when trying to access was a TLS handshake error:

* Host projectifyapp.com:443 was resolved.
* IPv6: (none)
* IPv4: 216.24.57.1
*   Trying 216.24.57.1:443...
* Connected to projectifyapp.com (216.24.57.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure

This issue resolved after a 10 min wait. Finally, I migrate the www domain to Render. A while later I run a smoke test and confirm that the following still work:

  1. Sign up for a new account
  2. Confirm email address
  3. Log in
  4. Go through onboarding and create a new workspace
  5. Test CSRF protection, CORS, and WebSocket connectivity by opening the site in several browser windows and creating and modifying tasks.
  6. Log out and log in again
  7. Test picture upload
  8. Test WebSocket connectivity again

I disconnect all remaining staging domains, and shut down the old Heroku application. That’s it!

Upcoming challenges

Some things remain. There are a few remaining tasks:

Once the remaining tasks are cleared, the Projectify app will reduce its privacy footprint a lot. Netlify, all third-party providers on Heroku, and the Heroku instance itself will be replaced by first-party dependencies on Render.com instead. The only addition is CloudFlare, which is enabled by Render.com by default.