JWP Consulting GK

Migrating Projectify to use Render

Written on 2024-07-23

[Record Scratch] [Freeze Frame] Yep, my SPA is slow.

You’re probably wondering how I ended up in this situation.

The Projectify web app has recently migrated to use Render.com instead of Heroku. The reasons for that are complex. Figuring out which platform to migrate to is challenging by itself, but planning a safe migration with minimal downtime is even more challenging.

The Projectify web app has components that all need to work together, otherwise the app doesn’t function correctly. There’s a SvelteKit frontend, a Django backend, a Celery worker, a Redis cache and message queue, and the whole thing uses PostgreSQL as its RDBMS. Cloudinary stores image assets, but even here I am looking to migrate the app away.

Projectify architecture until now

For a long time, Netlify has been serving a static Projectify frontend. This caused an initial-load bottleneck.

The Projectify frontend is a complex SPA, and only sometimes a slightly faster multi-page app. This is because the Projectify frontend hides most of its useful features behind a user authentication check. Because the frontend doesn’t know who’s logged in at the time of pre-rendering, it just can’t be pre-rendered.

SvelteKit has great support for pre-rendering parent layouts. Many times, SvelteKit can only render the whole parent if it has already loaded made sure that a user has logged in correctly. Then, that page has to stay blank until SvelteKit has made sure that the user is authenticated. If a user isn’t logged in, the Projectify frontend then has to redirect them to a log in page.

You can “optimize” this behavior in many ways. It probably all started with YouTube lazy-loading its frontend and showing us these frustrating gray placeholders (sigh). They could have also just invested more time in making the first render faster server-side.

Some may interject big-tech-scalability and cutting-edge-frontend technology arguments here. And I can live with that. The (decidedly FOSS culture) guiding principle behind the development of the Projectify app is:

Practical solutions to practical problems

Developing the Projectify frontend as a solo developer doesn’t give me so much time to invent a whole secondary UI. You need to account for many failure scenarios in SPAs, for when your user’s internet connection is slow or unreliable. Or, perhaps your app isn’t so reliable either and maybe just slow.

Where does that leave a project that uses SvelteKit?

SvelteKit’ server side rendering

Everything old is new.

You can co-locate your frontend with your backend, and make it pre-render different things based on who is talking to it. If you do that, you’re recreating old-school service side rendering. And I don’t like that I myself think of it as old-school. Some also call this pattern Backend for Frontend (BFF) and the SvelteKit developers are aware of this trend as well. Everything old is new, and HTMX seems to be well received. Even I spend some evenings staring into a corner thinking, if only I had used HTMX.

I remind myself of how pleasant writing even just HTML with SvelteKit is. If only I could use Svelte components and pages in Django. Yes, some ideas around there exist. There’s dxsvelte and it looks fun to use.

Using SvelteKit itself is already straying from the norm, and I don’t want to spend my innovation tokens on even less common things. I also don’t intend to rewrite the backend in SvelteKit. Django’s batteries-included approach works well for me. Reviewing all the CSRF security options before the migration reminded me of how well this approach works. Switching to SvelteKit would require me to re-test every single security issue from scratch.

Instead, I embrace SvelteKit’s server side rendering (SSR). Reviewing the Projectify source code you can see how many times I’ve turned off SSR, and how many opportunities for optimizing page load times present themselves.

Caddy

Here’s the crucial thing with serving an SSR-based frontend: some requests still go to the backend and SvelteKit and adapter-node. These by themselves aren’t suited to handling incoming requests from the web.

The solution is to place a reverse proxy in front of frontend/backend and route requests accordingly. A reverse proxy was already needed just for running Django, and Heroku’s integrated routing has performed this role so far.

I’m familiar with Caddy. I’ve decided to provision Caddy as the “missing link” to kick off SvelteKit’s server side rendering. Caddy can forward requests to the backend as well, based on the path matched inside Caddy’s reverse proxy handler.

The advantage when working with this, is that cross-domain cookies become so much easier. Projectify serves all requests through the same domain. With this, Projectify can serve its cookies with the same site policy cookie variable set to samesite=Strict.

Caddy matches and routes requests (see the Caddyfile) like so:

Caddy also adds security related headers, and Projectify now has a 105 out of 100 score (wow?) on the MDN HTTP Observatory.

Supervisor, Podman, and Docker

I’ve spent a significant amount of time making sure that you can run each part of the Projectify web app with a single command using Nix.

I’ve also created container build Nix derivations to deploy the Projectify app inside a Podman environment. You can find the relevant Nix flake files right here:

Since I prefer rootless containers over Docker’s approach, and prefer entirely Free Software over Docker’s intellectual property mishmash, I’ve embraced Podman and Podman-compose. Nix excels here: not only does it build Docker-compatible containers much, much faster, it also gives you streaming builds that use less CPU and memory on compression.

This is useful if you upload containers to a registry anyway. Projectify uses Skopeo to upload all Projectify containers to the GitHub container registry after a build finishes.

This is the ideal buikd [sic] process. You may not like it, but this is what peak container build performance looks like.

Volunteers develop and maintain Nix, NixOS, and nixpkgs. The fact that the Nix community provides so much support is humbling.

Now, anyone can spin up a complete Projectify deployment with just a few commands on their local machine. This includes a Redis compatible cache (keydb) and a PostgreSQL instance.

Since container build times can still be somewhat long, I’ve also added a Supervisor configuration and added the necessary binaries to the Projectify repository’s root folder flake file. You can set up and run the Projectify web app using Supervisor with just a few commands. Nix caches all builds and they’re fast.

Configuration prep-work

The Projectify backend was recently changed to let you add more than one security origin, both for CSRF and CORS origins. You can set all security origins using SECURITY_ORIGINS variable.

Serving Projectify through a reverse proxy removes the need to support more than one origin. When everything goes through the same reverse proxy, there is only one origin. This means that I need this settings only for migration. I can remove it after the migration is over.

I still wanted to make sure I don’t miss anything. I set the new Render deployment’s SECURITY_ORIGINS to include these two domains:

https://STAGING_HOST_NAME.onrender.com,https://www.projectifyapp.com

I updated ALLOWED_HOSTS in the Render deployment to match the needed origin domains and set it to the following value:

STAGING_HOST_NAME.onrender.com,www.projectifyapp.com,api.projectifyapp.com

I set the FRONTEND_URL address to be https://www.projectifyapp.com, as this environment variable only exists for cosmetic purposes. This address is only used for inserting links to Projectify into emails that the Projectify backend sends.

PostgreSQL migration steps

Since database migrations are always exciting, I am writing down the PostgreSQL migration steps here for future reference:

# Make sure app is in maintenance mode
# Run inside nix shell nixpkgs#heroku
heroku maintenance:on --app projectifyapp
heroku pg:backups:capture --app projectifyapp
# Ensure backups are ready
heroku pg:backups:info --app projectifyapp
# Download backup
heroku pg:backups:download --app projectifyapp

A file called latest.dump appears in the current working directory. I add my own workstation’s IP to the Render.com PostgreSQL instance access control list.

Next, following the migration tutorial from Render, I use pg_restore to upload the dump. The PostgreSQL instance’s settings page shows the connection string that I can use to connect to it from the Projectify backend.

I adjust the invocation by adding the --clean flag to make sure that I wipe out previous DB migrations.

Please: think before running pg_restore with --clean.

read -l CONNECTION_STRING
pg_restore --verbose --no-acl --no-owner \
    --clean \
    -d $CONNECTION_STRING \
    latest.dump

Server migration

With the database restored, I first update the backend at api.projectifyapp.com as an intermediate step. Since I’ve set the ALLOWED_HOSTS settings variable to let both requests to www and api go through, I can decouple migrating Netlify and Heroku.

After adding the www and api sub domains as custom domains in the Render app, I update the DNS records to point to the new domain. I then wait for the changes to propagate.

I update the frontend hosted on Netlify to direct API requests to https://api.projectifyapp.com/api.

It works well and I follow up by updating the @ record (projectifyapp.com) first and waiting for the DNS records to propagate. An intermittent error I encountered when trying to access was a TLS handshake error:

* Host projectifyapp.com:443 was resolved.
* IPv6: (none)
* IPv4: 216.24.57.1
*   Trying 216.24.57.1:443...
* Connected to projectifyapp.com (216.24.57.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure

This issue resolved after a 10 min wait. Finally, I migrate the www domain to Render. A while later I run a smoke test and confirm that the following still work:

  1. Sign up for a new account
  2. Confirm email address
  3. Log in
  4. Go through onboarding and create a new workspace
  5. Test CSRF protection, CORS, and WebSocket connectivity by opening the site in more than one browser window and creating and modifying tasks.
  6. Log out and log in again
  7. Test picture upload
  8. Test WebSocket connectivity again

I disconnect all remaining staging domains, and shut down the old Heroku app. That’s it.

Upcoming challenges

Some things remain. These are the remaining tasks:

Once I’ve cleared these remaining tasks, the Projectify app is going to reduce its privacy footprint. Future iterations are going to replace Netlify, all third-party providers on Heroku, and the Heroku instance itself with first-party dependencies on Render.com. The only addition is CloudFlare, which is enabled by Render.com by default.