JWP Consulting GK

Migrating Projectify to use Render

Written on 2024-07-23

[Record Scratch] [Freeze Frame] Yep, my SPA is slow.

You’re probably wondering how I ended up in this situation.

The Projectify web app has recently migrated to use Render.com instead of Heroku. The reasons for that are complex. Figuring out which platform to migrate to is challenging by itself, but planning a safe migration with minimal downtime is even more challenging.

The Projectify web app has components that all need to work together, otherwise the app doesn’t function correctly. There’s a SvelteKit frontend, a Django backend, a Celery worker, a Redis cache and message queue, and the whole thing uses PostgreSQL as its RDBMS. Image assets are stored with Cloudinary so far, but even here I am looking to migrate the app away.

Projectify architecture until now

For a long time, the frontend has been served statically from Netlify. This caused an initial-load bottleneck.

The Projectify frontend is a complex SPA, and only sometimes a slightly faster multi-page app. This is because most of the useful frontend features are hidden behind a user authentication check and can’t be pre-rendered.

SvelteKit has great support for pre-rendering parent layouts. But, if the whole parent can only be rendered if a user is loaded, then that page stays blank until SvelteKit has made sure that the user is authenticated. If a user isn’t logged in, the Projectify frontend then has to redirect them to a log in page.

You can “optimize” this behavior in many ways. It probably all started with YouTube lazy-loading its frontend and showing us these frustrating gray placeholders (sigh). They could have also just invested more time in making the first render faster server-side.

Some may interject big-tech-scalability and cutting-edge-frontend technology arguments here. And I can live with that. The (decidedly FOSS culture) guiding principle behind the development of the Projectify app is:

Practical solutions to practical problems

Developing the Projectify frontend as a solo developer doesn’t give me so much time to invent a whole secondary UI. You need to account for many failure scenarios in SPAs, for when your user’s internet connection is slow or unreliable. Or, perhaps your app isn’t so reliable either and maybe just slow.

Where does that leave a project that uses SvelteKit?

SvelteKit’ server side rendering

Everything old is new.

You can co-locate your frontend with your backend, and make it pre-render different things based on who is talking to it. If you do that, you’re recreating old-school service side rendering. And I don’t like that I myself think of it as old-school. Some also call this pattern Backend for Frontend (BFF) and the SvelteKit developers are aware of this trend as well. Everything old is new, and HTMX seems to be well received. Even I spend some evenings staring into a corner thinking, if only I had used HTMX.

I remind myself of how pleasant writing even just HTML with SvelteKit is. If only I could use Svelte components and pages in Django. Yes, some ideas around there exist. There’s dxsvelte and it looks fun to use.

Using SvelteKit itself is already straying from the norm, and I don’t want to spend my innovation tokens on even less common things. I also don’t intend to rewrite the backend in SvelteKit. Django’s batteries-included approach works well for me. Reviewing all the CSRF security options before the migration reminded me of how well this approach works. Switching to SvelteKit would require me to re-test every single security issue from scratch.

Instead, I embrace SvelteKit’s server side rendering (SSR). Reviewing the Projectify source code you can see how many times I’ve turned off SSR, and how many opportunities for optimizing page load times present themselves.

Caddy

Here’s the crucial thing with serving an SSR-based frontend: some requests still go to the backend and SvelteKit and adapter-node. These by themselves aren’t suited to handling incoming requests from the web.

The solution is to place a reverse proxy in front of frontend/backend and route requests accordingly. A reverse proxy was already needed just for running Django, and Heroku’s integrated routing has performed this role so far.

I’m familiar with Caddy. I’ve decided to provision Caddy as the “missing link” to kick off SvelteKit’s server side rendering. Caddy can forward requests to the backend as well, based on the path matched inside Caddy’s reverse proxy handler.

The advantage when working with this, is that cross-domain cookies become so much easier. All requests are served through the same domain. Cookies can be set to samesite=Strict.

Caddy matches and routes requests (see the Caddyfile) like so:

Caddy also adds security related headers, and Projectify now has a 105 out of 100 score (wow?) on the MDN HTTP Observatory.

Supervisor, Podman, and Docker

I’ve spent a significant amount of time making sure that each part of the Projectify web app can be run as a single command using Nix.

I’ve also created container build Nix derivations to deploy the Projectify app inside a Podman environment. The relevant Nix flake files can be found here:

Since I prefer rootless containers over Docker’s approach, and prefer entirely Free Software over Docker’s intellectual property mishmash, I’ve embraced Podman and Podman-compose. Nix excels here: not only does it build Docker-compatible containers much, much faster, it also gives you streaming builds that use less CPU and memory on compression.

This is useful if a container is uploaded to a registry anyway. Skopeo is used to upload all Projectify containers to the GitHub container registry after a build is finished.

This is the ideal buikd [sic] process. You may not like it, but this is what peak container build performance looks like.

The fact that Nix, NixOS and nixpkgs are developed and maintained by volunteers, and that the Nix community provides so much support is humbling.

Now, anyone can spin up a complete Projectify deployment with just a few commands on their local machine. This includes a Redis compatible cache (keydb) and a PostgreSQL instance.

Since container build times can still be somewhat long, I’ve also added a Supervisor configuration and added the necessary binaries to the Projectify repository’s root folder flake file. Setting up and running the Projectify web app using Supervisor can be achieved with just a few commands. All builds are cached by Nix and they are fast.

Configuration prep-work

The Projectify backend was recently changed to let you add more than one security origin, both for CSRF and CORS origins. You can set all security origins using SECURITY_ORIGINS variable.

Serving Projectify through a reverse proxy removes the need to support more than one origin. When everything goes through the same reverse proxy, there is only one origin. This means that I need this settings only for migration. I can remove it after the migration is over.

I still wanted to make sure I don’t miss anything. I set the new Render deployment’s SECURITY_ORIGINS to include these two domains:

https://STAGING_HOST_NAME.onrender.com,https://www.projectifyapp.com

I updated ALLOWED_HOSTS in the Render deployment to match the needed origin domains and set it to the following value:

STAGING_HOST_NAME.onrender.com,www.projectifyapp.com,api.projectifyapp.com

I set FRONTEND_URL to be https://www.projectifyapp.com, as this environment variable only exists for cosmetic purposes. That is, it’s used for inserting full URLs into backend-sent emails.

PostgreSQL migration steps

Since database migrations are always exciting, I am writing down the PostgreSQL migration steps here for future reference:

# Make sure app is in maintenance mode
# Run inside nix shell nixpkgs#heroku
heroku maintenance:on --app projectifyapp
heroku pg:backups:capture --app projectifyapp
# Ensure backups are ready
heroku pg:backups:info --app projectifyapp
# Download backup
heroku pg:backups:download --app projectifyapp

A file called latest.dump appears in the current working directory. I add my own workstation’s IP to the Render.com PostgreSQL instance access control list.

Next, following the migration tutorial from Render, I use pg_restore to upload the dump. The connection string can be retrieved from the PostgreSQL instance’s settings page.

I adjust the invocation by adding the --clean flag, making sure that previous DB migrations are wiped out.

Please: think before running pg_restore with --clean.

read -l CONNECTION_STRING
pg_restore --verbose --no-acl --no-owner \
    --clean \
    -d $CONNECTION_STRING \
    latest.dump

Server migration

With the database restored, I first update the backend at api.projectifyapp.com as an intermediate step. Since ALLOWED_HOSTS has been set to let both requests to www and api go through, I can decouple migrating Netlify and Heroku.

After adding the www and api sub domains as custom domains in the Render app, I update the DNS records to point to the new domain. I then wait for the changes to propagate.

I update the frontend hosted on Netlify to direct API requests to https://api.projectifyapp.com/api.

It works well and I follow up by updating the @ record (projectifyapp.com) first and waiting for the DNS records to propagate. An intermittent error I encountered when trying to access was a TLS handshake error:

* Host projectifyapp.com:443 was resolved.
* IPv6: (none)
* IPv4: 216.24.57.1
*   Trying 216.24.57.1:443...
* Connected to projectifyapp.com (216.24.57.1) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS alert, handshake failure (552):
* OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000410:SSL routines::sslv3 alert handshake failure

This issue resolved after a 10 min wait. Finally, I migrate the www domain to Render. A while later I run a smoke test and confirm that the following still work:

  1. Sign up for a new account
  2. Confirm email address
  3. Log in
  4. Go through onboarding and create a new workspace
  5. Test CSRF protection, CORS, and WebSocket connectivity by opening the site in more than one browser window and creating and modifying tasks.
  6. Log out and log in again
  7. Test picture upload
  8. Test WebSocket connectivity again

I disconnect all remaining staging domains, and shut down the old Heroku app. That’s it.

Upcoming challenges

Some things remain. These are the remaining tasks:

Once the remaining tasks are cleared, the Projectify app is going to reduce its privacy footprint a lot. Netlify, all third-party providers on Heroku, and the Heroku instance itself is going to be replaced by first-party dependencies on Render.com instead. The only addition is CloudFlare, which is enabled by Render.com by default.