Gift list dev diary: deployment

Sat, Dec 26, 2020

All development for this project is shared on github at https://github.com/codonnell/mygiftlist-blog. I will endeavor to focus on the most relevant snippets of code rather than go over all of them on this blog. For interested parties, the github repo is available to survey all of the minor details.

In this post we’ll deploy the application to dokku. Dokku is a self-hosted PaaS solution which is great for small personal projects. Its API is very similar to Heroku’s, but allows hosting memory-hungry applications more cheaply. To package up our app into an uberjar, we’ll use depstar.

A number of steps from this post are pulled from this blog post, and the author has my deep gratitude for helping me avoid some deployment trial and error.

We’ll cover most of the code changes in this post, but you can see the full diff in this commit.

Migrations

We will run migrations as a release command. In order for this to work, we can’t run the migrations in a flyway docker container. This is because dokku release tasks are run in a container bootstrapped by the node.js and clojure buildpacks. Fortunately, we can pull in flyway’s java library as a dependency and run the migrations in clojure with a minimum of hassle.

(ns rocks.mygiftlist.migrate
  (:require [clojure.string :as str])
  (:import [java.net URI]
           [org.flywaydb.core Flyway]
           [org.flywaydb.core.api Location]))

(defn database-url->datasource-args [database-url]
  (let [{:keys [userInfo host port path]} (bean (URI. database-url))
        [username password] (str/split userInfo #":")]
    {:jdbc-url (str "jdbc:postgresql://" host ":" port path)
     :username username
     :password password}))

(defn migrate [{:keys [database-url]}]
  (let [{:keys [jdbc-url username password]}
        (database-url->datasource-args
          (or database-url (System/getenv "DATABASE_URL")))]
    (.. (Flyway/configure)
      (dataSource jdbc-url username password)
      (locations (into-array Location
                   [(Location. "filesystem:./migrations")]))
      (load)
      (migrate))))

We changed to using a database url instead of separately specifying the username, password, and other parameters. We did this to satisfy the API of dokku-postgres, which sets a DATABASE_URL environment variable for the application. We can now run migrations with clojure -X:migrate, optionally passing an explicit :database-url '"postgresql://user:password@my-host:5432/my-database"'.

Packaging

As is common in the Clojure world, we will package up our app into an uberjar to be run in production. In order to do this, our app now needs an entrypoint. We need to require every namespace that has an integrant component in order to load their multimethods.

(ns rocks.mygiftlist.main
  (:require rocks.mygiftlist.server
            rocks.mygiftlist.parser
            rocks.mygiftlist.db
            rocks.mygiftlist.config
            rocks.mygiftlist.authentication
            [integrant.core :as ig]
            [clojure.java.io :as io]))

(defn -main [& _args]
  (-> "system.edn"
    io/resource
    slurp
    ig/read-string
    ig/init))

We will create our uberjar following the depstar readme by adding a depstar alias to our deps.edn file and adding a make step which installs javascript dependencies, compiles clojurescript, compiles scss, and finally creates an uberjar with our backend app and frontend assets.

:depstar   {:extra-deps {seancorfield/depstar {:mvn/version "2.0.161"}}
            :ns-default hf.depstar
            :exec-args  {}}

uberjar:
	npm install
	npx shadow-cljs release prod
	npm run css-build
	clojure -X:depstar uberjar :jar target/mygiftlistrocks.jar :aliases '[:backend]'

You may notice that we changed our aliases to differentiate frontend, backend, development, and test dependencies. We also distributed our source files into clj, cljc, and cljs folders. This is to avoid unecessarily bloating our uberjar with clojurescript source files.

With this in place, after creating an uberjar with make uberjar we can run our app with java -cp target/mygiftlistrocks.jar clojure.main -m rocks.mygiftlist.main.

Deployment Configuration

There are a few small files we need to create in order to deploy successfully to dokku. First, we will write a Procfile which declares how to run our app and our migrations.

web: java -cp target/mygiftlistrocks.jar clojure.main -m rocks.mygiftlist.main
release: clojure -X:migrate

Second, we’ll write an executable bin/build file that declares how to build our app.

#!/usr/bin/env bash
make uberjar

(Don’t forget to make the file executable with chmod a+x bin/build!)

In order for dokku to register our project as a Clojure project, we need a project.clj file. Fortunately, it doesn’t have to have anything in it. We can create an empty one with touch project.clj.

To run our app with Java 14, we will create a system.properties file.

java.runtime.version=14

Lastly, it’s helpful to have a ready check which verifies that a newly deployed container is ready before directing traffic to it. By default dokku waits 10 seconds after starting an app to direct traffic to it, which is generally not enough time. We will add a custom ready check by writing a CHECKS file. The CHECKS file records a string that should be present on the homepage when our app is ready.

/ My Gift List Rocks

Provisioning

The first thing we will do is deploy a dokku instance. Digital Ocean offers a self-hosted, one-click Dokku deployment through their marketplace. (The link is my affiliate link, which will give you $100 in credit over 60 days if you’re a first-time user. If you spend $25, I get $25.) I have a larger dokku instance on which I host multiple personal projects, but a $5/mo. droplet suffices to host the gift list app.

I generally only write out the most salient snippets, but I will try to be exhaustive in writing out the commands to set up dokku. If I’ve left anything out, please let me know; I will do my best to keep this post updated.

NOTE: We will set up an SSL certificate for the app, and I’ll assume the reader has a subdomain she can use if she wants to follow along.

After the dokku instance is deployed, we will get its IP address and create an A record pointing from our subdomain of choice to the dokku instance’s IP address. If possible, we’ll use the subdomain mygiftlist.domain.tld. It makes the subsequent process slightly easier. After creating the A record, we will go to our subdomain to set up dokku. The form should already have an ssh key entered. We’ll enter our domain name (not subdomain) as the dokku host and select “Use virtualhost naming for apps”, then click “Finish Setup”.

We’ll quickly add a new non-root user so we don’t have to do everything as root, giving the user sudo and docker permissions. The docker permissions aren’t strictly necessary, but they can be useful for debugging. We’ll also copy the root user’s ssh config to the new user so we can ssh in as the new user using the same key.

adduser chris
usermod -aG sudo chris
usermod -aG docker chris
rsync --archive --chown=chris:chris ~/.ssh /home/chris

It’s useful to be able to run dokku commands from our local machine. Following the instructions, we will create a host in our ssh config.

Host dokku
     HostName <DOKKU DROPLET IP ADDRESS>
     User chris
     RequestTTY yes

After this, we’ll install the dokku client following the OS-specific instructions. Subsequently, we should be able to run dokku apps:list on our local machine and see that we have not yet deployed any apps.

With this done, we’ll create an app.

dokku apps:create mygiftlist

If we run git remote -v and dokku apps:list we should see that a new git remote and a new app have been created. We’ll also need a postgres database. To create one, we’ll need to ssh in to dokku and install the postgres plugin. Then we’ll create our database and link it to our app.

ssh dokku
# On dokku remote server
sudo dokku plugin:install https://github.com/dokku/dokku-postgres.git
exit
# On local machine
dokku postgres:create mygiftlist-database --image-version 13.1
dokku postgres:link mygiftlist-database mygiftlist

In order to build our app, we need a recent version of the clojure CLI. We can control this with the CLOJURE_CLI_VERSION environment variable, which we’ll set to the most recent version as of this writing.

# From repository root
dokku config:set CLOJURE_CLI_VERSION=1.10.1.763

We’ll also need to increase the number of times the health check is retried. Five retries is not always enough, especially on the $5/mo. droplet. The documentation explains how to do this.

# From repository root
dokku config:set DOKKU_CHECKS_ATTEMPTS=20

Next we’ll need to add buildpacks that give us access to node and the clojure CLI in our build process.

# From repository root
dokku buildpacks:add https://github.com/heroku/heroku-buildpack-nodejs.git
dokku buildpacks:add https://github.com/heroku/heroku-buildpack-clojure.git

Now we are ready to deploy an initial version of the app!

# From repository root
git push dokku master

Once the deploy has finished, we should be able to access our app at http://mygiftlist.domain.tld. If we didn’t name the subdomain mygiftlist, we can run dokku domains:add othersubdomain.domain.tld to make the app accessible at the appropriate url.

Our final task is to set up an SSL cert with letsencrypt. This is a relatively straightforward process using the letsencrypt plugin. First we need to install the plugin on the dokku server. We will also set an email address to use for all of our letsencrypt certificates.

NOTE: As part of this setup, we will set a long HSTS policy on the subdomain, causing future requests to go over https instead of http.

ssh dokku
# On dokku remote server
sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git
dokku config:set --global DOKKU_LETSENCRYPT_EMAIL=myemail@example.com
exit

It’s good practice to test out certificate retrieval on the staging server first, so that’s what we’ll do. (If we ran dokku domains:add othersubdomain.domain.tld earlier, we will need to remove the default domain with dokku domains:remove mygiftlist.domain.tld.)

# From repository root
dokku config:set --no-restart DOKKU_LETSENCRYPT_SERVER=staging

Now when we visit our app at http://mygiftlist.domain.tld, it should redirect to https and complain about an invalid certificate authority. That means we’re all set to do this for real!

# From repository root
dokku config:unset --no-restart DOKKU_LETSENCRYPT_SERVER
dokku letsencrypt

And we want our certificate to auto-renew, so we’ll add a cron job which does that, too.

# From repository root
dokku letsencrypt:cron-job --add

And we’re done! We have our app deployed, and we can deploy any further updates with a git push dokku master. We can even check the security of our SSL configuration on ssllabs and see that it gets an A+.

Prev: Gift list dev diary: gift list navigation