Two Ghosts, One Host

A friend of mine who runs a small publication, On Fire In AK, recently realized they wouldn't have time to manage the site's infrastructure, and was going to shut it down. So they asked if I'd be interested in maintaining it.

It seemed a shame to waste all that hard work, and trash the content written over the last few years by our talented writers. So I accepted.

Paying for the hosting, however, seemed pretty unappealing. With present traffic levels, even the 1GB DigitalOcean droplet it ran on was overkill. A single instance of an ultra-efficient app like Ghost didn't need all those resources.

I briefly considered trying to shoehorn another instance of the application natively onto the same server that ran my website. Unfortunately, this came with some other implications, like a messy upgrade path which involved trying to manage multiple installations of Node.js.

I went with a containerized setup, putting the two instances of Ghost behind an Nginx reverse proxy.

Prerequisites

• A linux server (what server and what distro is immaterial).
• DNS A records for both domains pointing at the server.
• Docker and docker-compose installed.
• A mail service supporting SMTP. I (and, the developers of Ghost ) recommend Mailgun.

Architecture

This project is open source. You can check out the repository here.

My setup consists of four components, each made up of containers:

1. Ghost #1 (noahsbwilliams.com)(tilde)
2. Ghost #2 (onfireinak.com)(onfireinak)
3. Nginx (reverse proxy)
4. Certbot (SSL certificates from LetsEncrypt)

A quick note before we continue: You will see throughout this post, that "Tilde" is the refers to things related to noahsbwilliams.com.

It's a nickname given because, you know...

~/🏠 = whereis ❤️

Docker compose

Managing containers from a command line sucks, and proper, production-ready container orchestration tools like Kubernetes are overkill for a project like this.

Enter: Docker compose.

For the uninitiated: Docker compose a frontend for Docker that lets you write your container infrastructure in a single, Linux-familiar, everything-is-a-file docker-compose.yaml file. Once your file is written, all it takes to start up the containers is running the command docker-compose up in working directory.

Side note: IMHO, for Linux SysAdmins, it's actually a good idea to learn Docker compose before the Docker CLI. But that's another rant, for another day.

Ghost #1 (Tilde)

Let's start at the goal and work backwards from there.

Right off the bat - here's the configuration for Tilde:

tilde:
    image: ghost:3-alpine
    restart: always
    volumes:
      - ./content/tilde:/var/lib/ghost/content
    environment:
      database__client: 'sqlite3'
      url: 'https://noahsbwilliams.com'
      mail__transport: 'SMTP'
      mail__options__service: 'Mailgun'
      mail__options__host: 'smtp.mailgun.org'
      mail__options__port: '465'
      mail__options__secureConnection: 'true'
      mail__options__auth__user: 'postmaster@email.noahsbwilliams.com'
      mail__options__auth__pass: ${PW_TILDE}
    networks:
      - reverse_proxy_tilde

We start from a base image of the latest version of Ghost v3 using image: ghost:3.

We then setup a volume, mapping the content/tilde folder inside the current working directory to the content content location in the container, /var/lib/ghost/content. This allows our data to persist, as the rest of the container will be regularly removed and rebuilt. After all, a CMS that doesn't store your site's content isn't very useful! Don't worry about creating these folders on the host, as Docker will do that automatically when the container starts.

Now let's have a look at the environment variables.

The Ghost container uses the syntax of double underscores (key__subkey__subsubkey: value) for nested configuration options.

So to convert an existing config.production.json file, like this one from my old, non-containerized blog...

{
  "url": "https://noahsbwilliams.com",
  "server": {
    "port": 2368,
    "host": "127.0.0.1"
  },
  "database": {
    "client": "sqlite3"
  },
  "mail": {
    "transport": "SMTP",
    "options": {
      "service": "Mailgun",
      "port": "465",
      "secureConnection": "true",
      "host": "smtp.mailgun.org",
      "auth": {
        "user": "ghost@email.noahsbwilliams.com",
        "pass": "<smtp password>"
      }
    }
  }
}

...we'll need to input the variables in the new format into the docker-compose file.

First, we set the URL for our blog:

url: 'https://noahsbwilliams.com'

Next, Ghost needs to know what kind of database you want it to use. Since we've chosen SQLite, we simply set:

database__client: 'sqlite3'

If we wanted to use MySQL instead, however, the config would be slightly more complicated, something like this:

database__client: 'mysql'
database__connection__host: 'db_container'
database__connection__user: 'ghost'
database__connection__password: 'correct-horse-battery-staple'
database__connection__database: 'tilde'

Finally, we'll setup SMTP, so Ghost can send emails:

mail__transport: 'SMTP'
mail__options__service: 'Mailgun'
mail__options__host: 'smtp.mailgun.org'
mail__options__port: '465'
mail__options__secureConnection: 'true'
mail__options__auth__user: 'ghost@email.noahsbwilliams.com'
mail__options__auth__pass: ${PW_TILDE}

Wait, but what on earth is that ${PW_TILDE} thing?

That, my friend, is a reference to an environment variable.

Since I've decided to control this whole setup with Git, I wanted to avoid putting the password in the compose file itself (committing your creds to a public repository = bad).

Docker has implemented several confusing ways to use runtime environment variables; here, I decided to go with the "Dot-Env" method.

What this entails is simply creating a .env file in the working directory and adding it to the .gitignore to avoid committing it. If you're not using git to version the Docker-compose, you can probably skip this step.

All the file contains, in our case, is the passwords to our SMTP services. Below, is the full contents of the file.

PW_TILDE="mailgun-apikey-goes-here"
PW_ONFIRE="sendgrid-apikey-goes-here"

Ghost #1 is complete!

Note: Setting up this container was a pain!

The docs for the "Official" (actually unofficial, but maintained by the community in the _ repo) Ghost image are a bit vague. When I'm in a nicer mood after lunch, I'll quit complaining and submit a pull request to fill in the gaps.

Ghost #2 (OnFireInAK)

This one was fairly easy; a copy-paste of the first one with some changes to the SMTP settings, a different URL, and another Docker network.

Currently, On Fire In AK is using SendGrid, but I might switch it over to Mailgun at some point in the near future. As of Feb 22, 2019, SendGrid rate limits at 100 messages/day on the free tier, whereas Mailgun lumps this into a simply monthly 3,000.

Since blasting out new articles to subscribers at publish time is a sprint rather than a marathon, I find Mailgun's approach to be better for our use case.

No matter. I'll leave the Sendgrid config in for educational purposes.

Here's the config for Ghost #2:

onfireinak:
    image: ghost:3-alpine
    restart: always
    volumes:
      - ./content/onfireinak:/var/lib/ghost/content
    environment:
      database__client: 'sqlite3'
      url: 'https://onfireinak.com'
      mail__transport: 'SMTP'
      mail__options__service: 'Sendgrid'
      mail__options__port: '465'
      mail__options__secureConnection: 'true'
      mail__options__host: 'smtp.sendgrid.net'
      mail__options__auth__user: 'apikey'
      mail__options__auth__pass: ${PW_ONFIRE}
    networks:
      - reverse_proxy_onfireinak

Ghost #2: Done!

Nginx

I decided to put the Ghost instances behind an Nginx server, partially because it's the officially supported web server for Ghost, and yes, partially out of habit. I'll do a project with HAproxy sometime, I promise, @HusseinNasser!

I wrote up the following config for the Nginx container in the docker-compose:

nginx:
    depends_on:
      - tilde
      - onfireinak
    image: nginx
    restart: always
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - ./nginx/:/etc/nginx/
      - ./letsencrypt:/etc/letsencrypt
    networks:
      - reverse_proxy_tilde
      - reverse_proxy_onfireinak

Dependancies

We want Docker to wait until the two Ghost containers are up and running to start up the reverse proxy container; other will terminate in error once it begins receiving HTTP requests with no backend to pass them to.

Docker compose provides some rudimentary orchestration logic via the depends_on policy. Setting this is quite simple:

depends_on:
      - tilde
      - onfireinak

External Ports

Docker has a pretty cool firewall policy for incoming traffic: "Not unless I said so!"

This means we need to explicitly open ports 80 and 443 (HTTP & HTTPS respectively).

We do this with:

ports:
      - '80:80'
      - '443:443'

And yes - we do need both of them open, even though we intend to follow best practices and make our site to be HTTPS-only.

This is because when someone types in noahsbwilliams.com to their browser for the first time, their initial request is sent unencrypted (this sucks, but will be remedied later; stay with me).

Internal networking

Another way Docker is cool is how it handles container-to-container networking for you. No fumbling with NAT tables or adapters, you just name the damn things whatever you like and Docker sets them up for you.

In this case, looking at the bottom of our Docker compose file, we've got two docker networks:

networks:
  reverse_proxy_tilde:
  reverse_proxy_onfireinak:

What do these mean?

Both of these are connected to the Nginx reverse proxy, but only one is connected to each Ghost container.

Is this necessary? Eh. Maybe? Maybe not. But it seemed like a good application of the rule of least privilege, and required very little extra work to implement. So why not?

Nginx Configuration

This part calls to the SysAmin in me, as it's the most "classic" part of the project.

At the time of this writing, Nginx doesn't support many Docker environment variables for its' own configuration, so we'll have to mount a volume containing configuration files.

We also mount the ./letsencrypt folder as /etc/letsencrypt, where Nginx can expect to find our TLS certificates. We'll get to those later.

First, the main nginx.conf file for the application itself:

# Set Nginx to run as the www-data user in the Debian-based Nginx container
user                    www-data;  

worker_processes        auto;  
pid                     /var/run/nginx.pid;

events {  
                        worker_connections  1024;
}

# Setup content compression
http {  
    gzip                on;
    gzip_proxied        expired no-cache no-store private auth;
    gzip_types          text/plain text/css application/json application/javascript text/xml application/xml
                        application/xml+rss text/javascript image/svg+xml application/vnd.ms-fontobject
                        application/x-woff;
	# Enable support for webp media files
    map                 $http_accept  $webp_suffix {
                        default "";
                        "~*webp" ".webp";
    }
	
    # Include all Nginx configurations in sites-enabled 
    include             /etc/nginx/sites-enabled/*;
    
    # Set a reasonable max upload size of 10 MB for images etc
    client_max_body_size 10M;

}

Next, the meat and potatoes - the reverse proxy configuration file.

This is a site (host)-specific file. We'll make two of these, one for each site, in the nginx/sites-enabled folder.

The name of the files can be whatever you like; I named this first one tilde.conf, in accordance with project convention.

# Redirect unencrypted HTTP requests to HTTPS
server {  
  listen              80;
  listen		      [::]:80;
  server_name         noahsbwilliams.com;
  return              301 https://$server_name$request_uri;
}


server {  
    
    # Add HSTS headers to help prevent HTTPS downgrade attacks
    add_header Strict-Transport-Security "max-age=63072000; preload";    	 
    
    # Listen for HTTPS traffic
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name         noahsbwilliams.com;

	# Specify path to TLS certificate within container
    ssl_certificate /etc/letsencrypt/live/noahsbwilliams.com/fullchain.pem;
    # Specify path to TLS private key within container
    ssl_certificate_key /etc/letsencrypt/live/noahsbwilliams.com/privkey.pem;
    
    # Specify protocols to use for HTTPS
    ssl_protocols TLSv1.2 TLSv1.3;
    # You really shouldn't be using anything
    # Lower than TLSv1.2 in the year 2020

	# Reverse proxy configuration
    location / {
    # ⬇️ Tell Nginx to set the appropriate headers for Ghost to interperet
    proxy_set_header  Host                $host;
    proxy_set_header  X-Forwarded-Proto   $scheme;
    proxy_set_header  X-Real-IP           $remote_addr;
    proxy_set_header  X-Forwarded-For     $proxy_add_x_forwarded_for;
    # ⬇️ Tell Nginx the name of the container to pass it to
    proxy_pass                            http://tilde:2368;
  }
}

For On Fire In AK, I made an identical one in the same folder, which I named onfireinak.conf. I then did a simple find-and-replace in my text editor to replace every instance of noahsbwilliams.com in the file with onfireinak.com.

Certbot (LetsEncrypt)

Can't have HTTPS without TLS - can't have TLS without some certificates!

We can get some for free from LetsEncrypt using Certbot - specifically, the conveniently published Certbot container.

You might think think we're going to add that to our docker-compose file - but you'd be wrong. You'll see why in just a sec.

Instead - we're gonna do it with a few bash scripts.

Here's the first script - the one for Tilde:

docker run -it --rm --name certbot \

  # Map a volume to a container on the host so the certs actually get saved
  -v /home/git/websites/letsencrypt/:/etc/letsencrypt \
  -v /home/git/websites/letsencrypt/log:/var/log/letsencrypt \
  
  # Expose port 80 which is needed to run Cerbot in standalone mode
  -p 80:80 \
  
  # Specify the official Certbot image
  certbot/certbot -t certonly \
  
  # Spin up Certbot's own webserver to do this, as our instance of Nginx will be offline
  --standalone \
  
  # You need to provide your email for certbot to function
  --email youremail@domain.com \
  
  # Choose not to subscribe to the EFF newsletteer
  --no-eff-email \
  
  # Duh
  --agree-tos \
  
  # On next run, look for certs and attempt to renew them instead of issuing new ones. 
  --renew-by-default \
  
  # Domains to obtain certs for
  -d noahsbwilliams.com,www.noahsbwilliams.com;

Here's the one for OnFireInAK - this time, without comments:

docker run -it --rm --name certbot \
  -v /home/git/websites/letsencrypt/:/etc/letsencrypt \
  -v /home/git/websites/letsencrypt/log:/var/log/letsencrypt \
  -p 80:80 \
  certbot/certbot -t certonly \
  --standalone \
  --email public@noahsbwilliams.com \
  --no-eff-email \
  --agree-tos --renew-by-default \
  -d onfireinak.com,www.onfireinak.com;

Now, we need to make these suckers executable...

sudo chmod +x letsencrypt-tilde.sh letsencrypt-onfireinak.sh

...and run 'em!

./letsencrypt-tilde.sh; ./letsencrypt-onfireinak.sh

Standing up the cluster

It's time to watch our hard work pay off!

In the project directory, we run:

docker-compose up -d

to start the containers in a session detached from your shell.

We can now open a browser, and setup the associated Ghost instances: (https://noahsbwilliams.com/ghost, https://onfireinak.com/ghost).

Cert Renewal and Updates

TLS certificates obtained from LetsEncrypt have a three-month shelf life, and must be periodically renewed.

Unfortunately, this part is tricky, as it's also where where Docker compose runs out of tricks.

Compose is only a rudimentary orchestration tool, and features no automatic update features like Kubernetes.

At this stage of the project, I got a bit sick of dealing with the intricacies of containers, and decided to just setup a cron job on the host machine to renew the damn things.

Before I explain what I did, let me add a couple of quick notes for my conscience:

• Certificate management: Do it better in prod!
  • Please don't use this hacked-together method in production at your company! Have some decent, dedicated way of managing certificates! This is just a small personal project that I didn't want to overcomplicate.
• Cron job security: Don't fsck it up!
  • With any script you intend to run in a privileged setting, be mindful about who can write to it! Whoever has such permissions can escalate privileges by simply modifying it to execute their arbitrary code (think: useradd badguy, ufw disable, etc).
• Automatic updates: We got em!
  • This solution actually gives us automatic updates for our containers, as it runs docker-compose up, automatically pulling the latest versions of Ghost v3 and Nginx.
  • You can disable this if you want by simply specifying a minor version, or even point release of Ghost in the docker-compose. You can also do the same for Nginx - but probably shouldn't, for security reasons.

Got it? Cool. Now let's hack.

First, let's create a renew script that runs the other two Certbot scripts we wrote sequentially.

Here's the code:

# Change to correct working directory
cd /home/git/websites/;

# Pull the latest patch versions of containers
docker-compose pull;

# Bring down the cluster to free up port 80 for Certbot
docker-compose down;

# Run the two LetsEncrypt containers sequentially
./letsencrypt-tilde.sh;
./letsencrypt-onfireinak.sh;

# Raise the cluster back up
docker-compose up -d;

Could you further simplify this by mashing all three of these scripts into one? Yeah, sure! I prefer not to, for the sake of modularity, but you can if you like.

Next, we need to create a cron job. This must be run in a privileged setting to be capable of running Docker commands.

So let's make a new privileged cron job with the command:

sudo crontab -e

Choose your preferred text editor at the prompt - I'll use Nano here:

no crontab for root - using an empty one

Select an editor.  To change later, run 'select-editor'.
  1. /bin/nano        <---- easiest
  2. /usr/bin/vim.basic
  3. /usr/bin/vim.tiny
  4. /bin/ed

We're gonna add the line:

# m h  dom mon dow   command
  0 0  1   *   *     /home/git/websites/letsencrypt-renew.sh

Quit Nano with ctrl + x and say y at the prompt to save the file.

The finished product

Boom! You're now officially done! Your Ghost sites are online, and will update themselves automatically!

Here is the resulting file structure:

git@websites:~/websites$ sudo tree -L 3
.
├── README.md
├── content
│   ├── onfireinak
│   │   ├── apps
│   │   ├── data
│   │   ├── images
│   │   ├── logs
│   │   ├── settings
│   │   └── themes
│   └── tilde
│       ├── apps
│       ├── data
│       ├── images
│       ├── logs
│       ├── settings
│       └── themes
├── docker-compose.yml
├── letsencrypt
│   ├── accounts
│   │   └── acme-v02.api.letsencrypt.org
│   ├── archive
│   │   ├── noahsbwilliams.com
│   │   └── onfireinak.com
│   ├── csr
│   │   ├── 0000_csr-certbot.pem
│   │   ├── 0001_csr-certbot.pem
│   │   ├── 0002_csr-certbot.pem
│   │   └── 0003_csr-certbot.pem
│   ├── keys
│   │   ├── 0000_key-certbot.pem
│   │   ├── 0001_key-certbot.pem
│   │   ├── 0002_key-certbot.pem
│   │   └── 0003_key-certbot.pem
│   ├── live
│   │   ├── README
│   │   ├── noahsbwilliams.com
│   │   └── onfireinak.com
│   ├── log
│   │   ├── letsencrypt.log
│   │   ├── letsencrypt.log.1
│   │   ├── letsencrypt.log.2
│   │   ├── letsencrypt.log.3
│   │   └── letsencrypt.log.4
│   ├── renewal
│   │   ├── noahsbwilliams.com.conf
│   │   └── onfireinak.com.conf
│   └── renewal-hooks
│       ├── deploy
│       ├── post
│       └── pre
├── letsencrypt-onfireinak.sh
├── letsencrypt-renew.sh
├── letsencrypt-tilde.sh
└── nginx
    ├── nginx.conf
    └── sites-enabled
        ├── onfireinak.conf
        └── tilde.conf

34 directories, 24 files

The finished docker-compose.yaml file:

version: '3.7'

services:
  tilde:
    image: ghost:3-alpine
    restart: always
    volumes:
      - ./content/tilde:/var/lib/ghost/content
    environment:
      database__client: 'sqlite3'
      url: 'https://noahsbwilliams.com'
      mail__transport: 'SMTP'
      mail__options__service: 'Mailgun'
      mail__options__host: 'smtp.mailgun.org'
      mail__options__port: '465'
      mail__options__secureConnection: 'true'
      mail__options__auth__user: 'postmaster@email.noahsbwilliams.com'
      mail__options__auth__pass: ${PW_TILDE}
    networks:
      - reverse_proxy_tilde
  onfireinak:
    image: ghost:3-alpine
    restart: always
    volumes:
      - ./content/onfireinak:/var/lib/ghost/content
    environment:
      database__client: 'sqlite3'
      url: 'https://onfireinak.com'
      mail__transport: 'SMTP'
      mail__options__service: 'Sendgrid'
      mail__options__port: '465'
      mail__options__secureConnection: 'true'
      mail__options__host: 'smtp.sendgrid.net'
      mail__options__auth__user: 'apikey'
      mail__options__auth__pass: ${PW_ONFIRE}
    networks:
      - reverse_proxy_onfireinak
  nginx:
    depends_on:
      - tilde
      - onfireinak
    image: nginx
    restart: always
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - ./nginx/:/etc/nginx/
      - ./letsencrypt:/etc/letsencrypt
    networks:
      - reverse_proxy_tilde
      - reverse_proxy_onfireinak

networks:
  reverse_proxy_tilde:
  reverse_proxy_onfireinak: