How to Deploy GoBuffalo on Google's App Engine

This guide walks you through the process of deploying a Gobuffalo app to Google’s App Engine. It assumes usage of PostgreSQL as a database.

Background & Introduction

A bit of background first. App Engine standard environment had a lot of gotchas and system limitations in the past. With the introduction of go111 environment, running a golang app is more or less as running it anywhere else (well, with few tidbits). This article focuses on the standard environment. To find out about the differences between flex and standard environments, head over to G Cloud’s article.

Deploying a Buffalo app to GAE has been historically a challenging endeavour, due to mentioned system limitations and older version of go (e.g. usage of syscall and context.Context was required by Buffalo, but not supported on GAE). To make Buffalo GAE compatible is one of the longest open issues (2 years+).

Today, deploying a Buffalo app to the App Engine is possible. So let’s go on with it.

Requirements

1. A working Buffalo app (if not, you have some catching up to do). Preferably based on the latest v0.14+.
2. PostgreSQL as a database (others are not covered in this article)
3. Have GCP account and development environment set up
4. Created a Google Cloud Platform project
5. Have a running instance of Cloud SQL for PostgreSQL and have created a database. Quickstart

Cloud ignore file

GCP has a file similar to the .gitignore. What it does it ignores the files and folders specified in it when you want to upload them. There are certainly some that we don’t want to be uploaded when deploying to GAE.

In the root of your project create a .cloudignore file with the following content:

# GCloud / Git
.gcloudignore
.git
.gitignore

# Binaries for programs and plugins
*.exe
*.exe~
*.dll
*.so
*.dylib
# Test binary, build with `go test -c`
*.test
# Output of the go coverage tool, specifically when used with LiteIDE
*.out

# Logs and misc editors
**/*.log
**/*.sqlite
.idea/
.vscode/

# Buffalo specific ignores
bin/
tmp/
node_modules/
.sass-cache/
.grifter/

# App's binary filename - change to yours
golangtesting

App Engine Application file

app.yaml is the file that defines the App Engine app. Place it in the root of your project. It has many different options, but here is the minimal setup for the Buffalo app:

runtime: go111

#[START env_variables]
env_variables:
  PORT: 8080
  GO_ENV: 'production'
#[END env_variables]

handlers:
  - url: /robots.txt
    static_files: public/robots.txt
    upload: public/robots.txt
  - url: /assets
    static_dir: public/assets
  - url: /.*
    script: auto

First, we define the runtime by setting it to go111. Environment variables are set via env_variables option. We need to say to Buffalo to use the port 8080 (which GAE expects) and that the go environment is production.

App engine has an option to serve static files, so why bother if that is sorted for us the Google way? We have to define a couple of handlers; one for robots.txt (file definition) and one for all the assets (folder definition). In the end, we define a handler to direct all the other requests to our app by specifying script: auto.

The Environment Variables

The App Engine way

As you could see above, defining the environment variables within the app.yaml is pretty simple. This file can be safely versioned in git since it doesn’t contain any secrets.

The Buffalo way

Buffalo has a docs section about configuration. We are going to use .env file to store our secrets. The file is git-ignored as it shouldn’t end up in git. However, it is not cloud-ignored. This file will be uploaded to the App Engine, since we need it to define our database connection string and other secrets. In an ideal scenario, when deploying to the production, your CI/CD pipeline should generate this file after pulling secrets from KMS (Key Management Service). But secrets management is a topic beyond this guide.

.env example:

DATABASE_URL="user=postgres password=Im4P4$$w0Rd dbname=golangtesting_production host=/cloudsql/my-project-id:europe-west1:my-production-instance"
SESSION_SECRET=yeahR1ght

# Add more secrets here

DATABASE_URL is expected to be in the following format:

user=USER password=PASSWORD host=/cloudsql/PROJECT_ID:REGION_ID:INSTANCE_ID/[ dbname=DB_NAME]

The Database

Cloud SQL requires a config URL specific to the PostgreSQL dialog, which we just defined as a DATABASE_URL variable. Buffalo did not support this format up until now. You can read more about it here: PostgreSQL dialog URL parser

What that means in practice is that you need Buffalo Pop to be set to v4.10.0 or later. Without this version you won’t be able to create a proper connection string Cloud SQL for PostgreSQL is expecting. Hence, you won’t be able to run DB powered Buffalo app on GAE.

To ensure you have this version, update your go.mod file to include:

require (
    ...
    github.com/gobuffalo/pop v4.10.0
    ...
)

replace github.com/gobuffalo/pop => github.com/gobuffalo/pop v4.10.0

and then tidy it up with: go mod tidy.

With Pop up-to-date, you are ready to set up your config file for production.

Here is the production section of the database.yml file:

production:
  url: { { env "DATABASE_URL" } }
  dialect: postgres
  pool: 1
  idle_pool: 1

As you can see, we are relying on the DATABASE_URL environment variable to provide us with the connection details. Dialect is set to postgres. Based on the best practices we also set the pool and idle pool to 1. Meaning each request to the App Engine uses 1 connection for all the queries executed within that request. Each GAE instance running in the standard environment can’t have more than 100 concurrent connections to a Cloud SQL instance. If we used a larger connection pool, we would hit the connection limit pretty soon.

Migrations

With all the pieces set up for the database, one not being covered is migrations.

Since we can’t run soda migrate up on the App Engine, we have a few options to get the database up to date.

1. Programmatically run migrations from the app

Each time when the app is starting (or better to say the new App Engine instance is spawned), we can programmatically run the migrations. That means the first request is going to be slower. Also, the requests on the App Engine are limited to 60s duration. So if your migrations are running longer than that expect the timeout.

2. Manually run queries by connecting to the database externally (e.g., by using a proxy)

It would not use the migrations system Buffalo has built in, but manually running queries have their place. For anything complex or importing/exporting large datasets that would the way to go.

3. Create a background task that won’t run on the App Engine

Background task would be ideal since it does the migrations automatically (unlike manual queries) and it is not limited with 60s request limit App Engine has.

However, for the sake of simplicity, we’ll cover running the migrations from the app. As long as you understand the limitations of this method, it should be good enough for something simple as this blog.

To add auto-migrations to your app, append the main.go file with:

import (
    "github.com/gobuffalo/pop"
    ...
    "bitbucket.org/deviseops/golangtesting/models"
)

func main() {
    // Execute database migrations
    mig, err := pop.NewFileMigrator("./migrations", models.DB)
    if err != nil {
        panic(err)
    }
    mig.Up()

    ...
}

Another limitation of this method is that the App Engine is a read-only system (apart from /tmp folder), so when running the migrations, you’ll get a warning like this:

[POP] 2019/03/12 13:28:36 warn - Migrator: unable to dump schema: open migrations/schema.sql: read-only file system

When the migrations are applied, migrator is unsuccessfully trying to dump the latest changes to schema.sql file. Which is something we can live with for now.

The Assets

For some reason which I didn’t have time to investigate, App Engine is looking into paths loaded from packr differently than when running in the dev environment.

Quick fix for this is changing the path based on the environment. For example, for loading localisation files in the app.go file, that would be something like:

// translations will load locale files, set up the translator `actions.T`,
// and will return a middleware to use to load the correct locale for each
// request.
// for more information: https://gobuffalo.io/en/docs/localization
func translations() buffalo.MiddlewareFunc {
    var err error
    // Path changed based on the ENV variable fixes
    // properly loading files for the App Engine
    path := "../locales"
    if ENV == "production" {
        path = "./locales"
    }
    if T, err = i18n.New(packr.New("Locales", path), "en-US");      err != nil {
            app.Stop(err)
    }
    return T.Middleware()
}

Similarity, this would be the content of the render.go file:

package actions

import (
    "github.com/gobuffalo/buffalo/render"
    "github.com/gobuffalo/packr/v2"
)

var r *render.Engine
var assetsBox *packr.Box

func init() {

    // Path changed for the App Engine
    if ENV == "production" {
        assetsBox = packr.New("Public", "./public")
    } else {
        assetsBox = packr.New("Public", "../public")
    }

    r = render.New(render.Options{
        // HTML layout to be used for all HTML requests:
        HTMLLayout: "application.html",

        // Box containing all of the templates:
        TemplatesBox: packr.New("Templates", "../templates"),
        AssetsBox:    assetsBox,

        // Add template helpers here:
        Helpers: render.Helpers{
        // uncomment for non-Bootstrap form helpers:
        // “form”:     plush.FormHelper,
        // “form_for”: plush.FormForHelper,
        },
    })
}

Notice that TemplatesBox doesn’t need a change. 🤷‍♂️

Again, didn’t have time to look into internals of packr and why GAE sees things differently. Probably there is a better solution. Nevertheless, this is working.

If you remember, we defined a handler in app.yaml file which will serve everything from the assets folder as static.

Deploying

Hopefully, by now your app should be ready to be deployed. Each time you want to deploy the app, run the buffalo build command first. That will prepare all the assets for you. The only remaining thing to do is deploying it to the App Engine, by using gcloud command:

gcloud app deploy

Bonus points if CI/CD is doing all this for you.