automagic sql db migrations (#461)

* adds migrations

closes #57

migrations only run if the database is not brand new. brand new
databases will contain all the right fields when CREATE TABLE is called,
this is for readability mostly more than efficiency (do not want to have
to go through all of the database migrations to ascertain what columns a table
has). upon startup of a new database, the migrations will be analyzed and the
highest version set, so that future migrations will be run. this should also
avoid running through all the migrations, which could bork db's easily enough
(if the user just exits from impatience, say).

otherwise, all migrations that a db has not yet seen will be run against it
upon startup, this should be seamless to the user whether they had a db that
had 0 migrations run on it before or N. this means users will not have to
explicitly run any migrations on their dbs nor see any errors when we upgrade
the db (so long as things go well). if migrations do not go so well, users
will have to manually repair dbs (this is the intention of the `migrate`
library and it seems sane), this should be rare, and I'm unsure myself how
best to resolve not having gone through this myself, I would assume it will
require running down migrations and then manually updating the migration
field; in any case, docs once one of us has to go through this.

migrations are written to files and checked into version control, and then use
go-bindata to generate those files into go code and compiled in to be consumed
by the migrate library (so that we don't have to put migration files on any
servers) -- this is also in vcs. this seems to work ok. I don't like having to
use the separate go-bindata tool but it wasn't really hard to install and then
go generate takes care of the args. adding migrations should be relatively
rare anyway, but tried to make it pretty painless.

1 migration to add created_at to the route is done here as an example of how
to do migrations, as well as testing these things ;) -- `created_at` will be
`0001-01-01T00:00:00.000Z` for any existing routes after a user runs this
version. could spend the extra time adding 'today's date to any outstanding
records, but that's not really accurate, the main thing is nobody will have to
nuke their db with the migrations in place & we don't have any prod clusters
really to worry about. all future routes will correctly have `created_at` set,
and plan to add other timestamps but wanted to keep this patch as small as
possible so only did routes.created_at.

there are tests that a spankin new db will work as expected as well as a db
after running all down & up migrations works. the latter tests only run on mysql
and postgres, since sqlite3 does not like ALTER TABLE DROP COLUMN; up
migrations will need to be tested manually for sqlite3 only, but in theory if
they are simple and work on postgres and mysql, there is a good likelihood of
success; the new migration from this patch works on sqlite3 fine.

for now, we need to use `github.com/rdallman/migrate` to move forward, as
getting integrated into upstream is proving difficult due to
`github.com/go-sql-driver/mysql` being broken on master (yay dependencies).
Fortunately for us, we vendor a version of the `mysql` bindings that actually
works, thus, we are capable of using the `mattes/migrate` library with success
due to that. this also will require go1.9 to use the new `database/sql.Conn`
type, CI has been updated accordingly.

some doc fixes too from testing.. and of course updated all deps.

anyway, whew. this should let us add fields to the db without busting
everybody's dbs. open to feedback on better ways, but this was overall pretty
simple despite futzing with mysql.

* add migrate pkg to deps, update deps

use rdallman/migrate until we resolve in mattes land

* add README in migrations package

* add ref to mattes lib

vendor/github.com/rdallman/migrate/database/postgres/README.md

@@ -0,0 +1,28 @@
+# postgres
+
+`postgres://user:password@host:port/dbname?query` (`postgresql://` works, too)
+
+| URL Query  | WithInstance Config | Description |
+|------------|---------------------|-------------|
+| `x-migrations-table` | `MigrationsTable` | Name of the migrations table |
+| `dbname` | `DatabaseName` | The name of the database to connect to |
+| `search_path` | | This variable specifies the order in which schemas are searched when an object is referenced by a simple name with no schema specified. |
+| `user` | | The user to sign in as |
+| `password` | | The user's password | 
+| `host` | | The host to connect to. Values that start with / are for unix domain sockets. (default is localhost) |
+| `port` | | The port to bind to. (default is 5432) |
+| `fallback_application_name` | | An application_name to fall back to if one isn't provided. |
+| `connect_timeout` | | Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. |
+| `sslcert` | | Cert file location. The file must contain PEM encoded data. |
+| `sslkey` | | Key file location. The file must contain PEM encoded data. |
+| `sslrootcert` | | The location of the root certificate file. The file must contain PEM encoded data. | 
+| `sslmode` | | Whether or not to use SSL (disable\|require\|verify-ca\|verify-full) |
+
+
+## Upgrading from v1
+
+1. Write down the current migration version from schema_migrations
+1. `DROP TABLE schema_migrations`
+2. Wrap your existing migrations in transactions ([BEGIN/COMMIT](https://www.postgresql.org/docs/current/static/transaction-iso.html)) if you use multiple statements within one migration.
+3. Download and install the latest migrate version.
+4. Force the current migration version with `migrate force <current_version>`.

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1085649617_create_users_table.down.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS users;

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1085649617_create_users_table.up.sql

@@ -0,0 +1,5 @@
+CREATE TABLE users (
+  user_id integer unique,
+  name    varchar(40),
+  email   varchar(40)
+);

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1185749658_add_city_to_users.down.sql

@@ -0,0 +1 @@
+ALTER TABLE users DROP COLUMN IF EXISTS city;

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1185749658_add_city_to_users.up.sql

@@ -0,0 +1,3 @@
+ALTER TABLE users ADD COLUMN city varchar(100);
+
+

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1285849751_add_index_on_user_emails.down.sql

@@ -0,0 +1 @@
+DROP INDEX IF EXISTS users_email_index;

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1285849751_add_index_on_user_emails.up.sql

@@ -0,0 +1,3 @@
+CREATE UNIQUE INDEX CONCURRENTLY users_email_index ON users (email);
+
+-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1385949617_create_books_table.down.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS books;

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1385949617_create_books_table.up.sql

@@ -0,0 +1,5 @@
+CREATE TABLE books (
+  user_id integer,
+  name    varchar(40),
+  author  varchar(40)
+);

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1485949617_create_movies_table.down.sql

@@ -0,0 +1 @@
+DROP TABLE IF EXISTS movies;

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1485949617_create_movies_table.up.sql

@@ -0,0 +1,5 @@
+CREATE TABLE movies (
+  user_id   integer,
+  name      varchar(40),
+  director  varchar(40)
+);

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1585849751_just_a_comment.up.sql

@@ -0,0 +1 @@
+-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1685849751_another_comment.up.sql

@@ -0,0 +1 @@
+-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1785849751_another_comment.up.sql

@@ -0,0 +1 @@
+-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.

vendor/github.com/rdallman/migrate/database/postgres/examples/migrations/1885849751_another_comment.up.sql

@@ -0,0 +1 @@
+-- Lorem ipsum dolor sit amet, consectetur adipiscing elit. Aenean sed interdum velit, tristique iaculis justo. Pellentesque ut porttitor dolor. Donec sit amet pharetra elit. Cras vel ligula ex. Phasellus posuere.

vendor/github.com/rdallman/migrate/database/postgres/postgres.go

@@ -0,0 +1,273 @@
+package postgres
+
+import (
+	"database/sql"
+	"fmt"
+	"io"
+	"io/ioutil"
+	nurl "net/url"
+
+	"github.com/lib/pq"
+	"github.com/mattes/migrate"
+	"github.com/mattes/migrate/database"
+)
+
+func init() {
+	db := Postgres{}
+	database.Register("postgres", &db)
+	database.Register("postgresql", &db)
+}
+
+var DefaultMigrationsTable = "schema_migrations"
+
+var (
+	ErrNilConfig      = fmt.Errorf("no config")
+	ErrNoDatabaseName = fmt.Errorf("no database name")
+	ErrNoSchema       = fmt.Errorf("no schema")
+	ErrDatabaseDirty  = fmt.Errorf("database is dirty")
+)
+
+type Config struct {
+	MigrationsTable string
+	DatabaseName    string
+}
+
+type Postgres struct {
+	db       *sql.DB
+	isLocked bool
+
+	// Open and WithInstance need to garantuee that config is never nil
+	config *Config
+}
+
+func WithInstance(instance *sql.DB, config *Config) (database.Driver, error) {
+	if config == nil {
+		return nil, ErrNilConfig
+	}
+
+	if err := instance.Ping(); err != nil {
+		return nil, err
+	}
+
+	query := `SELECT CURRENT_DATABASE()`
+	var databaseName string
+	if err := instance.QueryRow(query).Scan(&databaseName); err != nil {
+		return nil, &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+
+	if len(databaseName) == 0 {
+		return nil, ErrNoDatabaseName
+	}
+
+	config.DatabaseName = databaseName
+
+	if len(config.MigrationsTable) == 0 {
+		config.MigrationsTable = DefaultMigrationsTable
+	}
+
+	px := &Postgres{
+		db:     instance,
+		config: config,
+	}
+
+	if err := px.ensureVersionTable(); err != nil {
+		return nil, err
+	}
+
+	return px, nil
+}
+
+func (p *Postgres) Open(url string) (database.Driver, error) {
+	purl, err := nurl.Parse(url)
+	if err != nil {
+		return nil, err
+	}
+
+	db, err := sql.Open("postgres", migrate.FilterCustomQuery(purl).String())
+	if err != nil {
+		return nil, err
+	}
+
+	migrationsTable := purl.Query().Get("x-migrations-table")
+	if len(migrationsTable) == 0 {
+		migrationsTable = DefaultMigrationsTable
+	}
+
+	px, err := WithInstance(db, &Config{
+		DatabaseName:    purl.Path,
+		MigrationsTable: migrationsTable,
+	})
+	if err != nil {
+		return nil, err
+	}
+
+	return px, nil
+}
+
+func (p *Postgres) Close() error {
+	return p.db.Close()
+}
+
+// https://www.postgresql.org/docs/9.6/static/explicit-locking.html#ADVISORY-LOCKS
+func (p *Postgres) Lock() error {
+	if p.isLocked {
+		return database.ErrLocked
+	}
+
+	aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
+	if err != nil {
+		return err
+	}
+
+	// This will either obtain the lock immediately and return true,
+	// or return false if the lock cannot be acquired immediately.
+	query := `SELECT pg_try_advisory_lock($1)`
+	var success bool
+	if err := p.db.QueryRow(query, aid).Scan(&success); err != nil {
+		return &database.Error{OrigErr: err, Err: "try lock failed", Query: []byte(query)}
+	}
+
+	if success {
+		p.isLocked = true
+		return nil
+	}
+
+	return database.ErrLocked
+}
+
+func (p *Postgres) Unlock() error {
+	if !p.isLocked {
+		return nil
+	}
+
+	aid, err := database.GenerateAdvisoryLockId(p.config.DatabaseName)
+	if err != nil {
+		return err
+	}
+
+	query := `SELECT pg_advisory_unlock($1)`
+	if _, err := p.db.Exec(query, aid); err != nil {
+		return &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+	p.isLocked = false
+	return nil
+}
+
+func (p *Postgres) Run(migration io.Reader) error {
+	migr, err := ioutil.ReadAll(migration)
+	if err != nil {
+		return err
+	}
+
+	// run migration
+	query := string(migr[:])
+	if _, err := p.db.Exec(query); err != nil {
+		// TODO: cast to postgress error and get line number
+		return database.Error{OrigErr: err, Err: "migration failed", Query: migr}
+	}
+
+	return nil
+}
+
+func (p *Postgres) SetVersion(version int, dirty bool) error {
+	tx, err := p.db.Begin()
+	if err != nil {
+		return &database.Error{OrigErr: err, Err: "transaction start failed"}
+	}
+
+	query := `TRUNCATE "` + p.config.MigrationsTable + `"`
+	if _, err := tx.Exec(query); err != nil {
+		tx.Rollback()
+		return &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+
+	if version >= 0 {
+		query = `INSERT INTO "` + p.config.MigrationsTable + `" (version, dirty) VALUES ($1, $2)`
+		if _, err := tx.Exec(query, version, dirty); err != nil {
+			tx.Rollback()
+			return &database.Error{OrigErr: err, Query: []byte(query)}
+		}
+	}
+
+	if err := tx.Commit(); err != nil {
+		return &database.Error{OrigErr: err, Err: "transaction commit failed"}
+	}
+
+	return nil
+}
+
+func (p *Postgres) Version() (version int, dirty bool, err error) {
+	query := `SELECT version, dirty FROM "` + p.config.MigrationsTable + `" LIMIT 1`
+	err = p.db.QueryRow(query).Scan(&version, &dirty)
+	switch {
+	case err == sql.ErrNoRows:
+		return database.NilVersion, false, nil
+
+	case err != nil:
+		if e, ok := err.(*pq.Error); ok {
+			if e.Code.Name() == "undefined_table" {
+				return database.NilVersion, false, nil
+			}
+		}
+		return 0, false, &database.Error{OrigErr: err, Query: []byte(query)}
+
+	default:
+		return version, dirty, nil
+	}
+}
+
+func (p *Postgres) Drop() error {
+	// select all tables in current schema
+	query := `SELECT table_name FROM information_schema.tables WHERE table_schema=(SELECT current_schema())`
+	tables, err := p.db.Query(query)
+	if err != nil {
+		return &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+	defer tables.Close()
+
+	// delete one table after another
+	tableNames := make([]string, 0)
+	for tables.Next() {
+		var tableName string
+		if err := tables.Scan(&tableName); err != nil {
+			return err
+		}
+		if len(tableName) > 0 {
+			tableNames = append(tableNames, tableName)
+		}
+	}
+
+	if len(tableNames) > 0 {
+		// delete one by one ...
+		for _, t := range tableNames {
+			query = `DROP TABLE IF EXISTS ` + t + ` CASCADE`
+			if _, err := p.db.Exec(query); err != nil {
+				return &database.Error{OrigErr: err, Query: []byte(query)}
+			}
+		}
+		if err := p.ensureVersionTable(); err != nil {
+			return err
+		}
+	}
+
+	return nil
+}
+
+func (p *Postgres) ensureVersionTable() error {
+	// check if migration table exists
+	var count int
+	query := `SELECT COUNT(1) FROM information_schema.tables WHERE table_name = $1 AND table_schema = (SELECT current_schema()) LIMIT 1`
+	if err := p.db.QueryRow(query, p.config.MigrationsTable).Scan(&count); err != nil {
+		return &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+	if count == 1 {
+		return nil
+	}
+
+	// if not, create the empty migration table
+	query = `CREATE TABLE "` + p.config.MigrationsTable + `" (version bigint not null primary key, dirty boolean not null)`
+	if _, err := p.db.Exec(query); err != nil {
+		return &database.Error{OrigErr: err, Query: []byte(query)}
+	}
+	return nil
+}

vendor/github.com/rdallman/migrate/database/postgres/postgres_test.go

@@ -0,0 +1,150 @@
+package postgres
+
+// error codes https://github.com/lib/pq/blob/master/error.go
+
+import (
+	"bytes"
+	"database/sql"
+	"fmt"
+	"io"
+	"testing"
+
+	"github.com/lib/pq"
+	dt "github.com/mattes/migrate/database/testing"
+	mt "github.com/mattes/migrate/testing"
+)
+
+var versions = []mt.Version{
+	{Image: "postgres:9.6"},
+	{Image: "postgres:9.5"},
+	{Image: "postgres:9.4"},
+	{Image: "postgres:9.3"},
+	{Image: "postgres:9.2"},
+}
+
+func isReady(i mt.Instance) bool {
+	db, err := sql.Open("postgres", fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable", i.Host(), i.Port()))
+	if err != nil {
+		return false
+	}
+	defer db.Close()
+	err = db.Ping()
+	if err == io.EOF {
+		return false
+
+	} else if e, ok := err.(*pq.Error); ok {
+		if e.Code.Name() == "cannot_connect_now" {
+			return false
+		}
+	}
+
+	return true
+}
+
+func Test(t *testing.T) {
+	mt.ParallelTest(t, versions, isReady,
+		func(t *testing.T, i mt.Instance) {
+			p := &Postgres{}
+			addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable", i.Host(), i.Port())
+			d, err := p.Open(addr)
+			if err != nil {
+				t.Fatalf("%v", err)
+			}
+			dt.Test(t, d, []byte("SELECT 1"))
+		})
+}
+
+func TestMultiStatement(t *testing.T) {
+	mt.ParallelTest(t, versions, isReady,
+		func(t *testing.T, i mt.Instance) {
+			p := &Postgres{}
+			addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable", i.Host(), i.Port())
+			d, err := p.Open(addr)
+			if err != nil {
+				t.Fatalf("%v", err)
+			}
+			if err := d.Run(bytes.NewReader([]byte("CREATE TABLE foo (foo text); CREATE TABLE bar (bar text);"))); err != nil {
+				t.Fatalf("expected err to be nil, got %v", err)
+			}
+
+			// make sure second table exists
+			var exists bool
+			if err := d.(*Postgres).db.QueryRow("SELECT EXISTS (SELECT 1 FROM information_schema.tables WHERE table_name = 'bar' AND table_schema = (SELECT current_schema()))").Scan(&exists); err != nil {
+				t.Fatal(err)
+			}
+			if !exists {
+				t.Fatalf("expected table bar to exist")
+			}
+		})
+}
+
+func TestFilterCustomQuery(t *testing.T) {
+	mt.ParallelTest(t, versions, isReady,
+		func(t *testing.T, i mt.Instance) {
+			p := &Postgres{}
+			addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&x-custom=foobar", i.Host(), i.Port())
+			_, err := p.Open(addr)
+			if err != nil {
+				t.Fatalf("%v", err)
+			}
+		})
+}
+
+func TestWithSchema(t *testing.T) {
+	mt.ParallelTest(t, versions, isReady,
+		func(t *testing.T, i mt.Instance) {
+			p := &Postgres{}
+			addr := fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable", i.Host(), i.Port())
+			d, err := p.Open(addr)
+			if err != nil {
+				t.Fatalf("%v", err)
+			}
+
+			// create foobar schema
+			if err := d.Run(bytes.NewReader([]byte("CREATE SCHEMA foobar AUTHORIZATION postgres"))); err != nil {
+				t.Fatal(err)
+			}
+			if err := d.SetVersion(1, false); err != nil {
+				t.Fatal(err)
+			}
+
+			// re-connect using that schema
+			d2, err := p.Open(fmt.Sprintf("postgres://postgres@%v:%v/postgres?sslmode=disable&search_path=foobar", i.Host(), i.Port()))
+			if err != nil {
+				t.Fatalf("%v", err)
+			}
+
+			version, _, err := d2.Version()
+			if err != nil {
+				t.Fatal(err)
+			}
+			if version != -1 {
+				t.Fatal("expected NilVersion")
+			}
+
+			// now update version and compare
+			if err := d2.SetVersion(2, false); err != nil {
+				t.Fatal(err)
+			}
+			version, _, err = d2.Version()
+			if err != nil {
+				t.Fatal(err)
+			}
+			if version != 2 {
+				t.Fatal("expected version 2")
+			}
+
+			// meanwhile, the public schema still has the other version
+			version, _, err = d.Version()
+			if err != nil {
+				t.Fatal(err)
+			}
+			if version != 1 {
+				t.Fatal("expected version 2")
+			}
+		})
+}
+
+func TestWithInstance(t *testing.T) {
+
+}